2023-12-16T21:19:59.332 INFO:root:teuthology version: 0.0.1.dev216+g5f3731e 2023-12-16T21:19:59.332 DEBUG:teuthology.run:Teuthology command: teuthology --verbose --archive /home/teuthworker/archive/yuriw-2023-12-16_15:45:14-fs-wip-yuri8-testing-2023-12-15-0911-distro-default-smithi/7494858 --owner scheduled_yuriw@teuthology --description fs/upgrade/mds_upgrade_sequence/{bluestore-bitmap centos_9.stream conf/{client mds mon osd} fail_fs/yes overrides/{ignorelist_health ignorelist_wrongly_marked_down pg-warn syntax} roles tasks/{0-from/quincy 1-volume/{0-create 1-ranks/1 2-allow_standby_replay/yes 3-inline/yes 4-verify} 2-client 3-upgrade-mgr-staggered 4-config-upgrade/{fail_fs} 5-upgrade-with-workload 6-verify}} --name yuriw-2023-12-16_15:45:14-fs-wip-yuri8-testing-2023-12-15-0911-distro-default-smithi -- /home/teuthworker/archive/yuriw-2023-12-16_15:45:14-fs-wip-yuri8-testing-2023-12-15-0911-distro-default-smithi/7494858/orig.config.yaml 2023-12-16T21:19:59.385 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2023-12-16T21:19:59.472 INFO:teuthology.run:Config: archive_path: /home/teuthworker/archive/yuriw-2023-12-16_15:45:14-fs-wip-yuri8-testing-2023-12-15-0911-distro-default-smithi/7494858 branch: wip-yuri8-testing-2023-12-15-0911 description: fs/upgrade/mds_upgrade_sequence/{bluestore-bitmap centos_9.stream conf/{client mds mon osd} fail_fs/yes overrides/{ignorelist_health ignorelist_wrongly_marked_down pg-warn syntax} roles tasks/{0-from/quincy 1-volume/{0-create 1-ranks/1 2-allow_standby_replay/yes 3-inline/yes 4-verify} 2-client 3-upgrade-mgr-staggered 4-config-upgrade/{fail_fs} 5-upgrade-with-workload 6-verify}} email: yweinste@redhat.com first_in_suite: false job_id: '7494858' kernel: kdb: true sha1: distro last_in_suite: false machine_type: smithi meta: - desc: 'setup ceph/quincy ' name: yuriw-2023-12-16_15:45:14-fs-wip-yuri8-testing-2023-12-15-0911-distro-default-smithi no_nested_subset: false nuke-on-error: true os_type: centos os_version: 9.stream overrides: admin_socket: branch: wip-yuri8-testing-2023-12-15-0911 ceph: conf: client: client mount timeout: 600 debug client: 20 debug ms: 1 rados mon op timeout: 900 rados osd op timeout: 900 global: mon pg warn min per osd: 0 mds: debug mds: 20 debug mds balancer: 20 debug ms: 1 mds debug frag: true mds debug scatterstat: true mds op complaint time: 180 mds verify scatter: true osd op complaint time: 180 rados mon op timeout: 900 rados osd op timeout: 900 mgr: debug mgr: 20 debug ms: 1 mon: debug mon: 20 debug ms: 1 debug paxos: 20 mon op complaint time: 120 osd: bdev async discard: true bdev enable discard: true bluestore allocator: bitmap bluestore block size: 96636764160 bluestore fsck on mount: true debug bluefs: 1/20 debug bluestore: 1/20 debug ms: 1 debug osd: 20 debug rocksdb: 4/10 mon osd backfillfull_ratio: 0.85 mon osd full ratio: 0.9 mon osd nearfull ratio: 0.8 osd failsafe full ratio: 0.95 osd objectstore: bluestore osd op complaint time: 180 flavor: default fs: xfs log-ignorelist: - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) - overall HEALTH_ - \(FS_DEGRADED\) - \(MDS_FAILED\) - \(MDS_DEGRADED\) - \(FS_WITH_FAILED_MDS\) - \(MDS_DAMAGE\) - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) - \(FS_INLINE_DATA_DEPRECATED\) - \(POOL_APP_NOT_ENABLED\) - overall HEALTH_ - \(OSD_DOWN\) - \(OSD_ - but it is still running - is not responding sha1: c7586e95519b7be76f3b6e3a4b333888baf46bc9 ceph-deploy: bluestore: true conf: client: log file: /var/log/ceph/ceph-$name.$pid.log mon: {} osd: bdev async discard: true bdev enable discard: true bluestore block size: 96636764160 bluestore fsck on mount: true debug bluefs: 1/20 debug bluestore: 1/20 debug rocksdb: 4/10 mon osd backfillfull_ratio: 0.85 mon osd full ratio: 0.9 mon osd nearfull ratio: 0.8 osd failsafe full ratio: 0.95 osd objectstore: bluestore fs: xfs install: ceph: flavor: default sha1: c7586e95519b7be76f3b6e3a4b333888baf46bc9 kclient: syntax: v1 selinux: whitelist: - scontext=system_u:system_r:logrotate_t:s0 thrashosds: bdev_inject_crash: 2 bdev_inject_crash_probability: 0.5 workunit: branch: wip-yuri8-testing-2023-12-15-0911 sha1: c7586e95519b7be76f3b6e3a4b333888baf46bc9 owner: scheduled_yuriw@teuthology priority: 75 repo: https://github.com/ceph/ceph-ci.git roles: - - host.a - client.0 - osd.0 - osd.1 - osd.2 - - host.b - client.1 - osd.3 - osd.4 - osd.5 seed: 2960 sha1: c7586e95519b7be76f3b6e3a4b333888baf46bc9 sleep_before_teardown: 0 subset: 111/12000 suite: fs suite_branch: wip-yuri8-testing-2023-12-15-0911 suite_path: /home/teuthworker/src/github.com_ceph_ceph-c_c7586e95519b7be76f3b6e3a4b333888baf46bc9/qa suite_relpath: qa suite_repo: https://github.com/ceph/ceph-ci.git suite_sha1: c7586e95519b7be76f3b6e3a4b333888baf46bc9 targets: smithi045.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBPsCK7s1vHn0T4FJqBKnHXIpKBUcg+SA1RW0IVqgduhRtZ+wdrh/8M1i96JDgGgUBfEwf4A9vlTbi2lG9J6V2oE= smithi169.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBDcLjeTZR2mTfVitxMoIj6Egtg+Gcrvh+9VO54SqCqzI0kSuJl14hyyfN8dF8vxDH8FuhaDPT12sO01w//NOn5s= tasks: - install: branch: quincy exclude_packages: - ceph-volume - print: '**** done install task...' - cephadm: cephadm_branch: quincy cephadm_git_url: https://github.com/ceph/ceph conf: osd: osd_class_default_list: '*' osd_class_load_list: '*' image: quay.ceph.io/ceph-ci/ceph:quincy roleless: true - print: '**** done end installing quincy cephadm ...' - cephadm.shell: host.a: - ceph config set mgr mgr/cephadm/use_repo_digest true --force - print: '**** done cephadm.shell ceph config set mgr...' - cephadm.shell: host.a: - ceph orch status - ceph orch ps - ceph orch ls - ceph orch host ls - ceph orch device ls - cephadm.shell: host.a: - ceph fs volume create cephfs --placement=4 - ceph fs dump - cephadm.shell: host.a: - ceph fs set cephfs max_mds 1 - cephadm.shell: host.a: - ceph fs set cephfs allow_standby_replay true - cephadm.shell: host.a: - ceph fs set cephfs inline_data true --yes-i-really-really-mean-it - cephadm.shell: host.a: - ceph fs dump - ceph --format=json fs dump | jq -e ".filesystems | length == 1" - while ! ceph --format=json mds versions | jq -e ". | add == 4"; do sleep 1; done - fs.pre_upgrade_save: null - kclient: null - print: '**** done client' - parallel: - upgrade-tasks - workload-tasks - cephadm.shell: host.a: - ceph fs dump - fs.post_upgrade_checks: null teuthology: fragments_dropped: [] meta: {} postmerge: [] variables: fail_fs: true teuthology_branch: main teuthology_sha1: 5f3731ee8689ef3b0ef7d571e4cc2cb968c78d73 timestamp: 2023-12-16_15:45:14 tube: smithi upgrade-tasks: sequential: - cephadm.shell: env: - sha1 host.a: - ceph config set mon mon_warn_on_insecure_global_id_reclaim false --force - ceph config set mon mon_warn_on_insecure_global_id_reclaim_allowed false --force - ceph config set global log_to_journald false --force - ceph orch upgrade start --image quay.ceph.io/ceph-ci/ceph:$sha1 --daemon-types mgr - while ceph orch upgrade status | jq '.in_progress' | grep true && ! ceph orch upgrade status | jq '.message' | grep Error ; do ceph orch ps ; ceph versions ; ceph orch upgrade status ; sleep 30 ; done - ceph versions | jq -e '.mgr | length == 1' - ceph versions | jq -e '.mgr | keys' | grep $sha1 - ceph versions | jq -e '.overall | length == 2' - ceph orch upgrade check quay.ceph.io/ceph-ci/ceph:$sha1 | jq -e '.up_to_date | length == 2' - ceph orch ps - cephadm.shell: env: - sha1 host.a: - ceph config set mgr mgr/orchestrator/fail_fs true - cephadm.shell: env: - sha1 host.a: - ceph config set mon mon_warn_on_insecure_global_id_reclaim false --force - ceph config set mon mon_warn_on_insecure_global_id_reclaim_allowed false --force - ceph config set global log_to_journald false --force - ceph orch upgrade start --image quay.ceph.io/ceph-ci/ceph:$sha1 - cephadm.shell: env: - sha1 host.a: - while ceph orch upgrade status | jq '.in_progress' | grep true && ! ceph orch upgrade status | jq '.message' | grep Error ; do ceph orch ps ; ceph versions ; ceph fs dump; ceph orch upgrade status ; ceph health detail ; sleep 30 ; done - ceph orch ps - ceph orch upgrade status - ceph health detail - ceph versions - echo "wait for servicemap items w/ changing names to refresh" - sleep 60 - ceph orch ps - ceph versions - ceph versions | jq -e '.overall | length == 1' - ceph versions | jq -e '.overall | keys' | grep $sha1 user: yuriw verbose: true worker_log: /home/teuthworker/archive/worker_logs/dispatcher.smithi.1485078 workload-tasks: sequential: - workunit: clients: all: - suites/fsstress.sh 2023-12-16T21:19:59.473 INFO:teuthology.run:suite_path is set to /home/teuthworker/src/github.com_ceph_ceph-c_c7586e95519b7be76f3b6e3a4b333888baf46bc9/qa; will attempt to use it 2023-12-16T21:19:59.473 INFO:teuthology.run:Found tasks at /home/teuthworker/src/github.com_ceph_ceph-c_c7586e95519b7be76f3b6e3a4b333888baf46bc9/qa/tasks 2023-12-16T21:19:59.474 INFO:teuthology.run_tasks:Running task internal.check_packages... 2023-12-16T21:19:59.475 INFO:teuthology.task.internal:Checking packages... 2023-12-16T21:19:59.498 INFO:teuthology.task.internal:Checking packages for os_type 'centos', flavor 'default' and ceph hash 'c7586e95519b7be76f3b6e3a4b333888baf46bc9' 2023-12-16T21:19:59.499 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2023-12-16T21:19:59.499 INFO:teuthology.packaging:ref: None 2023-12-16T21:19:59.499 INFO:teuthology.packaging:tag: None 2023-12-16T21:19:59.499 INFO:teuthology.packaging:branch: wip-yuri8-testing-2023-12-15-0911 2023-12-16T21:19:59.499 INFO:teuthology.packaging:sha1: c7586e95519b7be76f3b6e3a4b333888baf46bc9 2023-12-16T21:19:59.499 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&ref=wip-yuri8-testing-2023-12-15-0911 2023-12-16T21:19:59.921 INFO:teuthology.task.internal:Found packages for ceph version 19.0.0-202.gc7586e95 2023-12-16T21:19:59.922 INFO:teuthology.run_tasks:Running task internal.buildpackages_prep... 2023-12-16T21:19:59.928 INFO:teuthology.task.internal:no buildpackages task found 2023-12-16T21:19:59.928 INFO:teuthology.run_tasks:Running task internal.save_config... 2023-12-16T21:19:59.934 INFO:teuthology.task.internal:Saving configuration 2023-12-16T21:19:59.954 INFO:teuthology.run_tasks:Running task internal.check_lock... 2023-12-16T21:19:59.960 INFO:teuthology.task.internal.check_lock:Checking locks... 2023-12-16T21:19:59.987 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi045.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/yuriw-2023-12-16_15:45:14-fs-wip-yuri8-testing-2023-12-15-0911-distro-default-smithi/7494858', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2023-12-16 21:14:45.499527', 'locked_by': 'scheduled_yuriw@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBPsCK7s1vHn0T4FJqBKnHXIpKBUcg+SA1RW0IVqgduhRtZ+wdrh/8M1i96JDgGgUBfEwf4A9vlTbi2lG9J6V2oE='} 2023-12-16T21:20:00.015 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi169.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/yuriw-2023-12-16_15:45:14-fs-wip-yuri8-testing-2023-12-15-0911-distro-default-smithi/7494858', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2023-12-16 21:14:45.497652', 'locked_by': 'scheduled_yuriw@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBDcLjeTZR2mTfVitxMoIj6Egtg+Gcrvh+9VO54SqCqzI0kSuJl14hyyfN8dF8vxDH8FuhaDPT12sO01w//NOn5s='} 2023-12-16T21:20:00.015 INFO:teuthology.run_tasks:Running task internal.add_remotes... 2023-12-16T21:20:00.021 INFO:teuthology.task.internal:roles: ubuntu@smithi045.front.sepia.ceph.com - ['host.a', 'client.0', 'osd.0', 'osd.1', 'osd.2'] 2023-12-16T21:20:00.021 INFO:teuthology.task.internal:roles: ubuntu@smithi169.front.sepia.ceph.com - ['host.b', 'client.1', 'osd.3', 'osd.4', 'osd.5'] 2023-12-16T21:20:00.021 INFO:teuthology.run_tasks:Running task console_log... 2023-12-16T21:20:00.094 DEBUG:teuthology.exit:Installing handler: Handler(exiter=, func=.kill_console_loggers at 0x7ff6317a7f70>, signals=[15]) 2023-12-16T21:20:00.094 INFO:teuthology.run_tasks:Running task internal.connect... 2023-12-16T21:20:00.100 INFO:teuthology.task.internal:Opening connections... 2023-12-16T21:20:00.100 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi045.front.sepia.ceph.com 2023-12-16T21:20:00.102 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi045.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-12-16T21:20:00.198 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi169.front.sepia.ceph.com 2023-12-16T21:20:00.199 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi169.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-12-16T21:20:00.285 INFO:teuthology.run_tasks:Running task internal.push_inventory... 2023-12-16T21:20:00.292 DEBUG:teuthology.orchestra.run.smithi045:> uname -m 2023-12-16T21:20:00.310 INFO:teuthology.orchestra.run.smithi045.stdout:x86_64 2023-12-16T21:20:00.310 DEBUG:teuthology.orchestra.run.smithi045:> cat /etc/os-release 2023-12-16T21:20:00.368 INFO:teuthology.orchestra.run.smithi045.stdout:NAME="CentOS Stream" 2023-12-16T21:20:00.368 INFO:teuthology.orchestra.run.smithi045.stdout:VERSION="9" 2023-12-16T21:20:00.368 INFO:teuthology.orchestra.run.smithi045.stdout:ID="centos" 2023-12-16T21:20:00.368 INFO:teuthology.orchestra.run.smithi045.stdout:ID_LIKE="rhel fedora" 2023-12-16T21:20:00.368 INFO:teuthology.orchestra.run.smithi045.stdout:VERSION_ID="9" 2023-12-16T21:20:00.368 INFO:teuthology.orchestra.run.smithi045.stdout:PLATFORM_ID="platform:el9" 2023-12-16T21:20:00.368 INFO:teuthology.orchestra.run.smithi045.stdout:PRETTY_NAME="CentOS Stream 9" 2023-12-16T21:20:00.368 INFO:teuthology.orchestra.run.smithi045.stdout:ANSI_COLOR="0;31" 2023-12-16T21:20:00.368 INFO:teuthology.orchestra.run.smithi045.stdout:LOGO="fedora-logo-icon" 2023-12-16T21:20:00.368 INFO:teuthology.orchestra.run.smithi045.stdout:CPE_NAME="cpe:/o:centos:centos:9" 2023-12-16T21:20:00.368 INFO:teuthology.orchestra.run.smithi045.stdout:HOME_URL="https://centos.org/" 2023-12-16T21:20:00.368 INFO:teuthology.orchestra.run.smithi045.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2023-12-16T21:20:00.369 INFO:teuthology.orchestra.run.smithi045.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9" 2023-12-16T21:20:00.369 INFO:teuthology.orchestra.run.smithi045.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2023-12-16T21:20:00.369 INFO:teuthology.lock.ops:Updating smithi045.front.sepia.ceph.com on lock server 2023-12-16T21:20:00.393 DEBUG:teuthology.orchestra.run.smithi169:> uname -m 2023-12-16T21:20:00.414 INFO:teuthology.orchestra.run.smithi169.stdout:x86_64 2023-12-16T21:20:00.414 DEBUG:teuthology.orchestra.run.smithi169:> cat /etc/os-release 2023-12-16T21:20:00.474 INFO:teuthology.orchestra.run.smithi169.stdout:NAME="CentOS Stream" 2023-12-16T21:20:00.475 INFO:teuthology.orchestra.run.smithi169.stdout:VERSION="9" 2023-12-16T21:20:00.475 INFO:teuthology.orchestra.run.smithi169.stdout:ID="centos" 2023-12-16T21:20:00.475 INFO:teuthology.orchestra.run.smithi169.stdout:ID_LIKE="rhel fedora" 2023-12-16T21:20:00.475 INFO:teuthology.orchestra.run.smithi169.stdout:VERSION_ID="9" 2023-12-16T21:20:00.475 INFO:teuthology.orchestra.run.smithi169.stdout:PLATFORM_ID="platform:el9" 2023-12-16T21:20:00.475 INFO:teuthology.orchestra.run.smithi169.stdout:PRETTY_NAME="CentOS Stream 9" 2023-12-16T21:20:00.475 INFO:teuthology.orchestra.run.smithi169.stdout:ANSI_COLOR="0;31" 2023-12-16T21:20:00.475 INFO:teuthology.orchestra.run.smithi169.stdout:LOGO="fedora-logo-icon" 2023-12-16T21:20:00.475 INFO:teuthology.orchestra.run.smithi169.stdout:CPE_NAME="cpe:/o:centos:centos:9" 2023-12-16T21:20:00.476 INFO:teuthology.orchestra.run.smithi169.stdout:HOME_URL="https://centos.org/" 2023-12-16T21:20:00.476 INFO:teuthology.orchestra.run.smithi169.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2023-12-16T21:20:00.476 INFO:teuthology.orchestra.run.smithi169.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9" 2023-12-16T21:20:00.476 INFO:teuthology.orchestra.run.smithi169.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2023-12-16T21:20:00.476 INFO:teuthology.lock.ops:Updating smithi169.front.sepia.ceph.com on lock server 2023-12-16T21:20:00.501 INFO:teuthology.run_tasks:Running task internal.serialize_remote_roles... 2023-12-16T21:20:00.510 INFO:teuthology.run_tasks:Running task internal.check_conflict... 2023-12-16T21:20:00.516 INFO:teuthology.task.internal:Checking for old test directory... 2023-12-16T21:20:00.517 DEBUG:teuthology.orchestra.run.smithi045:> test '!' -e /home/ubuntu/cephtest 2023-12-16T21:20:00.519 DEBUG:teuthology.orchestra.run.smithi169:> test '!' -e /home/ubuntu/cephtest 2023-12-16T21:20:00.534 INFO:teuthology.run_tasks:Running task internal.check_ceph_data... 2023-12-16T21:20:00.540 INFO:teuthology.task.internal:Checking for non-empty /var/lib/ceph... 2023-12-16T21:20:00.540 DEBUG:teuthology.orchestra.run.smithi045:> test -z $(ls -A /var/lib/ceph) 2023-12-16T21:20:00.577 DEBUG:teuthology.orchestra.run.smithi169:> test -z $(ls -A /var/lib/ceph) 2023-12-16T21:20:00.645 INFO:teuthology.run_tasks:Running task internal.vm_setup... 2023-12-16T21:20:00.700 INFO:teuthology.run_tasks:Running task kernel... 2023-12-16T21:20:00.714 INFO:teuthology.task.kernel:normalize config orig: {'kdb': True, 'sha1': 'distro'} 2023-12-16T21:20:00.714 INFO:teuthology.task.kernel:config {'host.a': {'kdb': True, 'sha1': 'distro'}, 'host.b': {'kdb': True, 'sha1': 'distro'}}, timeout 300 2023-12-16T21:20:00.715 DEBUG:teuthology.orchestra.run.smithi045:> test -f /run/.containerenv -o -f /.dockerenv 2023-12-16T21:20:00.715 DEBUG:teuthology.orchestra.run.smithi169:> test -f /run/.containerenv -o -f /.dockerenv 2023-12-16T21:20:00.730 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-12-16T21:20:00.730 DEBUG:teuthology.orchestra.run.smithi169:> uname -r 2023-12-16T21:20:00.731 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-12-16T21:20:00.732 DEBUG:teuthology.orchestra.run.smithi045:> uname -r 2023-12-16T21:20:00.790 INFO:teuthology.orchestra.run.smithi045.stdout:5.14.0-361.el9.x86_64 2023-12-16T21:20:00.791 INFO:teuthology.orchestra.run.smithi169.stdout:5.14.0-361.el9.x86_64 2023-12-16T21:20:00.791 INFO:teuthology.task.kernel:Running kernel on smithi045: 5.14.0-361.el9.x86_64 2023-12-16T21:20:00.791 DEBUG:teuthology.orchestra.run.smithi045:> sudo yum install -y kernel 2023-12-16T21:20:00.791 INFO:teuthology.task.kernel:Running kernel on smithi169: 5.14.0-361.el9.x86_64 2023-12-16T21:20:00.791 DEBUG:teuthology.orchestra.run.smithi169:> sudo yum install -y kernel 2023-12-16T21:20:02.845 INFO:teuthology.orchestra.run.smithi169.stdout:CentOS-9 - AppStream 15 MB/s | 6.4 MB 00:00 2023-12-16T21:20:02.869 INFO:teuthology.orchestra.run.smithi045.stdout:CentOS-9 - AppStream 15 MB/s | 6.4 MB 00:00 2023-12-16T21:20:04.171 INFO:teuthology.orchestra.run.smithi169.stdout:CentOS-9 - BaseOS 7.2 MB/s | 2.0 MB 00:00 2023-12-16T21:20:04.269 INFO:teuthology.orchestra.run.smithi045.stdout:CentOS-9 - BaseOS 6.7 MB/s | 2.0 MB 00:00 2023-12-16T21:20:05.362 INFO:teuthology.orchestra.run.smithi169.stdout:CentOS Stream 9 - BaseOS 9.1 MB/s | 7.9 MB 00:00 2023-12-16T21:20:06.017 INFO:teuthology.orchestra.run.smithi045.stdout:CentOS Stream 9 - BaseOS 5.6 MB/s | 7.9 MB 00:01 2023-12-16T21:20:08.562 INFO:teuthology.orchestra.run.smithi169.stdout:CentOS Stream 9 - AppStream 8.4 MB/s | 18 MB 00:02 2023-12-16T21:20:11.876 INFO:teuthology.orchestra.run.smithi045.stdout:CentOS Stream 9 - AppStream 3.8 MB/s | 18 MB 00:04 2023-12-16T21:20:13.391 INFO:teuthology.orchestra.run.smithi169.stdout:CentOS Stream 9 - CRB 4.0 MB/s | 5.8 MB 00:01 2023-12-16T21:20:15.123 INFO:teuthology.orchestra.run.smithi169.stdout:CentOS Stream 9 - Extras packages 22 kB/s | 15 kB 00:00 2023-12-16T21:20:16.169 INFO:teuthology.orchestra.run.smithi045.stdout:CentOS Stream 9 - CRB 8.5 MB/s | 5.8 MB 00:00 2023-12-16T21:20:16.807 INFO:teuthology.orchestra.run.smithi169.stdout:Extra Packages for Enterprise Linux 13 MB/s | 20 MB 00:01 2023-12-16T21:20:17.552 INFO:teuthology.orchestra.run.smithi045.stdout:CentOS Stream 9 - Extras packages 61 kB/s | 15 kB 00:00 2023-12-16T21:20:18.514 INFO:teuthology.orchestra.run.smithi045.stdout:Extra Packages for Enterprise Linux 23 MB/s | 20 MB 00:00 2023-12-16T21:20:21.598 INFO:teuthology.orchestra.run.smithi169.stdout:lab-extras 12 kB/s | 1.7 kB 00:00 2023-12-16T21:20:23.462 INFO:teuthology.orchestra.run.smithi045.stdout:lab-extras 13 kB/s | 1.7 kB 00:00 2023-12-16T21:20:23.730 INFO:teuthology.orchestra.run.smithi169.stdout:Package kernel-5.14.0-239.el9.x86_64 is already installed. 2023-12-16T21:20:23.730 INFO:teuthology.orchestra.run.smithi169.stdout:Package kernel-5.14.0-361.el9.x86_64 is already installed. 2023-12-16T21:20:23.823 INFO:teuthology.orchestra.run.smithi169.stdout:Dependencies resolved. 2023-12-16T21:20:23.834 INFO:teuthology.orchestra.run.smithi169.stdout:================================================================================ 2023-12-16T21:20:23.835 INFO:teuthology.orchestra.run.smithi169.stdout: Package Architecture Version Repository Size 2023-12-16T21:20:23.835 INFO:teuthology.orchestra.run.smithi169.stdout:================================================================================ 2023-12-16T21:20:23.835 INFO:teuthology.orchestra.run.smithi169.stdout:Installing: 2023-12-16T21:20:23.835 INFO:teuthology.orchestra.run.smithi169.stdout: kernel x86_64 5.14.0-391.el9 baseos 5.6 M 2023-12-16T21:20:23.835 INFO:teuthology.orchestra.run.smithi169.stdout:Installing dependencies: 2023-12-16T21:20:23.835 INFO:teuthology.orchestra.run.smithi169.stdout: kernel-core x86_64 5.14.0-391.el9 baseos 20 M 2023-12-16T21:20:23.835 INFO:teuthology.orchestra.run.smithi169.stdout: kernel-modules x86_64 5.14.0-391.el9 baseos 38 M 2023-12-16T21:20:23.835 INFO:teuthology.orchestra.run.smithi169.stdout: kernel-modules-core x86_64 5.14.0-391.el9 baseos 32 M 2023-12-16T21:20:23.835 INFO:teuthology.orchestra.run.smithi169.stdout: 2023-12-16T21:20:23.836 INFO:teuthology.orchestra.run.smithi169.stdout:Transaction Summary 2023-12-16T21:20:23.836 INFO:teuthology.orchestra.run.smithi169.stdout:================================================================================ 2023-12-16T21:20:23.836 INFO:teuthology.orchestra.run.smithi169.stdout:Install 4 Packages 2023-12-16T21:20:23.836 INFO:teuthology.orchestra.run.smithi169.stdout: 2023-12-16T21:20:23.837 INFO:teuthology.orchestra.run.smithi169.stdout:Total download size: 96 M 2023-12-16T21:20:23.837 INFO:teuthology.orchestra.run.smithi169.stdout:Installed size: 121 M 2023-12-16T21:20:23.837 INFO:teuthology.orchestra.run.smithi169.stdout:Downloading Packages: 2023-12-16T21:20:25.241 INFO:teuthology.orchestra.run.smithi169.stdout:(1/4): kernel-5.14.0-391.el9.x86_64.rpm 4.2 MB/s | 5.6 MB 00:01 2023-12-16T21:20:25.644 INFO:teuthology.orchestra.run.smithi045.stdout:Package kernel-5.14.0-239.el9.x86_64 is already installed. 2023-12-16T21:20:25.644 INFO:teuthology.orchestra.run.smithi045.stdout:Package kernel-5.14.0-361.el9.x86_64 is already installed. 2023-12-16T21:20:25.732 INFO:teuthology.orchestra.run.smithi045.stdout:Dependencies resolved. 2023-12-16T21:20:25.751 INFO:teuthology.orchestra.run.smithi045.stdout:================================================================================ 2023-12-16T21:20:25.752 INFO:teuthology.orchestra.run.smithi045.stdout: Package Architecture Version Repository Size 2023-12-16T21:20:25.752 INFO:teuthology.orchestra.run.smithi045.stdout:================================================================================ 2023-12-16T21:20:25.752 INFO:teuthology.orchestra.run.smithi045.stdout:Installing: 2023-12-16T21:20:25.752 INFO:teuthology.orchestra.run.smithi045.stdout: kernel x86_64 5.14.0-391.el9 baseos 5.6 M 2023-12-16T21:20:25.752 INFO:teuthology.orchestra.run.smithi045.stdout:Installing dependencies: 2023-12-16T21:20:25.752 INFO:teuthology.orchestra.run.smithi045.stdout: kernel-core x86_64 5.14.0-391.el9 baseos 20 M 2023-12-16T21:20:25.752 INFO:teuthology.orchestra.run.smithi045.stdout: kernel-modules x86_64 5.14.0-391.el9 baseos 38 M 2023-12-16T21:20:25.752 INFO:teuthology.orchestra.run.smithi045.stdout: kernel-modules-core x86_64 5.14.0-391.el9 baseos 32 M 2023-12-16T21:20:25.752 INFO:teuthology.orchestra.run.smithi045.stdout: 2023-12-16T21:20:25.752 INFO:teuthology.orchestra.run.smithi045.stdout:Transaction Summary 2023-12-16T21:20:25.752 INFO:teuthology.orchestra.run.smithi045.stdout:================================================================================ 2023-12-16T21:20:25.752 INFO:teuthology.orchestra.run.smithi045.stdout:Install 4 Packages 2023-12-16T21:20:25.752 INFO:teuthology.orchestra.run.smithi045.stdout: 2023-12-16T21:20:25.753 INFO:teuthology.orchestra.run.smithi045.stdout:Total download size: 96 M 2023-12-16T21:20:25.753 INFO:teuthology.orchestra.run.smithi045.stdout:Installed size: 121 M 2023-12-16T21:20:25.753 INFO:teuthology.orchestra.run.smithi045.stdout:Downloading Packages: 2023-12-16T21:20:25.900 INFO:teuthology.orchestra.run.smithi169.stdout:(2/4): kernel-core-5.14.0-391.el9.x86_64.rpm 10 MB/s | 20 MB 00:01 2023-12-16T21:20:26.675 INFO:teuthology.orchestra.run.smithi169.stdout:(3/4): kernel-modules-5.14.0-391.el9.x86_64.rpm 14 MB/s | 38 MB 00:02 2023-12-16T21:20:27.738 INFO:teuthology.orchestra.run.smithi045.stdout:(1/4): kernel-5.14.0-391.el9.x86_64.rpm 3.0 MB/s | 5.6 MB 00:01 2023-12-16T21:20:27.958 INFO:teuthology.orchestra.run.smithi169.stdout:(4/4): kernel-modules-core-5.14.0-391.el9.x86_6 12 MB/s | 32 MB 00:02 2023-12-16T21:20:27.959 INFO:teuthology.orchestra.run.smithi169.stdout:-------------------------------------------------------------------------------- 2023-12-16T21:20:27.959 INFO:teuthology.orchestra.run.smithi169.stdout:Total 23 MB/s | 96 MB 00:04 2023-12-16T21:20:28.816 INFO:teuthology.orchestra.run.smithi169.stdout:Running transaction check 2023-12-16T21:20:29.204 INFO:teuthology.orchestra.run.smithi169.stdout:Transaction check succeeded. 2023-12-16T21:20:29.204 INFO:teuthology.orchestra.run.smithi169.stdout:Running transaction test 2023-12-16T21:20:30.512 INFO:teuthology.orchestra.run.smithi169.stdout:Transaction test succeeded. 2023-12-16T21:20:30.513 INFO:teuthology.orchestra.run.smithi169.stdout:Running transaction 2023-12-16T21:20:33.404 INFO:teuthology.orchestra.run.smithi169.stdout: Preparing : 1/1 2023-12-16T21:20:34.055 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : kernel-modules-core-5.14.0-391.el9.x86_64 1/4 2023-12-16T21:20:34.200 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : kernel-core-5.14.0-391.el9.x86_64 2/4 2023-12-16T21:20:35.171 INFO:teuthology.orchestra.run.smithi045.stdout:(2/4): kernel-core-5.14.0-391.el9.x86_64.rpm 2.2 MB/s | 20 MB 00:09 2023-12-16T21:20:35.654 INFO:teuthology.orchestra.run.smithi045.stdout:(3/4): kernel-modules-core-5.14.0-391.el9.x86_6 4.1 MB/s | 32 MB 00:07 2023-12-16T21:20:36.986 INFO:teuthology.orchestra.run.smithi169.stdout: Running scriptlet: kernel-core-5.14.0-391.el9.x86_64 2/4 2023-12-16T21:20:37.221 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : kernel-modules-5.14.0-391.el9.x86_64 3/4 2023-12-16T21:20:37.613 INFO:teuthology.orchestra.run.smithi045.stdout:(4/4): kernel-modules-5.14.0-391.el9.x86_64.rpm 3.2 MB/s | 38 MB 00:11 2023-12-16T21:20:37.613 INFO:teuthology.orchestra.run.smithi045.stdout:-------------------------------------------------------------------------------- 2023-12-16T21:20:37.613 INFO:teuthology.orchestra.run.smithi045.stdout:Total 8.1 MB/s | 96 MB 00:11 2023-12-16T21:20:38.507 INFO:teuthology.orchestra.run.smithi045.stdout:Running transaction check 2023-12-16T21:20:38.930 INFO:teuthology.orchestra.run.smithi045.stdout:Transaction check succeeded. 2023-12-16T21:20:38.931 INFO:teuthology.orchestra.run.smithi045.stdout:Running transaction test 2023-12-16T21:20:40.324 INFO:teuthology.orchestra.run.smithi045.stdout:Transaction test succeeded. 2023-12-16T21:20:40.324 INFO:teuthology.orchestra.run.smithi045.stdout:Running transaction 2023-12-16T21:20:41.845 INFO:teuthology.orchestra.run.smithi169.stdout: Running scriptlet: kernel-modules-5.14.0-391.el9.x86_64 3/4 2023-12-16T21:20:41.971 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : kernel-5.14.0-391.el9.x86_64 4/4 2023-12-16T21:20:43.129 INFO:teuthology.orchestra.run.smithi045.stdout: Preparing : 1/1 2023-12-16T21:20:43.792 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : kernel-modules-core-5.14.0-391.el9.x86_64 1/4 2023-12-16T21:20:43.922 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : kernel-core-5.14.0-391.el9.x86_64 2/4 2023-12-16T21:20:46.591 INFO:teuthology.orchestra.run.smithi169.stdout: Running scriptlet: kernel-modules-core-5.14.0-391.el9.x86_64 4/4 2023-12-16T21:20:46.804 INFO:teuthology.orchestra.run.smithi045.stdout: Running scriptlet: kernel-core-5.14.0-391.el9.x86_64 2/4 2023-12-16T21:20:47.064 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : kernel-modules-5.14.0-391.el9.x86_64 3/4 2023-12-16T21:20:52.023 INFO:teuthology.orchestra.run.smithi045.stdout: Running scriptlet: kernel-modules-5.14.0-391.el9.x86_64 3/4 2023-12-16T21:20:52.240 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : kernel-5.14.0-391.el9.x86_64 4/4 2023-12-16T21:20:57.222 INFO:teuthology.orchestra.run.smithi045.stdout: Running scriptlet: kernel-modules-core-5.14.0-391.el9.x86_64 4/4 2023-12-16T21:21:15.328 INFO:teuthology.orchestra.run.smithi169.stdout: Running scriptlet: kernel-core-5.14.0-391.el9.x86_64 4/4 2023-12-16T21:21:15.372 INFO:teuthology.orchestra.run.smithi169.stdout: Running scriptlet: kernel-modules-5.14.0-391.el9.x86_64 4/4 2023-12-16T21:21:15.757 INFO:teuthology.orchestra.run.smithi169.stdout: Running scriptlet: kernel-5.14.0-391.el9.x86_64 4/4 2023-12-16T21:21:15.757 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : kernel-5.14.0-391.el9.x86_64 1/4 2023-12-16T21:21:15.757 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : kernel-core-5.14.0-391.el9.x86_64 2/4 2023-12-16T21:21:15.757 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : kernel-modules-5.14.0-391.el9.x86_64 3/4 2023-12-16T21:21:16.211 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : kernel-modules-core-5.14.0-391.el9.x86_64 4/4 2023-12-16T21:21:16.211 INFO:teuthology.orchestra.run.smithi169.stdout: 2023-12-16T21:21:16.211 INFO:teuthology.orchestra.run.smithi169.stdout:Installed: 2023-12-16T21:21:16.211 INFO:teuthology.orchestra.run.smithi169.stdout: kernel-5.14.0-391.el9.x86_64 2023-12-16T21:21:16.211 INFO:teuthology.orchestra.run.smithi169.stdout: kernel-core-5.14.0-391.el9.x86_64 2023-12-16T21:21:16.211 INFO:teuthology.orchestra.run.smithi169.stdout: kernel-modules-5.14.0-391.el9.x86_64 2023-12-16T21:21:16.211 INFO:teuthology.orchestra.run.smithi169.stdout: kernel-modules-core-5.14.0-391.el9.x86_64 2023-12-16T21:21:16.211 INFO:teuthology.orchestra.run.smithi169.stdout: 2023-12-16T21:21:16.211 INFO:teuthology.orchestra.run.smithi169.stdout:Complete! 2023-12-16T21:21:16.695 DEBUG:teuthology.orchestra.run.smithi169:> rpm -q kernel | sort -rV | head -n 1 2023-12-16T21:21:16.837 INFO:teuthology.orchestra.run.smithi169.stdout:kernel-5.14.0-391.el9.x86_64 2023-12-16T21:21:16.837 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-391.el9.x86_64 2023-12-16T21:21:16.837 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 5.14.0-361.el9.x86_64 Expected: 5.14.0-391.el9.x86_64 2023-12-16T21:21:16.838 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2023-12-16T21:21:16.838 DEBUG:teuthology.task.kernel:src is distro, skipping download 2023-12-16T21:21:16.838 INFO:teuthology.task.kernel:Installing distro kernel on host.b... 2023-12-16T21:21:16.838 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi169.front.sepia.ceph.com, path=None, version=distro) 2023-12-16T21:21:16.838 DEBUG:teuthology.orchestra.run.smithi169:> rpm -q kernel | sort -rV | head -n 1 2023-12-16T21:21:16.955 INFO:teuthology.orchestra.run.smithi169.stdout:kernel-5.14.0-391.el9.x86_64 2023-12-16T21:21:16.956 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-391.el9.x86_64 2023-12-16T21:21:16.956 DEBUG:teuthology.orchestra.run.smithi169:> sudo rpm -qi grub2-tools 2023-12-16T21:21:16.989 INFO:teuthology.orchestra.run.smithi169.stdout:Name : grub2-tools 2023-12-16T21:21:16.989 INFO:teuthology.orchestra.run.smithi169.stdout:Epoch : 1 2023-12-16T21:21:16.990 INFO:teuthology.orchestra.run.smithi169.stdout:Version : 2.06 2023-12-16T21:21:16.990 INFO:teuthology.orchestra.run.smithi169.stdout:Release : 68.el9 2023-12-16T21:21:16.990 INFO:teuthology.orchestra.run.smithi169.stdout:Architecture: x86_64 2023-12-16T21:21:16.990 INFO:teuthology.orchestra.run.smithi169.stdout:Install Date: Fri 01 Sep 2023 09:14:30 PM UTC 2023-12-16T21:21:16.990 INFO:teuthology.orchestra.run.smithi169.stdout:Group : Unspecified 2023-12-16T21:21:16.990 INFO:teuthology.orchestra.run.smithi169.stdout:Size : 8264254 2023-12-16T21:21:16.990 INFO:teuthology.orchestra.run.smithi169.stdout:License : GPLv3+ 2023-12-16T21:21:16.990 INFO:teuthology.orchestra.run.smithi169.stdout:Signature : RSA/SHA256, Mon 07 Aug 2023 01:57:16 PM UTC, Key ID 05b555b38483c65d 2023-12-16T21:21:16.990 INFO:teuthology.orchestra.run.smithi169.stdout:Source RPM : grub2-2.06-68.el9.src.rpm 2023-12-16T21:21:16.990 INFO:teuthology.orchestra.run.smithi169.stdout:Build Date : Wed 02 Aug 2023 03:15:48 PM UTC 2023-12-16T21:21:16.990 INFO:teuthology.orchestra.run.smithi169.stdout:Build Host : x86-05.stream.rdu2.redhat.com 2023-12-16T21:21:16.990 INFO:teuthology.orchestra.run.smithi169.stdout:Packager : builder@centos.org 2023-12-16T21:21:16.990 INFO:teuthology.orchestra.run.smithi169.stdout:Vendor : CentOS 2023-12-16T21:21:16.990 INFO:teuthology.orchestra.run.smithi169.stdout:URL : http://www.gnu.org/software/grub/ 2023-12-16T21:21:16.990 INFO:teuthology.orchestra.run.smithi169.stdout:Summary : Support tools for GRUB. 2023-12-16T21:21:16.991 INFO:teuthology.orchestra.run.smithi169.stdout:Description : 2023-12-16T21:21:16.991 INFO:teuthology.orchestra.run.smithi169.stdout: 2023-12-16T21:21:16.991 INFO:teuthology.orchestra.run.smithi169.stdout:The GRand Unified Bootloader (GRUB) is a highly configurable and 2023-12-16T21:21:16.991 INFO:teuthology.orchestra.run.smithi169.stdout:customizable bootloader with modular architecture. It supports a rich 2023-12-16T21:21:16.991 INFO:teuthology.orchestra.run.smithi169.stdout:variety of kernel formats, file systems, computer architectures and 2023-12-16T21:21:16.991 INFO:teuthology.orchestra.run.smithi169.stdout:hardware devices. 2023-12-16T21:21:16.991 INFO:teuthology.orchestra.run.smithi169.stdout: 2023-12-16T21:21:16.991 INFO:teuthology.orchestra.run.smithi169.stdout:This subpackage provides tools for support of all platforms. 2023-12-16T21:21:16.992 INFO:teuthology.task.kernel:Updating Grub Version: grub2 2023-12-16T21:21:16.992 INFO:teuthology.task.kernel:Updating grub on smithi169 to boot 5.14.0-391.el9.x86_64 2023-12-16T21:21:16.992 DEBUG:teuthology.orchestra.run.smithi169:> sudo grub2-mkconfig -o /boot/grub2/grub.cfg 2023-12-16T21:21:17.657 INFO:teuthology.orchestra.run.smithi169.stderr:Generating grub configuration file ... 2023-12-16T21:21:19.041 INFO:teuthology.orchestra.run.smithi169.stderr:Adding boot menu entry for UEFI Firmware Settings ... 2023-12-16T21:21:19.082 INFO:teuthology.orchestra.run.smithi169.stderr:done 2023-12-16T21:21:19.085 DEBUG:teuthology.orchestra.run.smithi169:> mktemp 2023-12-16T21:21:19.100 INFO:teuthology.orchestra.run.smithi169.stdout:/tmp/tmp.hwu1XX3Tak 2023-12-16T21:21:19.101 DEBUG:teuthology.orchestra.run.smithi169:> sudo cp /boot/grub2/grub.cfg /tmp/tmp.hwu1XX3Tak 2023-12-16T21:21:19.169 DEBUG:teuthology.orchestra.run.smithi169:> sudo chmod 0666 /tmp/tmp.hwu1XX3Tak 2023-12-16T21:21:19.307 DEBUG:teuthology.orchestra.remote:smithi169:/tmp/tmp.hwu1XX3Tak is 6KB 2023-12-16T21:21:19.357 DEBUG:teuthology.orchestra.run.smithi169:> rm -fr /tmp/tmp.hwu1XX3Tak 2023-12-16T21:21:19.371 DEBUG:teuthology.orchestra.run.smithi169:> sudo /bin/ls /boot/loader/entries || true 2023-12-16T21:21:19.436 INFO:teuthology.orchestra.run.smithi169.stdout:15949e3c88704300882482eddfd7baae-0-rescue.conf 2023-12-16T21:21:19.436 INFO:teuthology.orchestra.run.smithi169.stdout:15949e3c88704300882482eddfd7baae-5.14.0-239.el9.x86_64.conf 2023-12-16T21:21:19.436 INFO:teuthology.orchestra.run.smithi169.stdout:15949e3c88704300882482eddfd7baae-5.14.0-361.el9.x86_64.conf 2023-12-16T21:21:19.436 INFO:teuthology.orchestra.run.smithi169.stdout:15949e3c88704300882482eddfd7baae-5.14.0-391.el9.x86_64.conf 2023-12-16T21:21:19.437 DEBUG:teuthology.orchestra.run.smithi169:> sudo grub2-set-default 15949e3c88704300882482eddfd7baae-5.14.0-391.el9.x86_64 2023-12-16T21:21:19.577 DEBUG:teuthology.orchestra.run.smithi169:> sudo shutdown -r now 2023-12-16T21:21:27.664 INFO:teuthology.orchestra.run.smithi045.stdout: Running scriptlet: kernel-core-5.14.0-391.el9.x86_64 4/4 2023-12-16T21:21:27.709 INFO:teuthology.orchestra.run.smithi045.stdout: Running scriptlet: kernel-modules-5.14.0-391.el9.x86_64 4/4 2023-12-16T21:21:28.146 INFO:teuthology.orchestra.run.smithi045.stdout: Running scriptlet: kernel-5.14.0-391.el9.x86_64 4/4 2023-12-16T21:21:28.146 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : kernel-5.14.0-391.el9.x86_64 1/4 2023-12-16T21:21:28.146 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : kernel-core-5.14.0-391.el9.x86_64 2/4 2023-12-16T21:21:28.146 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : kernel-modules-5.14.0-391.el9.x86_64 3/4 2023-12-16T21:21:28.618 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : kernel-modules-core-5.14.0-391.el9.x86_64 4/4 2023-12-16T21:21:28.618 INFO:teuthology.orchestra.run.smithi045.stdout: 2023-12-16T21:21:28.619 INFO:teuthology.orchestra.run.smithi045.stdout:Installed: 2023-12-16T21:21:28.619 INFO:teuthology.orchestra.run.smithi045.stdout: kernel-5.14.0-391.el9.x86_64 2023-12-16T21:21:28.619 INFO:teuthology.orchestra.run.smithi045.stdout: kernel-core-5.14.0-391.el9.x86_64 2023-12-16T21:21:28.619 INFO:teuthology.orchestra.run.smithi045.stdout: kernel-modules-5.14.0-391.el9.x86_64 2023-12-16T21:21:28.619 INFO:teuthology.orchestra.run.smithi045.stdout: kernel-modules-core-5.14.0-391.el9.x86_64 2023-12-16T21:21:28.619 INFO:teuthology.orchestra.run.smithi045.stdout: 2023-12-16T21:21:28.619 INFO:teuthology.orchestra.run.smithi045.stdout:Complete! 2023-12-16T21:21:29.114 DEBUG:teuthology.orchestra.run.smithi045:> rpm -q kernel | sort -rV | head -n 1 2023-12-16T21:21:29.263 INFO:teuthology.orchestra.run.smithi045.stdout:kernel-5.14.0-391.el9.x86_64 2023-12-16T21:21:29.264 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-391.el9.x86_64 2023-12-16T21:21:29.264 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 5.14.0-361.el9.x86_64 Expected: 5.14.0-391.el9.x86_64 2023-12-16T21:21:29.264 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2023-12-16T21:21:29.264 DEBUG:teuthology.task.kernel:src is distro, skipping download 2023-12-16T21:21:29.264 INFO:teuthology.task.kernel:Installing distro kernel on host.a... 2023-12-16T21:21:29.265 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi045.front.sepia.ceph.com, path=None, version=distro) 2023-12-16T21:21:29.265 DEBUG:teuthology.orchestra.run.smithi045:> rpm -q kernel | sort -rV | head -n 1 2023-12-16T21:21:29.387 INFO:teuthology.orchestra.run.smithi045.stdout:kernel-5.14.0-391.el9.x86_64 2023-12-16T21:21:29.388 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-391.el9.x86_64 2023-12-16T21:21:29.388 DEBUG:teuthology.orchestra.run.smithi045:> sudo rpm -qi grub2-tools 2023-12-16T21:21:29.428 INFO:teuthology.orchestra.run.smithi045.stdout:Name : grub2-tools 2023-12-16T21:21:29.428 INFO:teuthology.orchestra.run.smithi045.stdout:Epoch : 1 2023-12-16T21:21:29.428 INFO:teuthology.orchestra.run.smithi045.stdout:Version : 2.06 2023-12-16T21:21:29.428 INFO:teuthology.orchestra.run.smithi045.stdout:Release : 68.el9 2023-12-16T21:21:29.429 INFO:teuthology.orchestra.run.smithi045.stdout:Architecture: x86_64 2023-12-16T21:21:29.429 INFO:teuthology.orchestra.run.smithi045.stdout:Install Date: Fri 01 Sep 2023 09:14:30 PM UTC 2023-12-16T21:21:29.429 INFO:teuthology.orchestra.run.smithi045.stdout:Group : Unspecified 2023-12-16T21:21:29.429 INFO:teuthology.orchestra.run.smithi045.stdout:Size : 8264254 2023-12-16T21:21:29.429 INFO:teuthology.orchestra.run.smithi045.stdout:License : GPLv3+ 2023-12-16T21:21:29.429 INFO:teuthology.orchestra.run.smithi045.stdout:Signature : RSA/SHA256, Mon 07 Aug 2023 01:57:16 PM UTC, Key ID 05b555b38483c65d 2023-12-16T21:21:29.429 INFO:teuthology.orchestra.run.smithi045.stdout:Source RPM : grub2-2.06-68.el9.src.rpm 2023-12-16T21:21:29.429 INFO:teuthology.orchestra.run.smithi045.stdout:Build Date : Wed 02 Aug 2023 03:15:48 PM UTC 2023-12-16T21:21:29.429 INFO:teuthology.orchestra.run.smithi045.stdout:Build Host : x86-05.stream.rdu2.redhat.com 2023-12-16T21:21:29.429 INFO:teuthology.orchestra.run.smithi045.stdout:Packager : builder@centos.org 2023-12-16T21:21:29.429 INFO:teuthology.orchestra.run.smithi045.stdout:Vendor : CentOS 2023-12-16T21:21:29.429 INFO:teuthology.orchestra.run.smithi045.stdout:URL : http://www.gnu.org/software/grub/ 2023-12-16T21:21:29.429 INFO:teuthology.orchestra.run.smithi045.stdout:Summary : Support tools for GRUB. 2023-12-16T21:21:29.429 INFO:teuthology.orchestra.run.smithi045.stdout:Description : 2023-12-16T21:21:29.430 INFO:teuthology.orchestra.run.smithi045.stdout: 2023-12-16T21:21:29.430 INFO:teuthology.orchestra.run.smithi045.stdout:The GRand Unified Bootloader (GRUB) is a highly configurable and 2023-12-16T21:21:29.430 INFO:teuthology.orchestra.run.smithi045.stdout:customizable bootloader with modular architecture. It supports a rich 2023-12-16T21:21:29.430 INFO:teuthology.orchestra.run.smithi045.stdout:variety of kernel formats, file systems, computer architectures and 2023-12-16T21:21:29.430 INFO:teuthology.orchestra.run.smithi045.stdout:hardware devices. 2023-12-16T21:21:29.430 INFO:teuthology.orchestra.run.smithi045.stdout: 2023-12-16T21:21:29.430 INFO:teuthology.orchestra.run.smithi045.stdout:This subpackage provides tools for support of all platforms. 2023-12-16T21:21:29.431 INFO:teuthology.task.kernel:Updating Grub Version: grub2 2023-12-16T21:21:29.431 INFO:teuthology.task.kernel:Updating grub on smithi045 to boot 5.14.0-391.el9.x86_64 2023-12-16T21:21:29.431 DEBUG:teuthology.orchestra.run.smithi045:> sudo grub2-mkconfig -o /boot/grub2/grub.cfg 2023-12-16T21:21:30.076 INFO:teuthology.orchestra.run.smithi045.stderr:Generating grub configuration file ... 2023-12-16T21:21:31.445 INFO:teuthology.orchestra.run.smithi045.stderr:Adding boot menu entry for UEFI Firmware Settings ... 2023-12-16T21:21:31.481 INFO:teuthology.orchestra.run.smithi045.stderr:done 2023-12-16T21:21:31.483 DEBUG:teuthology.orchestra.run.smithi045:> mktemp 2023-12-16T21:21:31.498 INFO:teuthology.orchestra.run.smithi045.stdout:/tmp/tmp.mSQX5qzeXk 2023-12-16T21:21:31.499 DEBUG:teuthology.orchestra.run.smithi045:> sudo cp /boot/grub2/grub.cfg /tmp/tmp.mSQX5qzeXk 2023-12-16T21:21:31.570 DEBUG:teuthology.orchestra.run.smithi045:> sudo chmod 0666 /tmp/tmp.mSQX5qzeXk 2023-12-16T21:21:31.714 DEBUG:teuthology.orchestra.remote:smithi045:/tmp/tmp.mSQX5qzeXk is 6KB 2023-12-16T21:21:31.762 DEBUG:teuthology.orchestra.run.smithi045:> rm -fr /tmp/tmp.mSQX5qzeXk 2023-12-16T21:21:31.777 DEBUG:teuthology.orchestra.run.smithi045:> sudo /bin/ls /boot/loader/entries || true 2023-12-16T21:21:31.845 INFO:teuthology.orchestra.run.smithi045.stdout:15949e3c88704300882482eddfd7baae-0-rescue.conf 2023-12-16T21:21:31.846 INFO:teuthology.orchestra.run.smithi045.stdout:15949e3c88704300882482eddfd7baae-5.14.0-239.el9.x86_64.conf 2023-12-16T21:21:31.846 INFO:teuthology.orchestra.run.smithi045.stdout:15949e3c88704300882482eddfd7baae-5.14.0-361.el9.x86_64.conf 2023-12-16T21:21:31.846 INFO:teuthology.orchestra.run.smithi045.stdout:15949e3c88704300882482eddfd7baae-5.14.0-391.el9.x86_64.conf 2023-12-16T21:21:31.847 DEBUG:teuthology.orchestra.run.smithi045:> sudo grub2-set-default 15949e3c88704300882482eddfd7baae-5.14.0-391.el9.x86_64 2023-12-16T21:21:32.017 DEBUG:teuthology.orchestra.run.smithi045:> sudo shutdown -r now 2023-12-16T21:21:49.596 INFO:teuthology.task.kernel:Checking client host.b for new kernel version... 2023-12-16T21:21:49.597 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi169.front.sepia.ceph.com' 2023-12-16T21:21:49.597 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi169.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-12-16T21:22:02.029 INFO:teuthology.task.kernel:Checking client host.a for new kernel version... 2023-12-16T21:22:02.030 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi045.front.sepia.ceph.com' 2023-12-16T21:22:02.030 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi045.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-12-16T21:22:08.021 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.169 2023-12-16T21:22:17.029 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi169.front.sepia.ceph.com' 2023-12-16T21:22:17.030 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi169.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-12-16T21:22:20.081 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.169 2023-12-16T21:22:20.561 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.45 2023-12-16T21:22:29.569 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi045.front.sepia.ceph.com' 2023-12-16T21:22:29.570 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi045.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-12-16T21:22:32.085 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi169.front.sepia.ceph.com' 2023-12-16T21:22:32.090 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi169.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-12-16T21:23:29.630 DEBUG:teuthology.orchestra.remote:timed out 2023-12-16T21:23:32.093 DEBUG:teuthology.orchestra.remote:timed out 2023-12-16T21:23:41.637 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi045.front.sepia.ceph.com' 2023-12-16T21:23:41.638 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi045.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-12-16T21:23:41.949 DEBUG:teuthology.orchestra.run.smithi045:> true 2023-12-16T21:23:42.328 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi045.front.sepia.ceph.com' 2023-12-16T21:23:42.329 INFO:teuthology.task.kernel:Checking kernel version of host.a, want "5.14.0-391.el9.x86_64"... 2023-12-16T21:23:42.329 DEBUG:teuthology.orchestra.run.smithi045:> uname -r 2023-12-16T21:23:42.344 INFO:teuthology.orchestra.run.smithi045.stdout:5.14.0-391.el9.x86_64 2023-12-16T21:23:42.344 DEBUG:teuthology.task.kernel:current kernel version is 5.14.0-391.el9.x86_64 vs 5.14.0-391.el9.x86_64 2023-12-16T21:23:42.345 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2023-12-16T21:23:42.345 DEBUG:teuthology.task.kernel:Distro of this test job: centos 2023-12-16T21:23:43.346 INFO:teuthology.task.kernel:Enabling kdb on host.a... 2023-12-16T21:23:43.346 DEBUG:teuthology.orchestra.run.smithi045:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2023-12-16T21:23:43.432 INFO:teuthology.orchestra.run.smithi045.stdout:ttyS1 2023-12-16T21:23:43.474 DEBUG:teuthology.parallel:result is None 2023-12-16T21:23:47.094 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi169.front.sepia.ceph.com' 2023-12-16T21:23:47.094 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi169.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-12-16T21:23:47.519 DEBUG:teuthology.orchestra.run.smithi169:> true 2023-12-16T21:23:47.908 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi169.front.sepia.ceph.com' 2023-12-16T21:23:47.908 INFO:teuthology.task.kernel:Checking kernel version of host.b, want "5.14.0-391.el9.x86_64"... 2023-12-16T21:23:47.908 DEBUG:teuthology.orchestra.run.smithi169:> uname -r 2023-12-16T21:23:47.924 INFO:teuthology.orchestra.run.smithi169.stdout:5.14.0-391.el9.x86_64 2023-12-16T21:23:47.924 DEBUG:teuthology.task.kernel:current kernel version is 5.14.0-391.el9.x86_64 vs 5.14.0-391.el9.x86_64 2023-12-16T21:23:47.924 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2023-12-16T21:23:47.924 DEBUG:teuthology.task.kernel:Distro of this test job: centos 2023-12-16T21:23:48.924 INFO:teuthology.task.kernel:Enabling kdb on host.b... 2023-12-16T21:23:48.925 DEBUG:teuthology.orchestra.run.smithi169:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2023-12-16T21:23:48.996 INFO:teuthology.orchestra.run.smithi169.stdout:ttyS1 2023-12-16T21:23:49.038 DEBUG:teuthology.parallel:result is None 2023-12-16T21:23:49.038 INFO:teuthology.run_tasks:Running task internal.base... 2023-12-16T21:23:49.046 INFO:teuthology.task.internal:Creating test directory... 2023-12-16T21:23:49.046 DEBUG:teuthology.orchestra.run.smithi045:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2023-12-16T21:23:49.049 DEBUG:teuthology.orchestra.run.smithi169:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2023-12-16T21:23:49.066 INFO:teuthology.run_tasks:Running task internal.archive_upload... 2023-12-16T21:23:49.073 INFO:teuthology.run_tasks:Running task internal.archive... 2023-12-16T21:23:49.085 INFO:teuthology.task.internal:Creating archive directory... 2023-12-16T21:23:49.085 DEBUG:teuthology.orchestra.run.smithi045:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2023-12-16T21:23:49.109 DEBUG:teuthology.orchestra.run.smithi169:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2023-12-16T21:23:49.138 INFO:teuthology.run_tasks:Running task internal.coredump... 2023-12-16T21:23:49.145 INFO:teuthology.task.internal:Enabling coredump saving... 2023-12-16T21:23:49.145 DEBUG:teuthology.orchestra.run.smithi045:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2023-12-16T21:23:49.171 DEBUG:teuthology.orchestra.run.smithi169:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2023-12-16T21:23:49.212 INFO:teuthology.orchestra.run.smithi169.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2023-12-16T21:23:49.215 INFO:teuthology.orchestra.run.smithi045.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2023-12-16T21:23:49.226 INFO:teuthology.orchestra.run.smithi169.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2023-12-16T21:23:49.232 INFO:teuthology.orchestra.run.smithi045.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2023-12-16T21:23:49.234 INFO:teuthology.run_tasks:Running task internal.sudo... 2023-12-16T21:23:49.241 INFO:teuthology.task.internal:Configuring sudo... 2023-12-16T21:23:49.241 DEBUG:teuthology.orchestra.run.smithi045:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2023-12-16T21:23:49.245 DEBUG:teuthology.orchestra.run.smithi169:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2023-12-16T21:23:49.296 INFO:teuthology.run_tasks:Running task internal.syslog... 2023-12-16T21:23:49.305 INFO:teuthology.task.internal.syslog:Starting syslog monitoring... 2023-12-16T21:23:49.305 DEBUG:teuthology.orchestra.run.smithi045:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2023-12-16T21:23:49.318 DEBUG:teuthology.orchestra.run.smithi169:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2023-12-16T21:23:49.352 DEBUG:teuthology.orchestra.run.smithi045:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2023-12-16T21:23:49.418 DEBUG:teuthology.orchestra.run.smithi045:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2023-12-16T21:23:49.469 DEBUG:teuthology.orchestra.run.smithi045:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2023-12-16T21:23:49.553 DEBUG:teuthology.orchestra.run.smithi045:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2023-12-16T21:23:49.599 DEBUG:teuthology.orchestra.run.smithi045:> set -ex 2023-12-16T21:23:49.599 DEBUG:teuthology.orchestra.run.smithi045:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2023-12-16T21:23:49.670 DEBUG:teuthology.orchestra.run.smithi169:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2023-12-16T21:23:49.711 DEBUG:teuthology.orchestra.run.smithi169:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2023-12-16T21:23:49.774 DEBUG:teuthology.orchestra.run.smithi169:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2023-12-16T21:23:49.816 DEBUG:teuthology.orchestra.run.smithi169:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2023-12-16T21:23:49.860 DEBUG:teuthology.orchestra.run.smithi169:> set -ex 2023-12-16T21:23:49.860 DEBUG:teuthology.orchestra.run.smithi169:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2023-12-16T21:23:49.928 DEBUG:teuthology.orchestra.run.smithi045:> sudo service rsyslog restart 2023-12-16T21:23:49.931 DEBUG:teuthology.orchestra.run.smithi169:> sudo service rsyslog restart 2023-12-16T21:23:49.975 INFO:teuthology.orchestra.run.smithi045.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2023-12-16T21:23:50.011 INFO:teuthology.orchestra.run.smithi169.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2023-12-16T21:23:50.345 INFO:teuthology.run_tasks:Running task internal.timer... 2023-12-16T21:23:50.353 INFO:teuthology.task.internal:Starting timer... 2023-12-16T21:23:50.353 INFO:teuthology.run_tasks:Running task pcp... 2023-12-16T21:23:50.364 INFO:teuthology.run_tasks:Running task selinux... 2023-12-16T21:23:50.372 DEBUG:teuthology.task:Applying overrides for task selinux: {'whitelist': ['scontext=system_u:system_r:logrotate_t:s0']} 2023-12-16T21:23:50.372 DEBUG:teuthology.orchestra.run.smithi045:> sudo service auditd rotate 2023-12-16T21:23:50.431 INFO:teuthology.orchestra.run.smithi045.stdout:Rotating logs: 2023-12-16T21:23:50.433 DEBUG:teuthology.orchestra.run.smithi169:> sudo service auditd rotate 2023-12-16T21:23:50.480 INFO:teuthology.orchestra.run.smithi169.stdout:Rotating logs: 2023-12-16T21:23:50.481 DEBUG:teuthology.task.selinux:Getting current SELinux state 2023-12-16T21:23:50.482 DEBUG:teuthology.orchestra.run.smithi045:> /usr/sbin/getenforce 2023-12-16T21:23:50.501 INFO:teuthology.orchestra.run.smithi045.stdout:Permissive 2023-12-16T21:23:50.501 DEBUG:teuthology.orchestra.run.smithi169:> /usr/sbin/getenforce 2023-12-16T21:23:50.536 INFO:teuthology.orchestra.run.smithi169.stdout:Permissive 2023-12-16T21:23:50.536 DEBUG:teuthology.task.selinux:Existing SELinux modes: {'ubuntu@smithi045.front.sepia.ceph.com': 'permissive', 'ubuntu@smithi169.front.sepia.ceph.com': 'permissive'} 2023-12-16T21:23:50.537 DEBUG:teuthology.orchestra.run.smithi045:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2023-12-16T21:23:50.574 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-12-16T21:23:50.574 DEBUG:teuthology.orchestra.run.smithi169:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2023-12-16T21:23:50.604 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-12-16T21:23:50.604 INFO:teuthology.task.selinux:Putting SELinux into permissive mode 2023-12-16T21:23:50.604 DEBUG:teuthology.orchestra.run.smithi045:> sudo /usr/sbin/setenforce permissive 2023-12-16T21:23:50.661 DEBUG:teuthology.orchestra.run.smithi169:> sudo /usr/sbin/setenforce permissive 2023-12-16T21:23:50.697 INFO:teuthology.run_tasks:Running task ansible.cephlab... 2023-12-16T21:23:50.707 DEBUG:teuthology.repo_utils:Setting repo remote to https://git.ceph.com/ceph-cm-ansible.git 2023-12-16T21:23:50.712 INFO:teuthology.repo_utils:Fetching main from origin 2023-12-16T21:23:50.835 INFO:teuthology.repo_utils:Resetting repo at /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main to origin/main 2023-12-16T21:23:50.857 INFO:teuthology.task.ansible:Playbook: [{'import_playbook': 'ansible_managed.yml'}, {'import_playbook': 'teuthology.yml'}, {'hosts': 'testnodes', 'tasks': [{'set_fact': {'ran_from_cephlab_playbook': True}}]}, {'import_playbook': 'testnodes.yml'}, {'import_playbook': 'container-host.yml'}, {'import_playbook': 'cobbler.yml'}, {'import_playbook': 'paddles.yml'}, {'import_playbook': 'pulpito.yml'}, {'hosts': 'testnodes', 'become': True, 'tasks': [{'name': 'Touch /ceph-qa-ready', 'file': {'path': '/ceph-qa-ready', 'state': 'touch'}, 'when': 'ran_from_cephlab_playbook|bool'}]}] 2023-12-16T21:23:50.858 DEBUG:teuthology.task.ansible:Running ansible-playbook -v --extra-vars '{"ansible_ssh_user": "ubuntu"}' -i /etc/ansible/hosts --limit smithi045.front.sepia.ceph.com,smithi169.front.sepia.ceph.com /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main/cephlab.yml 2023-12-16T21:29:39.715 DEBUG:teuthology.task.ansible:Reconnecting to [Remote(name='ubuntu@smithi045.front.sepia.ceph.com'), Remote(name='ubuntu@smithi169.front.sepia.ceph.com')] 2023-12-16T21:29:39.716 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi045.front.sepia.ceph.com' 2023-12-16T21:29:39.717 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi045.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-12-16T21:29:39.813 DEBUG:teuthology.orchestra.run.smithi045:> true 2023-12-16T21:29:39.884 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi045.front.sepia.ceph.com' 2023-12-16T21:29:39.885 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi169.front.sepia.ceph.com' 2023-12-16T21:29:39.885 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi169.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-12-16T21:29:39.967 DEBUG:teuthology.orchestra.run.smithi169:> true 2023-12-16T21:29:40.035 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi169.front.sepia.ceph.com' 2023-12-16T21:29:40.036 INFO:teuthology.run_tasks:Running task clock... 2023-12-16T21:29:40.047 INFO:teuthology.task.clock:Syncing clocks and checking initial clock skew... 2023-12-16T21:29:40.047 INFO:teuthology.orchestra.run:Running command with timeout 360 2023-12-16T21:29:40.047 DEBUG:teuthology.orchestra.run.smithi045:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2023-12-16T21:29:40.050 INFO:teuthology.orchestra.run:Running command with timeout 360 2023-12-16T21:29:40.050 DEBUG:teuthology.orchestra.run.smithi169:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2023-12-16T21:29:40.083 INFO:teuthology.orchestra.run.smithi045.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2023-12-16T21:29:40.104 INFO:teuthology.orchestra.run.smithi045.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2023-12-16T21:29:40.105 INFO:teuthology.orchestra.run.smithi169.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2023-12-16T21:29:40.122 INFO:teuthology.orchestra.run.smithi169.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2023-12-16T21:29:40.140 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: ntpd: command not found 2023-12-16T21:29:40.151 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: ntpd: command not found 2023-12-16T21:29:40.157 INFO:teuthology.orchestra.run.smithi045.stdout:506 Cannot talk to daemon 2023-12-16T21:29:40.164 INFO:teuthology.orchestra.run.smithi169.stdout:506 Cannot talk to daemon 2023-12-16T21:29:40.177 INFO:teuthology.orchestra.run.smithi045.stderr:Failed to start ntp.service: Unit ntp.service not found. 2023-12-16T21:29:40.182 INFO:teuthology.orchestra.run.smithi169.stderr:Failed to start ntp.service: Unit ntp.service not found. 2023-12-16T21:29:40.197 INFO:teuthology.orchestra.run.smithi045.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2023-12-16T21:29:40.200 INFO:teuthology.orchestra.run.smithi169.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2023-12-16T21:29:40.244 INFO:teuthology.orchestra.run.smithi045.stderr:bash: line 1: ntpq: command not found 2023-12-16T21:29:40.246 INFO:teuthology.orchestra.run.smithi169.stderr:bash: line 1: ntpq: command not found 2023-12-16T21:29:40.247 INFO:teuthology.orchestra.run.smithi045.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2023-12-16T21:29:40.247 INFO:teuthology.orchestra.run.smithi045.stdout:=============================================================================== 2023-12-16T21:29:40.247 INFO:teuthology.orchestra.run.smithi045.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2023-12-16T21:29:40.247 INFO:teuthology.orchestra.run.smithi045.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2023-12-16T21:29:40.247 INFO:teuthology.orchestra.run.smithi045.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2023-12-16T21:29:40.248 INFO:teuthology.orchestra.run.smithi045.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2023-12-16T21:29:40.249 INFO:teuthology.orchestra.run.smithi169.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2023-12-16T21:29:40.249 INFO:teuthology.orchestra.run.smithi169.stdout:=============================================================================== 2023-12-16T21:29:40.249 INFO:teuthology.orchestra.run.smithi169.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2023-12-16T21:29:40.249 INFO:teuthology.orchestra.run.smithi169.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2023-12-16T21:29:40.249 INFO:teuthology.orchestra.run.smithi169.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2023-12-16T21:29:40.249 INFO:teuthology.orchestra.run.smithi169.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2023-12-16T21:29:40.250 INFO:teuthology.run_tasks:Running task install... 2023-12-16T21:29:40.258 DEBUG:teuthology.task.install:project ceph 2023-12-16T21:29:40.259 DEBUG:teuthology.task.install:INSTALL overrides: {'ceph': {'flavor': 'default', 'sha1': 'c7586e95519b7be76f3b6e3a4b333888baf46bc9'}} 2023-12-16T21:29:40.259 DEBUG:teuthology.task.install:config {'branch': 'quincy', 'exclude_packages': ['ceph-volume'], 'flavor': 'default', 'sha1': 'c7586e95519b7be76f3b6e3a4b333888baf46bc9'} 2023-12-16T21:29:40.259 INFO:teuthology.task.install:Using flavor: default 2023-12-16T21:29:40.264 DEBUG:teuthology.task.install:Package list is: {'deb': ['ceph', 'cephadm', 'ceph-mds', 'ceph-mgr', 'ceph-common', 'ceph-fuse', 'ceph-test', 'radosgw', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'libcephfs2', 'libcephfs-dev', 'librados2', 'librbd1', 'rbd-fuse'], 'rpm': ['ceph-radosgw', 'ceph-test', 'ceph', 'ceph-base', 'cephadm', 'ceph-immutable-object-cache', 'ceph-mgr', 'ceph-mgr-dashboard', 'ceph-mgr-diskprediction-local', 'ceph-mgr-rook', 'ceph-mgr-cephadm', 'ceph-fuse', 'librados-devel', 'libcephfs2', 'libcephfs-devel', 'librados2', 'librbd1', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'rbd-fuse', 'rbd-mirror', 'rbd-nbd']} 2023-12-16T21:29:40.265 INFO:teuthology.task.install:extra packages: [] 2023-12-16T21:29:40.265 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': 'quincy', 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': ['ceph-volume'], 'extra_packages': [], 'extra_system_packages': [], 'extras': None, 'enable_coprs': [], 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': 'c7586e95519b7be76f3b6e3a4b333888baf46bc9', 'tag': None, 'wait_for_package': False} 2023-12-16T21:29:40.265 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2023-12-16T21:29:40.265 INFO:teuthology.packaging:ref: None 2023-12-16T21:29:40.265 INFO:teuthology.packaging:tag: None 2023-12-16T21:29:40.265 INFO:teuthology.packaging:branch: quincy 2023-12-16T21:29:40.266 INFO:teuthology.packaging:sha1: c7586e95519b7be76f3b6e3a4b333888baf46bc9 2023-12-16T21:29:40.266 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&ref=quincy 2023-12-16T21:29:40.268 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': 'quincy', 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': ['ceph-volume'], 'extra_packages': [], 'extra_system_packages': [], 'extras': None, 'enable_coprs': [], 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': 'c7586e95519b7be76f3b6e3a4b333888baf46bc9', 'tag': None, 'wait_for_package': False} 2023-12-16T21:29:40.268 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2023-12-16T21:29:40.268 INFO:teuthology.packaging:ref: None 2023-12-16T21:29:40.268 INFO:teuthology.packaging:tag: None 2023-12-16T21:29:40.268 INFO:teuthology.packaging:branch: quincy 2023-12-16T21:29:40.268 INFO:teuthology.packaging:sha1: c7586e95519b7be76f3b6e3a4b333888baf46bc9 2023-12-16T21:29:40.269 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&ref=quincy 2023-12-16T21:29:40.475 INFO:teuthology.task.install.rpm:Pulling from https://2.chacra.ceph.com/r/ceph/quincy/1f2f55109dc1ed2dc2dda9de1db3c70923ba0dfd/centos/9/flavors/default/ 2023-12-16T21:29:40.475 INFO:teuthology.task.install.rpm:Package version is 17.2.7-125.g1f2f5510 2023-12-16T21:29:40.623 INFO:teuthology.task.install.rpm:Pulling from https://2.chacra.ceph.com/r/ceph/quincy/1f2f55109dc1ed2dc2dda9de1db3c70923ba0dfd/centos/9/flavors/default/ 2023-12-16T21:29:40.623 INFO:teuthology.task.install.rpm:Package version is 17.2.7-125.g1f2f5510 2023-12-16T21:29:40.728 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://2.chacra.ceph.com/r/ceph/quincy/1f2f55109dc1ed2dc2dda9de1db3c70923ba0dfd/centos/9/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://2.chacra.ceph.com/r/ceph/quincy/1f2f55109dc1ed2dc2dda9de1db3c70923ba0dfd/centos/9/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://2.chacra.ceph.com/r/ceph/quincy/1f2f55109dc1ed2dc2dda9de1db3c70923ba0dfd/centos/9/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2023-12-16T21:29:40.728 DEBUG:teuthology.orchestra.run.smithi045:> set -ex 2023-12-16T21:29:40.728 DEBUG:teuthology.orchestra.run.smithi045:> sudo dd of=/etc/yum.repos.d/ceph.repo 2023-12-16T21:29:40.731 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://2.chacra.ceph.com/r/ceph/quincy/1f2f55109dc1ed2dc2dda9de1db3c70923ba0dfd/centos/9/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://2.chacra.ceph.com/r/ceph/quincy/1f2f55109dc1ed2dc2dda9de1db3c70923ba0dfd/centos/9/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://2.chacra.ceph.com/r/ceph/quincy/1f2f55109dc1ed2dc2dda9de1db3c70923ba0dfd/centos/9/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2023-12-16T21:29:40.731 DEBUG:teuthology.orchestra.run.smithi169:> set -ex 2023-12-16T21:29:40.731 DEBUG:teuthology.orchestra.run.smithi169:> sudo dd of=/etc/yum.repos.d/ceph.repo 2023-12-16T21:29:40.762 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd on remote rpm x86_64 2023-12-16T21:29:40.762 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2023-12-16T21:29:40.762 INFO:teuthology.packaging:ref: None 2023-12-16T21:29:40.762 INFO:teuthology.packaging:tag: None 2023-12-16T21:29:40.762 INFO:teuthology.packaging:branch: quincy 2023-12-16T21:29:40.762 INFO:teuthology.packaging:sha1: c7586e95519b7be76f3b6e3a4b333888baf46bc9 2023-12-16T21:29:40.762 DEBUG:teuthology.orchestra.run.smithi169:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;ref/quincy/;g' /etc/yum.repos.d/ceph.repo ; fi 2023-12-16T21:29:40.763 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd on remote rpm x86_64 2023-12-16T21:29:40.763 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2023-12-16T21:29:40.763 INFO:teuthology.packaging:ref: None 2023-12-16T21:29:40.763 INFO:teuthology.packaging:tag: None 2023-12-16T21:29:40.763 INFO:teuthology.packaging:branch: quincy 2023-12-16T21:29:40.763 INFO:teuthology.packaging:sha1: c7586e95519b7be76f3b6e3a4b333888baf46bc9 2023-12-16T21:29:40.764 DEBUG:teuthology.orchestra.run.smithi045:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;ref/quincy/;g' /etc/yum.repos.d/ceph.repo ; fi 2023-12-16T21:29:40.831 DEBUG:teuthology.orchestra.run.smithi169:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2023-12-16T21:29:40.834 DEBUG:teuthology.orchestra.run.smithi045:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2023-12-16T21:29:40.913 DEBUG:teuthology.orchestra.run.smithi169:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2023-12-16T21:29:40.925 DEBUG:teuthology.orchestra.run.smithi045:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2023-12-16T21:29:40.956 INFO:teuthology.orchestra.run.smithi045.stdout:check_obsoletes = 1 2023-12-16T21:29:40.957 DEBUG:teuthology.orchestra.run.smithi045:> sudo yum clean all 2023-12-16T21:29:40.980 INFO:teuthology.orchestra.run.smithi169.stdout:check_obsoletes = 1 2023-12-16T21:29:40.981 DEBUG:teuthology.orchestra.run.smithi169:> sudo yum clean all 2023-12-16T21:29:41.225 INFO:teuthology.orchestra.run.smithi169.stdout:73 files removed 2023-12-16T21:29:41.232 INFO:teuthology.orchestra.run.smithi045.stdout:73 files removed 2023-12-16T21:29:41.252 DEBUG:teuthology.orchestra.run.smithi169:> sudo yum -y install ceph-radosgw 2023-12-16T21:29:41.258 DEBUG:teuthology.orchestra.run.smithi045:> sudo yum -y install ceph-radosgw 2023-12-16T21:29:41.975 INFO:teuthology.orchestra.run.smithi169.stdout:ceph packages for x86_64 191 kB/s | 77 kB 00:00 2023-12-16T21:29:42.011 INFO:teuthology.orchestra.run.smithi045.stdout:ceph packages for x86_64 185 kB/s | 77 kB 00:00 2023-12-16T21:29:42.404 INFO:teuthology.orchestra.run.smithi045.stdout:ceph noarch packages 27 kB/s | 9.1 kB 00:00 2023-12-16T21:29:42.409 INFO:teuthology.orchestra.run.smithi169.stdout:ceph noarch packages 24 kB/s | 9.1 kB 00:00 2023-12-16T21:29:42.738 INFO:teuthology.orchestra.run.smithi045.stdout:ceph source packages 6.1 kB/s | 1.8 kB 00:00 2023-12-16T21:29:42.843 INFO:teuthology.orchestra.run.smithi169.stdout:ceph source packages 4.5 kB/s | 1.8 kB 00:00 2023-12-16T21:29:43.132 INFO:teuthology.orchestra.run.smithi045.stdout:CentOS-9 - AppStream 18 MB/s | 6.4 MB 00:00 2023-12-16T21:29:43.279 INFO:teuthology.orchestra.run.smithi169.stdout:CentOS-9 - AppStream 16 MB/s | 6.4 MB 00:00 2023-12-16T21:29:44.507 INFO:teuthology.orchestra.run.smithi045.stdout:CentOS-9 - BaseOS 7.0 MB/s | 2.0 MB 00:00 2023-12-16T21:29:44.604 INFO:teuthology.orchestra.run.smithi169.stdout:CentOS-9 - BaseOS 7.6 MB/s | 2.0 MB 00:00 2023-12-16T21:29:45.495 INFO:teuthology.orchestra.run.smithi169.stdout:CentOS Stream 9 - BaseOS 14 MB/s | 7.9 MB 00:00 2023-12-16T21:29:46.309 INFO:teuthology.orchestra.run.smithi045.stdout:CentOS Stream 9 - BaseOS 5.4 MB/s | 7.9 MB 00:01 2023-12-16T21:29:49.823 INFO:teuthology.orchestra.run.smithi045.stdout:CentOS Stream 9 - AppStream 7.6 MB/s | 18 MB 00:02 2023-12-16T21:29:53.816 INFO:teuthology.orchestra.run.smithi045.stdout:CentOS Stream 9 - CRB 13 MB/s | 5.8 MB 00:00 2023-12-16T21:29:55.250 INFO:teuthology.orchestra.run.smithi045.stdout:CentOS Stream 9 - Extras packages 53 kB/s | 15 kB 00:00 2023-12-16T21:29:58.377 INFO:teuthology.orchestra.run.smithi045.stdout:Extra Packages for Enterprise Linux 6.6 MB/s | 20 MB 00:03 2023-12-16T21:30:01.878 INFO:teuthology.orchestra.run.smithi169.stdout:CentOS Stream 9 - AppStream 1.2 MB/s | 18 MB 00:15 2023-12-16T21:30:03.332 INFO:teuthology.orchestra.run.smithi045.stdout:lab-extras 27 kB/s | 1.7 kB 00:00 2023-12-16T21:30:05.754 INFO:teuthology.orchestra.run.smithi169.stdout:CentOS Stream 9 - CRB 11 MB/s | 5.8 MB 00:00 2023-12-16T21:30:05.903 INFO:teuthology.orchestra.run.smithi045.stdout:Dependencies resolved. 2023-12-16T21:30:05.916 INFO:teuthology.orchestra.run.smithi045.stdout:================================================================================ 2023-12-16T21:30:05.916 INFO:teuthology.orchestra.run.smithi045.stdout: Package Arch Version Repository Size 2023-12-16T21:30:05.916 INFO:teuthology.orchestra.run.smithi045.stdout:================================================================================ 2023-12-16T21:30:05.916 INFO:teuthology.orchestra.run.smithi045.stdout:Installing: 2023-12-16T21:30:05.916 INFO:teuthology.orchestra.run.smithi045.stdout: ceph-radosgw x86_64 2:17.2.7-125.g1f2f5510.el9 ceph 14 M 2023-12-16T21:30:05.916 INFO:teuthology.orchestra.run.smithi045.stdout:Upgrading: 2023-12-16T21:30:05.917 INFO:teuthology.orchestra.run.smithi045.stdout: librados2 x86_64 2:17.2.7-125.g1f2f5510.el9 ceph 3.1 M 2023-12-16T21:30:05.917 INFO:teuthology.orchestra.run.smithi045.stdout: librbd1 x86_64 2:17.2.7-125.g1f2f5510.el9 ceph 2.9 M 2023-12-16T21:30:05.917 INFO:teuthology.orchestra.run.smithi045.stdout:Installing dependencies: 2023-12-16T21:30:05.917 INFO:teuthology.orchestra.run.smithi045.stdout: ceph-base x86_64 2:17.2.7-125.g1f2f5510.el9 ceph 5.0 M 2023-12-16T21:30:05.917 INFO:teuthology.orchestra.run.smithi045.stdout: ceph-common x86_64 2:17.2.7-125.g1f2f5510.el9 ceph 23 M 2023-12-16T21:30:05.917 INFO:teuthology.orchestra.run.smithi045.stdout: ceph-selinux x86_64 2:17.2.7-125.g1f2f5510.el9 ceph 24 k 2023-12-16T21:30:05.917 INFO:teuthology.orchestra.run.smithi045.stdout: gperftools-libs x86_64 2.9.1-2.el9 epel 309 k 2023-12-16T21:30:05.917 INFO:teuthology.orchestra.run.smithi045.stdout: libcephfs2 x86_64 2:17.2.7-125.g1f2f5510.el9 ceph 636 k 2023-12-16T21:30:05.917 INFO:teuthology.orchestra.run.smithi045.stdout: liboath x86_64 2.6.7-2.el9 epel 49 k 2023-12-16T21:30:05.917 INFO:teuthology.orchestra.run.smithi045.stdout: libpmemobj x86_64 1.12.1-1.el9 CentOS-AppStream 159 k 2023-12-16T21:30:05.917 INFO:teuthology.orchestra.run.smithi045.stdout: librabbitmq x86_64 0.11.0-7.el9 CentOS-AppStream 44 k 2023-12-16T21:30:05.917 INFO:teuthology.orchestra.run.smithi045.stdout: libradosstriper1 x86_64 2:17.2.7-125.g1f2f5510.el9 ceph 458 k 2023-12-16T21:30:05.918 INFO:teuthology.orchestra.run.smithi045.stdout: librdkafka x86_64 1.6.1-102.el9 CentOS-AppStream 661 k 2023-12-16T21:30:05.918 INFO:teuthology.orchestra.run.smithi045.stdout: librgw2 x86_64 2:17.2.7-125.g1f2f5510.el9 ceph 4.9 M 2023-12-16T21:30:05.918 INFO:teuthology.orchestra.run.smithi045.stdout: libunwind x86_64 1.6.2-1.el9 epel 67 k 2023-12-16T21:30:05.918 INFO:teuthology.orchestra.run.smithi045.stdout: lttng-ust x86_64 2.12.0-6.el9 CentOS-AppStream 282 k 2023-12-16T21:30:05.918 INFO:teuthology.orchestra.run.smithi045.stdout: python3-ceph-argparse x86_64 2:17.2.7-125.g1f2f5510.el9 ceph 45 k 2023-12-16T21:30:05.918 INFO:teuthology.orchestra.run.smithi045.stdout: python3-ceph-common x86_64 2:17.2.7-125.g1f2f5510.el9 ceph 116 k 2023-12-16T21:30:05.918 INFO:teuthology.orchestra.run.smithi045.stdout: python3-cephfs x86_64 2:17.2.7-125.g1f2f5510.el9 ceph 156 k 2023-12-16T21:30:05.918 INFO:teuthology.orchestra.run.smithi045.stdout: python3-prettytable noarch 0.7.2-27.el9 CentOS-AppStream 41 k 2023-12-16T21:30:05.918 INFO:teuthology.orchestra.run.smithi045.stdout: python3-rados x86_64 2:17.2.7-125.g1f2f5510.el9 ceph 320 k 2023-12-16T21:30:05.918 INFO:teuthology.orchestra.run.smithi045.stdout: python3-rbd x86_64 2:17.2.7-125.g1f2f5510.el9 ceph 292 k 2023-12-16T21:30:05.918 INFO:teuthology.orchestra.run.smithi045.stdout: python3-rgw x86_64 2:17.2.7-125.g1f2f5510.el9 ceph 99 k 2023-12-16T21:30:05.918 INFO:teuthology.orchestra.run.smithi045.stdout: thrift x86_64 0.15.0-2.el9 epel 1.6 M 2023-12-16T21:30:05.918 INFO:teuthology.orchestra.run.smithi045.stdout: 2023-12-16T21:30:05.919 INFO:teuthology.orchestra.run.smithi045.stdout:Transaction Summary 2023-12-16T21:30:05.919 INFO:teuthology.orchestra.run.smithi045.stdout:================================================================================ 2023-12-16T21:30:05.919 INFO:teuthology.orchestra.run.smithi045.stdout:Install 22 Packages 2023-12-16T21:30:05.919 INFO:teuthology.orchestra.run.smithi045.stdout:Upgrade 2 Packages 2023-12-16T21:30:05.919 INFO:teuthology.orchestra.run.smithi045.stdout: 2023-12-16T21:30:05.919 INFO:teuthology.orchestra.run.smithi045.stdout:Total download size: 58 M 2023-12-16T21:30:05.919 INFO:teuthology.orchestra.run.smithi045.stdout:Downloading Packages: 2023-12-16T21:30:07.538 INFO:teuthology.orchestra.run.smithi169.stdout:CentOS Stream 9 - Extras packages 34 kB/s | 15 kB 00:00 2023-12-16T21:30:07.709 INFO:teuthology.orchestra.run.smithi045.stdout:(1/24): ceph-common-17.2.7-125.g1f2f5510.el9.x8 13 MB/s | 23 MB 00:01 2023-12-16T21:30:07.867 INFO:teuthology.orchestra.run.smithi045.stdout:(2/24): ceph-radosgw-17.2.7-125.g1f2f5510.el9.x 7.5 MB/s | 14 MB 00:01 2023-12-16T21:30:07.892 INFO:teuthology.orchestra.run.smithi045.stdout:(3/24): ceph-selinux-17.2.7-125.g1f2f5510.el9.x 131 kB/s | 24 kB 00:00 2023-12-16T21:30:07.926 INFO:teuthology.orchestra.run.smithi045.stdout:(4/24): libcephfs2-17.2.7-125.g1f2f5510.el9.x86 11 MB/s | 636 kB 00:00 2023-12-16T21:30:07.960 INFO:teuthology.orchestra.run.smithi045.stdout:(5/24): libradosstriper1-17.2.7-125.g1f2f5510.e 6.7 MB/s | 458 kB 00:00 2023-12-16T21:30:08.018 INFO:teuthology.orchestra.run.smithi045.stdout:(6/24): python3-ceph-argparse-17.2.7-125.g1f2f5 776 kB/s | 45 kB 00:00 2023-12-16T21:30:08.102 INFO:teuthology.orchestra.run.smithi045.stdout:(7/24): ceph-base-17.2.7-125.g1f2f5510.el9.x86_ 2.4 MB/s | 5.0 MB 00:02 2023-12-16T21:30:08.127 INFO:teuthology.orchestra.run.smithi045.stdout:(8/24): python3-ceph-common-17.2.7-125.g1f2f551 1.0 MB/s | 116 kB 00:00 2023-12-16T21:30:08.178 INFO:teuthology.orchestra.run.smithi045.stdout:(9/24): python3-cephfs-17.2.7-125.g1f2f5510.el9 2.0 MB/s | 156 kB 00:00 2023-12-16T21:30:08.203 INFO:teuthology.orchestra.run.smithi045.stdout:(10/24): python3-rados-17.2.7-125.g1f2f5510.el9 4.1 MB/s | 320 kB 00:00 2023-12-16T21:30:08.287 INFO:teuthology.orchestra.run.smithi045.stdout:(11/24): librgw2-17.2.7-125.g1f2f5510.el9.x86_6 14 MB/s | 4.9 MB 00:00 2023-12-16T21:30:08.312 INFO:teuthology.orchestra.run.smithi045.stdout:(12/24): python3-rgw-17.2.7-125.g1f2f5510.el9.x 916 kB/s | 99 kB 00:00 2023-12-16T21:30:08.362 INFO:teuthology.orchestra.run.smithi045.stdout:(13/24): python3-rbd-17.2.7-125.g1f2f5510.el9.x 1.5 MB/s | 292 kB 00:00 2023-12-16T21:30:08.429 INFO:teuthology.orchestra.run.smithi045.stdout:(14/24): libpmemobj-1.12.1-1.el9.x86_64.rpm 1.1 MB/s | 159 kB 00:00 2023-12-16T21:30:08.446 INFO:teuthology.orchestra.run.smithi045.stdout:(15/24): librabbitmq-0.11.0-7.el9.x86_64.rpm 415 kB/s | 44 kB 00:00 2023-12-16T21:30:08.513 INFO:teuthology.orchestra.run.smithi045.stdout:(16/24): librdkafka-1.6.1-102.el9.x86_64.rpm 5.3 MB/s | 661 kB 00:00 2023-12-16T21:30:08.538 INFO:teuthology.orchestra.run.smithi045.stdout:(17/24): lttng-ust-2.12.0-6.el9.x86_64.rpm 2.5 MB/s | 282 kB 00:00 2023-12-16T21:30:08.555 INFO:teuthology.orchestra.run.smithi045.stdout:(18/24): python3-prettytable-0.7.2-27.el9.noarc 380 kB/s | 41 kB 00:00 2023-12-16T21:30:09.000 INFO:teuthology.orchestra.run.smithi169.stdout:Extra Packages for Enterprise Linux 15 MB/s | 20 MB 00:01 2023-12-16T21:30:09.013 INFO:teuthology.orchestra.run.smithi045.stdout:(19/24): liboath-2.6.7-2.el9.x86_64.rpm 106 kB/s | 49 kB 00:00 2023-12-16T21:30:09.064 INFO:teuthology.orchestra.run.smithi045.stdout:(20/24): libunwind-1.6.2-1.el9.x86_64.rpm 132 kB/s | 67 kB 00:00 2023-12-16T21:30:09.197 INFO:teuthology.orchestra.run.smithi045.stdout:(21/24): gperftools-libs-2.9.1-2.el9.x86_64.rpm 452 kB/s | 309 kB 00:00 2023-12-16T21:30:09.406 INFO:teuthology.orchestra.run.smithi045.stdout:(22/24): thrift-0.15.0-2.el9.x86_64.rpm 4.0 MB/s | 1.6 MB 00:00 2023-12-16T21:30:09.448 INFO:teuthology.orchestra.run.smithi045.stdout:(23/24): librbd1-17.2.7-125.g1f2f5510.el9.x86_6 12 MB/s | 2.9 MB 00:00 2023-12-16T21:30:09.615 INFO:teuthology.orchestra.run.smithi045.stdout:(24/24): librados2-17.2.7-125.g1f2f5510.el9.x86 5.6 MB/s | 3.1 MB 00:00 2023-12-16T21:30:09.622 INFO:teuthology.orchestra.run.smithi045.stdout:-------------------------------------------------------------------------------- 2023-12-16T21:30:09.622 INFO:teuthology.orchestra.run.smithi045.stdout:Total 16 MB/s | 58 MB 00:03 2023-12-16T21:30:09.622 INFO:teuthology.orchestra.run.smithi045.stdout:Running transaction check 2023-12-16T21:30:09.660 INFO:teuthology.orchestra.run.smithi045.stdout:Transaction check succeeded. 2023-12-16T21:30:09.660 INFO:teuthology.orchestra.run.smithi045.stdout:Running transaction test 2023-12-16T21:30:10.142 INFO:teuthology.orchestra.run.smithi045.stdout:Transaction test succeeded. 2023-12-16T21:30:10.142 INFO:teuthology.orchestra.run.smithi045.stdout:Running transaction 2023-12-16T21:30:11.518 INFO:teuthology.orchestra.run.smithi045.stdout: Preparing : 1/1 2023-12-16T21:30:11.885 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : lttng-ust-2.12.0-6.el9.x86_64 1/26 2023-12-16T21:30:11.933 INFO:teuthology.orchestra.run.smithi045.stdout: Upgrading : librados2-2:17.2.7-125.g1f2f5510.el9.x86_64 2/26 2023-12-16T21:30:12.026 INFO:teuthology.orchestra.run.smithi045.stdout: Running scriptlet: librados2-2:17.2.7-125.g1f2f5510.el9.x86_64 2/26 2023-12-16T21:30:12.108 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : python3-rados-2:17.2.7-125.g1f2f5510.el9.x86_64 3/26 2023-12-16T21:30:12.135 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : libcephfs2-2:17.2.7-125.g1f2f5510.el9.x86_64 4/26 2023-12-16T21:30:12.215 INFO:teuthology.orchestra.run.smithi045.stdout: Running scriptlet: libcephfs2-2:17.2.7-125.g1f2f5510.el9.x86_64 4/26 2023-12-16T21:30:12.264 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : thrift-0.15.0-2.el9.x86_64 5/26 2023-12-16T21:30:12.303 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : librdkafka-1.6.1-102.el9.x86_64 6/26 2023-12-16T21:30:12.773 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : librabbitmq-0.11.0-7.el9.x86_64 7/26 2023-12-16T21:30:12.818 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : librgw2-2:17.2.7-125.g1f2f5510.el9.x86_64 8/26 2023-12-16T21:30:12.881 INFO:teuthology.orchestra.run.smithi045.stdout: Running scriptlet: librgw2-2:17.2.7-125.g1f2f5510.el9.x86_64 8/26 2023-12-16T21:30:12.910 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : liboath-2.6.7-2.el9.x86_64 9/26 2023-12-16T21:30:12.959 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : python3-ceph-argparse-2:17.2.7-125.g1f2f5510.el9.x 10/26 2023-12-16T21:30:12.998 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : python3-cephfs-2:17.2.7-125.g1f2f5510.el9.x86_64 11/26 2023-12-16T21:30:13.062 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : python3-rgw-2:17.2.7-125.g1f2f5510.el9.x86_64 12/26 2023-12-16T21:30:13.087 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : libradosstriper1-2:17.2.7-125.g1f2f5510.el9.x86_64 13/26 2023-12-16T21:30:13.162 INFO:teuthology.orchestra.run.smithi045.stdout: Running scriptlet: libradosstriper1-2:17.2.7-125.g1f2f5510.el9.x86_64 13/26 2023-12-16T21:30:13.202 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : libunwind-1.6.2-1.el9.x86_64 14/26 2023-12-16T21:30:13.246 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : gperftools-libs-2.9.1-2.el9.x86_64 15/26 2023-12-16T21:30:13.279 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : python3-prettytable-0.7.2-27.el9.noarch 16/26 2023-12-16T21:30:13.588 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : libpmemobj-1.12.1-1.el9.x86_64 17/26 2023-12-16T21:30:13.617 INFO:teuthology.orchestra.run.smithi045.stdout: Upgrading : librbd1-2:17.2.7-125.g1f2f5510.el9.x86_64 18/26 2023-12-16T21:30:13.707 INFO:teuthology.orchestra.run.smithi045.stdout: Running scriptlet: librbd1-2:17.2.7-125.g1f2f5510.el9.x86_64 18/26 2023-12-16T21:30:13.764 INFO:teuthology.orchestra.run.smithi169.stdout:lab-extras 11 kB/s | 1.7 kB 00:00 2023-12-16T21:30:13.783 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : python3-rbd-2:17.2.7-125.g1f2f5510.el9.x86_64 19/26 2023-12-16T21:30:13.820 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : python3-ceph-common-2:17.2.7-125.g1f2f5510.el9.x86 20/26 2023-12-16T21:30:16.256 INFO:teuthology.orchestra.run.smithi169.stdout:Dependencies resolved. 2023-12-16T21:30:16.264 INFO:teuthology.orchestra.run.smithi169.stdout:================================================================================ 2023-12-16T21:30:16.265 INFO:teuthology.orchestra.run.smithi169.stdout: Package Arch Version Repository Size 2023-12-16T21:30:16.265 INFO:teuthology.orchestra.run.smithi169.stdout:================================================================================ 2023-12-16T21:30:16.265 INFO:teuthology.orchestra.run.smithi169.stdout:Installing: 2023-12-16T21:30:16.265 INFO:teuthology.orchestra.run.smithi169.stdout: ceph-radosgw x86_64 2:17.2.7-125.g1f2f5510.el9 ceph 14 M 2023-12-16T21:30:16.265 INFO:teuthology.orchestra.run.smithi169.stdout:Upgrading: 2023-12-16T21:30:16.265 INFO:teuthology.orchestra.run.smithi169.stdout: librados2 x86_64 2:17.2.7-125.g1f2f5510.el9 ceph 3.1 M 2023-12-16T21:30:16.265 INFO:teuthology.orchestra.run.smithi169.stdout: librbd1 x86_64 2:17.2.7-125.g1f2f5510.el9 ceph 2.9 M 2023-12-16T21:30:16.265 INFO:teuthology.orchestra.run.smithi169.stdout:Installing dependencies: 2023-12-16T21:30:16.265 INFO:teuthology.orchestra.run.smithi169.stdout: ceph-base x86_64 2:17.2.7-125.g1f2f5510.el9 ceph 5.0 M 2023-12-16T21:30:16.265 INFO:teuthology.orchestra.run.smithi169.stdout: ceph-common x86_64 2:17.2.7-125.g1f2f5510.el9 ceph 23 M 2023-12-16T21:30:16.265 INFO:teuthology.orchestra.run.smithi169.stdout: ceph-selinux x86_64 2:17.2.7-125.g1f2f5510.el9 ceph 24 k 2023-12-16T21:30:16.265 INFO:teuthology.orchestra.run.smithi169.stdout: gperftools-libs x86_64 2.9.1-2.el9 epel 309 k 2023-12-16T21:30:16.266 INFO:teuthology.orchestra.run.smithi169.stdout: libcephfs2 x86_64 2:17.2.7-125.g1f2f5510.el9 ceph 636 k 2023-12-16T21:30:16.266 INFO:teuthology.orchestra.run.smithi169.stdout: liboath x86_64 2.6.7-2.el9 epel 49 k 2023-12-16T21:30:16.266 INFO:teuthology.orchestra.run.smithi169.stdout: libpmemobj x86_64 1.12.1-1.el9 CentOS-AppStream 159 k 2023-12-16T21:30:16.266 INFO:teuthology.orchestra.run.smithi169.stdout: librabbitmq x86_64 0.11.0-7.el9 CentOS-AppStream 44 k 2023-12-16T21:30:16.266 INFO:teuthology.orchestra.run.smithi169.stdout: libradosstriper1 x86_64 2:17.2.7-125.g1f2f5510.el9 ceph 458 k 2023-12-16T21:30:16.266 INFO:teuthology.orchestra.run.smithi169.stdout: librdkafka x86_64 1.6.1-102.el9 CentOS-AppStream 661 k 2023-12-16T21:30:16.266 INFO:teuthology.orchestra.run.smithi169.stdout: librgw2 x86_64 2:17.2.7-125.g1f2f5510.el9 ceph 4.9 M 2023-12-16T21:30:16.266 INFO:teuthology.orchestra.run.smithi169.stdout: libunwind x86_64 1.6.2-1.el9 epel 67 k 2023-12-16T21:30:16.266 INFO:teuthology.orchestra.run.smithi169.stdout: lttng-ust x86_64 2.12.0-6.el9 CentOS-AppStream 282 k 2023-12-16T21:30:16.266 INFO:teuthology.orchestra.run.smithi169.stdout: python3-ceph-argparse x86_64 2:17.2.7-125.g1f2f5510.el9 ceph 45 k 2023-12-16T21:30:16.266 INFO:teuthology.orchestra.run.smithi169.stdout: python3-ceph-common x86_64 2:17.2.7-125.g1f2f5510.el9 ceph 116 k 2023-12-16T21:30:16.266 INFO:teuthology.orchestra.run.smithi169.stdout: python3-cephfs x86_64 2:17.2.7-125.g1f2f5510.el9 ceph 156 k 2023-12-16T21:30:16.266 INFO:teuthology.orchestra.run.smithi169.stdout: python3-prettytable noarch 0.7.2-27.el9 CentOS-AppStream 41 k 2023-12-16T21:30:16.266 INFO:teuthology.orchestra.run.smithi169.stdout: python3-rados x86_64 2:17.2.7-125.g1f2f5510.el9 ceph 320 k 2023-12-16T21:30:16.267 INFO:teuthology.orchestra.run.smithi169.stdout: python3-rbd x86_64 2:17.2.7-125.g1f2f5510.el9 ceph 292 k 2023-12-16T21:30:16.267 INFO:teuthology.orchestra.run.smithi169.stdout: python3-rgw x86_64 2:17.2.7-125.g1f2f5510.el9 ceph 99 k 2023-12-16T21:30:16.267 INFO:teuthology.orchestra.run.smithi169.stdout: thrift x86_64 0.15.0-2.el9 epel 1.6 M 2023-12-16T21:30:16.267 INFO:teuthology.orchestra.run.smithi169.stdout: 2023-12-16T21:30:16.267 INFO:teuthology.orchestra.run.smithi169.stdout:Transaction Summary 2023-12-16T21:30:16.267 INFO:teuthology.orchestra.run.smithi169.stdout:================================================================================ 2023-12-16T21:30:16.267 INFO:teuthology.orchestra.run.smithi169.stdout:Install 22 Packages 2023-12-16T21:30:16.267 INFO:teuthology.orchestra.run.smithi169.stdout:Upgrade 2 Packages 2023-12-16T21:30:16.267 INFO:teuthology.orchestra.run.smithi169.stdout: 2023-12-16T21:30:16.268 INFO:teuthology.orchestra.run.smithi169.stdout:Total download size: 58 M 2023-12-16T21:30:16.268 INFO:teuthology.orchestra.run.smithi169.stdout:Downloading Packages: 2023-12-16T21:30:16.298 INFO:teuthology.orchestra.run.smithi045.stdout: Running scriptlet: ceph-common-2:17.2.7-125.g1f2f5510.el9.x86_64 21/26 2023-12-16T21:30:16.333 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : ceph-common-2:17.2.7-125.g1f2f5510.el9.x86_64 21/26 2023-12-16T21:30:16.890 INFO:teuthology.orchestra.run.smithi045.stdout: Running scriptlet: ceph-common-2:17.2.7-125.g1f2f5510.el9.x86_64 21/26 2023-12-16T21:30:16.928 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : ceph-base-2:17.2.7-125.g1f2f5510.el9.x86_64 22/26 2023-12-16T21:30:17.018 INFO:teuthology.orchestra.run.smithi045.stdout: Running scriptlet: ceph-base-2:17.2.7-125.g1f2f5510.el9.x86_64 22/26 2023-12-16T21:30:17.018 INFO:teuthology.orchestra.run.smithi045.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /usr/lib/systemd/system/ceph.target. 2023-12-16T21:30:17.018 INFO:teuthology.orchestra.run.smithi045.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service → /usr/lib/systemd/system/ceph-crash.service. 2023-12-16T21:30:17.019 INFO:teuthology.orchestra.run.smithi045.stdout: 2023-12-16T21:30:17.044 INFO:teuthology.orchestra.run.smithi169.stdout:(1/24): ceph-base-17.2.7-125.g1f2f5510.el9.x86_ 7.2 MB/s | 5.0 MB 00:00 2023-12-16T21:30:17.046 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : ceph-selinux-2:17.2.7-125.g1f2f5510.el9.x86_64 23/26 2023-12-16T21:30:17.103 INFO:teuthology.orchestra.run.smithi169.stdout:(2/24): ceph-selinux-17.2.7-125.g1f2f5510.el9.x 413 kB/s | 24 kB 00:00 2023-12-16T21:30:17.228 INFO:teuthology.orchestra.run.smithi169.stdout:(3/24): libcephfs2-17.2.7-125.g1f2f5510.el9.x86 5.0 MB/s | 636 kB 00:00 2023-12-16T21:30:17.295 INFO:teuthology.orchestra.run.smithi169.stdout:(4/24): libradosstriper1-17.2.7-125.g1f2f5510.e 6.7 MB/s | 458 kB 00:00 2023-12-16T21:30:17.820 INFO:teuthology.orchestra.run.smithi169.stdout:(5/24): ceph-radosgw-17.2.7-125.g1f2f5510.el9.x 9.6 MB/s | 14 MB 00:01 2023-12-16T21:30:17.896 INFO:teuthology.orchestra.run.smithi169.stdout:(6/24): librgw2-17.2.7-125.g1f2f5510.el9.x86_64 8.2 MB/s | 4.9 MB 00:00 2023-12-16T21:30:17.921 INFO:teuthology.orchestra.run.smithi169.stdout:(7/24): python3-ceph-argparse-17.2.7-125.g1f2f5 451 kB/s | 45 kB 00:00 2023-12-16T21:30:17.954 INFO:teuthology.orchestra.run.smithi169.stdout:(8/24): python3-ceph-common-17.2.7-125.g1f2f551 1.9 MB/s | 116 kB 00:00 2023-12-16T21:30:17.980 INFO:teuthology.orchestra.run.smithi169.stdout:(9/24): python3-cephfs-17.2.7-125.g1f2f5510.el9 2.6 MB/s | 156 kB 00:00 2023-12-16T21:30:18.014 INFO:teuthology.orchestra.run.smithi169.stdout:(10/24): python3-rados-17.2.7-125.g1f2f5510.el9 5.3 MB/s | 320 kB 00:00 2023-12-16T21:30:18.039 INFO:teuthology.orchestra.run.smithi169.stdout:(11/24): python3-rbd-17.2.7-125.g1f2f5510.el9.x 4.8 MB/s | 292 kB 00:00 2023-12-16T21:30:18.064 INFO:teuthology.orchestra.run.smithi169.stdout:(12/24): python3-rgw-17.2.7-125.g1f2f5510.el9.x 1.9 MB/s | 99 kB 00:00 2023-12-16T21:30:18.190 INFO:teuthology.orchestra.run.smithi169.stdout:(13/24): librabbitmq-0.11.0-7.el9.x86_64.rpm 353 kB/s | 44 kB 00:00 2023-12-16T21:30:18.215 INFO:teuthology.orchestra.run.smithi169.stdout:(14/24): libpmemobj-1.12.1-1.el9.x86_64.rpm 906 kB/s | 159 kB 00:00 2023-12-16T21:30:18.265 INFO:teuthology.orchestra.run.smithi169.stdout:(15/24): librdkafka-1.6.1-102.el9.x86_64.rpm 8.6 MB/s | 661 kB 00:00 2023-12-16T21:30:18.290 INFO:teuthology.orchestra.run.smithi169.stdout:(16/24): lttng-ust-2.12.0-6.el9.x86_64.rpm 3.7 MB/s | 282 kB 00:00 2023-12-16T21:30:18.315 INFO:teuthology.orchestra.run.smithi169.stdout:(17/24): python3-prettytable-0.7.2-27.el9.noarc 826 kB/s | 41 kB 00:00 2023-12-16T21:30:18.474 INFO:teuthology.orchestra.run.smithi169.stdout:(18/24): liboath-2.6.7-2.el9.x86_64.rpm 306 kB/s | 49 kB 00:00 2023-12-16T21:30:18.733 INFO:teuthology.orchestra.run.smithi169.stdout:(19/24): ceph-common-17.2.7-125.g1f2f5510.el9.x 9.5 MB/s | 23 MB 00:02 2023-12-16T21:30:18.750 INFO:teuthology.orchestra.run.smithi169.stdout:(20/24): libunwind-1.6.2-1.el9.x86_64.rpm 245 kB/s | 67 kB 00:00 2023-12-16T21:30:18.775 INFO:teuthology.orchestra.run.smithi169.stdout:(21/24): gperftools-libs-2.9.1-2.el9.x86_64.rpm 638 kB/s | 309 kB 00:00 2023-12-16T21:30:19.025 INFO:teuthology.orchestra.run.smithi169.stdout:(22/24): librbd1-17.2.7-125.g1f2f5510.el9.x86_6 12 MB/s | 2.9 MB 00:00 2023-12-16T21:30:19.085 INFO:teuthology.orchestra.run.smithi169.stdout:(23/24): librados2-17.2.7-125.g1f2f5510.el9.x86 9.2 MB/s | 3.1 MB 00:00 2023-12-16T21:30:19.134 INFO:teuthology.orchestra.run.smithi169.stdout:(24/24): thrift-0.15.0-2.el9.x86_64.rpm 3.9 MB/s | 1.6 MB 00:00 2023-12-16T21:30:19.138 INFO:teuthology.orchestra.run.smithi169.stdout:-------------------------------------------------------------------------------- 2023-12-16T21:30:19.138 INFO:teuthology.orchestra.run.smithi169.stdout:Total 20 MB/s | 58 MB 00:02 2023-12-16T21:30:19.138 INFO:teuthology.orchestra.run.smithi169.stdout:Running transaction check 2023-12-16T21:30:19.169 INFO:teuthology.orchestra.run.smithi169.stdout:Transaction check succeeded. 2023-12-16T21:30:19.169 INFO:teuthology.orchestra.run.smithi169.stdout:Running transaction test 2023-12-16T21:30:19.635 INFO:teuthology.orchestra.run.smithi169.stdout:Transaction test succeeded. 2023-12-16T21:30:19.635 INFO:teuthology.orchestra.run.smithi169.stdout:Running transaction 2023-12-16T21:30:20.868 INFO:teuthology.orchestra.run.smithi169.stdout: Preparing : 1/1 2023-12-16T21:30:21.215 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : lttng-ust-2.12.0-6.el9.x86_64 1/26 2023-12-16T21:30:21.278 INFO:teuthology.orchestra.run.smithi169.stdout: Upgrading : librados2-2:17.2.7-125.g1f2f5510.el9.x86_64 2/26 2023-12-16T21:30:21.354 INFO:teuthology.orchestra.run.smithi169.stdout: Running scriptlet: librados2-2:17.2.7-125.g1f2f5510.el9.x86_64 2/26 2023-12-16T21:30:21.434 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : python3-rados-2:17.2.7-125.g1f2f5510.el9.x86_64 3/26 2023-12-16T21:30:21.480 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : libcephfs2-2:17.2.7-125.g1f2f5510.el9.x86_64 4/26 2023-12-16T21:30:21.552 INFO:teuthology.orchestra.run.smithi169.stdout: Running scriptlet: libcephfs2-2:17.2.7-125.g1f2f5510.el9.x86_64 4/26 2023-12-16T21:30:21.605 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : thrift-0.15.0-2.el9.x86_64 5/26 2023-12-16T21:30:21.635 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : librdkafka-1.6.1-102.el9.x86_64 6/26 2023-12-16T21:30:22.102 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : librabbitmq-0.11.0-7.el9.x86_64 7/26 2023-12-16T21:30:22.133 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : librgw2-2:17.2.7-125.g1f2f5510.el9.x86_64 8/26 2023-12-16T21:30:22.195 INFO:teuthology.orchestra.run.smithi169.stdout: Running scriptlet: librgw2-2:17.2.7-125.g1f2f5510.el9.x86_64 8/26 2023-12-16T21:30:22.230 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : liboath-2.6.7-2.el9.x86_64 9/26 2023-12-16T21:30:22.279 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : python3-ceph-argparse-2:17.2.7-125.g1f2f5510.el9.x 10/26 2023-12-16T21:30:22.318 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : python3-cephfs-2:17.2.7-125.g1f2f5510.el9.x86_64 11/26 2023-12-16T21:30:22.380 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : python3-rgw-2:17.2.7-125.g1f2f5510.el9.x86_64 12/26 2023-12-16T21:30:22.408 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : libradosstriper1-2:17.2.7-125.g1f2f5510.el9.x86_64 13/26 2023-12-16T21:30:22.474 INFO:teuthology.orchestra.run.smithi169.stdout: Running scriptlet: libradosstriper1-2:17.2.7-125.g1f2f5510.el9.x86_64 13/26 2023-12-16T21:30:22.515 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : libunwind-1.6.2-1.el9.x86_64 14/26 2023-12-16T21:30:22.549 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : gperftools-libs-2.9.1-2.el9.x86_64 15/26 2023-12-16T21:30:22.590 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : python3-prettytable-0.7.2-27.el9.noarch 16/26 2023-12-16T21:30:22.885 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : libpmemobj-1.12.1-1.el9.x86_64 17/26 2023-12-16T21:30:22.912 INFO:teuthology.orchestra.run.smithi169.stdout: Upgrading : librbd1-2:17.2.7-125.g1f2f5510.el9.x86_64 18/26 2023-12-16T21:30:22.994 INFO:teuthology.orchestra.run.smithi169.stdout: Running scriptlet: librbd1-2:17.2.7-125.g1f2f5510.el9.x86_64 18/26 2023-12-16T21:30:23.066 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : python3-rbd-2:17.2.7-125.g1f2f5510.el9.x86_64 19/26 2023-12-16T21:30:23.090 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : python3-ceph-common-2:17.2.7-125.g1f2f5510.el9.x86 20/26 2023-12-16T21:30:25.467 INFO:teuthology.orchestra.run.smithi169.stdout: Running scriptlet: ceph-common-2:17.2.7-125.g1f2f5510.el9.x86_64 21/26 2023-12-16T21:30:25.511 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : ceph-common-2:17.2.7-125.g1f2f5510.el9.x86_64 21/26 2023-12-16T21:30:26.048 INFO:teuthology.orchestra.run.smithi169.stdout: Running scriptlet: ceph-common-2:17.2.7-125.g1f2f5510.el9.x86_64 21/26 2023-12-16T21:30:26.081 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : ceph-base-2:17.2.7-125.g1f2f5510.el9.x86_64 22/26 2023-12-16T21:30:26.167 INFO:teuthology.orchestra.run.smithi169.stdout: Running scriptlet: ceph-base-2:17.2.7-125.g1f2f5510.el9.x86_64 22/26 2023-12-16T21:30:26.167 INFO:teuthology.orchestra.run.smithi169.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /usr/lib/systemd/system/ceph.target. 2023-12-16T21:30:26.167 INFO:teuthology.orchestra.run.smithi169.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service → /usr/lib/systemd/system/ceph-crash.service. 2023-12-16T21:30:26.167 INFO:teuthology.orchestra.run.smithi169.stdout: 2023-12-16T21:30:26.207 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : ceph-selinux-2:17.2.7-125.g1f2f5510.el9.x86_64 23/26 2023-12-16T21:30:27.079 INFO:teuthology.orchestra.run.smithi045.stdout: Running scriptlet: ceph-selinux-2:17.2.7-125.g1f2f5510.el9.x86_64 23/26 2023-12-16T21:30:27.079 INFO:teuthology.orchestra.run.smithi045.stdout:skipping the directory /sys 2023-12-16T21:30:27.079 INFO:teuthology.orchestra.run.smithi045.stdout:skipping the directory /proc 2023-12-16T21:30:27.079 INFO:teuthology.orchestra.run.smithi045.stdout:skipping the directory /mnt 2023-12-16T21:30:27.079 INFO:teuthology.orchestra.run.smithi045.stdout:skipping the directory /var/tmp 2023-12-16T21:30:27.079 INFO:teuthology.orchestra.run.smithi045.stdout:skipping the directory /home 2023-12-16T21:30:27.079 INFO:teuthology.orchestra.run.smithi045.stdout:skipping the directory /root 2023-12-16T21:30:27.079 INFO:teuthology.orchestra.run.smithi045.stdout:skipping the directory /tmp 2023-12-16T21:30:27.080 INFO:teuthology.orchestra.run.smithi045.stdout: 2023-12-16T21:30:28.389 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : ceph-radosgw-2:17.2.7-125.g1f2f5510.el9.x86_64 24/26 2023-12-16T21:30:28.466 INFO:teuthology.orchestra.run.smithi045.stdout: Running scriptlet: ceph-radosgw-2:17.2.7-125.g1f2f5510.el9.x86_64 24/26 2023-12-16T21:30:28.466 INFO:teuthology.orchestra.run.smithi045.stdout:Glob pattern passed to enable, but globs are not supported for this. 2023-12-16T21:30:28.466 INFO:teuthology.orchestra.run.smithi045.stdout:Invalid unit name "ceph-radosgw@*.service" escaped as "ceph-radosgw@\x2a.service". 2023-12-16T21:30:28.466 INFO:teuthology.orchestra.run.smithi045.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2023-12-16T21:30:28.466 INFO:teuthology.orchestra.run.smithi045.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2023-12-16T21:30:28.467 INFO:teuthology.orchestra.run.smithi045.stdout: 2023-12-16T21:30:28.467 INFO:teuthology.orchestra.run.smithi045.stdout: Cleanup : librbd1-2:16.2.4-5.el9.x86_64 25/26 2023-12-16T21:30:28.675 INFO:teuthology.orchestra.run.smithi045.stdout: Running scriptlet: librbd1-2:16.2.4-5.el9.x86_64 25/26 2023-12-16T21:30:28.675 INFO:teuthology.orchestra.run.smithi045.stdout: Cleanup : librados2-2:16.2.4-5.el9.x86_64 26/26 2023-12-16T21:30:29.875 INFO:teuthology.orchestra.run.smithi045.stdout: Running scriptlet: librados2-2:16.2.4-5.el9.x86_64 26/26 2023-12-16T21:30:29.875 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : ceph-base-2:17.2.7-125.g1f2f5510.el9.x86_64 1/26 2023-12-16T21:30:29.875 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : ceph-common-2:17.2.7-125.g1f2f5510.el9.x86_64 2/26 2023-12-16T21:30:29.875 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : ceph-radosgw-2:17.2.7-125.g1f2f5510.el9.x86_64 3/26 2023-12-16T21:30:29.875 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : ceph-selinux-2:17.2.7-125.g1f2f5510.el9.x86_64 4/26 2023-12-16T21:30:29.876 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : libcephfs2-2:17.2.7-125.g1f2f5510.el9.x86_64 5/26 2023-12-16T21:30:29.876 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : libradosstriper1-2:17.2.7-125.g1f2f5510.el9.x86_64 6/26 2023-12-16T21:30:29.876 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : librgw2-2:17.2.7-125.g1f2f5510.el9.x86_64 7/26 2023-12-16T21:30:29.876 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : python3-ceph-argparse-2:17.2.7-125.g1f2f5510.el9.x 8/26 2023-12-16T21:30:29.876 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : python3-ceph-common-2:17.2.7-125.g1f2f5510.el9.x86 9/26 2023-12-16T21:30:29.876 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : python3-cephfs-2:17.2.7-125.g1f2f5510.el9.x86_64 10/26 2023-12-16T21:30:29.876 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : python3-rados-2:17.2.7-125.g1f2f5510.el9.x86_64 11/26 2023-12-16T21:30:29.876 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : python3-rbd-2:17.2.7-125.g1f2f5510.el9.x86_64 12/26 2023-12-16T21:30:29.876 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : python3-rgw-2:17.2.7-125.g1f2f5510.el9.x86_64 13/26 2023-12-16T21:30:29.876 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : libpmemobj-1.12.1-1.el9.x86_64 14/26 2023-12-16T21:30:29.876 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : librabbitmq-0.11.0-7.el9.x86_64 15/26 2023-12-16T21:30:29.876 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : librdkafka-1.6.1-102.el9.x86_64 16/26 2023-12-16T21:30:29.876 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : lttng-ust-2.12.0-6.el9.x86_64 17/26 2023-12-16T21:30:29.876 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : python3-prettytable-0.7.2-27.el9.noarch 18/26 2023-12-16T21:30:29.877 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : gperftools-libs-2.9.1-2.el9.x86_64 19/26 2023-12-16T21:30:29.877 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : liboath-2.6.7-2.el9.x86_64 20/26 2023-12-16T21:30:29.877 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : libunwind-1.6.2-1.el9.x86_64 21/26 2023-12-16T21:30:29.877 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : thrift-0.15.0-2.el9.x86_64 22/26 2023-12-16T21:30:29.877 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : librados2-2:17.2.7-125.g1f2f5510.el9.x86_64 23/26 2023-12-16T21:30:29.877 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : librados2-2:16.2.4-5.el9.x86_64 24/26 2023-12-16T21:30:29.877 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : librbd1-2:17.2.7-125.g1f2f5510.el9.x86_64 25/26 2023-12-16T21:30:30.842 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : librbd1-2:16.2.4-5.el9.x86_64 26/26 2023-12-16T21:30:30.842 INFO:teuthology.orchestra.run.smithi045.stdout: 2023-12-16T21:30:30.842 INFO:teuthology.orchestra.run.smithi045.stdout:Upgraded: 2023-12-16T21:30:30.842 INFO:teuthology.orchestra.run.smithi045.stdout: librados2-2:17.2.7-125.g1f2f5510.el9.x86_64 2023-12-16T21:30:30.842 INFO:teuthology.orchestra.run.smithi045.stdout: librbd1-2:17.2.7-125.g1f2f5510.el9.x86_64 2023-12-16T21:30:30.842 INFO:teuthology.orchestra.run.smithi045.stdout:Installed: 2023-12-16T21:30:30.842 INFO:teuthology.orchestra.run.smithi045.stdout: ceph-base-2:17.2.7-125.g1f2f5510.el9.x86_64 2023-12-16T21:30:30.842 INFO:teuthology.orchestra.run.smithi045.stdout: ceph-common-2:17.2.7-125.g1f2f5510.el9.x86_64 2023-12-16T21:30:30.842 INFO:teuthology.orchestra.run.smithi045.stdout: ceph-radosgw-2:17.2.7-125.g1f2f5510.el9.x86_64 2023-12-16T21:30:30.843 INFO:teuthology.orchestra.run.smithi045.stdout: ceph-selinux-2:17.2.7-125.g1f2f5510.el9.x86_64 2023-12-16T21:30:30.843 INFO:teuthology.orchestra.run.smithi045.stdout: gperftools-libs-2.9.1-2.el9.x86_64 2023-12-16T21:30:30.843 INFO:teuthology.orchestra.run.smithi045.stdout: libcephfs2-2:17.2.7-125.g1f2f5510.el9.x86_64 2023-12-16T21:30:30.843 INFO:teuthology.orchestra.run.smithi045.stdout: liboath-2.6.7-2.el9.x86_64 2023-12-16T21:30:30.843 INFO:teuthology.orchestra.run.smithi045.stdout: libpmemobj-1.12.1-1.el9.x86_64 2023-12-16T21:30:30.843 INFO:teuthology.orchestra.run.smithi045.stdout: librabbitmq-0.11.0-7.el9.x86_64 2023-12-16T21:30:30.843 INFO:teuthology.orchestra.run.smithi045.stdout: libradosstriper1-2:17.2.7-125.g1f2f5510.el9.x86_64 2023-12-16T21:30:30.843 INFO:teuthology.orchestra.run.smithi045.stdout: librdkafka-1.6.1-102.el9.x86_64 2023-12-16T21:30:30.843 INFO:teuthology.orchestra.run.smithi045.stdout: librgw2-2:17.2.7-125.g1f2f5510.el9.x86_64 2023-12-16T21:30:30.843 INFO:teuthology.orchestra.run.smithi045.stdout: libunwind-1.6.2-1.el9.x86_64 2023-12-16T21:30:30.843 INFO:teuthology.orchestra.run.smithi045.stdout: lttng-ust-2.12.0-6.el9.x86_64 2023-12-16T21:30:30.843 INFO:teuthology.orchestra.run.smithi045.stdout: python3-ceph-argparse-2:17.2.7-125.g1f2f5510.el9.x86_64 2023-12-16T21:30:30.843 INFO:teuthology.orchestra.run.smithi045.stdout: python3-ceph-common-2:17.2.7-125.g1f2f5510.el9.x86_64 2023-12-16T21:30:30.843 INFO:teuthology.orchestra.run.smithi045.stdout: python3-cephfs-2:17.2.7-125.g1f2f5510.el9.x86_64 2023-12-16T21:30:30.844 INFO:teuthology.orchestra.run.smithi045.stdout: python3-prettytable-0.7.2-27.el9.noarch 2023-12-16T21:30:30.844 INFO:teuthology.orchestra.run.smithi045.stdout: python3-rados-2:17.2.7-125.g1f2f5510.el9.x86_64 2023-12-16T21:30:30.844 INFO:teuthology.orchestra.run.smithi045.stdout: python3-rbd-2:17.2.7-125.g1f2f5510.el9.x86_64 2023-12-16T21:30:30.844 INFO:teuthology.orchestra.run.smithi045.stdout: python3-rgw-2:17.2.7-125.g1f2f5510.el9.x86_64 2023-12-16T21:30:30.844 INFO:teuthology.orchestra.run.smithi045.stdout: thrift-0.15.0-2.el9.x86_64 2023-12-16T21:30:30.844 INFO:teuthology.orchestra.run.smithi045.stdout: 2023-12-16T21:30:30.844 INFO:teuthology.orchestra.run.smithi045.stdout:Complete! 2023-12-16T21:30:31.213 DEBUG:teuthology.orchestra.run.smithi045:> sudo yum -y install ceph-test 2023-12-16T21:30:31.793 INFO:teuthology.orchestra.run.smithi045.stdout:Last metadata expiration check: 0:00:28 ago on Sat 16 Dec 2023 09:30:03 PM UTC. 2023-12-16T21:30:31.944 INFO:teuthology.orchestra.run.smithi045.stdout:Dependencies resolved. 2023-12-16T21:30:31.945 INFO:teuthology.orchestra.run.smithi045.stdout:================================================================================ 2023-12-16T21:30:31.945 INFO:teuthology.orchestra.run.smithi045.stdout: Package Arch Version Repository Size 2023-12-16T21:30:31.945 INFO:teuthology.orchestra.run.smithi045.stdout:================================================================================ 2023-12-16T21:30:31.945 INFO:teuthology.orchestra.run.smithi045.stdout:Installing: 2023-12-16T21:30:31.945 INFO:teuthology.orchestra.run.smithi045.stdout: ceph-test x86_64 2:17.2.7-125.g1f2f5510.el9 ceph 48 M 2023-12-16T21:30:31.945 INFO:teuthology.orchestra.run.smithi045.stdout:Installing dependencies: 2023-12-16T21:30:31.945 INFO:teuthology.orchestra.run.smithi045.stdout: jq x86_64 1.6-16.el9 CentOS-AppStream 186 k 2023-12-16T21:30:31.945 INFO:teuthology.orchestra.run.smithi045.stdout: libcephsqlite x86_64 2:17.2.7-125.g1f2f5510.el9 ceph 159 k 2023-12-16T21:30:31.945 INFO:teuthology.orchestra.run.smithi045.stdout: oniguruma x86_64 6.9.6-1.el9.5 CentOS-AppStream 217 k 2023-12-16T21:30:31.945 INFO:teuthology.orchestra.run.smithi045.stdout: socat x86_64 1.7.4.1-5.el9 CentOS-AppStream 300 k 2023-12-16T21:30:31.945 INFO:teuthology.orchestra.run.smithi045.stdout: xmlstarlet x86_64 1.6.1-20.el9 CentOS-AppStream 63 k 2023-12-16T21:30:31.946 INFO:teuthology.orchestra.run.smithi045.stdout: 2023-12-16T21:30:31.946 INFO:teuthology.orchestra.run.smithi045.stdout:Transaction Summary 2023-12-16T21:30:31.946 INFO:teuthology.orchestra.run.smithi045.stdout:================================================================================ 2023-12-16T21:30:31.946 INFO:teuthology.orchestra.run.smithi045.stdout:Install 6 Packages 2023-12-16T21:30:31.946 INFO:teuthology.orchestra.run.smithi045.stdout: 2023-12-16T21:30:31.946 INFO:teuthology.orchestra.run.smithi045.stdout:Total download size: 49 M 2023-12-16T21:30:31.946 INFO:teuthology.orchestra.run.smithi045.stdout:Installed size: 195 M 2023-12-16T21:30:31.946 INFO:teuthology.orchestra.run.smithi045.stdout:Downloading Packages: 2023-12-16T21:30:32.173 INFO:teuthology.orchestra.run.smithi045.stdout:(1/6): jq-1.6-16.el9.x86_64.rpm 824 kB/s | 186 kB 00:00 2023-12-16T21:30:32.240 INFO:teuthology.orchestra.run.smithi045.stdout:(2/6): libcephsqlite-17.2.7-125.g1f2f5510.el9.x 542 kB/s | 159 kB 00:00 2023-12-16T21:30:32.266 INFO:teuthology.orchestra.run.smithi045.stdout:(3/6): oniguruma-6.9.6-1.el9.5.x86_64.rpm 2.3 MB/s | 217 kB 00:00 2023-12-16T21:30:32.308 INFO:teuthology.orchestra.run.smithi045.stdout:(4/6): socat-1.7.4.1-5.el9.x86_64.rpm 4.4 MB/s | 300 kB 00:00 2023-12-16T21:30:32.325 INFO:teuthology.orchestra.run.smithi045.stdout:(5/6): xmlstarlet-1.6.1-20.el9.x86_64.rpm 1.0 MB/s | 63 kB 00:00 2023-12-16T21:30:34.583 INFO:teuthology.orchestra.run.smithi045.stdout:(6/6): ceph-test-17.2.7-125.g1f2f5510.el9.x86_6 18 MB/s | 48 MB 00:02 2023-12-16T21:30:34.586 INFO:teuthology.orchestra.run.smithi045.stdout:-------------------------------------------------------------------------------- 2023-12-16T21:30:34.586 INFO:teuthology.orchestra.run.smithi045.stdout:Total 18 MB/s | 49 MB 00:02 2023-12-16T21:30:34.587 INFO:teuthology.orchestra.run.smithi045.stdout:Running transaction check 2023-12-16T21:30:34.608 INFO:teuthology.orchestra.run.smithi045.stdout:Transaction check succeeded. 2023-12-16T21:30:34.609 INFO:teuthology.orchestra.run.smithi045.stdout:Running transaction test 2023-12-16T21:30:35.015 INFO:teuthology.orchestra.run.smithi045.stdout:Transaction test succeeded. 2023-12-16T21:30:35.015 INFO:teuthology.orchestra.run.smithi045.stdout:Running transaction 2023-12-16T21:30:35.626 INFO:teuthology.orchestra.run.smithi045.stdout: Preparing : 1/1 2023-12-16T21:30:35.702 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : xmlstarlet-1.6.1-20.el9.x86_64 1/6 2023-12-16T21:30:35.761 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : socat-1.7.4.1-5.el9.x86_64 2/6 2023-12-16T21:30:35.790 INFO:teuthology.orchestra.run.smithi169.stdout: Running scriptlet: ceph-selinux-2:17.2.7-125.g1f2f5510.el9.x86_64 23/26 2023-12-16T21:30:35.790 INFO:teuthology.orchestra.run.smithi169.stdout:skipping the directory /sys 2023-12-16T21:30:35.791 INFO:teuthology.orchestra.run.smithi169.stdout:skipping the directory /proc 2023-12-16T21:30:35.791 INFO:teuthology.orchestra.run.smithi169.stdout:skipping the directory /mnt 2023-12-16T21:30:35.791 INFO:teuthology.orchestra.run.smithi169.stdout:skipping the directory /var/tmp 2023-12-16T21:30:35.791 INFO:teuthology.orchestra.run.smithi169.stdout:skipping the directory /home 2023-12-16T21:30:35.791 INFO:teuthology.orchestra.run.smithi169.stdout:skipping the directory /root 2023-12-16T21:30:35.791 INFO:teuthology.orchestra.run.smithi169.stdout:skipping the directory /tmp 2023-12-16T21:30:35.791 INFO:teuthology.orchestra.run.smithi169.stdout: 2023-12-16T21:30:35.791 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : oniguruma-6.9.6-1.el9.5.x86_64 3/6 2023-12-16T21:30:35.842 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : jq-1.6-16.el9.x86_64 4/6 2023-12-16T21:30:35.870 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : libcephsqlite-2:17.2.7-125.g1f2f5510.el9.x86_64 5/6 2023-12-16T21:30:37.082 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : ceph-radosgw-2:17.2.7-125.g1f2f5510.el9.x86_64 24/26 2023-12-16T21:30:37.160 INFO:teuthology.orchestra.run.smithi169.stdout: Running scriptlet: ceph-radosgw-2:17.2.7-125.g1f2f5510.el9.x86_64 24/26 2023-12-16T21:30:37.160 INFO:teuthology.orchestra.run.smithi169.stdout:Glob pattern passed to enable, but globs are not supported for this. 2023-12-16T21:30:37.160 INFO:teuthology.orchestra.run.smithi169.stdout:Invalid unit name "ceph-radosgw@*.service" escaped as "ceph-radosgw@\x2a.service". 2023-12-16T21:30:37.160 INFO:teuthology.orchestra.run.smithi169.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2023-12-16T21:30:37.160 INFO:teuthology.orchestra.run.smithi169.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2023-12-16T21:30:37.160 INFO:teuthology.orchestra.run.smithi169.stdout: 2023-12-16T21:30:37.162 INFO:teuthology.orchestra.run.smithi169.stdout: Cleanup : librbd1-2:16.2.4-5.el9.x86_64 25/26 2023-12-16T21:30:37.252 INFO:teuthology.orchestra.run.smithi169.stdout: Running scriptlet: librbd1-2:16.2.4-5.el9.x86_64 25/26 2023-12-16T21:30:37.252 INFO:teuthology.orchestra.run.smithi169.stdout: Cleanup : librados2-2:16.2.4-5.el9.x86_64 26/26 2023-12-16T21:30:38.436 INFO:teuthology.orchestra.run.smithi169.stdout: Running scriptlet: librados2-2:16.2.4-5.el9.x86_64 26/26 2023-12-16T21:30:38.436 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : ceph-base-2:17.2.7-125.g1f2f5510.el9.x86_64 1/26 2023-12-16T21:30:38.437 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : ceph-common-2:17.2.7-125.g1f2f5510.el9.x86_64 2/26 2023-12-16T21:30:38.437 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : ceph-radosgw-2:17.2.7-125.g1f2f5510.el9.x86_64 3/26 2023-12-16T21:30:38.437 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : ceph-selinux-2:17.2.7-125.g1f2f5510.el9.x86_64 4/26 2023-12-16T21:30:38.437 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : libcephfs2-2:17.2.7-125.g1f2f5510.el9.x86_64 5/26 2023-12-16T21:30:38.437 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : libradosstriper1-2:17.2.7-125.g1f2f5510.el9.x86_64 6/26 2023-12-16T21:30:38.437 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : librgw2-2:17.2.7-125.g1f2f5510.el9.x86_64 7/26 2023-12-16T21:30:38.437 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : python3-ceph-argparse-2:17.2.7-125.g1f2f5510.el9.x 8/26 2023-12-16T21:30:38.437 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : python3-ceph-common-2:17.2.7-125.g1f2f5510.el9.x86 9/26 2023-12-16T21:30:38.437 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : python3-cephfs-2:17.2.7-125.g1f2f5510.el9.x86_64 10/26 2023-12-16T21:30:38.437 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : python3-rados-2:17.2.7-125.g1f2f5510.el9.x86_64 11/26 2023-12-16T21:30:38.437 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : python3-rbd-2:17.2.7-125.g1f2f5510.el9.x86_64 12/26 2023-12-16T21:30:38.437 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : python3-rgw-2:17.2.7-125.g1f2f5510.el9.x86_64 13/26 2023-12-16T21:30:38.438 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : libpmemobj-1.12.1-1.el9.x86_64 14/26 2023-12-16T21:30:38.438 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : librabbitmq-0.11.0-7.el9.x86_64 15/26 2023-12-16T21:30:38.438 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : librdkafka-1.6.1-102.el9.x86_64 16/26 2023-12-16T21:30:38.438 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : lttng-ust-2.12.0-6.el9.x86_64 17/26 2023-12-16T21:30:38.438 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : python3-prettytable-0.7.2-27.el9.noarch 18/26 2023-12-16T21:30:38.438 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : gperftools-libs-2.9.1-2.el9.x86_64 19/26 2023-12-16T21:30:38.438 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : liboath-2.6.7-2.el9.x86_64 20/26 2023-12-16T21:30:38.438 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : libunwind-1.6.2-1.el9.x86_64 21/26 2023-12-16T21:30:38.438 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : thrift-0.15.0-2.el9.x86_64 22/26 2023-12-16T21:30:38.438 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : librados2-2:17.2.7-125.g1f2f5510.el9.x86_64 23/26 2023-12-16T21:30:38.438 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : librados2-2:16.2.4-5.el9.x86_64 24/26 2023-12-16T21:30:38.438 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : librbd1-2:17.2.7-125.g1f2f5510.el9.x86_64 25/26 2023-12-16T21:30:39.273 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : librbd1-2:16.2.4-5.el9.x86_64 26/26 2023-12-16T21:30:39.273 INFO:teuthology.orchestra.run.smithi169.stdout: 2023-12-16T21:30:39.273 INFO:teuthology.orchestra.run.smithi169.stdout:Upgraded: 2023-12-16T21:30:39.273 INFO:teuthology.orchestra.run.smithi169.stdout: librados2-2:17.2.7-125.g1f2f5510.el9.x86_64 2023-12-16T21:30:39.273 INFO:teuthology.orchestra.run.smithi169.stdout: librbd1-2:17.2.7-125.g1f2f5510.el9.x86_64 2023-12-16T21:30:39.273 INFO:teuthology.orchestra.run.smithi169.stdout:Installed: 2023-12-16T21:30:39.273 INFO:teuthology.orchestra.run.smithi169.stdout: ceph-base-2:17.2.7-125.g1f2f5510.el9.x86_64 2023-12-16T21:30:39.273 INFO:teuthology.orchestra.run.smithi169.stdout: ceph-common-2:17.2.7-125.g1f2f5510.el9.x86_64 2023-12-16T21:30:39.273 INFO:teuthology.orchestra.run.smithi169.stdout: ceph-radosgw-2:17.2.7-125.g1f2f5510.el9.x86_64 2023-12-16T21:30:39.274 INFO:teuthology.orchestra.run.smithi169.stdout: ceph-selinux-2:17.2.7-125.g1f2f5510.el9.x86_64 2023-12-16T21:30:39.274 INFO:teuthology.orchestra.run.smithi169.stdout: gperftools-libs-2.9.1-2.el9.x86_64 2023-12-16T21:30:39.274 INFO:teuthology.orchestra.run.smithi169.stdout: libcephfs2-2:17.2.7-125.g1f2f5510.el9.x86_64 2023-12-16T21:30:39.274 INFO:teuthology.orchestra.run.smithi169.stdout: liboath-2.6.7-2.el9.x86_64 2023-12-16T21:30:39.274 INFO:teuthology.orchestra.run.smithi169.stdout: libpmemobj-1.12.1-1.el9.x86_64 2023-12-16T21:30:39.274 INFO:teuthology.orchestra.run.smithi169.stdout: librabbitmq-0.11.0-7.el9.x86_64 2023-12-16T21:30:39.274 INFO:teuthology.orchestra.run.smithi169.stdout: libradosstriper1-2:17.2.7-125.g1f2f5510.el9.x86_64 2023-12-16T21:30:39.274 INFO:teuthology.orchestra.run.smithi169.stdout: librdkafka-1.6.1-102.el9.x86_64 2023-12-16T21:30:39.274 INFO:teuthology.orchestra.run.smithi169.stdout: librgw2-2:17.2.7-125.g1f2f5510.el9.x86_64 2023-12-16T21:30:39.274 INFO:teuthology.orchestra.run.smithi169.stdout: libunwind-1.6.2-1.el9.x86_64 2023-12-16T21:30:39.274 INFO:teuthology.orchestra.run.smithi169.stdout: lttng-ust-2.12.0-6.el9.x86_64 2023-12-16T21:30:39.274 INFO:teuthology.orchestra.run.smithi169.stdout: python3-ceph-argparse-2:17.2.7-125.g1f2f5510.el9.x86_64 2023-12-16T21:30:39.274 INFO:teuthology.orchestra.run.smithi169.stdout: python3-ceph-common-2:17.2.7-125.g1f2f5510.el9.x86_64 2023-12-16T21:30:39.274 INFO:teuthology.orchestra.run.smithi169.stdout: python3-cephfs-2:17.2.7-125.g1f2f5510.el9.x86_64 2023-12-16T21:30:39.274 INFO:teuthology.orchestra.run.smithi169.stdout: python3-prettytable-0.7.2-27.el9.noarch 2023-12-16T21:30:39.275 INFO:teuthology.orchestra.run.smithi169.stdout: python3-rados-2:17.2.7-125.g1f2f5510.el9.x86_64 2023-12-16T21:30:39.275 INFO:teuthology.orchestra.run.smithi169.stdout: python3-rbd-2:17.2.7-125.g1f2f5510.el9.x86_64 2023-12-16T21:30:39.275 INFO:teuthology.orchestra.run.smithi169.stdout: python3-rgw-2:17.2.7-125.g1f2f5510.el9.x86_64 2023-12-16T21:30:39.275 INFO:teuthology.orchestra.run.smithi169.stdout: thrift-0.15.0-2.el9.x86_64 2023-12-16T21:30:39.275 INFO:teuthology.orchestra.run.smithi169.stdout: 2023-12-16T21:30:39.275 INFO:teuthology.orchestra.run.smithi169.stdout:Complete! 2023-12-16T21:30:39.635 DEBUG:teuthology.orchestra.run.smithi169:> sudo yum -y install ceph-test 2023-12-16T21:30:40.197 INFO:teuthology.orchestra.run.smithi169.stdout:Last metadata expiration check: 0:00:27 ago on Sat 16 Dec 2023 09:30:13 PM UTC. 2023-12-16T21:30:40.342 INFO:teuthology.orchestra.run.smithi169.stdout:Dependencies resolved. 2023-12-16T21:30:40.343 INFO:teuthology.orchestra.run.smithi169.stdout:================================================================================ 2023-12-16T21:30:40.343 INFO:teuthology.orchestra.run.smithi169.stdout: Package Arch Version Repository Size 2023-12-16T21:30:40.343 INFO:teuthology.orchestra.run.smithi169.stdout:================================================================================ 2023-12-16T21:30:40.343 INFO:teuthology.orchestra.run.smithi169.stdout:Installing: 2023-12-16T21:30:40.343 INFO:teuthology.orchestra.run.smithi169.stdout: ceph-test x86_64 2:17.2.7-125.g1f2f5510.el9 ceph 48 M 2023-12-16T21:30:40.343 INFO:teuthology.orchestra.run.smithi169.stdout:Installing dependencies: 2023-12-16T21:30:40.343 INFO:teuthology.orchestra.run.smithi169.stdout: jq x86_64 1.6-16.el9 CentOS-AppStream 186 k 2023-12-16T21:30:40.344 INFO:teuthology.orchestra.run.smithi169.stdout: libcephsqlite x86_64 2:17.2.7-125.g1f2f5510.el9 ceph 159 k 2023-12-16T21:30:40.344 INFO:teuthology.orchestra.run.smithi169.stdout: oniguruma x86_64 6.9.6-1.el9.5 CentOS-AppStream 217 k 2023-12-16T21:30:40.344 INFO:teuthology.orchestra.run.smithi169.stdout: socat x86_64 1.7.4.1-5.el9 CentOS-AppStream 300 k 2023-12-16T21:30:40.344 INFO:teuthology.orchestra.run.smithi169.stdout: xmlstarlet x86_64 1.6.1-20.el9 CentOS-AppStream 63 k 2023-12-16T21:30:40.344 INFO:teuthology.orchestra.run.smithi169.stdout: 2023-12-16T21:30:40.344 INFO:teuthology.orchestra.run.smithi169.stdout:Transaction Summary 2023-12-16T21:30:40.344 INFO:teuthology.orchestra.run.smithi169.stdout:================================================================================ 2023-12-16T21:30:40.344 INFO:teuthology.orchestra.run.smithi169.stdout:Install 6 Packages 2023-12-16T21:30:40.344 INFO:teuthology.orchestra.run.smithi169.stdout: 2023-12-16T21:30:40.345 INFO:teuthology.orchestra.run.smithi169.stdout:Total download size: 49 M 2023-12-16T21:30:40.345 INFO:teuthology.orchestra.run.smithi169.stdout:Installed size: 195 M 2023-12-16T21:30:40.345 INFO:teuthology.orchestra.run.smithi169.stdout:Downloading Packages: 2023-12-16T21:30:40.497 INFO:teuthology.orchestra.run.smithi169.stdout:(1/6): jq-1.6-16.el9.x86_64.rpm 1.2 MB/s | 186 kB 00:00 2023-12-16T21:30:40.536 INFO:teuthology.orchestra.run.smithi045.stdout: Running scriptlet: libcephsqlite-2:17.2.7-125.g1f2f5510.el9.x86_64 5/6 2023-12-16T21:30:40.539 INFO:teuthology.orchestra.run.smithi169.stdout:(2/6): oniguruma-6.9.6-1.el9.5.x86_64.rpm 5.1 MB/s | 217 kB 00:00 2023-12-16T21:30:40.580 INFO:teuthology.orchestra.run.smithi169.stdout:(3/6): socat-1.7.4.1-5.el9.x86_64.rpm 7.1 MB/s | 300 kB 00:00 2023-12-16T21:30:40.614 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : ceph-test-2:17.2.7-125.g1f2f5510.el9.x86_64 6/6 2023-12-16T21:30:40.614 INFO:teuthology.orchestra.run.smithi169.stdout:(4/6): xmlstarlet-1.6.1-20.el9.x86_64.rpm 1.8 MB/s | 63 kB 00:00 2023-12-16T21:30:40.648 INFO:teuthology.orchestra.run.smithi169.stdout:(5/6): libcephsqlite-17.2.7-125.g1f2f5510.el9.x 525 kB/s | 159 kB 00:00 2023-12-16T21:30:42.666 INFO:teuthology.orchestra.run.smithi169.stdout:(6/6): ceph-test-17.2.7-125.g1f2f5510.el9.x86_6 21 MB/s | 48 MB 00:02 2023-12-16T21:30:42.707 INFO:teuthology.orchestra.run.smithi169.stdout:-------------------------------------------------------------------------------- 2023-12-16T21:30:42.707 INFO:teuthology.orchestra.run.smithi169.stdout:Total 21 MB/s | 49 MB 00:02 2023-12-16T21:30:42.707 INFO:teuthology.orchestra.run.smithi169.stdout:Running transaction check 2023-12-16T21:30:42.708 INFO:teuthology.orchestra.run.smithi169.stdout:Transaction check succeeded. 2023-12-16T21:30:42.708 INFO:teuthology.orchestra.run.smithi169.stdout:Running transaction test 2023-12-16T21:30:42.969 INFO:teuthology.orchestra.run.smithi045.stdout: Running scriptlet: ceph-test-2:17.2.7-125.g1f2f5510.el9.x86_64 6/6 2023-12-16T21:30:42.969 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : ceph-test-2:17.2.7-125.g1f2f5510.el9.x86_64 1/6 2023-12-16T21:30:42.969 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : libcephsqlite-2:17.2.7-125.g1f2f5510.el9.x86_64 2/6 2023-12-16T21:30:42.969 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : jq-1.6-16.el9.x86_64 3/6 2023-12-16T21:30:42.969 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : oniguruma-6.9.6-1.el9.5.x86_64 4/6 2023-12-16T21:30:42.969 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : socat-1.7.4.1-5.el9.x86_64 5/6 2023-12-16T21:30:43.073 INFO:teuthology.orchestra.run.smithi169.stdout:Transaction test succeeded. 2023-12-16T21:30:43.073 INFO:teuthology.orchestra.run.smithi169.stdout:Running transaction 2023-12-16T21:30:43.367 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : xmlstarlet-1.6.1-20.el9.x86_64 6/6 2023-12-16T21:30:43.368 INFO:teuthology.orchestra.run.smithi045.stdout: 2023-12-16T21:30:43.368 INFO:teuthology.orchestra.run.smithi045.stdout:Installed: 2023-12-16T21:30:43.368 INFO:teuthology.orchestra.run.smithi045.stdout: ceph-test-2:17.2.7-125.g1f2f5510.el9.x86_64 2023-12-16T21:30:43.368 INFO:teuthology.orchestra.run.smithi045.stdout: jq-1.6-16.el9.x86_64 2023-12-16T21:30:43.368 INFO:teuthology.orchestra.run.smithi045.stdout: libcephsqlite-2:17.2.7-125.g1f2f5510.el9.x86_64 2023-12-16T21:30:43.369 INFO:teuthology.orchestra.run.smithi045.stdout: oniguruma-6.9.6-1.el9.5.x86_64 2023-12-16T21:30:43.369 INFO:teuthology.orchestra.run.smithi045.stdout: socat-1.7.4.1-5.el9.x86_64 2023-12-16T21:30:43.369 INFO:teuthology.orchestra.run.smithi045.stdout: xmlstarlet-1.6.1-20.el9.x86_64 2023-12-16T21:30:43.369 INFO:teuthology.orchestra.run.smithi045.stdout: 2023-12-16T21:30:43.369 INFO:teuthology.orchestra.run.smithi045.stdout:Complete! 2023-12-16T21:30:43.658 DEBUG:teuthology.orchestra.run.smithi045:> sudo yum -y install ceph 2023-12-16T21:30:43.686 INFO:teuthology.orchestra.run.smithi169.stdout: Preparing : 1/1 2023-12-16T21:30:43.769 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : xmlstarlet-1.6.1-20.el9.x86_64 1/6 2023-12-16T21:30:43.810 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : socat-1.7.4.1-5.el9.x86_64 2/6 2023-12-16T21:30:43.847 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : oniguruma-6.9.6-1.el9.5.x86_64 3/6 2023-12-16T21:30:43.889 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : jq-1.6-16.el9.x86_64 4/6 2023-12-16T21:30:43.917 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : libcephsqlite-2:17.2.7-125.g1f2f5510.el9.x86_64 5/6 2023-12-16T21:30:44.232 INFO:teuthology.orchestra.run.smithi045.stdout:Last metadata expiration check: 0:00:41 ago on Sat 16 Dec 2023 09:30:03 PM UTC. 2023-12-16T21:30:44.385 INFO:teuthology.orchestra.run.smithi045.stdout:Dependencies resolved. 2023-12-16T21:30:44.387 INFO:teuthology.orchestra.run.smithi045.stdout:======================================================================================== 2023-12-16T21:30:44.388 INFO:teuthology.orchestra.run.smithi045.stdout: Package Arch Version Repository Size 2023-12-16T21:30:44.388 INFO:teuthology.orchestra.run.smithi045.stdout:======================================================================================== 2023-12-16T21:30:44.388 INFO:teuthology.orchestra.run.smithi045.stdout:Installing: 2023-12-16T21:30:44.388 INFO:teuthology.orchestra.run.smithi045.stdout: ceph x86_64 2:17.2.7-125.g1f2f5510.el9 ceph 6.5 k 2023-12-16T21:30:44.388 INFO:teuthology.orchestra.run.smithi045.stdout:Installing dependencies: 2023-12-16T21:30:44.388 INFO:teuthology.orchestra.run.smithi045.stdout: ceph-mds x86_64 2:17.2.7-125.g1f2f5510.el9 ceph 2.1 M 2023-12-16T21:30:44.388 INFO:teuthology.orchestra.run.smithi045.stdout: ceph-mgr x86_64 2:17.2.7-125.g1f2f5510.el9 ceph 1.4 M 2023-12-16T21:30:44.388 INFO:teuthology.orchestra.run.smithi045.stdout: ceph-mgr-modules-core noarch 2:17.2.7-125.g1f2f5510.el9 ceph-noarch 238 k 2023-12-16T21:30:44.388 INFO:teuthology.orchestra.run.smithi045.stdout: ceph-mon x86_64 2:17.2.7-125.g1f2f5510.el9 ceph 3.7 M 2023-12-16T21:30:44.388 INFO:teuthology.orchestra.run.smithi045.stdout: ceph-osd x86_64 2:17.2.7-125.g1f2f5510.el9 ceph 15 M 2023-12-16T21:30:44.388 INFO:teuthology.orchestra.run.smithi045.stdout: python3-bcrypt x86_64 3.2.2-1.el9 epel 43 k 2023-12-16T21:30:44.388 INFO:teuthology.orchestra.run.smithi045.stdout: python3-cffi x86_64 1.14.5-5.el9 CentOS-BaseOS 241 k 2023-12-16T21:30:44.388 INFO:teuthology.orchestra.run.smithi045.stdout: python3-cheroot noarch 8.6.0-4.el9 epel 172 k 2023-12-16T21:30:44.389 INFO:teuthology.orchestra.run.smithi045.stdout: python3-cherrypy noarch 18.6.1-2.el9 epel 358 k 2023-12-16T21:30:44.389 INFO:teuthology.orchestra.run.smithi045.stdout: python3-cryptography x86_64 36.0.1-4.el9 CentOS-BaseOS 1.1 M 2023-12-16T21:30:44.389 INFO:teuthology.orchestra.run.smithi045.stdout: python3-jaraco noarch 8.2.1-3.el9 epel 11 k 2023-12-16T21:30:44.389 INFO:teuthology.orchestra.run.smithi045.stdout: python3-jaraco-classes noarch 3.2.1-5.el9 epel 18 k 2023-12-16T21:30:44.389 INFO:teuthology.orchestra.run.smithi045.stdout: python3-jaraco-collections noarch 3.0.0-8.el9 epel 23 k 2023-12-16T21:30:44.389 INFO:teuthology.orchestra.run.smithi045.stdout: python3-jaraco-functools noarch 3.5.0-2.el9 epel 19 k 2023-12-16T21:30:44.389 INFO:teuthology.orchestra.run.smithi045.stdout: python3-jaraco-text noarch 3.2.0-6.el9 epel 20 k 2023-12-16T21:30:44.389 INFO:teuthology.orchestra.run.smithi045.stdout: python3-logutils noarch 0.3.5-21.el9 epel 46 k 2023-12-16T21:30:44.389 INFO:teuthology.orchestra.run.smithi045.stdout: python3-mako noarch 1.1.4-6.el9 CentOS-AppStream 144 k 2023-12-16T21:30:44.389 INFO:teuthology.orchestra.run.smithi045.stdout: python3-markupsafe x86_64 1.1.1-12.el9 CentOS-AppStream 32 k 2023-12-16T21:30:44.389 INFO:teuthology.orchestra.run.smithi045.stdout: python3-more-itertools noarch 8.12.0-2.el9 epel 79 k 2023-12-16T21:30:44.389 INFO:teuthology.orchestra.run.smithi045.stdout: python3-pecan noarch 1.4.2-3.el9 epel 272 k 2023-12-16T21:30:44.389 INFO:teuthology.orchestra.run.smithi045.stdout: python3-ply noarch 3.11-14.el9 CentOS-BaseOS 103 k 2023-12-16T21:30:44.390 INFO:teuthology.orchestra.run.smithi045.stdout: python3-portend noarch 3.1.0-2.el9 epel 16 k 2023-12-16T21:30:44.390 INFO:teuthology.orchestra.run.smithi045.stdout: python3-pyOpenSSL noarch 21.0.0-1.el9 epel 90 k 2023-12-16T21:30:44.390 INFO:teuthology.orchestra.run.smithi045.stdout: python3-pycparser noarch 2.20-6.el9 CentOS-BaseOS 124 k 2023-12-16T21:30:44.390 INFO:teuthology.orchestra.run.smithi045.stdout: python3-pytz noarch 2021.1-5.el9 CentOS-AppStream 47 k 2023-12-16T21:30:44.390 INFO:teuthology.orchestra.run.smithi045.stdout: python3-requests noarch 2.25.1-7.el9 CentOS-BaseOS 113 k 2023-12-16T21:30:44.390 INFO:teuthology.orchestra.run.smithi045.stdout: python3-tempora noarch 5.0.0-2.el9 epel 36 k 2023-12-16T21:30:44.390 INFO:teuthology.orchestra.run.smithi045.stdout: python3-toml noarch 0.10.2-6.el9 CentOS-AppStream 39 k 2023-12-16T21:30:44.390 INFO:teuthology.orchestra.run.smithi045.stdout: python3-urllib3 noarch 1.26.5-4.el9 CentOS-BaseOS 187 k 2023-12-16T21:30:44.390 INFO:teuthology.orchestra.run.smithi045.stdout: python3-webob noarch 1.8.7-6.el9 epel 230 k 2023-12-16T21:30:44.390 INFO:teuthology.orchestra.run.smithi045.stdout: python3-werkzeug noarch 2.0.3-3.el9.1 epel 427 k 2023-12-16T21:30:44.390 INFO:teuthology.orchestra.run.smithi045.stdout: python3-zc-lockfile noarch 2.0-10.el9 epel 20 k 2023-12-16T21:30:44.390 INFO:teuthology.orchestra.run.smithi045.stdout: 2023-12-16T21:30:44.390 INFO:teuthology.orchestra.run.smithi045.stdout:Transaction Summary 2023-12-16T21:30:44.390 INFO:teuthology.orchestra.run.smithi045.stdout:======================================================================================== 2023-12-16T21:30:44.391 INFO:teuthology.orchestra.run.smithi045.stdout:Install 33 Packages 2023-12-16T21:30:44.391 INFO:teuthology.orchestra.run.smithi045.stdout: 2023-12-16T21:30:44.391 INFO:teuthology.orchestra.run.smithi045.stdout:Total download size: 27 M 2023-12-16T21:30:44.391 INFO:teuthology.orchestra.run.smithi045.stdout:Installed size: 99 M 2023-12-16T21:30:44.391 INFO:teuthology.orchestra.run.smithi045.stdout:Downloading Packages: 2023-12-16T21:30:44.738 INFO:teuthology.orchestra.run.smithi045.stdout:(1/33): ceph-17.2.7-125.g1f2f5510.el9.x86_64.rp 35 kB/s | 6.5 kB 00:00 2023-12-16T21:30:45.063 INFO:teuthology.orchestra.run.smithi045.stdout:(2/33): ceph-mon-17.2.7-125.g1f2f5510.el9.x86_6 11 MB/s | 3.7 MB 00:00 2023-12-16T21:30:45.222 INFO:teuthology.orchestra.run.smithi045.stdout:(3/33): ceph-mgr-17.2.7-125.g1f2f5510.el9.x86_6 2.1 MB/s | 1.4 MB 00:00 2023-12-16T21:30:45.287 INFO:teuthology.orchestra.run.smithi045.stdout:(4/33): ceph-mds-17.2.7-125.g1f2f5510.el9.x86_6 2.8 MB/s | 2.1 MB 00:00 2023-12-16T21:30:45.312 INFO:teuthology.orchestra.run.smithi045.stdout:(5/33): ceph-mgr-modules-core-17.2.7-125.g1f2f5 2.6 MB/s | 238 kB 00:00 2023-12-16T21:30:45.421 INFO:teuthology.orchestra.run.smithi045.stdout:(6/33): python3-markupsafe-1.1.1-12.el9.x86_64. 374 kB/s | 32 kB 00:00 2023-12-16T21:30:45.446 INFO:teuthology.orchestra.run.smithi045.stdout:(7/33): python3-mako-1.1.4-6.el9.noarch.rpm 908 kB/s | 144 kB 00:00 2023-12-16T21:30:45.471 INFO:teuthology.orchestra.run.smithi045.stdout:(8/33): python3-pytz-2021.1-5.el9.noarch.rpm 943 kB/s | 47 kB 00:00 2023-12-16T21:30:45.497 INFO:teuthology.orchestra.run.smithi045.stdout:(9/33): python3-toml-0.10.2-6.el9.noarch.rpm 766 kB/s | 39 kB 00:00 2023-12-16T21:30:45.522 INFO:teuthology.orchestra.run.smithi045.stdout:(10/33): python3-cffi-1.14.5-5.el9.x86_64.rpm 4.7 MB/s | 241 kB 00:00 2023-12-16T21:30:45.581 INFO:teuthology.orchestra.run.smithi045.stdout:(11/33): python3-cryptography-36.0.1-4.el9.x86_ 14 MB/s | 1.1 MB 00:00 2023-12-16T21:30:45.606 INFO:teuthology.orchestra.run.smithi045.stdout:(12/33): python3-ply-3.11-14.el9.noarch.rpm 1.2 MB/s | 103 kB 00:00 2023-12-16T21:30:45.632 INFO:teuthology.orchestra.run.smithi045.stdout:(13/33): python3-pycparser-2.20-6.el9.noarch.rp 2.4 MB/s | 124 kB 00:00 2023-12-16T21:30:45.657 INFO:teuthology.orchestra.run.smithi045.stdout:(14/33): python3-requests-2.25.1-7.el9.noarch.r 2.2 MB/s | 113 kB 00:00 2023-12-16T21:30:45.682 INFO:teuthology.orchestra.run.smithi045.stdout:(15/33): python3-urllib3-1.26.5-4.el9.noarch.rp 3.6 MB/s | 187 kB 00:00 2023-12-16T21:30:45.850 INFO:teuthology.orchestra.run.smithi045.stdout:(16/33): ceph-osd-17.2.7-125.g1f2f5510.el9.x86_ 19 MB/s | 15 MB 00:00 2023-12-16T21:30:45.866 INFO:teuthology.orchestra.run.smithi045.stdout:(17/33): python3-bcrypt-3.2.2-1.el9.x86_64.rpm 208 kB/s | 43 kB 00:00 2023-12-16T21:30:45.908 INFO:teuthology.orchestra.run.smithi045.stdout:(18/33): python3-jaraco-8.2.1-3.el9.noarch.rpm 256 kB/s | 11 kB 00:00 2023-12-16T21:30:45.950 INFO:teuthology.orchestra.run.smithi045.stdout:(19/33): python3-jaraco-classes-3.2.1-5.el9.noa 425 kB/s | 18 kB 00:00 2023-12-16T21:30:45.976 INFO:teuthology.orchestra.run.smithi045.stdout:(20/33): python3-cheroot-8.6.0-4.el9.noarch.rpm 587 kB/s | 172 kB 00:00 2023-12-16T21:30:46.000 INFO:teuthology.orchestra.run.smithi045.stdout:(21/33): python3-jaraco-collections-3.0.0-8.el9 460 kB/s | 23 kB 00:00 2023-12-16T21:30:46.026 INFO:teuthology.orchestra.run.smithi045.stdout:(22/33): python3-jaraco-functools-3.5.0-2.el9.n 388 kB/s | 19 kB 00:00 2023-12-16T21:30:46.051 INFO:teuthology.orchestra.run.smithi045.stdout:(23/33): python3-jaraco-text-3.2.0-6.el9.noarch 391 kB/s | 20 kB 00:00 2023-12-16T21:30:46.076 INFO:teuthology.orchestra.run.smithi045.stdout:(24/33): python3-logutils-0.3.5-21.el9.noarch.r 921 kB/s | 46 kB 00:00 2023-12-16T21:30:46.102 INFO:teuthology.orchestra.run.smithi045.stdout:(25/33): python3-more-itertools-8.12.0-2.el9.no 1.5 MB/s | 79 kB 00:00 2023-12-16T21:30:46.127 INFO:teuthology.orchestra.run.smithi045.stdout:(26/33): python3-cherrypy-18.6.1-2.el9.noarch.r 1.3 MB/s | 358 kB 00:00 2023-12-16T21:30:46.153 INFO:teuthology.orchestra.run.smithi045.stdout:(27/33): python3-portend-3.1.0-2.el9.noarch.rpm 327 kB/s | 16 kB 00:00 2023-12-16T21:30:46.194 INFO:teuthology.orchestra.run.smithi045.stdout:(28/33): python3-pecan-1.4.2-3.el9.noarch.rpm 2.3 MB/s | 272 kB 00:00 2023-12-16T21:30:46.219 INFO:teuthology.orchestra.run.smithi045.stdout:(29/33): python3-tempora-5.0.0-2.el9.noarch.rpm 536 kB/s | 36 kB 00:00 2023-12-16T21:30:46.245 INFO:teuthology.orchestra.run.smithi045.stdout:(30/33): python3-pyOpenSSL-21.0.0-1.el9.noarch. 766 kB/s | 90 kB 00:00 2023-12-16T21:30:46.278 INFO:teuthology.orchestra.run.smithi045.stdout:(31/33): python3-webob-1.8.7-6.el9.noarch.rpm 3.8 MB/s | 230 kB 00:00 2023-12-16T21:30:46.303 INFO:teuthology.orchestra.run.smithi045.stdout:(32/33): python3-zc-lockfile-2.0-10.el9.noarch. 342 kB/s | 20 kB 00:00 2023-12-16T21:30:46.329 INFO:teuthology.orchestra.run.smithi045.stdout:(33/33): python3-werkzeug-2.0.3-3.el9.1.noarch. 3.8 MB/s | 427 kB 00:00 2023-12-16T21:30:46.333 INFO:teuthology.orchestra.run.smithi045.stdout:-------------------------------------------------------------------------------- 2023-12-16T21:30:46.334 INFO:teuthology.orchestra.run.smithi045.stdout:Total 14 MB/s | 27 MB 00:01 2023-12-16T21:30:46.334 INFO:teuthology.orchestra.run.smithi045.stdout:Running transaction check 2023-12-16T21:30:46.376 INFO:teuthology.orchestra.run.smithi045.stdout:Transaction check succeeded. 2023-12-16T21:30:46.377 INFO:teuthology.orchestra.run.smithi045.stdout:Running transaction test 2023-12-16T21:30:46.733 INFO:teuthology.orchestra.run.smithi045.stdout:Transaction test succeeded. 2023-12-16T21:30:46.733 INFO:teuthology.orchestra.run.smithi045.stdout:Running transaction 2023-12-16T21:30:47.967 INFO:teuthology.orchestra.run.smithi045.stdout: Preparing : 1/1 2023-12-16T21:30:48.015 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : python3-more-itertools-8.12.0-2.el9.noarch 1/33 2023-12-16T21:30:48.057 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : python3-jaraco-8.2.1-3.el9.noarch 2/33 2023-12-16T21:30:48.089 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : python3-jaraco-classes-3.2.1-5.el9.noarch 3/33 2023-12-16T21:30:48.185 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : python3-zc-lockfile-2.0-10.el9.noarch 4/33 2023-12-16T21:30:48.261 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : python3-werkzeug-2.0.3-3.el9.1.noarch 5/33 2023-12-16T21:30:48.309 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : python3-webob-1.8.7-6.el9.noarch 6/33 2023-12-16T21:30:48.331 INFO:teuthology.orchestra.run.smithi169.stdout: Running scriptlet: libcephsqlite-2:17.2.7-125.g1f2f5510.el9.x86_64 5/6 2023-12-16T21:30:48.383 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : python3-logutils-0.3.5-21.el9.noarch 7/33 2023-12-16T21:30:48.407 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : ceph-test-2:17.2.7-125.g1f2f5510.el9.x86_64 6/6 2023-12-16T21:30:48.436 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : python3-urllib3-1.26.5-4.el9.noarch 8/33 2023-12-16T21:30:48.474 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : python3-requests-2.25.1-7.el9.noarch 9/33 2023-12-16T21:30:48.532 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : python3-ply-3.11-14.el9.noarch 10/33 2023-12-16T21:30:48.628 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : python3-pycparser-2.20-6.el9.noarch 11/33 2023-12-16T21:30:48.827 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : python3-cffi-1.14.5-5.el9.x86_64 12/33 2023-12-16T21:30:48.869 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : python3-cryptography-36.0.1-4.el9.x86_64 13/33 2023-12-16T21:30:48.908 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : python3-pyOpenSSL-21.0.0-1.el9.noarch 14/33 2023-12-16T21:30:48.942 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : python3-bcrypt-3.2.2-1.el9.x86_64 15/33 2023-12-16T21:30:48.972 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : python3-toml-0.10.2-6.el9.noarch 16/33 2023-12-16T21:30:49.044 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : python3-jaraco-functools-3.5.0-2.el9.noarch 17/33 2023-12-16T21:30:49.074 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : python3-cheroot-8.6.0-4.el9.noarch 18/33 2023-12-16T21:30:49.108 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : python3-jaraco-text-3.2.0-6.el9.noarch 19/33 2023-12-16T21:30:49.149 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : python3-jaraco-collections-3.0.0-8.el9.noarch 20/33 2023-12-16T21:30:49.192 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : python3-pytz-2021.1-5.el9.noarch 21/33 2023-12-16T21:30:49.226 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : python3-tempora-5.0.0-2.el9.noarch 22/33 2023-12-16T21:30:49.319 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : python3-portend-3.1.0-2.el9.noarch 23/33 2023-12-16T21:30:49.366 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : python3-cherrypy-18.6.1-2.el9.noarch 24/33 2023-12-16T21:30:49.441 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : python3-markupsafe-1.1.1-12.el9.x86_64 25/33 2023-12-16T21:30:49.582 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : python3-mako-1.1.4-6.el9.noarch 26/33 2023-12-16T21:30:49.723 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : python3-pecan-1.4.2-3.el9.noarch 27/33 2023-12-16T21:30:49.879 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : ceph-mgr-modules-core-2:17.2.7-125.g1f2f5510.el9.n 28/33 2023-12-16T21:30:49.904 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : ceph-mgr-2:17.2.7-125.g1f2f5510.el9.x86_64 29/33 2023-12-16T21:30:49.930 INFO:teuthology.orchestra.run.smithi045.stdout: Running scriptlet: ceph-mgr-2:17.2.7-125.g1f2f5510.el9.x86_64 29/33 2023-12-16T21:30:49.930 INFO:teuthology.orchestra.run.smithi045.stdout:Glob pattern passed to enable, but globs are not supported for this. 2023-12-16T21:30:49.931 INFO:teuthology.orchestra.run.smithi045.stdout:Invalid unit name "ceph-mgr@*.service" escaped as "ceph-mgr@\x2a.service". 2023-12-16T21:30:49.931 INFO:teuthology.orchestra.run.smithi045.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2023-12-16T21:30:49.931 INFO:teuthology.orchestra.run.smithi045.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2023-12-16T21:30:49.931 INFO:teuthology.orchestra.run.smithi045.stdout: 2023-12-16T21:30:50.675 INFO:teuthology.orchestra.run.smithi169.stdout: Running scriptlet: ceph-test-2:17.2.7-125.g1f2f5510.el9.x86_64 6/6 2023-12-16T21:30:50.675 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : ceph-test-2:17.2.7-125.g1f2f5510.el9.x86_64 1/6 2023-12-16T21:30:50.675 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : libcephsqlite-2:17.2.7-125.g1f2f5510.el9.x86_64 2/6 2023-12-16T21:30:50.675 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : jq-1.6-16.el9.x86_64 3/6 2023-12-16T21:30:50.675 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : oniguruma-6.9.6-1.el9.5.x86_64 4/6 2023-12-16T21:30:50.675 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : socat-1.7.4.1-5.el9.x86_64 5/6 2023-12-16T21:30:51.094 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : xmlstarlet-1.6.1-20.el9.x86_64 6/6 2023-12-16T21:30:51.094 INFO:teuthology.orchestra.run.smithi169.stdout: 2023-12-16T21:30:51.094 INFO:teuthology.orchestra.run.smithi169.stdout:Installed: 2023-12-16T21:30:51.094 INFO:teuthology.orchestra.run.smithi169.stdout: ceph-test-2:17.2.7-125.g1f2f5510.el9.x86_64 2023-12-16T21:30:51.094 INFO:teuthology.orchestra.run.smithi169.stdout: jq-1.6-16.el9.x86_64 2023-12-16T21:30:51.094 INFO:teuthology.orchestra.run.smithi169.stdout: libcephsqlite-2:17.2.7-125.g1f2f5510.el9.x86_64 2023-12-16T21:30:51.095 INFO:teuthology.orchestra.run.smithi169.stdout: oniguruma-6.9.6-1.el9.5.x86_64 2023-12-16T21:30:51.095 INFO:teuthology.orchestra.run.smithi169.stdout: socat-1.7.4.1-5.el9.x86_64 2023-12-16T21:30:51.095 INFO:teuthology.orchestra.run.smithi169.stdout: xmlstarlet-1.6.1-20.el9.x86_64 2023-12-16T21:30:51.095 INFO:teuthology.orchestra.run.smithi169.stdout: 2023-12-16T21:30:51.095 INFO:teuthology.orchestra.run.smithi169.stdout:Complete! 2023-12-16T21:30:51.347 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : ceph-osd-2:17.2.7-125.g1f2f5510.el9.x86_64 30/33 2023-12-16T21:30:51.374 INFO:teuthology.orchestra.run.smithi045.stdout: Running scriptlet: ceph-osd-2:17.2.7-125.g1f2f5510.el9.x86_64 30/33 2023-12-16T21:30:51.375 INFO:teuthology.orchestra.run.smithi045.stdout:Glob pattern passed to enable, but globs are not supported for this. 2023-12-16T21:30:51.375 INFO:teuthology.orchestra.run.smithi045.stdout:Invalid unit name "ceph-osd@*.service" escaped as "ceph-osd@\x2a.service". 2023-12-16T21:30:51.375 INFO:teuthology.orchestra.run.smithi045.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2023-12-16T21:30:51.375 INFO:teuthology.orchestra.run.smithi045.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2023-12-16T21:30:51.375 INFO:teuthology.orchestra.run.smithi045.stdout: 2023-12-16T21:30:51.391 DEBUG:teuthology.orchestra.run.smithi169:> sudo yum -y install ceph 2023-12-16T21:30:51.773 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : ceph-mon-2:17.2.7-125.g1f2f5510.el9.x86_64 31/33 2023-12-16T21:30:51.797 INFO:teuthology.orchestra.run.smithi045.stdout: Running scriptlet: ceph-mon-2:17.2.7-125.g1f2f5510.el9.x86_64 31/33 2023-12-16T21:30:51.797 INFO:teuthology.orchestra.run.smithi045.stdout:Glob pattern passed to enable, but globs are not supported for this. 2023-12-16T21:30:51.797 INFO:teuthology.orchestra.run.smithi045.stdout:Invalid unit name "ceph-mon@*.service" escaped as "ceph-mon@\x2a.service". 2023-12-16T21:30:51.798 INFO:teuthology.orchestra.run.smithi045.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2023-12-16T21:30:51.798 INFO:teuthology.orchestra.run.smithi045.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2023-12-16T21:30:51.798 INFO:teuthology.orchestra.run.smithi045.stdout: 2023-12-16T21:30:51.973 INFO:teuthology.orchestra.run.smithi169.stdout:Last metadata expiration check: 0:00:38 ago on Sat 16 Dec 2023 09:30:13 PM UTC. 2023-12-16T21:30:52.015 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : ceph-mds-2:17.2.7-125.g1f2f5510.el9.x86_64 32/33 2023-12-16T21:30:52.039 INFO:teuthology.orchestra.run.smithi045.stdout: Running scriptlet: ceph-mds-2:17.2.7-125.g1f2f5510.el9.x86_64 32/33 2023-12-16T21:30:52.040 INFO:teuthology.orchestra.run.smithi045.stdout:Glob pattern passed to enable, but globs are not supported for this. 2023-12-16T21:30:52.040 INFO:teuthology.orchestra.run.smithi045.stdout:Invalid unit name "ceph-mds@*.service" escaped as "ceph-mds@\x2a.service". 2023-12-16T21:30:52.040 INFO:teuthology.orchestra.run.smithi045.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2023-12-16T21:30:52.040 INFO:teuthology.orchestra.run.smithi045.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2023-12-16T21:30:52.040 INFO:teuthology.orchestra.run.smithi045.stdout: 2023-12-16T21:30:52.098 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : ceph-2:17.2.7-125.g1f2f5510.el9.x86_64 33/33 2023-12-16T21:30:52.137 INFO:teuthology.orchestra.run.smithi169.stdout:Dependencies resolved. 2023-12-16T21:30:52.139 INFO:teuthology.orchestra.run.smithi169.stdout:======================================================================================== 2023-12-16T21:30:52.139 INFO:teuthology.orchestra.run.smithi169.stdout: Package Arch Version Repository Size 2023-12-16T21:30:52.139 INFO:teuthology.orchestra.run.smithi169.stdout:======================================================================================== 2023-12-16T21:30:52.139 INFO:teuthology.orchestra.run.smithi169.stdout:Installing: 2023-12-16T21:30:52.140 INFO:teuthology.orchestra.run.smithi169.stdout: ceph x86_64 2:17.2.7-125.g1f2f5510.el9 ceph 6.5 k 2023-12-16T21:30:52.140 INFO:teuthology.orchestra.run.smithi169.stdout:Installing dependencies: 2023-12-16T21:30:52.140 INFO:teuthology.orchestra.run.smithi169.stdout: ceph-mds x86_64 2:17.2.7-125.g1f2f5510.el9 ceph 2.1 M 2023-12-16T21:30:52.140 INFO:teuthology.orchestra.run.smithi169.stdout: ceph-mgr x86_64 2:17.2.7-125.g1f2f5510.el9 ceph 1.4 M 2023-12-16T21:30:52.140 INFO:teuthology.orchestra.run.smithi169.stdout: ceph-mgr-modules-core noarch 2:17.2.7-125.g1f2f5510.el9 ceph-noarch 238 k 2023-12-16T21:30:52.140 INFO:teuthology.orchestra.run.smithi169.stdout: ceph-mon x86_64 2:17.2.7-125.g1f2f5510.el9 ceph 3.7 M 2023-12-16T21:30:52.140 INFO:teuthology.orchestra.run.smithi169.stdout: ceph-osd x86_64 2:17.2.7-125.g1f2f5510.el9 ceph 15 M 2023-12-16T21:30:52.140 INFO:teuthology.orchestra.run.smithi169.stdout: python3-bcrypt x86_64 3.2.2-1.el9 epel 43 k 2023-12-16T21:30:52.140 INFO:teuthology.orchestra.run.smithi169.stdout: python3-cffi x86_64 1.14.5-5.el9 CentOS-BaseOS 241 k 2023-12-16T21:30:52.140 INFO:teuthology.orchestra.run.smithi169.stdout: python3-cheroot noarch 8.6.0-4.el9 epel 172 k 2023-12-16T21:30:52.140 INFO:teuthology.orchestra.run.smithi169.stdout: python3-cherrypy noarch 18.6.1-2.el9 epel 358 k 2023-12-16T21:30:52.140 INFO:teuthology.orchestra.run.smithi169.stdout: python3-cryptography x86_64 36.0.1-4.el9 CentOS-BaseOS 1.1 M 2023-12-16T21:30:52.140 INFO:teuthology.orchestra.run.smithi169.stdout: python3-jaraco noarch 8.2.1-3.el9 epel 11 k 2023-12-16T21:30:52.140 INFO:teuthology.orchestra.run.smithi169.stdout: python3-jaraco-classes noarch 3.2.1-5.el9 epel 18 k 2023-12-16T21:30:52.141 INFO:teuthology.orchestra.run.smithi169.stdout: python3-jaraco-collections noarch 3.0.0-8.el9 epel 23 k 2023-12-16T21:30:52.141 INFO:teuthology.orchestra.run.smithi169.stdout: python3-jaraco-functools noarch 3.5.0-2.el9 epel 19 k 2023-12-16T21:30:52.141 INFO:teuthology.orchestra.run.smithi169.stdout: python3-jaraco-text noarch 3.2.0-6.el9 epel 20 k 2023-12-16T21:30:52.141 INFO:teuthology.orchestra.run.smithi169.stdout: python3-logutils noarch 0.3.5-21.el9 epel 46 k 2023-12-16T21:30:52.141 INFO:teuthology.orchestra.run.smithi169.stdout: python3-mako noarch 1.1.4-6.el9 CentOS-AppStream 144 k 2023-12-16T21:30:52.141 INFO:teuthology.orchestra.run.smithi169.stdout: python3-markupsafe x86_64 1.1.1-12.el9 CentOS-AppStream 32 k 2023-12-16T21:30:52.141 INFO:teuthology.orchestra.run.smithi169.stdout: python3-more-itertools noarch 8.12.0-2.el9 epel 79 k 2023-12-16T21:30:52.141 INFO:teuthology.orchestra.run.smithi169.stdout: python3-pecan noarch 1.4.2-3.el9 epel 272 k 2023-12-16T21:30:52.141 INFO:teuthology.orchestra.run.smithi169.stdout: python3-ply noarch 3.11-14.el9 CentOS-BaseOS 103 k 2023-12-16T21:30:52.141 INFO:teuthology.orchestra.run.smithi169.stdout: python3-portend noarch 3.1.0-2.el9 epel 16 k 2023-12-16T21:30:52.141 INFO:teuthology.orchestra.run.smithi169.stdout: python3-pyOpenSSL noarch 21.0.0-1.el9 epel 90 k 2023-12-16T21:30:52.141 INFO:teuthology.orchestra.run.smithi169.stdout: python3-pycparser noarch 2.20-6.el9 CentOS-BaseOS 124 k 2023-12-16T21:30:52.141 INFO:teuthology.orchestra.run.smithi169.stdout: python3-pytz noarch 2021.1-5.el9 CentOS-AppStream 47 k 2023-12-16T21:30:52.141 INFO:teuthology.orchestra.run.smithi169.stdout: python3-requests noarch 2.25.1-7.el9 CentOS-BaseOS 113 k 2023-12-16T21:30:52.142 INFO:teuthology.orchestra.run.smithi169.stdout: python3-tempora noarch 5.0.0-2.el9 epel 36 k 2023-12-16T21:30:52.142 INFO:teuthology.orchestra.run.smithi169.stdout: python3-toml noarch 0.10.2-6.el9 CentOS-AppStream 39 k 2023-12-16T21:30:52.142 INFO:teuthology.orchestra.run.smithi169.stdout: python3-urllib3 noarch 1.26.5-4.el9 CentOS-BaseOS 187 k 2023-12-16T21:30:52.142 INFO:teuthology.orchestra.run.smithi169.stdout: python3-webob noarch 1.8.7-6.el9 epel 230 k 2023-12-16T21:30:52.142 INFO:teuthology.orchestra.run.smithi169.stdout: python3-werkzeug noarch 2.0.3-3.el9.1 epel 427 k 2023-12-16T21:30:52.142 INFO:teuthology.orchestra.run.smithi169.stdout: python3-zc-lockfile noarch 2.0-10.el9 epel 20 k 2023-12-16T21:30:52.142 INFO:teuthology.orchestra.run.smithi169.stdout: 2023-12-16T21:30:52.142 INFO:teuthology.orchestra.run.smithi169.stdout:Transaction Summary 2023-12-16T21:30:52.142 INFO:teuthology.orchestra.run.smithi169.stdout:======================================================================================== 2023-12-16T21:30:52.142 INFO:teuthology.orchestra.run.smithi169.stdout:Install 33 Packages 2023-12-16T21:30:52.142 INFO:teuthology.orchestra.run.smithi169.stdout: 2023-12-16T21:30:52.143 INFO:teuthology.orchestra.run.smithi169.stdout:Total download size: 27 M 2023-12-16T21:30:52.143 INFO:teuthology.orchestra.run.smithi169.stdout:Installed size: 99 M 2023-12-16T21:30:52.143 INFO:teuthology.orchestra.run.smithi169.stdout:Downloading Packages: 2023-12-16T21:30:52.482 INFO:teuthology.orchestra.run.smithi169.stdout:(1/33): ceph-17.2.7-125.g1f2f5510.el9.x86_64.rp 39 kB/s | 6.5 kB 00:00 2023-12-16T21:30:52.699 INFO:teuthology.orchestra.run.smithi169.stdout:(2/33): ceph-mds-17.2.7-125.g1f2f5510.el9.x86_6 5.4 MB/s | 2.1 MB 00:00 2023-12-16T21:30:52.916 INFO:teuthology.orchestra.run.smithi169.stdout:(3/33): ceph-mon-17.2.7-125.g1f2f5510.el9.x86_6 8.5 MB/s | 3.7 MB 00:00 2023-12-16T21:30:53.000 INFO:teuthology.orchestra.run.smithi169.stdout:(4/33): ceph-mgr-modules-core-17.2.7-125.g1f2f5 2.8 MB/s | 238 kB 00:00 2023-12-16T21:30:53.100 INFO:teuthology.orchestra.run.smithi169.stdout:(5/33): python3-mako-1.1.4-6.el9.noarch.rpm 1.4 MB/s | 144 kB 00:00 2023-12-16T21:30:53.134 INFO:teuthology.orchestra.run.smithi169.stdout:(6/33): python3-markupsafe-1.1.1-12.el9.x86_64. 954 kB/s | 32 kB 00:00 2023-12-16T21:30:53.167 INFO:teuthology.orchestra.run.smithi169.stdout:(7/33): python3-pytz-2021.1-5.el9.noarch.rpm 1.4 MB/s | 47 kB 00:00 2023-12-16T21:30:53.201 INFO:teuthology.orchestra.run.smithi169.stdout:(8/33): ceph-mgr-17.2.7-125.g1f2f5510.el9.x86_6 1.6 MB/s | 1.4 MB 00:00 2023-12-16T21:30:53.226 INFO:teuthology.orchestra.run.smithi169.stdout:(9/33): python3-toml-0.10.2-6.el9.noarch.rpm 654 kB/s | 39 kB 00:00 2023-12-16T21:30:53.252 INFO:teuthology.orchestra.run.smithi169.stdout:(10/33): python3-cffi-1.14.5-5.el9.x86_64.rpm 4.7 MB/s | 241 kB 00:00 2023-12-16T21:30:53.302 INFO:teuthology.orchestra.run.smithi169.stdout:(11/33): python3-cryptography-36.0.1-4.el9.x86_ 15 MB/s | 1.1 MB 00:00 2023-12-16T21:30:53.319 INFO:teuthology.orchestra.run.smithi169.stdout:(12/33): python3-ply-3.11-14.el9.noarch.rpm 1.5 MB/s | 103 kB 00:00 2023-12-16T21:30:53.344 INFO:teuthology.orchestra.run.smithi169.stdout:(13/33): python3-pycparser-2.20-6.el9.noarch.rp 2.9 MB/s | 124 kB 00:00 2023-12-16T21:30:53.370 INFO:teuthology.orchestra.run.smithi169.stdout:(14/33): python3-requests-2.25.1-7.el9.noarch.r 2.2 MB/s | 113 kB 00:00 2023-12-16T21:30:53.395 INFO:teuthology.orchestra.run.smithi169.stdout:(15/33): python3-urllib3-1.26.5-4.el9.noarch.rp 3.6 MB/s | 187 kB 00:00 2023-12-16T21:30:53.545 INFO:teuthology.orchestra.run.smithi169.stdout:(16/33): ceph-osd-17.2.7-125.g1f2f5510.el9.x86_ 18 MB/s | 15 MB 00:00 2023-12-16T21:30:53.780 INFO:teuthology.orchestra.run.smithi169.stdout:(17/33): python3-bcrypt-3.2.2-1.el9.x86_64.rpm 106 kB/s | 43 kB 00:00 2023-12-16T21:30:53.783 INFO:teuthology.orchestra.run.smithi045.stdout: Running scriptlet: ceph-2:17.2.7-125.g1f2f5510.el9.x86_64 33/33 2023-12-16T21:30:53.783 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : ceph-2:17.2.7-125.g1f2f5510.el9.x86_64 1/33 2023-12-16T21:30:53.783 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : ceph-mds-2:17.2.7-125.g1f2f5510.el9.x86_64 2/33 2023-12-16T21:30:53.783 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : ceph-mgr-2:17.2.7-125.g1f2f5510.el9.x86_64 3/33 2023-12-16T21:30:53.783 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : ceph-mon-2:17.2.7-125.g1f2f5510.el9.x86_64 4/33 2023-12-16T21:30:53.783 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : ceph-osd-2:17.2.7-125.g1f2f5510.el9.x86_64 5/33 2023-12-16T21:30:53.784 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : ceph-mgr-modules-core-2:17.2.7-125.g1f2f5510.el9.n 6/33 2023-12-16T21:30:53.784 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : python3-mako-1.1.4-6.el9.noarch 7/33 2023-12-16T21:30:53.784 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : python3-markupsafe-1.1.1-12.el9.x86_64 8/33 2023-12-16T21:30:53.784 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : python3-pytz-2021.1-5.el9.noarch 9/33 2023-12-16T21:30:53.784 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : python3-toml-0.10.2-6.el9.noarch 10/33 2023-12-16T21:30:53.784 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : python3-cffi-1.14.5-5.el9.x86_64 11/33 2023-12-16T21:30:53.784 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : python3-cryptography-36.0.1-4.el9.x86_64 12/33 2023-12-16T21:30:53.784 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : python3-ply-3.11-14.el9.noarch 13/33 2023-12-16T21:30:53.784 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : python3-pycparser-2.20-6.el9.noarch 14/33 2023-12-16T21:30:53.784 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : python3-requests-2.25.1-7.el9.noarch 15/33 2023-12-16T21:30:53.784 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : python3-urllib3-1.26.5-4.el9.noarch 16/33 2023-12-16T21:30:53.784 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : python3-bcrypt-3.2.2-1.el9.x86_64 17/33 2023-12-16T21:30:53.784 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : python3-cheroot-8.6.0-4.el9.noarch 18/33 2023-12-16T21:30:53.784 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : python3-cherrypy-18.6.1-2.el9.noarch 19/33 2023-12-16T21:30:53.785 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : python3-jaraco-8.2.1-3.el9.noarch 20/33 2023-12-16T21:30:53.785 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : python3-jaraco-classes-3.2.1-5.el9.noarch 21/33 2023-12-16T21:30:53.785 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : python3-jaraco-collections-3.0.0-8.el9.noarch 22/33 2023-12-16T21:30:53.785 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : python3-jaraco-functools-3.5.0-2.el9.noarch 23/33 2023-12-16T21:30:53.785 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : python3-jaraco-text-3.2.0-6.el9.noarch 24/33 2023-12-16T21:30:53.785 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : python3-logutils-0.3.5-21.el9.noarch 25/33 2023-12-16T21:30:53.785 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : python3-more-itertools-8.12.0-2.el9.noarch 26/33 2023-12-16T21:30:53.785 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : python3-pecan-1.4.2-3.el9.noarch 27/33 2023-12-16T21:30:53.785 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : python3-portend-3.1.0-2.el9.noarch 28/33 2023-12-16T21:30:53.785 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : python3-pyOpenSSL-21.0.0-1.el9.noarch 29/33 2023-12-16T21:30:53.785 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : python3-tempora-5.0.0-2.el9.noarch 30/33 2023-12-16T21:30:53.785 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : python3-webob-1.8.7-6.el9.noarch 31/33 2023-12-16T21:30:53.785 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : python3-werkzeug-2.0.3-3.el9.1.noarch 32/33 2023-12-16T21:30:53.855 INFO:teuthology.orchestra.run.smithi169.stdout:(18/33): python3-jaraco-8.2.1-3.el9.noarch.rpm 142 kB/s | 11 kB 00:00 2023-12-16T21:30:53.922 INFO:teuthology.orchestra.run.smithi169.stdout:(19/33): python3-cheroot-8.6.0-4.el9.noarch.rpm 326 kB/s | 172 kB 00:00 2023-12-16T21:30:53.939 INFO:teuthology.orchestra.run.smithi169.stdout:(20/33): python3-jaraco-classes-3.2.1-5.el9.noa 212 kB/s | 18 kB 00:00 2023-12-16T21:30:53.989 INFO:teuthology.orchestra.run.smithi169.stdout:(21/33): python3-jaraco-collections-3.0.0-8.el9 346 kB/s | 23 kB 00:00 2023-12-16T21:30:54.014 INFO:teuthology.orchestra.run.smithi169.stdout:(22/33): python3-jaraco-functools-3.5.0-2.el9.n 258 kB/s | 19 kB 00:00 2023-12-16T21:30:54.040 INFO:teuthology.orchestra.run.smithi169.stdout:(23/33): python3-cherrypy-18.6.1-2.el9.noarch.r 803 kB/s | 358 kB 00:00 2023-12-16T21:30:54.065 INFO:teuthology.orchestra.run.smithi169.stdout:(24/33): python3-jaraco-text-3.2.0-6.el9.noarch 260 kB/s | 20 kB 00:00 2023-12-16T21:30:54.124 INFO:teuthology.orchestra.run.smithi169.stdout:(25/33): python3-more-itertools-8.12.0-2.el9.no 943 kB/s | 79 kB 00:00 2023-12-16T21:30:54.140 INFO:teuthology.orchestra.run.smithi169.stdout:(26/33): python3-logutils-0.3.5-21.el9.noarch.r 368 kB/s | 46 kB 00:00 2023-12-16T21:30:54.199 INFO:teuthology.orchestra.run.smithi169.stdout:(27/33): python3-pecan-1.4.2-3.el9.noarch.rpm 2.0 MB/s | 272 kB 00:00 2023-12-16T21:30:54.216 INFO:teuthology.orchestra.run.smithi169.stdout:(28/33): python3-portend-3.1.0-2.el9.noarch.rpm 178 kB/s | 16 kB 00:00 2023-12-16T21:30:54.241 INFO:teuthology.orchestra.run.smithi169.stdout:(29/33): python3-pyOpenSSL-21.0.0-1.el9.noarch. 894 kB/s | 90 kB 00:00 2023-12-16T21:30:54.275 INFO:teuthology.orchestra.run.smithi169.stdout:(30/33): python3-tempora-5.0.0-2.el9.noarch.rpm 475 kB/s | 36 kB 00:00 2023-12-16T21:30:54.350 INFO:teuthology.orchestra.run.smithi169.stdout:(31/33): python3-zc-lockfile-2.0-10.el9.noarch. 266 kB/s | 20 kB 00:00 2023-12-16T21:30:54.375 INFO:teuthology.orchestra.run.smithi169.stdout:(32/33): python3-werkzeug-2.0.3-3.el9.1.noarch. 3.1 MB/s | 427 kB 00:00 2023-12-16T21:30:54.401 INFO:teuthology.orchestra.run.smithi169.stdout:(33/33): python3-webob-1.8.7-6.el9.noarch.rpm 1.2 MB/s | 230 kB 00:00 2023-12-16T21:30:54.407 INFO:teuthology.orchestra.run.smithi169.stdout:-------------------------------------------------------------------------------- 2023-12-16T21:30:54.407 INFO:teuthology.orchestra.run.smithi169.stdout:Total 12 MB/s | 27 MB 00:02 2023-12-16T21:30:54.407 INFO:teuthology.orchestra.run.smithi169.stdout:Running transaction check 2023-12-16T21:30:54.443 INFO:teuthology.orchestra.run.smithi169.stdout:Transaction check succeeded. 2023-12-16T21:30:54.443 INFO:teuthology.orchestra.run.smithi169.stdout:Running transaction test 2023-12-16T21:30:54.692 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : python3-zc-lockfile-2.0-10.el9.noarch 33/33 2023-12-16T21:30:54.692 INFO:teuthology.orchestra.run.smithi045.stdout: 2023-12-16T21:30:54.692 INFO:teuthology.orchestra.run.smithi045.stdout:Installed: 2023-12-16T21:30:54.692 INFO:teuthology.orchestra.run.smithi045.stdout: ceph-2:17.2.7-125.g1f2f5510.el9.x86_64 2023-12-16T21:30:54.692 INFO:teuthology.orchestra.run.smithi045.stdout: ceph-mds-2:17.2.7-125.g1f2f5510.el9.x86_64 2023-12-16T21:30:54.692 INFO:teuthology.orchestra.run.smithi045.stdout: ceph-mgr-2:17.2.7-125.g1f2f5510.el9.x86_64 2023-12-16T21:30:54.692 INFO:teuthology.orchestra.run.smithi045.stdout: ceph-mgr-modules-core-2:17.2.7-125.g1f2f5510.el9.noarch 2023-12-16T21:30:54.692 INFO:teuthology.orchestra.run.smithi045.stdout: ceph-mon-2:17.2.7-125.g1f2f5510.el9.x86_64 2023-12-16T21:30:54.692 INFO:teuthology.orchestra.run.smithi045.stdout: ceph-osd-2:17.2.7-125.g1f2f5510.el9.x86_64 2023-12-16T21:30:54.692 INFO:teuthology.orchestra.run.smithi045.stdout: python3-bcrypt-3.2.2-1.el9.x86_64 2023-12-16T21:30:54.693 INFO:teuthology.orchestra.run.smithi045.stdout: python3-cffi-1.14.5-5.el9.x86_64 2023-12-16T21:30:54.693 INFO:teuthology.orchestra.run.smithi045.stdout: python3-cheroot-8.6.0-4.el9.noarch 2023-12-16T21:30:54.693 INFO:teuthology.orchestra.run.smithi045.stdout: python3-cherrypy-18.6.1-2.el9.noarch 2023-12-16T21:30:54.693 INFO:teuthology.orchestra.run.smithi045.stdout: python3-cryptography-36.0.1-4.el9.x86_64 2023-12-16T21:30:54.693 INFO:teuthology.orchestra.run.smithi045.stdout: python3-jaraco-8.2.1-3.el9.noarch 2023-12-16T21:30:54.693 INFO:teuthology.orchestra.run.smithi045.stdout: python3-jaraco-classes-3.2.1-5.el9.noarch 2023-12-16T21:30:54.693 INFO:teuthology.orchestra.run.smithi045.stdout: python3-jaraco-collections-3.0.0-8.el9.noarch 2023-12-16T21:30:54.693 INFO:teuthology.orchestra.run.smithi045.stdout: python3-jaraco-functools-3.5.0-2.el9.noarch 2023-12-16T21:30:54.693 INFO:teuthology.orchestra.run.smithi045.stdout: python3-jaraco-text-3.2.0-6.el9.noarch 2023-12-16T21:30:54.693 INFO:teuthology.orchestra.run.smithi045.stdout: python3-logutils-0.3.5-21.el9.noarch 2023-12-16T21:30:54.693 INFO:teuthology.orchestra.run.smithi045.stdout: python3-mako-1.1.4-6.el9.noarch 2023-12-16T21:30:54.694 INFO:teuthology.orchestra.run.smithi045.stdout: python3-markupsafe-1.1.1-12.el9.x86_64 2023-12-16T21:30:54.694 INFO:teuthology.orchestra.run.smithi045.stdout: python3-more-itertools-8.12.0-2.el9.noarch 2023-12-16T21:30:54.694 INFO:teuthology.orchestra.run.smithi045.stdout: python3-pecan-1.4.2-3.el9.noarch 2023-12-16T21:30:54.694 INFO:teuthology.orchestra.run.smithi045.stdout: python3-ply-3.11-14.el9.noarch 2023-12-16T21:30:54.694 INFO:teuthology.orchestra.run.smithi045.stdout: python3-portend-3.1.0-2.el9.noarch 2023-12-16T21:30:54.694 INFO:teuthology.orchestra.run.smithi045.stdout: python3-pyOpenSSL-21.0.0-1.el9.noarch 2023-12-16T21:30:54.694 INFO:teuthology.orchestra.run.smithi045.stdout: python3-pycparser-2.20-6.el9.noarch 2023-12-16T21:30:54.694 INFO:teuthology.orchestra.run.smithi045.stdout: python3-pytz-2021.1-5.el9.noarch 2023-12-16T21:30:54.694 INFO:teuthology.orchestra.run.smithi045.stdout: python3-requests-2.25.1-7.el9.noarch 2023-12-16T21:30:54.694 INFO:teuthology.orchestra.run.smithi045.stdout: python3-tempora-5.0.0-2.el9.noarch 2023-12-16T21:30:54.694 INFO:teuthology.orchestra.run.smithi045.stdout: python3-toml-0.10.2-6.el9.noarch 2023-12-16T21:30:54.694 INFO:teuthology.orchestra.run.smithi045.stdout: python3-urllib3-1.26.5-4.el9.noarch 2023-12-16T21:30:54.694 INFO:teuthology.orchestra.run.smithi045.stdout: python3-webob-1.8.7-6.el9.noarch 2023-12-16T21:30:54.695 INFO:teuthology.orchestra.run.smithi045.stdout: python3-werkzeug-2.0.3-3.el9.1.noarch 2023-12-16T21:30:54.695 INFO:teuthology.orchestra.run.smithi045.stdout: python3-zc-lockfile-2.0-10.el9.noarch 2023-12-16T21:30:54.695 INFO:teuthology.orchestra.run.smithi045.stdout: 2023-12-16T21:30:54.695 INFO:teuthology.orchestra.run.smithi045.stdout:Complete! 2023-12-16T21:30:54.774 INFO:teuthology.orchestra.run.smithi169.stdout:Transaction test succeeded. 2023-12-16T21:30:54.775 INFO:teuthology.orchestra.run.smithi169.stdout:Running transaction 2023-12-16T21:30:55.019 DEBUG:teuthology.orchestra.run.smithi045:> sudo yum -y install ceph-base 2023-12-16T21:30:55.599 INFO:teuthology.orchestra.run.smithi045.stdout:Last metadata expiration check: 0:00:52 ago on Sat 16 Dec 2023 09:30:03 PM UTC. 2023-12-16T21:30:55.694 INFO:teuthology.orchestra.run.smithi045.stdout:Package ceph-base-2:17.2.7-125.g1f2f5510.el9.x86_64 is already installed. 2023-12-16T21:30:55.747 INFO:teuthology.orchestra.run.smithi045.stdout:Dependencies resolved. 2023-12-16T21:30:55.748 INFO:teuthology.orchestra.run.smithi045.stdout:Nothing to do. 2023-12-16T21:30:55.748 INFO:teuthology.orchestra.run.smithi045.stdout:Complete! 2023-12-16T21:30:55.819 DEBUG:teuthology.orchestra.run.smithi045:> sudo yum -y install cephadm 2023-12-16T21:30:55.965 INFO:teuthology.orchestra.run.smithi169.stdout: Preparing : 1/1 2023-12-16T21:30:56.020 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : python3-more-itertools-8.12.0-2.el9.noarch 1/33 2023-12-16T21:30:56.052 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : python3-jaraco-8.2.1-3.el9.noarch 2/33 2023-12-16T21:30:56.085 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : python3-jaraco-classes-3.2.1-5.el9.noarch 3/33 2023-12-16T21:30:56.176 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : python3-zc-lockfile-2.0-10.el9.noarch 4/33 2023-12-16T21:30:56.255 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : python3-werkzeug-2.0.3-3.el9.1.noarch 5/33 2023-12-16T21:30:56.296 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : python3-webob-1.8.7-6.el9.noarch 6/33 2023-12-16T21:30:56.368 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : python3-logutils-0.3.5-21.el9.noarch 7/33 2023-12-16T21:30:56.396 INFO:teuthology.orchestra.run.smithi045.stdout:Last metadata expiration check: 0:00:53 ago on Sat 16 Dec 2023 09:30:03 PM UTC. 2023-12-16T21:30:56.424 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : python3-urllib3-1.26.5-4.el9.noarch 8/33 2023-12-16T21:30:56.460 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : python3-requests-2.25.1-7.el9.noarch 9/33 2023-12-16T21:30:56.509 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : python3-ply-3.11-14.el9.noarch 10/33 2023-12-16T21:30:56.544 INFO:teuthology.orchestra.run.smithi045.stdout:Dependencies resolved. 2023-12-16T21:30:56.545 INFO:teuthology.orchestra.run.smithi045.stdout:================================================================================ 2023-12-16T21:30:56.545 INFO:teuthology.orchestra.run.smithi045.stdout: Package Arch Version Repository Size 2023-12-16T21:30:56.545 INFO:teuthology.orchestra.run.smithi045.stdout:================================================================================ 2023-12-16T21:30:56.545 INFO:teuthology.orchestra.run.smithi045.stdout:Installing: 2023-12-16T21:30:56.545 INFO:teuthology.orchestra.run.smithi045.stdout: cephadm noarch 2:17.2.7-125.g1f2f5510.el9 ceph-noarch 80 k 2023-12-16T21:30:56.545 INFO:teuthology.orchestra.run.smithi045.stdout: 2023-12-16T21:30:56.545 INFO:teuthology.orchestra.run.smithi045.stdout:Transaction Summary 2023-12-16T21:30:56.545 INFO:teuthology.orchestra.run.smithi045.stdout:================================================================================ 2023-12-16T21:30:56.545 INFO:teuthology.orchestra.run.smithi045.stdout:Install 1 Package 2023-12-16T21:30:56.545 INFO:teuthology.orchestra.run.smithi045.stdout: 2023-12-16T21:30:56.546 INFO:teuthology.orchestra.run.smithi045.stdout:Total download size: 80 k 2023-12-16T21:30:56.546 INFO:teuthology.orchestra.run.smithi045.stdout:Installed size: 364 k 2023-12-16T21:30:56.546 INFO:teuthology.orchestra.run.smithi045.stdout:Downloading Packages: 2023-12-16T21:30:56.568 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : python3-pycparser-2.20-6.el9.noarch 11/33 2023-12-16T21:30:56.762 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : python3-cffi-1.14.5-5.el9.x86_64 12/33 2023-12-16T21:30:56.806 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : python3-cryptography-36.0.1-4.el9.x86_64 13/33 2023-12-16T21:30:56.836 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : python3-pyOpenSSL-21.0.0-1.el9.noarch 14/33 2023-12-16T21:30:56.848 INFO:teuthology.orchestra.run.smithi045.stdout:cephadm-17.2.7-125.g1f2f5510.el9.noarch.rpm 267 kB/s | 80 kB 00:00 2023-12-16T21:30:56.848 INFO:teuthology.orchestra.run.smithi045.stdout:-------------------------------------------------------------------------------- 2023-12-16T21:30:56.848 INFO:teuthology.orchestra.run.smithi045.stdout:Total 265 kB/s | 80 kB 00:00 2023-12-16T21:30:56.849 INFO:teuthology.orchestra.run.smithi045.stdout:Running transaction check 2023-12-16T21:30:56.854 INFO:teuthology.orchestra.run.smithi045.stdout:Transaction check succeeded. 2023-12-16T21:30:56.854 INFO:teuthology.orchestra.run.smithi045.stdout:Running transaction test 2023-12-16T21:30:56.856 INFO:teuthology.orchestra.run.smithi045.stdout:Transaction test succeeded. 2023-12-16T21:30:56.856 INFO:teuthology.orchestra.run.smithi045.stdout:Running transaction 2023-12-16T21:30:56.879 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : python3-bcrypt-3.2.2-1.el9.x86_64 15/33 2023-12-16T21:30:56.909 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : python3-toml-0.10.2-6.el9.noarch 16/33 2023-12-16T21:30:56.933 INFO:teuthology.orchestra.run.smithi045.stdout: Preparing : 1/1 2023-12-16T21:30:56.986 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : python3-jaraco-functools-3.5.0-2.el9.noarch 17/33 2023-12-16T21:30:57.020 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : python3-cheroot-8.6.0-4.el9.noarch 18/33 2023-12-16T21:30:57.053 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : python3-jaraco-text-3.2.0-6.el9.noarch 19/33 2023-12-16T21:30:57.093 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : python3-jaraco-collections-3.0.0-8.el9.noarch 20/33 2023-12-16T21:30:57.128 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : python3-pytz-2021.1-5.el9.noarch 21/33 2023-12-16T21:30:57.163 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : python3-tempora-5.0.0-2.el9.noarch 22/33 2023-12-16T21:30:57.283 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : python3-portend-3.1.0-2.el9.noarch 23/33 2023-12-16T21:30:57.315 INFO:teuthology.orchestra.run.smithi045.stdout: Running scriptlet: cephadm-2:17.2.7-125.g1f2f5510.el9.noarch 1/1 2023-12-16T21:30:57.319 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : python3-cherrypy-18.6.1-2.el9.noarch 24/33 2023-12-16T21:30:57.399 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : python3-markupsafe-1.1.1-12.el9.x86_64 25/33 2023-12-16T21:30:57.423 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : cephadm-2:17.2.7-125.g1f2f5510.el9.noarch 1/1 2023-12-16T21:30:57.535 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : python3-mako-1.1.4-6.el9.noarch 26/33 2023-12-16T21:30:57.660 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : python3-pecan-1.4.2-3.el9.noarch 27/33 2023-12-16T21:30:57.781 INFO:teuthology.orchestra.run.smithi045.stdout: Running scriptlet: cephadm-2:17.2.7-125.g1f2f5510.el9.noarch 1/1 2023-12-16T21:30:57.811 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : ceph-mgr-modules-core-2:17.2.7-125.g1f2f5510.el9.n 28/33 2023-12-16T21:30:57.840 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : ceph-mgr-2:17.2.7-125.g1f2f5510.el9.x86_64 29/33 2023-12-16T21:30:57.866 INFO:teuthology.orchestra.run.smithi169.stdout: Running scriptlet: ceph-mgr-2:17.2.7-125.g1f2f5510.el9.x86_64 29/33 2023-12-16T21:30:57.866 INFO:teuthology.orchestra.run.smithi169.stdout:Glob pattern passed to enable, but globs are not supported for this. 2023-12-16T21:30:57.866 INFO:teuthology.orchestra.run.smithi169.stdout:Invalid unit name "ceph-mgr@*.service" escaped as "ceph-mgr@\x2a.service". 2023-12-16T21:30:57.866 INFO:teuthology.orchestra.run.smithi169.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2023-12-16T21:30:57.866 INFO:teuthology.orchestra.run.smithi169.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2023-12-16T21:30:57.867 INFO:teuthology.orchestra.run.smithi169.stdout: 2023-12-16T21:30:58.090 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : cephadm-2:17.2.7-125.g1f2f5510.el9.noarch 1/1 2023-12-16T21:30:58.090 INFO:teuthology.orchestra.run.smithi045.stdout: 2023-12-16T21:30:58.090 INFO:teuthology.orchestra.run.smithi045.stdout:Installed: 2023-12-16T21:30:58.090 INFO:teuthology.orchestra.run.smithi045.stdout: cephadm-2:17.2.7-125.g1f2f5510.el9.noarch 2023-12-16T21:30:58.090 INFO:teuthology.orchestra.run.smithi045.stdout: 2023-12-16T21:30:58.090 INFO:teuthology.orchestra.run.smithi045.stdout:Complete! 2023-12-16T21:30:58.296 DEBUG:teuthology.orchestra.run.smithi045:> sudo yum -y install ceph-immutable-object-cache 2023-12-16T21:30:58.876 INFO:teuthology.orchestra.run.smithi045.stdout:Last metadata expiration check: 0:00:55 ago on Sat 16 Dec 2023 09:30:03 PM UTC. 2023-12-16T21:30:59.026 INFO:teuthology.orchestra.run.smithi045.stdout:Dependencies resolved. 2023-12-16T21:30:59.026 INFO:teuthology.orchestra.run.smithi045.stdout:================================================================================ 2023-12-16T21:30:59.027 INFO:teuthology.orchestra.run.smithi045.stdout: Package Arch Version Repo Size 2023-12-16T21:30:59.027 INFO:teuthology.orchestra.run.smithi045.stdout:================================================================================ 2023-12-16T21:30:59.027 INFO:teuthology.orchestra.run.smithi045.stdout:Installing: 2023-12-16T21:30:59.027 INFO:teuthology.orchestra.run.smithi045.stdout: ceph-immutable-object-cache x86_64 2:17.2.7-125.g1f2f5510.el9 ceph 137 k 2023-12-16T21:30:59.027 INFO:teuthology.orchestra.run.smithi045.stdout: 2023-12-16T21:30:59.027 INFO:teuthology.orchestra.run.smithi045.stdout:Transaction Summary 2023-12-16T21:30:59.027 INFO:teuthology.orchestra.run.smithi045.stdout:================================================================================ 2023-12-16T21:30:59.027 INFO:teuthology.orchestra.run.smithi045.stdout:Install 1 Package 2023-12-16T21:30:59.027 INFO:teuthology.orchestra.run.smithi045.stdout: 2023-12-16T21:30:59.027 INFO:teuthology.orchestra.run.smithi045.stdout:Total download size: 137 k 2023-12-16T21:30:59.027 INFO:teuthology.orchestra.run.smithi045.stdout:Installed size: 379 k 2023-12-16T21:30:59.027 INFO:teuthology.orchestra.run.smithi045.stdout:Downloading Packages: 2023-12-16T21:30:59.208 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : ceph-osd-2:17.2.7-125.g1f2f5510.el9.x86_64 30/33 2023-12-16T21:30:59.235 INFO:teuthology.orchestra.run.smithi169.stdout: Running scriptlet: ceph-osd-2:17.2.7-125.g1f2f5510.el9.x86_64 30/33 2023-12-16T21:30:59.235 INFO:teuthology.orchestra.run.smithi169.stdout:Glob pattern passed to enable, but globs are not supported for this. 2023-12-16T21:30:59.235 INFO:teuthology.orchestra.run.smithi169.stdout:Invalid unit name "ceph-osd@*.service" escaped as "ceph-osd@\x2a.service". 2023-12-16T21:30:59.236 INFO:teuthology.orchestra.run.smithi169.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2023-12-16T21:30:59.236 INFO:teuthology.orchestra.run.smithi169.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2023-12-16T21:30:59.236 INFO:teuthology.orchestra.run.smithi169.stdout: 2023-12-16T21:30:59.380 INFO:teuthology.orchestra.run.smithi045.stdout:ceph-immutable-object-cache-17.2.7-125.g1f2f551 389 kB/s | 137 kB 00:00 2023-12-16T21:30:59.380 INFO:teuthology.orchestra.run.smithi045.stdout:-------------------------------------------------------------------------------- 2023-12-16T21:30:59.381 INFO:teuthology.orchestra.run.smithi045.stdout:Total 388 kB/s | 137 kB 00:00 2023-12-16T21:30:59.381 INFO:teuthology.orchestra.run.smithi045.stdout:Running transaction check 2023-12-16T21:30:59.389 INFO:teuthology.orchestra.run.smithi045.stdout:Transaction check succeeded. 2023-12-16T21:30:59.389 INFO:teuthology.orchestra.run.smithi045.stdout:Running transaction test 2023-12-16T21:30:59.444 INFO:teuthology.orchestra.run.smithi045.stdout:Transaction test succeeded. 2023-12-16T21:30:59.445 INFO:teuthology.orchestra.run.smithi045.stdout:Running transaction 2023-12-16T21:30:59.586 INFO:teuthology.orchestra.run.smithi045.stdout: Preparing : 1/1 2023-12-16T21:30:59.608 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : ceph-mon-2:17.2.7-125.g1f2f5510.el9.x86_64 31/33 2023-12-16T21:30:59.632 INFO:teuthology.orchestra.run.smithi169.stdout: Running scriptlet: ceph-mon-2:17.2.7-125.g1f2f5510.el9.x86_64 31/33 2023-12-16T21:30:59.632 INFO:teuthology.orchestra.run.smithi169.stdout:Glob pattern passed to enable, but globs are not supported for this. 2023-12-16T21:30:59.632 INFO:teuthology.orchestra.run.smithi169.stdout:Invalid unit name "ceph-mon@*.service" escaped as "ceph-mon@\x2a.service". 2023-12-16T21:30:59.632 INFO:teuthology.orchestra.run.smithi169.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2023-12-16T21:30:59.632 INFO:teuthology.orchestra.run.smithi169.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2023-12-16T21:30:59.632 INFO:teuthology.orchestra.run.smithi169.stdout: 2023-12-16T21:30:59.656 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : ceph-immutable-object-cache-2:17.2.7-125.g1f2f5510.e 1/1 2023-12-16T21:30:59.681 INFO:teuthology.orchestra.run.smithi045.stdout: Running scriptlet: ceph-immutable-object-cache-2:17.2.7-125.g1f2f5510.e 1/1 2023-12-16T21:30:59.682 INFO:teuthology.orchestra.run.smithi045.stdout:Glob pattern passed to enable, but globs are not supported for this. 2023-12-16T21:30:59.682 INFO:teuthology.orchestra.run.smithi045.stdout:Invalid unit name "ceph-immutable-object-cache@*.service" escaped as "ceph-immutable-object-cache@\x2a.service". 2023-12-16T21:30:59.682 INFO:teuthology.orchestra.run.smithi045.stdout: 2023-12-16T21:30:59.834 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : ceph-mds-2:17.2.7-125.g1f2f5510.el9.x86_64 32/33 2023-12-16T21:30:59.858 INFO:teuthology.orchestra.run.smithi169.stdout: Running scriptlet: ceph-mds-2:17.2.7-125.g1f2f5510.el9.x86_64 32/33 2023-12-16T21:30:59.858 INFO:teuthology.orchestra.run.smithi169.stdout:Glob pattern passed to enable, but globs are not supported for this. 2023-12-16T21:30:59.858 INFO:teuthology.orchestra.run.smithi169.stdout:Invalid unit name "ceph-mds@*.service" escaped as "ceph-mds@\x2a.service". 2023-12-16T21:30:59.858 INFO:teuthology.orchestra.run.smithi169.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2023-12-16T21:30:59.858 INFO:teuthology.orchestra.run.smithi169.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2023-12-16T21:30:59.858 INFO:teuthology.orchestra.run.smithi169.stdout: 2023-12-16T21:30:59.924 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : ceph-2:17.2.7-125.g1f2f5510.el9.x86_64 33/33 2023-12-16T21:31:00.691 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : ceph-immutable-object-cache-2:17.2.7-125.g1f2f5510.e 1/1 2023-12-16T21:31:00.692 INFO:teuthology.orchestra.run.smithi045.stdout: 2023-12-16T21:31:00.692 INFO:teuthology.orchestra.run.smithi045.stdout:Installed: 2023-12-16T21:31:00.692 INFO:teuthology.orchestra.run.smithi045.stdout: ceph-immutable-object-cache-2:17.2.7-125.g1f2f5510.el9.x86_64 2023-12-16T21:31:00.692 INFO:teuthology.orchestra.run.smithi045.stdout: 2023-12-16T21:31:00.692 INFO:teuthology.orchestra.run.smithi045.stdout:Complete! 2023-12-16T21:31:00.903 DEBUG:teuthology.orchestra.run.smithi045:> sudo yum -y install ceph-mgr 2023-12-16T21:31:01.486 INFO:teuthology.orchestra.run.smithi045.stdout:Last metadata expiration check: 0:00:58 ago on Sat 16 Dec 2023 09:30:03 PM UTC. 2023-12-16T21:31:01.582 INFO:teuthology.orchestra.run.smithi045.stdout:Package ceph-mgr-2:17.2.7-125.g1f2f5510.el9.x86_64 is already installed. 2023-12-16T21:31:01.633 INFO:teuthology.orchestra.run.smithi045.stdout:Dependencies resolved. 2023-12-16T21:31:01.635 INFO:teuthology.orchestra.run.smithi045.stdout:Nothing to do. 2023-12-16T21:31:01.635 INFO:teuthology.orchestra.run.smithi045.stdout:Complete! 2023-12-16T21:31:01.698 INFO:teuthology.orchestra.run.smithi169.stdout: Running scriptlet: ceph-2:17.2.7-125.g1f2f5510.el9.x86_64 33/33 2023-12-16T21:31:01.698 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : ceph-2:17.2.7-125.g1f2f5510.el9.x86_64 1/33 2023-12-16T21:31:01.699 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : ceph-mds-2:17.2.7-125.g1f2f5510.el9.x86_64 2/33 2023-12-16T21:31:01.699 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : ceph-mgr-2:17.2.7-125.g1f2f5510.el9.x86_64 3/33 2023-12-16T21:31:01.699 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : ceph-mon-2:17.2.7-125.g1f2f5510.el9.x86_64 4/33 2023-12-16T21:31:01.699 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : ceph-osd-2:17.2.7-125.g1f2f5510.el9.x86_64 5/33 2023-12-16T21:31:01.699 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : ceph-mgr-modules-core-2:17.2.7-125.g1f2f5510.el9.n 6/33 2023-12-16T21:31:01.699 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : python3-mako-1.1.4-6.el9.noarch 7/33 2023-12-16T21:31:01.700 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : python3-markupsafe-1.1.1-12.el9.x86_64 8/33 2023-12-16T21:31:01.700 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : python3-pytz-2021.1-5.el9.noarch 9/33 2023-12-16T21:31:01.700 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : python3-toml-0.10.2-6.el9.noarch 10/33 2023-12-16T21:31:01.700 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : python3-cffi-1.14.5-5.el9.x86_64 11/33 2023-12-16T21:31:01.700 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : python3-cryptography-36.0.1-4.el9.x86_64 12/33 2023-12-16T21:31:01.700 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : python3-ply-3.11-14.el9.noarch 13/33 2023-12-16T21:31:01.700 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : python3-pycparser-2.20-6.el9.noarch 14/33 2023-12-16T21:31:01.701 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : python3-requests-2.25.1-7.el9.noarch 15/33 2023-12-16T21:31:01.701 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : python3-urllib3-1.26.5-4.el9.noarch 16/33 2023-12-16T21:31:01.701 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : python3-bcrypt-3.2.2-1.el9.x86_64 17/33 2023-12-16T21:31:01.701 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : python3-cheroot-8.6.0-4.el9.noarch 18/33 2023-12-16T21:31:01.701 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : python3-cherrypy-18.6.1-2.el9.noarch 19/33 2023-12-16T21:31:01.702 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : python3-jaraco-8.2.1-3.el9.noarch 20/33 2023-12-16T21:31:01.702 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : python3-jaraco-classes-3.2.1-5.el9.noarch 21/33 2023-12-16T21:31:01.702 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : python3-jaraco-collections-3.0.0-8.el9.noarch 22/33 2023-12-16T21:31:01.702 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : python3-jaraco-functools-3.5.0-2.el9.noarch 23/33 2023-12-16T21:31:01.702 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : python3-jaraco-text-3.2.0-6.el9.noarch 24/33 2023-12-16T21:31:01.702 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : python3-logutils-0.3.5-21.el9.noarch 25/33 2023-12-16T21:31:01.702 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : python3-more-itertools-8.12.0-2.el9.noarch 26/33 2023-12-16T21:31:01.702 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : python3-pecan-1.4.2-3.el9.noarch 27/33 2023-12-16T21:31:01.702 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : python3-portend-3.1.0-2.el9.noarch 28/33 2023-12-16T21:31:01.703 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : python3-pyOpenSSL-21.0.0-1.el9.noarch 29/33 2023-12-16T21:31:01.703 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : python3-tempora-5.0.0-2.el9.noarch 30/33 2023-12-16T21:31:01.703 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : python3-webob-1.8.7-6.el9.noarch 31/33 2023-12-16T21:31:01.703 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : python3-werkzeug-2.0.3-3.el9.1.noarch 32/33 2023-12-16T21:31:01.704 DEBUG:teuthology.orchestra.run.smithi045:> sudo yum -y install ceph-mgr-dashboard 2023-12-16T21:31:02.283 INFO:teuthology.orchestra.run.smithi045.stdout:Last metadata expiration check: 0:00:59 ago on Sat 16 Dec 2023 09:30:03 PM UTC. 2023-12-16T21:31:02.433 INFO:teuthology.orchestra.run.smithi045.stdout:Dependencies resolved. 2023-12-16T21:31:02.434 INFO:teuthology.orchestra.run.smithi045.stdout:================================================================================ 2023-12-16T21:31:02.434 INFO:teuthology.orchestra.run.smithi045.stdout: Package Arch Version Repository Size 2023-12-16T21:31:02.434 INFO:teuthology.orchestra.run.smithi045.stdout:================================================================================ 2023-12-16T21:31:02.434 INFO:teuthology.orchestra.run.smithi045.stdout:Installing: 2023-12-16T21:31:02.434 INFO:teuthology.orchestra.run.smithi045.stdout: ceph-mgr-dashboard noarch 2:17.2.7-125.g1f2f5510.el9 ceph-noarch 1.7 M 2023-12-16T21:31:02.434 INFO:teuthology.orchestra.run.smithi045.stdout:Installing dependencies: 2023-12-16T21:31:02.434 INFO:teuthology.orchestra.run.smithi045.stdout: ceph-grafana-dashboards noarch 2:17.2.7-125.g1f2f5510.el9 ceph-noarch 23 k 2023-12-16T21:31:02.435 INFO:teuthology.orchestra.run.smithi045.stdout: ceph-prometheus-alerts noarch 2:17.2.7-125.g1f2f5510.el9 ceph-noarch 15 k 2023-12-16T21:31:02.435 INFO:teuthology.orchestra.run.smithi045.stdout: python3-jwt noarch 2.4.0-1.el9 epel 41 k 2023-12-16T21:31:02.435 INFO:teuthology.orchestra.run.smithi045.stdout: python3-repoze-lru noarch 0.7-16.el9 epel 31 k 2023-12-16T21:31:02.435 INFO:teuthology.orchestra.run.smithi045.stdout: python3-routes noarch 2.5.1-5.el9 epel 188 k 2023-12-16T21:31:02.435 INFO:teuthology.orchestra.run.smithi045.stdout:Installing weak dependencies: 2023-12-16T21:31:02.435 INFO:teuthology.orchestra.run.smithi045.stdout: python3-jwt+crypto noarch 2.4.0-1.el9 epel 9.0 k 2023-12-16T21:31:02.435 INFO:teuthology.orchestra.run.smithi045.stdout: 2023-12-16T21:31:02.435 INFO:teuthology.orchestra.run.smithi045.stdout:Transaction Summary 2023-12-16T21:31:02.435 INFO:teuthology.orchestra.run.smithi045.stdout:================================================================================ 2023-12-16T21:31:02.435 INFO:teuthology.orchestra.run.smithi045.stdout:Install 7 Packages 2023-12-16T21:31:02.435 INFO:teuthology.orchestra.run.smithi045.stdout: 2023-12-16T21:31:02.436 INFO:teuthology.orchestra.run.smithi045.stdout:Total download size: 2.0 M 2023-12-16T21:31:02.436 INFO:teuthology.orchestra.run.smithi045.stdout:Installed size: 8.1 M 2023-12-16T21:31:02.436 INFO:teuthology.orchestra.run.smithi045.stdout:Downloading Packages: 2023-12-16T21:31:02.647 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : python3-zc-lockfile-2.0-10.el9.noarch 33/33 2023-12-16T21:31:02.647 INFO:teuthology.orchestra.run.smithi169.stdout: 2023-12-16T21:31:02.647 INFO:teuthology.orchestra.run.smithi169.stdout:Installed: 2023-12-16T21:31:02.647 INFO:teuthology.orchestra.run.smithi169.stdout: ceph-2:17.2.7-125.g1f2f5510.el9.x86_64 2023-12-16T21:31:02.647 INFO:teuthology.orchestra.run.smithi169.stdout: ceph-mds-2:17.2.7-125.g1f2f5510.el9.x86_64 2023-12-16T21:31:02.648 INFO:teuthology.orchestra.run.smithi169.stdout: ceph-mgr-2:17.2.7-125.g1f2f5510.el9.x86_64 2023-12-16T21:31:02.648 INFO:teuthology.orchestra.run.smithi169.stdout: ceph-mgr-modules-core-2:17.2.7-125.g1f2f5510.el9.noarch 2023-12-16T21:31:02.648 INFO:teuthology.orchestra.run.smithi169.stdout: ceph-mon-2:17.2.7-125.g1f2f5510.el9.x86_64 2023-12-16T21:31:02.648 INFO:teuthology.orchestra.run.smithi169.stdout: ceph-osd-2:17.2.7-125.g1f2f5510.el9.x86_64 2023-12-16T21:31:02.648 INFO:teuthology.orchestra.run.smithi169.stdout: python3-bcrypt-3.2.2-1.el9.x86_64 2023-12-16T21:31:02.648 INFO:teuthology.orchestra.run.smithi169.stdout: python3-cffi-1.14.5-5.el9.x86_64 2023-12-16T21:31:02.648 INFO:teuthology.orchestra.run.smithi169.stdout: python3-cheroot-8.6.0-4.el9.noarch 2023-12-16T21:31:02.648 INFO:teuthology.orchestra.run.smithi169.stdout: python3-cherrypy-18.6.1-2.el9.noarch 2023-12-16T21:31:02.648 INFO:teuthology.orchestra.run.smithi169.stdout: python3-cryptography-36.0.1-4.el9.x86_64 2023-12-16T21:31:02.648 INFO:teuthology.orchestra.run.smithi169.stdout: python3-jaraco-8.2.1-3.el9.noarch 2023-12-16T21:31:02.648 INFO:teuthology.orchestra.run.smithi169.stdout: python3-jaraco-classes-3.2.1-5.el9.noarch 2023-12-16T21:31:02.648 INFO:teuthology.orchestra.run.smithi169.stdout: python3-jaraco-collections-3.0.0-8.el9.noarch 2023-12-16T21:31:02.648 INFO:teuthology.orchestra.run.smithi169.stdout: python3-jaraco-functools-3.5.0-2.el9.noarch 2023-12-16T21:31:02.649 INFO:teuthology.orchestra.run.smithi169.stdout: python3-jaraco-text-3.2.0-6.el9.noarch 2023-12-16T21:31:02.649 INFO:teuthology.orchestra.run.smithi169.stdout: python3-logutils-0.3.5-21.el9.noarch 2023-12-16T21:31:02.649 INFO:teuthology.orchestra.run.smithi169.stdout: python3-mako-1.1.4-6.el9.noarch 2023-12-16T21:31:02.649 INFO:teuthology.orchestra.run.smithi169.stdout: python3-markupsafe-1.1.1-12.el9.x86_64 2023-12-16T21:31:02.649 INFO:teuthology.orchestra.run.smithi169.stdout: python3-more-itertools-8.12.0-2.el9.noarch 2023-12-16T21:31:02.649 INFO:teuthology.orchestra.run.smithi169.stdout: python3-pecan-1.4.2-3.el9.noarch 2023-12-16T21:31:02.649 INFO:teuthology.orchestra.run.smithi169.stdout: python3-ply-3.11-14.el9.noarch 2023-12-16T21:31:02.649 INFO:teuthology.orchestra.run.smithi169.stdout: python3-portend-3.1.0-2.el9.noarch 2023-12-16T21:31:02.649 INFO:teuthology.orchestra.run.smithi169.stdout: python3-pyOpenSSL-21.0.0-1.el9.noarch 2023-12-16T21:31:02.649 INFO:teuthology.orchestra.run.smithi169.stdout: python3-pycparser-2.20-6.el9.noarch 2023-12-16T21:31:02.649 INFO:teuthology.orchestra.run.smithi169.stdout: python3-pytz-2021.1-5.el9.noarch 2023-12-16T21:31:02.649 INFO:teuthology.orchestra.run.smithi169.stdout: python3-requests-2.25.1-7.el9.noarch 2023-12-16T21:31:02.649 INFO:teuthology.orchestra.run.smithi169.stdout: python3-tempora-5.0.0-2.el9.noarch 2023-12-16T21:31:02.649 INFO:teuthology.orchestra.run.smithi169.stdout: python3-toml-0.10.2-6.el9.noarch 2023-12-16T21:31:02.649 INFO:teuthology.orchestra.run.smithi169.stdout: python3-urllib3-1.26.5-4.el9.noarch 2023-12-16T21:31:02.650 INFO:teuthology.orchestra.run.smithi169.stdout: python3-webob-1.8.7-6.el9.noarch 2023-12-16T21:31:02.650 INFO:teuthology.orchestra.run.smithi169.stdout: python3-werkzeug-2.0.3-3.el9.1.noarch 2023-12-16T21:31:02.650 INFO:teuthology.orchestra.run.smithi169.stdout: python3-zc-lockfile-2.0-10.el9.noarch 2023-12-16T21:31:02.650 INFO:teuthology.orchestra.run.smithi169.stdout: 2023-12-16T21:31:02.650 INFO:teuthology.orchestra.run.smithi169.stdout:Complete! 2023-12-16T21:31:02.785 INFO:teuthology.orchestra.run.smithi045.stdout:(1/7): ceph-grafana-dashboards-17.2.7-125.g1f2f 119 kB/s | 23 kB 00:00 2023-12-16T21:31:02.810 INFO:teuthology.orchestra.run.smithi045.stdout:(2/7): ceph-prometheus-alerts-17.2.7-125.g1f2f5 66 kB/s | 15 kB 00:00 2023-12-16T21:31:02.977 INFO:teuthology.orchestra.run.smithi045.stdout:(3/7): ceph-mgr-dashboard-17.2.7-125.g1f2f5510. 4.4 MB/s | 1.7 MB 00:00 2023-12-16T21:31:03.044 INFO:teuthology.orchestra.run.smithi045.stdout:(4/7): python3-jwt+crypto-2.4.0-1.el9.noarch.rp 35 kB/s | 9.0 kB 00:00 2023-12-16T21:31:03.045 DEBUG:teuthology.orchestra.run.smithi169:> sudo yum -y install ceph-base 2023-12-16T21:31:03.112 INFO:teuthology.orchestra.run.smithi045.stdout:(5/7): python3-jwt-2.4.0-1.el9.noarch.rpm 136 kB/s | 41 kB 00:00 2023-12-16T21:31:03.228 INFO:teuthology.orchestra.run.smithi045.stdout:(6/7): python3-repoze-lru-0.7-16.el9.noarch.rpm 123 kB/s | 31 kB 00:00 2023-12-16T21:31:03.295 INFO:teuthology.orchestra.run.smithi045.stdout:(7/7): python3-routes-2.5.1-5.el9.noarch.rpm 753 kB/s | 188 kB 00:00 2023-12-16T21:31:03.297 INFO:teuthology.orchestra.run.smithi045.stdout:-------------------------------------------------------------------------------- 2023-12-16T21:31:03.298 INFO:teuthology.orchestra.run.smithi045.stdout:Total 2.3 MB/s | 2.0 MB 00:00 2023-12-16T21:31:03.298 INFO:teuthology.orchestra.run.smithi045.stdout:Running transaction check 2023-12-16T21:31:03.306 INFO:teuthology.orchestra.run.smithi045.stdout:Transaction check succeeded. 2023-12-16T21:31:03.306 INFO:teuthology.orchestra.run.smithi045.stdout:Running transaction test 2023-12-16T21:31:03.379 INFO:teuthology.orchestra.run.smithi045.stdout:Transaction test succeeded. 2023-12-16T21:31:03.380 INFO:teuthology.orchestra.run.smithi045.stdout:Running transaction 2023-12-16T21:31:03.637 INFO:teuthology.orchestra.run.smithi169.stdout:Last metadata expiration check: 0:00:50 ago on Sat 16 Dec 2023 09:30:13 PM UTC. 2023-12-16T21:31:03.686 INFO:teuthology.orchestra.run.smithi045.stdout: Preparing : 1/1 2023-12-16T21:31:03.730 INFO:teuthology.orchestra.run.smithi169.stdout:Package ceph-base-2:17.2.7-125.g1f2f5510.el9.x86_64 is already installed. 2023-12-16T21:31:03.755 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : python3-jwt-2.4.0-1.el9.noarch 1/7 2023-12-16T21:31:03.780 INFO:teuthology.orchestra.run.smithi169.stdout:Dependencies resolved. 2023-12-16T21:31:03.781 INFO:teuthology.orchestra.run.smithi169.stdout:Nothing to do. 2023-12-16T21:31:03.781 INFO:teuthology.orchestra.run.smithi169.stdout:Complete! 2023-12-16T21:31:03.789 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : python3-jwt+crypto-2.4.0-1.el9.noarch 2/7 2023-12-16T21:31:03.837 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : python3-repoze-lru-0.7-16.el9.noarch 3/7 2023-12-16T21:31:03.852 DEBUG:teuthology.orchestra.run.smithi169:> sudo yum -y install cephadm 2023-12-16T21:31:03.868 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : python3-routes-2.5.1-5.el9.noarch 4/7 2023-12-16T21:31:03.897 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : ceph-prometheus-alerts-2:17.2.7-125.g1f2f5510.el9.no 5/7 2023-12-16T21:31:04.137 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : ceph-grafana-dashboards-2:17.2.7-125.g1f2f5510.el9.n 6/7 2023-12-16T21:31:04.166 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : ceph-mgr-dashboard-2:17.2.7-125.g1f2f5510.el9.noarch 7/7 2023-12-16T21:31:04.413 INFO:teuthology.orchestra.run.smithi169.stdout:Last metadata expiration check: 0:00:51 ago on Sat 16 Dec 2023 09:30:13 PM UTC. 2023-12-16T21:31:04.554 INFO:teuthology.orchestra.run.smithi169.stdout:Dependencies resolved. 2023-12-16T21:31:04.555 INFO:teuthology.orchestra.run.smithi169.stdout:================================================================================ 2023-12-16T21:31:04.555 INFO:teuthology.orchestra.run.smithi169.stdout: Package Arch Version Repository Size 2023-12-16T21:31:04.555 INFO:teuthology.orchestra.run.smithi169.stdout:================================================================================ 2023-12-16T21:31:04.555 INFO:teuthology.orchestra.run.smithi169.stdout:Installing: 2023-12-16T21:31:04.555 INFO:teuthology.orchestra.run.smithi169.stdout: cephadm noarch 2:17.2.7-125.g1f2f5510.el9 ceph-noarch 80 k 2023-12-16T21:31:04.555 INFO:teuthology.orchestra.run.smithi169.stdout: 2023-12-16T21:31:04.556 INFO:teuthology.orchestra.run.smithi169.stdout:Transaction Summary 2023-12-16T21:31:04.556 INFO:teuthology.orchestra.run.smithi169.stdout:================================================================================ 2023-12-16T21:31:04.556 INFO:teuthology.orchestra.run.smithi169.stdout:Install 1 Package 2023-12-16T21:31:04.556 INFO:teuthology.orchestra.run.smithi169.stdout: 2023-12-16T21:31:04.556 INFO:teuthology.orchestra.run.smithi169.stdout:Total download size: 80 k 2023-12-16T21:31:04.556 INFO:teuthology.orchestra.run.smithi169.stdout:Installed size: 364 k 2023-12-16T21:31:04.556 INFO:teuthology.orchestra.run.smithi169.stdout:Downloading Packages: 2023-12-16T21:31:04.740 INFO:teuthology.orchestra.run.smithi045.stdout: Running scriptlet: ceph-mgr-dashboard-2:17.2.7-125.g1f2f5510.el9.noarch 7/7 2023-12-16T21:31:04.740 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : ceph-grafana-dashboards-2:17.2.7-125.g1f2f5510.el9.n 1/7 2023-12-16T21:31:04.740 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : ceph-mgr-dashboard-2:17.2.7-125.g1f2f5510.el9.noarch 2/7 2023-12-16T21:31:04.740 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : ceph-prometheus-alerts-2:17.2.7-125.g1f2f5510.el9.no 3/7 2023-12-16T21:31:04.741 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : python3-jwt+crypto-2.4.0-1.el9.noarch 4/7 2023-12-16T21:31:04.741 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : python3-jwt-2.4.0-1.el9.noarch 5/7 2023-12-16T21:31:04.741 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : python3-repoze-lru-0.7-16.el9.noarch 6/7 2023-12-16T21:31:04.958 INFO:teuthology.orchestra.run.smithi169.stdout:cephadm-17.2.7-125.g1f2f5510.el9.noarch.rpm 200 kB/s | 80 kB 00:00 2023-12-16T21:31:04.958 INFO:teuthology.orchestra.run.smithi169.stdout:-------------------------------------------------------------------------------- 2023-12-16T21:31:04.959 INFO:teuthology.orchestra.run.smithi169.stdout:Total 200 kB/s | 80 kB 00:00 2023-12-16T21:31:04.959 INFO:teuthology.orchestra.run.smithi169.stdout:Running transaction check 2023-12-16T21:31:04.964 INFO:teuthology.orchestra.run.smithi169.stdout:Transaction check succeeded. 2023-12-16T21:31:04.964 INFO:teuthology.orchestra.run.smithi169.stdout:Running transaction test 2023-12-16T21:31:04.965 INFO:teuthology.orchestra.run.smithi169.stdout:Transaction test succeeded. 2023-12-16T21:31:04.966 INFO:teuthology.orchestra.run.smithi169.stdout:Running transaction 2023-12-16T21:31:05.037 INFO:teuthology.orchestra.run.smithi169.stdout: Preparing : 1/1 2023-12-16T21:31:05.084 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : python3-routes-2.5.1-5.el9.noarch 7/7 2023-12-16T21:31:05.084 INFO:teuthology.orchestra.run.smithi045.stdout: 2023-12-16T21:31:05.084 INFO:teuthology.orchestra.run.smithi045.stdout:Installed: 2023-12-16T21:31:05.085 INFO:teuthology.orchestra.run.smithi045.stdout: ceph-grafana-dashboards-2:17.2.7-125.g1f2f5510.el9.noarch 2023-12-16T21:31:05.085 INFO:teuthology.orchestra.run.smithi045.stdout: ceph-mgr-dashboard-2:17.2.7-125.g1f2f5510.el9.noarch 2023-12-16T21:31:05.085 INFO:teuthology.orchestra.run.smithi045.stdout: ceph-prometheus-alerts-2:17.2.7-125.g1f2f5510.el9.noarch 2023-12-16T21:31:05.085 INFO:teuthology.orchestra.run.smithi045.stdout: python3-jwt-2.4.0-1.el9.noarch 2023-12-16T21:31:05.085 INFO:teuthology.orchestra.run.smithi045.stdout: python3-jwt+crypto-2.4.0-1.el9.noarch 2023-12-16T21:31:05.085 INFO:teuthology.orchestra.run.smithi045.stdout: python3-repoze-lru-0.7-16.el9.noarch 2023-12-16T21:31:05.085 INFO:teuthology.orchestra.run.smithi045.stdout: python3-routes-2.5.1-5.el9.noarch 2023-12-16T21:31:05.085 INFO:teuthology.orchestra.run.smithi045.stdout: 2023-12-16T21:31:05.085 INFO:teuthology.orchestra.run.smithi045.stdout:Complete! 2023-12-16T21:31:05.308 DEBUG:teuthology.orchestra.run.smithi045:> sudo yum -y install ceph-mgr-diskprediction-local 2023-12-16T21:31:05.472 INFO:teuthology.orchestra.run.smithi169.stdout: Running scriptlet: cephadm-2:17.2.7-125.g1f2f5510.el9.noarch 1/1 2023-12-16T21:31:05.562 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : cephadm-2:17.2.7-125.g1f2f5510.el9.noarch 1/1 2023-12-16T21:31:05.895 INFO:teuthology.orchestra.run.smithi045.stdout:Last metadata expiration check: 0:01:02 ago on Sat 16 Dec 2023 09:30:03 PM UTC. 2023-12-16T21:31:05.955 INFO:teuthology.orchestra.run.smithi169.stdout: Running scriptlet: cephadm-2:17.2.7-125.g1f2f5510.el9.noarch 1/1 2023-12-16T21:31:06.047 INFO:teuthology.orchestra.run.smithi045.stdout:Dependencies resolved. 2023-12-16T21:31:06.048 INFO:teuthology.orchestra.run.smithi045.stdout:=========================================================================================== 2023-12-16T21:31:06.048 INFO:teuthology.orchestra.run.smithi045.stdout: Package Arch Version Repository Size 2023-12-16T21:31:06.048 INFO:teuthology.orchestra.run.smithi045.stdout:=========================================================================================== 2023-12-16T21:31:06.048 INFO:teuthology.orchestra.run.smithi045.stdout:Installing: 2023-12-16T21:31:06.049 INFO:teuthology.orchestra.run.smithi045.stdout: ceph-mgr-diskprediction-local noarch 2:17.2.7-125.g1f2f5510.el9 ceph-noarch 7.4 M 2023-12-16T21:31:06.049 INFO:teuthology.orchestra.run.smithi045.stdout:Installing dependencies: 2023-12-16T21:31:06.049 INFO:teuthology.orchestra.run.smithi045.stdout: flexiblas x86_64 3.0.4-8.el9 CentOS-AppStream 32 k 2023-12-16T21:31:06.049 INFO:teuthology.orchestra.run.smithi045.stdout: flexiblas-netlib x86_64 3.0.4-8.el9 CentOS-AppStream 3.0 M 2023-12-16T21:31:06.049 INFO:teuthology.orchestra.run.smithi045.stdout: flexiblas-openblas-openmp x86_64 3.0.4-8.el9 CentOS-AppStream 17 k 2023-12-16T21:31:06.049 INFO:teuthology.orchestra.run.smithi045.stdout: libgfortran x86_64 11.4.1-2.3.el9 CentOS-BaseOS 803 k 2023-12-16T21:31:06.049 INFO:teuthology.orchestra.run.smithi045.stdout: libquadmath x86_64 11.4.1-2.3.el9 CentOS-BaseOS 195 k 2023-12-16T21:31:06.049 INFO:teuthology.orchestra.run.smithi045.stdout: openblas x86_64 0.3.21-2.el9 CentOS-AppStream 34 k 2023-12-16T21:31:06.049 INFO:teuthology.orchestra.run.smithi045.stdout: openblas-openmp x86_64 0.3.21-2.el9 CentOS-AppStream 4.7 M 2023-12-16T21:31:06.049 INFO:teuthology.orchestra.run.smithi045.stdout: python3-devel x86_64 3.9.17-2.el9 appstream 245 k 2023-12-16T21:31:06.049 INFO:teuthology.orchestra.run.smithi045.stdout: python3-numpy x86_64 1:1.20.1-5.el9 CentOS-AppStream 4.8 M 2023-12-16T21:31:06.049 INFO:teuthology.orchestra.run.smithi045.stdout: python3-numpy-f2py x86_64 1:1.20.1-5.el9 CentOS-AppStream 217 k 2023-12-16T21:31:06.049 INFO:teuthology.orchestra.run.smithi045.stdout: python3-scipy x86_64 1.6.2-8.el9 CentOS-AppStream 16 M 2023-12-16T21:31:06.049 INFO:teuthology.orchestra.run.smithi045.stdout: 2023-12-16T21:31:06.050 INFO:teuthology.orchestra.run.smithi045.stdout:Transaction Summary 2023-12-16T21:31:06.050 INFO:teuthology.orchestra.run.smithi045.stdout:=========================================================================================== 2023-12-16T21:31:06.050 INFO:teuthology.orchestra.run.smithi045.stdout:Install 12 Packages 2023-12-16T21:31:06.050 INFO:teuthology.orchestra.run.smithi045.stdout: 2023-12-16T21:31:06.050 INFO:teuthology.orchestra.run.smithi045.stdout:Total download size: 37 M 2023-12-16T21:31:06.050 INFO:teuthology.orchestra.run.smithi045.stdout:Installed size: 209 M 2023-12-16T21:31:06.050 INFO:teuthology.orchestra.run.smithi045.stdout:Downloading Packages: 2023-12-16T21:31:06.237 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : cephadm-2:17.2.7-125.g1f2f5510.el9.noarch 1/1 2023-12-16T21:31:06.238 INFO:teuthology.orchestra.run.smithi169.stdout: 2023-12-16T21:31:06.238 INFO:teuthology.orchestra.run.smithi169.stdout:Installed: 2023-12-16T21:31:06.238 INFO:teuthology.orchestra.run.smithi169.stdout: cephadm-2:17.2.7-125.g1f2f5510.el9.noarch 2023-12-16T21:31:06.238 INFO:teuthology.orchestra.run.smithi169.stdout: 2023-12-16T21:31:06.238 INFO:teuthology.orchestra.run.smithi169.stdout:Complete! 2023-12-16T21:31:06.253 INFO:teuthology.orchestra.run.smithi045.stdout:(1/12): flexiblas-3.0.4-8.el9.x86_64.rpm 245 kB/s | 32 kB 00:00 2023-12-16T21:31:06.286 INFO:teuthology.orchestra.run.smithi045.stdout:(2/12): flexiblas-openblas-openmp-3.0.4-8.el9.x 537 kB/s | 17 kB 00:00 2023-12-16T21:31:06.320 INFO:teuthology.orchestra.run.smithi045.stdout:(3/12): openblas-0.3.21-2.el9.x86_64.rpm 1.0 MB/s | 34 kB 00:00 2023-12-16T21:31:06.379 INFO:teuthology.orchestra.run.smithi045.stdout:(4/12): flexiblas-netlib-3.0.4-8.el9.x86_64.rpm 12 MB/s | 3.0 MB 00:00 2023-12-16T21:31:06.453 DEBUG:teuthology.orchestra.run.smithi169:> sudo yum -y install ceph-immutable-object-cache 2023-12-16T21:31:06.504 INFO:teuthology.orchestra.run.smithi045.stdout:(5/12): openblas-openmp-0.3.21-2.el9.x86_64.rpm 26 MB/s | 4.7 MB 00:00 2023-12-16T21:31:06.571 INFO:teuthology.orchestra.run.smithi045.stdout:(6/12): python3-numpy-1.20.1-5.el9.x86_64.rpm 25 MB/s | 4.8 MB 00:00 2023-12-16T21:31:06.621 INFO:teuthology.orchestra.run.smithi045.stdout:(7/12): python3-numpy-f2py-1.20.1-5.el9.x86_64. 1.9 MB/s | 217 kB 00:00 2023-12-16T21:31:06.813 INFO:teuthology.orchestra.run.smithi045.stdout:(8/12): python3-scipy-1.6.2-8.el9.x86_64.rpm 65 MB/s | 16 MB 00:00 2023-12-16T21:31:06.847 INFO:teuthology.orchestra.run.smithi045.stdout:(9/12): libgfortran-11.4.1-2.3.el9.x86_64.rpm 3.5 MB/s | 803 kB 00:00 2023-12-16T21:31:06.872 INFO:teuthology.orchestra.run.smithi045.stdout:(10/12): libquadmath-11.4.1-2.3.el9.x86_64.rpm 3.3 MB/s | 195 kB 00:00 2023-12-16T21:31:07.014 INFO:teuthology.orchestra.run.smithi045.stdout:(11/12): ceph-mgr-diskprediction-local-17.2.7-1 8.3 MB/s | 7.4 MB 00:00 2023-12-16T21:31:07.020 INFO:teuthology.orchestra.run.smithi169.stdout:Last metadata expiration check: 0:00:54 ago on Sat 16 Dec 2023 09:30:13 PM UTC. 2023-12-16T21:31:07.166 INFO:teuthology.orchestra.run.smithi169.stdout:Dependencies resolved. 2023-12-16T21:31:07.167 INFO:teuthology.orchestra.run.smithi169.stdout:================================================================================ 2023-12-16T21:31:07.167 INFO:teuthology.orchestra.run.smithi169.stdout: Package Arch Version Repo Size 2023-12-16T21:31:07.167 INFO:teuthology.orchestra.run.smithi169.stdout:================================================================================ 2023-12-16T21:31:07.167 INFO:teuthology.orchestra.run.smithi169.stdout:Installing: 2023-12-16T21:31:07.167 INFO:teuthology.orchestra.run.smithi169.stdout: ceph-immutable-object-cache x86_64 2:17.2.7-125.g1f2f5510.el9 ceph 137 k 2023-12-16T21:31:07.168 INFO:teuthology.orchestra.run.smithi169.stdout: 2023-12-16T21:31:07.168 INFO:teuthology.orchestra.run.smithi169.stdout:Transaction Summary 2023-12-16T21:31:07.168 INFO:teuthology.orchestra.run.smithi169.stdout:================================================================================ 2023-12-16T21:31:07.168 INFO:teuthology.orchestra.run.smithi169.stdout:Install 1 Package 2023-12-16T21:31:07.168 INFO:teuthology.orchestra.run.smithi169.stdout: 2023-12-16T21:31:07.168 INFO:teuthology.orchestra.run.smithi169.stdout:Total download size: 137 k 2023-12-16T21:31:07.168 INFO:teuthology.orchestra.run.smithi169.stdout:Installed size: 379 k 2023-12-16T21:31:07.168 INFO:teuthology.orchestra.run.smithi169.stdout:Downloading Packages: 2023-12-16T21:31:07.389 INFO:teuthology.orchestra.run.smithi045.stdout:(12/12): python3-devel-3.9.17-2.el9.x86_64.rpm 453 kB/s | 245 kB 00:00 2023-12-16T21:31:07.392 INFO:teuthology.orchestra.run.smithi045.stdout:-------------------------------------------------------------------------------- 2023-12-16T21:31:07.392 INFO:teuthology.orchestra.run.smithi045.stdout:Total 28 MB/s | 37 MB 00:01 2023-12-16T21:31:07.397 INFO:teuthology.orchestra.run.smithi169.stdout:ceph-immutable-object-cache-17.2.7-125.g1f2f551 600 kB/s | 137 kB 00:00 2023-12-16T21:31:07.397 INFO:teuthology.orchestra.run.smithi169.stdout:-------------------------------------------------------------------------------- 2023-12-16T21:31:07.398 INFO:teuthology.orchestra.run.smithi169.stdout:Total 596 kB/s | 137 kB 00:00 2023-12-16T21:31:07.398 INFO:teuthology.orchestra.run.smithi169.stdout:Running transaction check 2023-12-16T21:31:07.405 INFO:teuthology.orchestra.run.smithi169.stdout:Transaction check succeeded. 2023-12-16T21:31:07.405 INFO:teuthology.orchestra.run.smithi169.stdout:Running transaction test 2023-12-16T21:31:07.408 INFO:teuthology.orchestra.run.smithi045.stdout:Running transaction check 2023-12-16T21:31:07.436 INFO:teuthology.orchestra.run.smithi045.stdout:Transaction check succeeded. 2023-12-16T21:31:07.436 INFO:teuthology.orchestra.run.smithi045.stdout:Running transaction test 2023-12-16T21:31:07.457 INFO:teuthology.orchestra.run.smithi169.stdout:Transaction test succeeded. 2023-12-16T21:31:07.458 INFO:teuthology.orchestra.run.smithi169.stdout:Running transaction 2023-12-16T21:31:07.597 INFO:teuthology.orchestra.run.smithi169.stdout: Preparing : 1/1 2023-12-16T21:31:07.656 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : ceph-immutable-object-cache-2:17.2.7-125.g1f2f5510.e 1/1 2023-12-16T21:31:07.680 INFO:teuthology.orchestra.run.smithi169.stdout: Running scriptlet: ceph-immutable-object-cache-2:17.2.7-125.g1f2f5510.e 1/1 2023-12-16T21:31:07.680 INFO:teuthology.orchestra.run.smithi169.stdout:Glob pattern passed to enable, but globs are not supported for this. 2023-12-16T21:31:07.680 INFO:teuthology.orchestra.run.smithi169.stdout:Invalid unit name "ceph-immutable-object-cache@*.service" escaped as "ceph-immutable-object-cache@\x2a.service". 2023-12-16T21:31:07.680 INFO:teuthology.orchestra.run.smithi169.stdout: 2023-12-16T21:31:07.899 INFO:teuthology.orchestra.run.smithi045.stdout:Transaction test succeeded. 2023-12-16T21:31:07.900 INFO:teuthology.orchestra.run.smithi045.stdout:Running transaction 2023-12-16T21:31:08.621 INFO:teuthology.orchestra.run.smithi045.stdout: Preparing : 1/1 2023-12-16T21:31:08.626 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : ceph-immutable-object-cache-2:17.2.7-125.g1f2f5510.e 1/1 2023-12-16T21:31:08.626 INFO:teuthology.orchestra.run.smithi169.stdout: 2023-12-16T21:31:08.626 INFO:teuthology.orchestra.run.smithi169.stdout:Installed: 2023-12-16T21:31:08.626 INFO:teuthology.orchestra.run.smithi169.stdout: ceph-immutable-object-cache-2:17.2.7-125.g1f2f5510.el9.x86_64 2023-12-16T21:31:08.626 INFO:teuthology.orchestra.run.smithi169.stdout: 2023-12-16T21:31:08.627 INFO:teuthology.orchestra.run.smithi169.stdout:Complete! 2023-12-16T21:31:08.693 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : libquadmath-11.4.1-2.3.el9.x86_64 1/12 2023-12-16T21:31:08.719 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : libgfortran-11.4.1-2.3.el9.x86_64 2/12 2023-12-16T21:31:08.821 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : flexiblas-3.0.4-8.el9.x86_64 3/12 2023-12-16T21:31:08.856 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : python3-devel-3.9.17-2.el9.x86_64 4/12 2023-12-16T21:31:08.858 DEBUG:teuthology.orchestra.run.smithi169:> sudo yum -y install ceph-mgr 2023-12-16T21:31:09.046 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : openblas-0.3.21-2.el9.x86_64 5/12 2023-12-16T21:31:09.079 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : openblas-openmp-0.3.21-2.el9.x86_64 6/12 2023-12-16T21:31:09.161 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 7/12 2023-12-16T21:31:09.420 INFO:teuthology.orchestra.run.smithi169.stdout:Last metadata expiration check: 0:00:56 ago on Sat 16 Dec 2023 09:30:13 PM UTC. 2023-12-16T21:31:09.515 INFO:teuthology.orchestra.run.smithi169.stdout:Package ceph-mgr-2:17.2.7-125.g1f2f5510.el9.x86_64 is already installed. 2023-12-16T21:31:09.565 INFO:teuthology.orchestra.run.smithi169.stdout:Dependencies resolved. 2023-12-16T21:31:09.566 INFO:teuthology.orchestra.run.smithi169.stdout:Nothing to do. 2023-12-16T21:31:09.566 INFO:teuthology.orchestra.run.smithi169.stdout:Complete! 2023-12-16T21:31:09.634 DEBUG:teuthology.orchestra.run.smithi169:> sudo yum -y install ceph-mgr-dashboard 2023-12-16T21:31:09.866 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : flexiblas-netlib-3.0.4-8.el9.x86_64 8/12 2023-12-16T21:31:09.982 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : python3-numpy-1:1.20.1-5.el9.x86_64 9/12 2023-12-16T21:31:10.184 INFO:teuthology.orchestra.run.smithi169.stdout:Last metadata expiration check: 0:00:57 ago on Sat 16 Dec 2023 09:30:13 PM UTC. 2023-12-16T21:31:10.328 INFO:teuthology.orchestra.run.smithi169.stdout:Dependencies resolved. 2023-12-16T21:31:10.329 INFO:teuthology.orchestra.run.smithi169.stdout:================================================================================ 2023-12-16T21:31:10.329 INFO:teuthology.orchestra.run.smithi169.stdout: Package Arch Version Repository Size 2023-12-16T21:31:10.329 INFO:teuthology.orchestra.run.smithi169.stdout:================================================================================ 2023-12-16T21:31:10.329 INFO:teuthology.orchestra.run.smithi169.stdout:Installing: 2023-12-16T21:31:10.329 INFO:teuthology.orchestra.run.smithi169.stdout: ceph-mgr-dashboard noarch 2:17.2.7-125.g1f2f5510.el9 ceph-noarch 1.7 M 2023-12-16T21:31:10.329 INFO:teuthology.orchestra.run.smithi169.stdout:Installing dependencies: 2023-12-16T21:31:10.329 INFO:teuthology.orchestra.run.smithi169.stdout: ceph-grafana-dashboards noarch 2:17.2.7-125.g1f2f5510.el9 ceph-noarch 23 k 2023-12-16T21:31:10.329 INFO:teuthology.orchestra.run.smithi169.stdout: ceph-prometheus-alerts noarch 2:17.2.7-125.g1f2f5510.el9 ceph-noarch 15 k 2023-12-16T21:31:10.330 INFO:teuthology.orchestra.run.smithi169.stdout: python3-jwt noarch 2.4.0-1.el9 epel 41 k 2023-12-16T21:31:10.330 INFO:teuthology.orchestra.run.smithi169.stdout: python3-repoze-lru noarch 0.7-16.el9 epel 31 k 2023-12-16T21:31:10.330 INFO:teuthology.orchestra.run.smithi169.stdout: python3-routes noarch 2.5.1-5.el9 epel 188 k 2023-12-16T21:31:10.330 INFO:teuthology.orchestra.run.smithi169.stdout:Installing weak dependencies: 2023-12-16T21:31:10.330 INFO:teuthology.orchestra.run.smithi169.stdout: python3-jwt+crypto noarch 2.4.0-1.el9 epel 9.0 k 2023-12-16T21:31:10.330 INFO:teuthology.orchestra.run.smithi169.stdout: 2023-12-16T21:31:10.330 INFO:teuthology.orchestra.run.smithi169.stdout:Transaction Summary 2023-12-16T21:31:10.330 INFO:teuthology.orchestra.run.smithi169.stdout:================================================================================ 2023-12-16T21:31:10.330 INFO:teuthology.orchestra.run.smithi169.stdout:Install 7 Packages 2023-12-16T21:31:10.330 INFO:teuthology.orchestra.run.smithi169.stdout: 2023-12-16T21:31:10.331 INFO:teuthology.orchestra.run.smithi169.stdout:Total download size: 2.0 M 2023-12-16T21:31:10.331 INFO:teuthology.orchestra.run.smithi169.stdout:Installed size: 8.1 M 2023-12-16T21:31:10.331 INFO:teuthology.orchestra.run.smithi169.stdout:Downloading Packages: 2023-12-16T21:31:11.188 INFO:teuthology.orchestra.run.smithi169.stdout:(1/7): ceph-mgr-dashboard-17.2.7-125.g1f2f5510. 4.6 MB/s | 1.7 MB 00:00 2023-12-16T21:31:11.209 INFO:teuthology.orchestra.run.smithi169.stdout:(2/7): ceph-prometheus-alerts-17.2.7-125.g1f2f5 37 kB/s | 15 kB 00:00 2023-12-16T21:31:11.225 INFO:teuthology.orchestra.run.smithi169.stdout:(3/7): ceph-grafana-dashboards-17.2.7-125.g1f2f 57 kB/s | 23 kB 00:00 2023-12-16T21:31:11.287 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 10/12 2023-12-16T21:31:11.392 INFO:teuthology.orchestra.run.smithi169.stdout:(4/7): python3-jwt+crypto-2.4.0-1.el9.noarch.rp 44 kB/s | 9.0 kB 00:00 2023-12-16T21:31:11.493 INFO:teuthology.orchestra.run.smithi169.stdout:(5/7): python3-repoze-lru-0.7-16.el9.noarch.rpm 115 kB/s | 31 kB 00:00 2023-12-16T21:31:11.543 INFO:teuthology.orchestra.run.smithi169.stdout:(6/7): python3-jwt-2.4.0-1.el9.noarch.rpm 122 kB/s | 41 kB 00:00 2023-12-16T21:31:11.761 INFO:teuthology.orchestra.run.smithi169.stdout:(7/7): python3-routes-2.5.1-5.el9.noarch.rpm 514 kB/s | 188 kB 00:00 2023-12-16T21:31:11.762 INFO:teuthology.orchestra.run.smithi169.stdout:-------------------------------------------------------------------------------- 2023-12-16T21:31:11.762 INFO:teuthology.orchestra.run.smithi169.stdout:Total 1.4 MB/s | 2.0 MB 00:01 2023-12-16T21:31:11.762 INFO:teuthology.orchestra.run.smithi169.stdout:Running transaction check 2023-12-16T21:31:11.770 INFO:teuthology.orchestra.run.smithi169.stdout:Transaction check succeeded. 2023-12-16T21:31:11.770 INFO:teuthology.orchestra.run.smithi169.stdout:Running transaction test 2023-12-16T21:31:11.832 INFO:teuthology.orchestra.run.smithi169.stdout:Transaction test succeeded. 2023-12-16T21:31:11.832 INFO:teuthology.orchestra.run.smithi169.stdout:Running transaction 2023-12-16T21:31:12.136 INFO:teuthology.orchestra.run.smithi169.stdout: Preparing : 1/1 2023-12-16T21:31:12.186 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : python3-jwt-2.4.0-1.el9.noarch 1/7 2023-12-16T21:31:12.220 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : python3-jwt+crypto-2.4.0-1.el9.noarch 2/7 2023-12-16T21:31:12.274 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : python3-repoze-lru-0.7-16.el9.noarch 3/7 2023-12-16T21:31:12.315 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : python3-routes-2.5.1-5.el9.noarch 4/7 2023-12-16T21:31:12.344 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : ceph-prometheus-alerts-2:17.2.7-125.g1f2f5510.el9.no 5/7 2023-12-16T21:31:12.358 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : python3-scipy-1.6.2-8.el9.x86_64 11/12 2023-12-16T21:31:12.391 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : ceph-mgr-diskprediction-local-2:17.2.7-125.g1f2f55 12/12 2023-12-16T21:31:12.572 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : ceph-grafana-dashboards-2:17.2.7-125.g1f2f5510.el9.n 6/7 2023-12-16T21:31:12.606 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : ceph-mgr-dashboard-2:17.2.7-125.g1f2f5510.el9.noarch 7/7 2023-12-16T21:31:13.139 INFO:teuthology.orchestra.run.smithi169.stdout: Running scriptlet: ceph-mgr-dashboard-2:17.2.7-125.g1f2f5510.el9.noarch 7/7 2023-12-16T21:31:13.139 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : ceph-grafana-dashboards-2:17.2.7-125.g1f2f5510.el9.n 1/7 2023-12-16T21:31:13.139 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : ceph-mgr-dashboard-2:17.2.7-125.g1f2f5510.el9.noarch 2/7 2023-12-16T21:31:13.139 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : ceph-prometheus-alerts-2:17.2.7-125.g1f2f5510.el9.no 3/7 2023-12-16T21:31:13.139 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : python3-jwt+crypto-2.4.0-1.el9.noarch 4/7 2023-12-16T21:31:13.139 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : python3-jwt-2.4.0-1.el9.noarch 5/7 2023-12-16T21:31:13.139 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : python3-repoze-lru-0.7-16.el9.noarch 6/7 2023-12-16T21:31:13.458 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : python3-routes-2.5.1-5.el9.noarch 7/7 2023-12-16T21:31:13.458 INFO:teuthology.orchestra.run.smithi169.stdout: 2023-12-16T21:31:13.458 INFO:teuthology.orchestra.run.smithi169.stdout:Installed: 2023-12-16T21:31:13.458 INFO:teuthology.orchestra.run.smithi169.stdout: ceph-grafana-dashboards-2:17.2.7-125.g1f2f5510.el9.noarch 2023-12-16T21:31:13.458 INFO:teuthology.orchestra.run.smithi169.stdout: ceph-mgr-dashboard-2:17.2.7-125.g1f2f5510.el9.noarch 2023-12-16T21:31:13.458 INFO:teuthology.orchestra.run.smithi169.stdout: ceph-prometheus-alerts-2:17.2.7-125.g1f2f5510.el9.noarch 2023-12-16T21:31:13.458 INFO:teuthology.orchestra.run.smithi169.stdout: python3-jwt-2.4.0-1.el9.noarch 2023-12-16T21:31:13.458 INFO:teuthology.orchestra.run.smithi169.stdout: python3-jwt+crypto-2.4.0-1.el9.noarch 2023-12-16T21:31:13.459 INFO:teuthology.orchestra.run.smithi169.stdout: python3-repoze-lru-0.7-16.el9.noarch 2023-12-16T21:31:13.459 INFO:teuthology.orchestra.run.smithi169.stdout: python3-routes-2.5.1-5.el9.noarch 2023-12-16T21:31:13.459 INFO:teuthology.orchestra.run.smithi169.stdout: 2023-12-16T21:31:13.459 INFO:teuthology.orchestra.run.smithi169.stdout:Complete! 2023-12-16T21:31:13.679 DEBUG:teuthology.orchestra.run.smithi169:> sudo yum -y install ceph-mgr-diskprediction-local 2023-12-16T21:31:14.238 INFO:teuthology.orchestra.run.smithi169.stdout:Last metadata expiration check: 0:01:01 ago on Sat 16 Dec 2023 09:30:13 PM UTC. 2023-12-16T21:31:14.383 INFO:teuthology.orchestra.run.smithi169.stdout:Dependencies resolved. 2023-12-16T21:31:14.384 INFO:teuthology.orchestra.run.smithi169.stdout:=========================================================================================== 2023-12-16T21:31:14.384 INFO:teuthology.orchestra.run.smithi169.stdout: Package Arch Version Repository Size 2023-12-16T21:31:14.384 INFO:teuthology.orchestra.run.smithi169.stdout:=========================================================================================== 2023-12-16T21:31:14.384 INFO:teuthology.orchestra.run.smithi169.stdout:Installing: 2023-12-16T21:31:14.384 INFO:teuthology.orchestra.run.smithi169.stdout: ceph-mgr-diskprediction-local noarch 2:17.2.7-125.g1f2f5510.el9 ceph-noarch 7.4 M 2023-12-16T21:31:14.384 INFO:teuthology.orchestra.run.smithi169.stdout:Installing dependencies: 2023-12-16T21:31:14.384 INFO:teuthology.orchestra.run.smithi169.stdout: flexiblas x86_64 3.0.4-8.el9 CentOS-AppStream 32 k 2023-12-16T21:31:14.385 INFO:teuthology.orchestra.run.smithi169.stdout: flexiblas-netlib x86_64 3.0.4-8.el9 CentOS-AppStream 3.0 M 2023-12-16T21:31:14.385 INFO:teuthology.orchestra.run.smithi169.stdout: flexiblas-openblas-openmp x86_64 3.0.4-8.el9 CentOS-AppStream 17 k 2023-12-16T21:31:14.385 INFO:teuthology.orchestra.run.smithi169.stdout: libgfortran x86_64 11.4.1-2.3.el9 CentOS-BaseOS 803 k 2023-12-16T21:31:14.385 INFO:teuthology.orchestra.run.smithi169.stdout: libquadmath x86_64 11.4.1-2.3.el9 CentOS-BaseOS 195 k 2023-12-16T21:31:14.385 INFO:teuthology.orchestra.run.smithi169.stdout: openblas x86_64 0.3.21-2.el9 CentOS-AppStream 34 k 2023-12-16T21:31:14.385 INFO:teuthology.orchestra.run.smithi169.stdout: openblas-openmp x86_64 0.3.21-2.el9 CentOS-AppStream 4.7 M 2023-12-16T21:31:14.385 INFO:teuthology.orchestra.run.smithi169.stdout: python3-devel x86_64 3.9.17-2.el9 appstream 245 k 2023-12-16T21:31:14.385 INFO:teuthology.orchestra.run.smithi169.stdout: python3-numpy x86_64 1:1.20.1-5.el9 CentOS-AppStream 4.8 M 2023-12-16T21:31:14.385 INFO:teuthology.orchestra.run.smithi169.stdout: python3-numpy-f2py x86_64 1:1.20.1-5.el9 CentOS-AppStream 217 k 2023-12-16T21:31:14.385 INFO:teuthology.orchestra.run.smithi169.stdout: python3-scipy x86_64 1.6.2-8.el9 CentOS-AppStream 16 M 2023-12-16T21:31:14.385 INFO:teuthology.orchestra.run.smithi169.stdout: 2023-12-16T21:31:14.385 INFO:teuthology.orchestra.run.smithi169.stdout:Transaction Summary 2023-12-16T21:31:14.385 INFO:teuthology.orchestra.run.smithi169.stdout:=========================================================================================== 2023-12-16T21:31:14.386 INFO:teuthology.orchestra.run.smithi169.stdout:Install 12 Packages 2023-12-16T21:31:14.386 INFO:teuthology.orchestra.run.smithi169.stdout: 2023-12-16T21:31:14.386 INFO:teuthology.orchestra.run.smithi169.stdout:Total download size: 37 M 2023-12-16T21:31:14.386 INFO:teuthology.orchestra.run.smithi169.stdout:Installed size: 209 M 2023-12-16T21:31:14.386 INFO:teuthology.orchestra.run.smithi169.stdout:Downloading Packages: 2023-12-16T21:31:14.642 INFO:teuthology.orchestra.run.smithi169.stdout:(1/12): flexiblas-3.0.4-8.el9.x86_64.rpm 217 kB/s | 32 kB 00:00 2023-12-16T21:31:14.675 INFO:teuthology.orchestra.run.smithi169.stdout:(2/12): flexiblas-openblas-openmp-3.0.4-8.el9.x 538 kB/s | 17 kB 00:00 2023-12-16T21:31:14.709 INFO:teuthology.orchestra.run.smithi169.stdout:(3/12): openblas-0.3.21-2.el9.x86_64.rpm 1.0 MB/s | 34 kB 00:00 2023-12-16T21:31:14.809 INFO:teuthology.orchestra.run.smithi169.stdout:(4/12): flexiblas-netlib-3.0.4-8.el9.x86_64.rpm 9.5 MB/s | 3.0 MB 00:00 2023-12-16T21:31:14.876 INFO:teuthology.orchestra.run.smithi169.stdout:(5/12): openblas-openmp-0.3.21-2.el9.x86_64.rpm 28 MB/s | 4.7 MB 00:00 2023-12-16T21:31:14.951 INFO:teuthology.orchestra.run.smithi169.stdout:(6/12): python3-numpy-1.20.1-5.el9.x86_64.rpm 34 MB/s | 4.8 MB 00:00 2023-12-16T21:31:14.977 INFO:teuthology.orchestra.run.smithi169.stdout:(7/12): python3-numpy-f2py-1.20.1-5.el9.x86_64. 2.2 MB/s | 217 kB 00:00 2023-12-16T21:31:14.992 INFO:teuthology.orchestra.run.smithi045.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:17.2.7-125.g1f2f55 12/12 2023-12-16T21:31:14.992 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : ceph-mgr-diskprediction-local-2:17.2.7-125.g1f2f55 1/12 2023-12-16T21:31:14.992 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : flexiblas-3.0.4-8.el9.x86_64 2/12 2023-12-16T21:31:14.993 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : flexiblas-netlib-3.0.4-8.el9.x86_64 3/12 2023-12-16T21:31:14.993 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 4/12 2023-12-16T21:31:14.993 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : openblas-0.3.21-2.el9.x86_64 5/12 2023-12-16T21:31:14.993 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : openblas-openmp-0.3.21-2.el9.x86_64 6/12 2023-12-16T21:31:14.993 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : python3-numpy-1:1.20.1-5.el9.x86_64 7/12 2023-12-16T21:31:14.993 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 8/12 2023-12-16T21:31:14.993 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : python3-scipy-1.6.2-8.el9.x86_64 9/12 2023-12-16T21:31:14.993 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : libgfortran-11.4.1-2.3.el9.x86_64 10/12 2023-12-16T21:31:14.993 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : libquadmath-11.4.1-2.3.el9.x86_64 11/12 2023-12-16T21:31:15.037 INFO:teuthology.orchestra.run.smithi169.stdout:(8/12): libgfortran-11.4.1-2.3.el9.x86_64.rpm 14 MB/s | 803 kB 00:00 2023-12-16T21:31:15.194 INFO:teuthology.orchestra.run.smithi169.stdout:(9/12): python3-scipy-1.6.2-8.el9.x86_64.rpm 65 MB/s | 16 MB 00:00 2023-12-16T21:31:15.220 INFO:teuthology.orchestra.run.smithi169.stdout:(10/12): libquadmath-11.4.1-2.3.el9.x86_64.rpm 1.0 MB/s | 195 kB 00:00 2023-12-16T21:31:15.452 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : python3-devel-3.9.17-2.el9.x86_64 12/12 2023-12-16T21:31:15.452 INFO:teuthology.orchestra.run.smithi045.stdout: 2023-12-16T21:31:15.452 INFO:teuthology.orchestra.run.smithi045.stdout:Installed: 2023-12-16T21:31:15.452 INFO:teuthology.orchestra.run.smithi045.stdout: ceph-mgr-diskprediction-local-2:17.2.7-125.g1f2f5510.el9.noarch 2023-12-16T21:31:15.452 INFO:teuthology.orchestra.run.smithi045.stdout: flexiblas-3.0.4-8.el9.x86_64 2023-12-16T21:31:15.452 INFO:teuthology.orchestra.run.smithi045.stdout: flexiblas-netlib-3.0.4-8.el9.x86_64 2023-12-16T21:31:15.452 INFO:teuthology.orchestra.run.smithi045.stdout: flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 2023-12-16T21:31:15.452 INFO:teuthology.orchestra.run.smithi045.stdout: libgfortran-11.4.1-2.3.el9.x86_64 2023-12-16T21:31:15.452 INFO:teuthology.orchestra.run.smithi045.stdout: libquadmath-11.4.1-2.3.el9.x86_64 2023-12-16T21:31:15.452 INFO:teuthology.orchestra.run.smithi045.stdout: openblas-0.3.21-2.el9.x86_64 2023-12-16T21:31:15.453 INFO:teuthology.orchestra.run.smithi045.stdout: openblas-openmp-0.3.21-2.el9.x86_64 2023-12-16T21:31:15.453 INFO:teuthology.orchestra.run.smithi045.stdout: python3-devel-3.9.17-2.el9.x86_64 2023-12-16T21:31:15.453 INFO:teuthology.orchestra.run.smithi045.stdout: python3-numpy-1:1.20.1-5.el9.x86_64 2023-12-16T21:31:15.453 INFO:teuthology.orchestra.run.smithi045.stdout: python3-numpy-f2py-1:1.20.1-5.el9.x86_64 2023-12-16T21:31:15.453 INFO:teuthology.orchestra.run.smithi045.stdout: python3-scipy-1.6.2-8.el9.x86_64 2023-12-16T21:31:15.453 INFO:teuthology.orchestra.run.smithi045.stdout: 2023-12-16T21:31:15.453 INFO:teuthology.orchestra.run.smithi045.stdout:Complete! 2023-12-16T21:31:15.536 INFO:teuthology.orchestra.run.smithi169.stdout:(11/12): ceph-mgr-diskprediction-local-17.2.7-1 7.1 MB/s | 7.4 MB 00:01 2023-12-16T21:31:15.834 DEBUG:teuthology.orchestra.run.smithi045:> sudo yum -y install ceph-mgr-rook 2023-12-16T21:31:15.903 INFO:teuthology.orchestra.run.smithi169.stdout:(12/12): python3-devel-3.9.17-2.el9.x86_64.rpm 347 kB/s | 245 kB 00:00 2023-12-16T21:31:15.906 INFO:teuthology.orchestra.run.smithi169.stdout:-------------------------------------------------------------------------------- 2023-12-16T21:31:15.907 INFO:teuthology.orchestra.run.smithi169.stdout:Total 24 MB/s | 37 MB 00:01 2023-12-16T21:31:15.922 INFO:teuthology.orchestra.run.smithi169.stdout:Running transaction check 2023-12-16T21:31:15.948 INFO:teuthology.orchestra.run.smithi169.stdout:Transaction check succeeded. 2023-12-16T21:31:15.948 INFO:teuthology.orchestra.run.smithi169.stdout:Running transaction test 2023-12-16T21:31:16.399 INFO:teuthology.orchestra.run.smithi169.stdout:Transaction test succeeded. 2023-12-16T21:31:16.399 INFO:teuthology.orchestra.run.smithi169.stdout:Running transaction 2023-12-16T21:31:16.420 INFO:teuthology.orchestra.run.smithi045.stdout:Last metadata expiration check: 0:01:13 ago on Sat 16 Dec 2023 09:30:03 PM UTC. 2023-12-16T21:31:16.571 INFO:teuthology.orchestra.run.smithi045.stdout:Dependencies resolved. 2023-12-16T21:31:16.572 INFO:teuthology.orchestra.run.smithi045.stdout:======================================================================================= 2023-12-16T21:31:16.572 INFO:teuthology.orchestra.run.smithi045.stdout: Package Arch Version Repository Size 2023-12-16T21:31:16.572 INFO:teuthology.orchestra.run.smithi045.stdout:======================================================================================= 2023-12-16T21:31:16.572 INFO:teuthology.orchestra.run.smithi045.stdout:Installing: 2023-12-16T21:31:16.572 INFO:teuthology.orchestra.run.smithi045.stdout: ceph-mgr-rook noarch 2:17.2.7-125.g1f2f5510.el9 ceph-noarch 47 k 2023-12-16T21:31:16.572 INFO:teuthology.orchestra.run.smithi045.stdout:Installing dependencies: 2023-12-16T21:31:16.573 INFO:teuthology.orchestra.run.smithi045.stdout: python3-cachetools noarch 4.2.4-1.el9 epel 32 k 2023-12-16T21:31:16.573 INFO:teuthology.orchestra.run.smithi045.stdout: python3-certifi noarch 2021.10.8-2.el9 epel 15 k 2023-12-16T21:31:16.573 INFO:teuthology.orchestra.run.smithi045.stdout: python3-google-auth noarch 1:2.23.4-1.el9 epel 197 k 2023-12-16T21:31:16.573 INFO:teuthology.orchestra.run.smithi045.stdout: python3-jsonpatch noarch 1.21-16.el9 CentOS-AppStream 24 k 2023-12-16T21:31:16.573 INFO:teuthology.orchestra.run.smithi045.stdout: python3-jsonpointer noarch 2.0-4.el9 CentOS-AppStream 18 k 2023-12-16T21:31:16.573 INFO:teuthology.orchestra.run.smithi045.stdout: python3-kubernetes noarch 1:26.1.0-2.el9 epel 1.0 M 2023-12-16T21:31:16.573 INFO:teuthology.orchestra.run.smithi045.stdout: python3-oauthlib noarch 3.1.1-5.el9 CentOS-AppStream 174 k 2023-12-16T21:31:16.573 INFO:teuthology.orchestra.run.smithi045.stdout: python3-pyasn1 noarch 0.4.8-6.el9 CentOS-AppStream 132 k 2023-12-16T21:31:16.573 INFO:teuthology.orchestra.run.smithi045.stdout: python3-pyasn1-modules noarch 0.4.8-6.el9 CentOS-AppStream 211 k 2023-12-16T21:31:16.573 INFO:teuthology.orchestra.run.smithi045.stdout: python3-requests-oauthlib noarch 1.3.0-12.el9 CentOS-AppStream 43 k 2023-12-16T21:31:16.573 INFO:teuthology.orchestra.run.smithi045.stdout: python3-rsa noarch 4.9-2.el9 epel 59 k 2023-12-16T21:31:16.574 INFO:teuthology.orchestra.run.smithi045.stdout: python3-websocket-client noarch 1.2.3-2.el9 epel 90 k 2023-12-16T21:31:16.574 INFO:teuthology.orchestra.run.smithi045.stdout: 2023-12-16T21:31:16.574 INFO:teuthology.orchestra.run.smithi045.stdout:Transaction Summary 2023-12-16T21:31:16.574 INFO:teuthology.orchestra.run.smithi045.stdout:======================================================================================= 2023-12-16T21:31:16.574 INFO:teuthology.orchestra.run.smithi045.stdout:Install 13 Packages 2023-12-16T21:31:16.574 INFO:teuthology.orchestra.run.smithi045.stdout: 2023-12-16T21:31:16.574 INFO:teuthology.orchestra.run.smithi045.stdout:Total download size: 2.0 M 2023-12-16T21:31:16.574 INFO:teuthology.orchestra.run.smithi045.stdout:Installed size: 26 M 2023-12-16T21:31:16.575 INFO:teuthology.orchestra.run.smithi045.stdout:Downloading Packages: 2023-12-16T21:31:16.851 INFO:teuthology.orchestra.run.smithi045.stdout:(1/13): python3-jsonpatch-1.21-16.el9.noarch.rp 200 kB/s | 24 kB 00:00 2023-12-16T21:31:16.877 INFO:teuthology.orchestra.run.smithi045.stdout:(2/13): python3-jsonpointer-2.0-4.el9.noarch.rp 123 kB/s | 18 kB 00:00 2023-12-16T21:31:16.918 INFO:teuthology.orchestra.run.smithi045.stdout:(3/13): python3-oauthlib-3.1.1-5.el9.noarch.rpm 2.6 MB/s | 174 kB 00:00 2023-12-16T21:31:16.944 INFO:teuthology.orchestra.run.smithi045.stdout:(4/13): python3-pyasn1-0.4.8-6.el9.noarch.rpm 1.9 MB/s | 132 kB 00:00 2023-12-16T21:31:16.969 INFO:teuthology.orchestra.run.smithi045.stdout:(5/13): ceph-mgr-rook-17.2.7-125.g1f2f5510.el9. 198 kB/s | 47 kB 00:00 2023-12-16T21:31:16.994 INFO:teuthology.orchestra.run.smithi045.stdout:(6/13): python3-pyasn1-modules-0.4.8-6.el9.noar 2.7 MB/s | 211 kB 00:00 2023-12-16T21:31:17.019 INFO:teuthology.orchestra.run.smithi045.stdout:(7/13): python3-requests-oauthlib-1.3.0-12.el9. 568 kB/s | 43 kB 00:00 2023-12-16T21:31:17.100 INFO:teuthology.orchestra.run.smithi169.stdout: Preparing : 1/1 2023-12-16T21:31:17.185 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : libquadmath-11.4.1-2.3.el9.x86_64 1/12 2023-12-16T21:31:17.215 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : libgfortran-11.4.1-2.3.el9.x86_64 2/12 2023-12-16T21:31:17.245 INFO:teuthology.orchestra.run.smithi045.stdout:(8/13): python3-certifi-2021.10.8-2.el9.noarch. 67 kB/s | 15 kB 00:00 2023-12-16T21:31:17.274 INFO:teuthology.orchestra.run.smithi045.stdout:(9/13): python3-cachetools-4.2.4-1.el9.noarch.r 107 kB/s | 32 kB 00:00 2023-12-16T21:31:17.307 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : flexiblas-3.0.4-8.el9.x86_64 3/12 2023-12-16T21:31:17.344 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : python3-devel-3.9.17-2.el9.x86_64 4/12 2023-12-16T21:31:17.420 INFO:teuthology.orchestra.run.smithi045.stdout:(10/13): python3-rsa-4.9-2.el9.noarch.rpm 393 kB/s | 59 kB 00:00 2023-12-16T21:31:17.522 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : openblas-0.3.21-2.el9.x86_64 5/12 2023-12-16T21:31:17.551 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : openblas-openmp-0.3.21-2.el9.x86_64 6/12 2023-12-16T21:31:17.562 INFO:teuthology.orchestra.run.smithi045.stdout:(11/13): python3-websocket-client-1.2.3-2.el9.n 634 kB/s | 90 kB 00:00 2023-12-16T21:31:17.588 INFO:teuthology.orchestra.run.smithi045.stdout:(12/13): python3-google-auth-2.23.4-1.el9.noarc 348 kB/s | 197 kB 00:00 2023-12-16T21:31:17.633 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 7/12 2023-12-16T21:31:18.121 INFO:teuthology.orchestra.run.smithi045.stdout:(13/13): python3-kubernetes-26.1.0-2.el9.noarch 1.2 MB/s | 1.0 MB 00:00 2023-12-16T21:31:18.123 INFO:teuthology.orchestra.run.smithi045.stdout:-------------------------------------------------------------------------------- 2023-12-16T21:31:18.123 INFO:teuthology.orchestra.run.smithi045.stdout:Total 1.3 MB/s | 2.0 MB 00:01 2023-12-16T21:31:18.124 INFO:teuthology.orchestra.run.smithi045.stdout:Running transaction check 2023-12-16T21:31:18.139 INFO:teuthology.orchestra.run.smithi045.stdout:Transaction check succeeded. 2023-12-16T21:31:18.140 INFO:teuthology.orchestra.run.smithi045.stdout:Running transaction test 2023-12-16T21:31:18.253 INFO:teuthology.orchestra.run.smithi045.stdout:Transaction test succeeded. 2023-12-16T21:31:18.253 INFO:teuthology.orchestra.run.smithi045.stdout:Running transaction 2023-12-16T21:31:18.304 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : flexiblas-netlib-3.0.4-8.el9.x86_64 8/12 2023-12-16T21:31:18.405 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : python3-numpy-1:1.20.1-5.el9.x86_64 9/12 2023-12-16T21:31:18.789 INFO:teuthology.orchestra.run.smithi045.stdout: Preparing : 1/1 2023-12-16T21:31:18.958 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : python3-pyasn1-0.4.8-6.el9.noarch 1/13 2023-12-16T21:31:19.015 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : python3-pyasn1-modules-0.4.8-6.el9.noarch 2/13 2023-12-16T21:31:19.071 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : python3-rsa-4.9-2.el9.noarch 3/13 2023-12-16T21:31:19.107 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : python3-websocket-client-1.2.3-2.el9.noarch 4/13 2023-12-16T21:31:19.149 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : python3-certifi-2021.10.8-2.el9.noarch 5/13 2023-12-16T21:31:19.253 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : python3-cachetools-4.2.4-1.el9.noarch 6/13 2023-12-16T21:31:19.362 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : python3-google-auth-1:2.23.4-1.el9.noarch 7/13 2023-12-16T21:31:19.417 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : python3-oauthlib-3.1.1-5.el9.noarch 8/13 2023-12-16T21:31:19.647 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 10/12 2023-12-16T21:31:20.176 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : python3-requests-oauthlib-1.3.0-12.el9.noarch 9/13 2023-12-16T21:31:20.240 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : python3-kubernetes-1:26.1.0-2.el9.noarch 10/13 2023-12-16T21:31:20.274 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : python3-jsonpointer-2.0-4.el9.noarch 11/13 2023-12-16T21:31:20.320 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : python3-jsonpatch-1.21-16.el9.noarch 12/13 2023-12-16T21:31:20.343 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : ceph-mgr-rook-2:17.2.7-125.g1f2f5510.el9.noarch 13/13 2023-12-16T21:31:20.673 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : python3-scipy-1.6.2-8.el9.x86_64 11/12 2023-12-16T21:31:20.700 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : ceph-mgr-diskprediction-local-2:17.2.7-125.g1f2f55 12/12 2023-12-16T21:31:21.138 INFO:teuthology.orchestra.run.smithi045.stdout: Running scriptlet: ceph-mgr-rook-2:17.2.7-125.g1f2f5510.el9.noarch 13/13 2023-12-16T21:31:21.138 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : ceph-mgr-rook-2:17.2.7-125.g1f2f5510.el9.noarch 1/13 2023-12-16T21:31:21.138 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : python3-jsonpatch-1.21-16.el9.noarch 2/13 2023-12-16T21:31:21.138 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : python3-jsonpointer-2.0-4.el9.noarch 3/13 2023-12-16T21:31:21.138 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : python3-oauthlib-3.1.1-5.el9.noarch 4/13 2023-12-16T21:31:21.138 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : python3-pyasn1-0.4.8-6.el9.noarch 5/13 2023-12-16T21:31:21.139 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : python3-pyasn1-modules-0.4.8-6.el9.noarch 6/13 2023-12-16T21:31:21.139 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : python3-requests-oauthlib-1.3.0-12.el9.noarch 7/13 2023-12-16T21:31:21.139 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : python3-cachetools-4.2.4-1.el9.noarch 8/13 2023-12-16T21:31:21.139 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : python3-certifi-2021.10.8-2.el9.noarch 9/13 2023-12-16T21:31:21.139 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : python3-google-auth-1:2.23.4-1.el9.noarch 10/13 2023-12-16T21:31:21.139 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : python3-kubernetes-1:26.1.0-2.el9.noarch 11/13 2023-12-16T21:31:21.139 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : python3-rsa-4.9-2.el9.noarch 12/13 2023-12-16T21:31:21.533 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : python3-websocket-client-1.2.3-2.el9.noarch 13/13 2023-12-16T21:31:21.534 INFO:teuthology.orchestra.run.smithi045.stdout: 2023-12-16T21:31:21.534 INFO:teuthology.orchestra.run.smithi045.stdout:Installed: 2023-12-16T21:31:21.534 INFO:teuthology.orchestra.run.smithi045.stdout: ceph-mgr-rook-2:17.2.7-125.g1f2f5510.el9.noarch 2023-12-16T21:31:21.534 INFO:teuthology.orchestra.run.smithi045.stdout: python3-cachetools-4.2.4-1.el9.noarch 2023-12-16T21:31:21.534 INFO:teuthology.orchestra.run.smithi045.stdout: python3-certifi-2021.10.8-2.el9.noarch 2023-12-16T21:31:21.534 INFO:teuthology.orchestra.run.smithi045.stdout: python3-google-auth-1:2.23.4-1.el9.noarch 2023-12-16T21:31:21.534 INFO:teuthology.orchestra.run.smithi045.stdout: python3-jsonpatch-1.21-16.el9.noarch 2023-12-16T21:31:21.534 INFO:teuthology.orchestra.run.smithi045.stdout: python3-jsonpointer-2.0-4.el9.noarch 2023-12-16T21:31:21.534 INFO:teuthology.orchestra.run.smithi045.stdout: python3-kubernetes-1:26.1.0-2.el9.noarch 2023-12-16T21:31:21.534 INFO:teuthology.orchestra.run.smithi045.stdout: python3-oauthlib-3.1.1-5.el9.noarch 2023-12-16T21:31:21.535 INFO:teuthology.orchestra.run.smithi045.stdout: python3-pyasn1-0.4.8-6.el9.noarch 2023-12-16T21:31:21.535 INFO:teuthology.orchestra.run.smithi045.stdout: python3-pyasn1-modules-0.4.8-6.el9.noarch 2023-12-16T21:31:21.535 INFO:teuthology.orchestra.run.smithi045.stdout: python3-requests-oauthlib-1.3.0-12.el9.noarch 2023-12-16T21:31:21.535 INFO:teuthology.orchestra.run.smithi045.stdout: python3-rsa-4.9-2.el9.noarch 2023-12-16T21:31:21.535 INFO:teuthology.orchestra.run.smithi045.stdout: python3-websocket-client-1.2.3-2.el9.noarch 2023-12-16T21:31:21.535 INFO:teuthology.orchestra.run.smithi045.stdout: 2023-12-16T21:31:21.535 INFO:teuthology.orchestra.run.smithi045.stdout:Complete! 2023-12-16T21:31:21.776 DEBUG:teuthology.orchestra.run.smithi045:> sudo yum -y install ceph-mgr-cephadm 2023-12-16T21:31:22.359 INFO:teuthology.orchestra.run.smithi045.stdout:Last metadata expiration check: 0:01:19 ago on Sat 16 Dec 2023 09:30:03 PM UTC. 2023-12-16T21:31:22.511 INFO:teuthology.orchestra.run.smithi045.stdout:Dependencies resolved. 2023-12-16T21:31:22.511 INFO:teuthology.orchestra.run.smithi045.stdout:======================================================================================= 2023-12-16T21:31:22.511 INFO:teuthology.orchestra.run.smithi045.stdout: Package Arch Version Repository Size 2023-12-16T21:31:22.511 INFO:teuthology.orchestra.run.smithi045.stdout:======================================================================================= 2023-12-16T21:31:22.511 INFO:teuthology.orchestra.run.smithi045.stdout:Installing: 2023-12-16T21:31:22.512 INFO:teuthology.orchestra.run.smithi045.stdout: ceph-mgr-cephadm noarch 2:17.2.7-125.g1f2f5510.el9 ceph-noarch 121 k 2023-12-16T21:31:22.512 INFO:teuthology.orchestra.run.smithi045.stdout:Installing dependencies: 2023-12-16T21:31:22.512 INFO:teuthology.orchestra.run.smithi045.stdout: python3-asyncssh noarch 2.13.2-3.el9 epel 548 k 2023-12-16T21:31:22.512 INFO:teuthology.orchestra.run.smithi045.stdout: python3-babel noarch 2.9.1-2.el9 CentOS-AppStream 5.8 M 2023-12-16T21:31:22.512 INFO:teuthology.orchestra.run.smithi045.stdout: python3-jinja2 noarch 2.11.3-4.el9 CentOS-AppStream 228 k 2023-12-16T21:31:22.512 INFO:teuthology.orchestra.run.smithi045.stdout: python3-natsort noarch 7.1.1-5.el9 epel 58 k 2023-12-16T21:31:22.512 INFO:teuthology.orchestra.run.smithi045.stdout: python3-typing-extensions noarch 4.4.0-2.el9 epel 51 k 2023-12-16T21:31:22.512 INFO:teuthology.orchestra.run.smithi045.stdout: 2023-12-16T21:31:22.512 INFO:teuthology.orchestra.run.smithi045.stdout:Transaction Summary 2023-12-16T21:31:22.512 INFO:teuthology.orchestra.run.smithi045.stdout:======================================================================================= 2023-12-16T21:31:22.512 INFO:teuthology.orchestra.run.smithi045.stdout:Install 6 Packages 2023-12-16T21:31:22.513 INFO:teuthology.orchestra.run.smithi045.stdout: 2023-12-16T21:31:22.513 INFO:teuthology.orchestra.run.smithi045.stdout:Total download size: 6.8 M 2023-12-16T21:31:22.513 INFO:teuthology.orchestra.run.smithi045.stdout:Installed size: 33 M 2023-12-16T21:31:22.513 INFO:teuthology.orchestra.run.smithi045.stdout:Downloading Packages: 2023-12-16T21:31:22.725 INFO:teuthology.orchestra.run.smithi045.stdout:(1/6): python3-jinja2-2.11.3-4.el9.noarch.rpm 1.4 MB/s | 228 kB 00:00 2023-12-16T21:31:22.850 INFO:teuthology.orchestra.run.smithi045.stdout:(2/6): python3-babel-2.9.1-2.el9.noarch.rpm 20 MB/s | 5.8 MB 00:00 2023-12-16T21:31:22.876 INFO:teuthology.orchestra.run.smithi045.stdout:(3/6): ceph-mgr-cephadm-17.2.7-125.g1f2f5510.el 392 kB/s | 121 kB 00:00 2023-12-16T21:31:23.370 INFO:teuthology.orchestra.run.smithi169.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:17.2.7-125.g1f2f55 12/12 2023-12-16T21:31:23.370 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : ceph-mgr-diskprediction-local-2:17.2.7-125.g1f2f55 1/12 2023-12-16T21:31:23.370 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : flexiblas-3.0.4-8.el9.x86_64 2/12 2023-12-16T21:31:23.370 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : flexiblas-netlib-3.0.4-8.el9.x86_64 3/12 2023-12-16T21:31:23.370 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 4/12 2023-12-16T21:31:23.370 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : openblas-0.3.21-2.el9.x86_64 5/12 2023-12-16T21:31:23.370 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : openblas-openmp-0.3.21-2.el9.x86_64 6/12 2023-12-16T21:31:23.370 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : python3-numpy-1:1.20.1-5.el9.x86_64 7/12 2023-12-16T21:31:23.370 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 8/12 2023-12-16T21:31:23.371 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : python3-scipy-1.6.2-8.el9.x86_64 9/12 2023-12-16T21:31:23.371 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : libgfortran-11.4.1-2.3.el9.x86_64 10/12 2023-12-16T21:31:23.371 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : libquadmath-11.4.1-2.3.el9.x86_64 11/12 2023-12-16T21:31:23.693 INFO:teuthology.orchestra.run.smithi045.stdout:(4/6): python3-typing-extensions-4.4.0-2.el9.no 62 kB/s | 51 kB 00:00 2023-12-16T21:31:23.718 INFO:teuthology.orchestra.run.smithi045.stdout:(5/6): python3-natsort-7.1.1-5.el9.noarch.rpm 67 kB/s | 58 kB 00:00 2023-12-16T21:31:23.818 INFO:teuthology.orchestra.run.smithi045.stdout:(6/6): python3-asyncssh-2.13.2-3.el9.noarch.rpm 502 kB/s | 548 kB 00:01 2023-12-16T21:31:23.820 INFO:teuthology.orchestra.run.smithi045.stdout:-------------------------------------------------------------------------------- 2023-12-16T21:31:23.820 INFO:teuthology.orchestra.run.smithi045.stdout:Total 5.2 MB/s | 6.8 MB 00:01 2023-12-16T21:31:23.821 INFO:teuthology.orchestra.run.smithi045.stdout:Running transaction check 2023-12-16T21:31:23.827 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : python3-devel-3.9.17-2.el9.x86_64 12/12 2023-12-16T21:31:23.827 INFO:teuthology.orchestra.run.smithi169.stdout: 2023-12-16T21:31:23.827 INFO:teuthology.orchestra.run.smithi169.stdout:Installed: 2023-12-16T21:31:23.828 INFO:teuthology.orchestra.run.smithi169.stdout: ceph-mgr-diskprediction-local-2:17.2.7-125.g1f2f5510.el9.noarch 2023-12-16T21:31:23.828 INFO:teuthology.orchestra.run.smithi169.stdout: flexiblas-3.0.4-8.el9.x86_64 2023-12-16T21:31:23.828 INFO:teuthology.orchestra.run.smithi169.stdout: flexiblas-netlib-3.0.4-8.el9.x86_64 2023-12-16T21:31:23.828 INFO:teuthology.orchestra.run.smithi169.stdout: flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 2023-12-16T21:31:23.828 INFO:teuthology.orchestra.run.smithi169.stdout: libgfortran-11.4.1-2.3.el9.x86_64 2023-12-16T21:31:23.828 INFO:teuthology.orchestra.run.smithi169.stdout: libquadmath-11.4.1-2.3.el9.x86_64 2023-12-16T21:31:23.828 INFO:teuthology.orchestra.run.smithi169.stdout: openblas-0.3.21-2.el9.x86_64 2023-12-16T21:31:23.828 INFO:teuthology.orchestra.run.smithi169.stdout: openblas-openmp-0.3.21-2.el9.x86_64 2023-12-16T21:31:23.828 INFO:teuthology.orchestra.run.smithi169.stdout: python3-devel-3.9.17-2.el9.x86_64 2023-12-16T21:31:23.828 INFO:teuthology.orchestra.run.smithi169.stdout: python3-numpy-1:1.20.1-5.el9.x86_64 2023-12-16T21:31:23.828 INFO:teuthology.orchestra.run.smithi169.stdout: python3-numpy-f2py-1:1.20.1-5.el9.x86_64 2023-12-16T21:31:23.829 INFO:teuthology.orchestra.run.smithi169.stdout: python3-scipy-1.6.2-8.el9.x86_64 2023-12-16T21:31:23.829 INFO:teuthology.orchestra.run.smithi169.stdout: 2023-12-16T21:31:23.829 INFO:teuthology.orchestra.run.smithi169.stdout:Complete! 2023-12-16T21:31:23.832 INFO:teuthology.orchestra.run.smithi045.stdout:Transaction check succeeded. 2023-12-16T21:31:23.832 INFO:teuthology.orchestra.run.smithi045.stdout:Running transaction test 2023-12-16T21:31:23.931 INFO:teuthology.orchestra.run.smithi045.stdout:Transaction test succeeded. 2023-12-16T21:31:23.931 INFO:teuthology.orchestra.run.smithi045.stdout:Running transaction 2023-12-16T21:31:24.214 DEBUG:teuthology.orchestra.run.smithi169:> sudo yum -y install ceph-mgr-rook 2023-12-16T21:31:24.241 INFO:teuthology.orchestra.run.smithi045.stdout: Preparing : 1/1 2023-12-16T21:31:24.406 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : python3-typing-extensions-4.4.0-2.el9.noarch 1/6 2023-12-16T21:31:24.471 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : python3-asyncssh-2.13.2-3.el9.noarch 2/6 2023-12-16T21:31:24.775 INFO:teuthology.orchestra.run.smithi169.stdout:Last metadata expiration check: 0:01:11 ago on Sat 16 Dec 2023 09:30:13 PM UTC. 2023-12-16T21:31:24.920 INFO:teuthology.orchestra.run.smithi169.stdout:Dependencies resolved. 2023-12-16T21:31:24.921 INFO:teuthology.orchestra.run.smithi169.stdout:======================================================================================= 2023-12-16T21:31:24.921 INFO:teuthology.orchestra.run.smithi169.stdout: Package Arch Version Repository Size 2023-12-16T21:31:24.921 INFO:teuthology.orchestra.run.smithi169.stdout:======================================================================================= 2023-12-16T21:31:24.921 INFO:teuthology.orchestra.run.smithi169.stdout:Installing: 2023-12-16T21:31:24.921 INFO:teuthology.orchestra.run.smithi169.stdout: ceph-mgr-rook noarch 2:17.2.7-125.g1f2f5510.el9 ceph-noarch 47 k 2023-12-16T21:31:24.921 INFO:teuthology.orchestra.run.smithi169.stdout:Installing dependencies: 2023-12-16T21:31:24.921 INFO:teuthology.orchestra.run.smithi169.stdout: python3-cachetools noarch 4.2.4-1.el9 epel 32 k 2023-12-16T21:31:24.922 INFO:teuthology.orchestra.run.smithi169.stdout: python3-certifi noarch 2021.10.8-2.el9 epel 15 k 2023-12-16T21:31:24.922 INFO:teuthology.orchestra.run.smithi169.stdout: python3-google-auth noarch 1:2.23.4-1.el9 epel 197 k 2023-12-16T21:31:24.922 INFO:teuthology.orchestra.run.smithi169.stdout: python3-jsonpatch noarch 1.21-16.el9 CentOS-AppStream 24 k 2023-12-16T21:31:24.922 INFO:teuthology.orchestra.run.smithi169.stdout: python3-jsonpointer noarch 2.0-4.el9 CentOS-AppStream 18 k 2023-12-16T21:31:24.922 INFO:teuthology.orchestra.run.smithi169.stdout: python3-kubernetes noarch 1:26.1.0-2.el9 epel 1.0 M 2023-12-16T21:31:24.922 INFO:teuthology.orchestra.run.smithi169.stdout: python3-oauthlib noarch 3.1.1-5.el9 CentOS-AppStream 174 k 2023-12-16T21:31:24.922 INFO:teuthology.orchestra.run.smithi169.stdout: python3-pyasn1 noarch 0.4.8-6.el9 CentOS-AppStream 132 k 2023-12-16T21:31:24.922 INFO:teuthology.orchestra.run.smithi169.stdout: python3-pyasn1-modules noarch 0.4.8-6.el9 CentOS-AppStream 211 k 2023-12-16T21:31:24.922 INFO:teuthology.orchestra.run.smithi169.stdout: python3-requests-oauthlib noarch 1.3.0-12.el9 CentOS-AppStream 43 k 2023-12-16T21:31:24.923 INFO:teuthology.orchestra.run.smithi169.stdout: python3-rsa noarch 4.9-2.el9 epel 59 k 2023-12-16T21:31:24.923 INFO:teuthology.orchestra.run.smithi169.stdout: python3-websocket-client noarch 1.2.3-2.el9 epel 90 k 2023-12-16T21:31:24.923 INFO:teuthology.orchestra.run.smithi169.stdout: 2023-12-16T21:31:24.923 INFO:teuthology.orchestra.run.smithi169.stdout:Transaction Summary 2023-12-16T21:31:24.923 INFO:teuthology.orchestra.run.smithi169.stdout:======================================================================================= 2023-12-16T21:31:24.923 INFO:teuthology.orchestra.run.smithi169.stdout:Install 13 Packages 2023-12-16T21:31:24.923 INFO:teuthology.orchestra.run.smithi169.stdout: 2023-12-16T21:31:24.924 INFO:teuthology.orchestra.run.smithi169.stdout:Total download size: 2.0 M 2023-12-16T21:31:24.924 INFO:teuthology.orchestra.run.smithi169.stdout:Installed size: 26 M 2023-12-16T21:31:24.924 INFO:teuthology.orchestra.run.smithi169.stdout:Downloading Packages: 2023-12-16T21:31:25.027 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : python3-natsort-7.1.1-5.el9.noarch 3/6 2023-12-16T21:31:25.116 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : python3-babel-2.9.1-2.el9.noarch 4/6 2023-12-16T21:31:25.125 INFO:teuthology.orchestra.run.smithi169.stdout:(1/13): python3-jsonpatch-1.21-16.el9.noarch.rp 205 kB/s | 24 kB 00:00 2023-12-16T21:31:25.142 INFO:teuthology.orchestra.run.smithi169.stdout:(2/13): python3-jsonpointer-2.0-4.el9.noarch.rp 133 kB/s | 18 kB 00:00 2023-12-16T21:31:25.171 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : python3-jinja2-2.11.3-4.el9.noarch 5/6 2023-12-16T21:31:25.184 INFO:teuthology.orchestra.run.smithi169.stdout:(3/13): python3-oauthlib-3.1.1-5.el9.noarch.rpm 3.0 MB/s | 174 kB 00:00 2023-12-16T21:31:25.199 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : ceph-mgr-cephadm-2:17.2.7-125.g1f2f5510.el9.noarch 6/6 2023-12-16T21:31:25.201 INFO:teuthology.orchestra.run.smithi169.stdout:(4/13): python3-pyasn1-0.4.8-6.el9.noarch.rpm 2.2 MB/s | 132 kB 00:00 2023-12-16T21:31:25.241 INFO:teuthology.orchestra.run.smithi169.stdout:(5/13): python3-pyasn1-modules-0.4.8-6.el9.noar 3.6 MB/s | 211 kB 00:00 2023-12-16T21:31:25.271 INFO:teuthology.orchestra.run.smithi169.stdout:(6/13): python3-requests-oauthlib-1.3.0-12.el9. 623 kB/s | 43 kB 00:00 2023-12-16T21:31:25.296 INFO:teuthology.orchestra.run.smithi169.stdout:(7/13): ceph-mgr-rook-17.2.7-125.g1f2f5510.el9. 164 kB/s | 47 kB 00:00 2023-12-16T21:31:25.530 INFO:teuthology.orchestra.run.smithi169.stdout:(8/13): python3-cachetools-4.2.4-1.el9.noarch.r 112 kB/s | 32 kB 00:00 2023-12-16T21:31:25.555 INFO:teuthology.orchestra.run.smithi169.stdout:(9/13): python3-certifi-2021.10.8-2.el9.noarch. 53 kB/s | 15 kB 00:00 2023-12-16T21:31:25.613 INFO:teuthology.orchestra.run.smithi169.stdout:(10/13): python3-google-auth-2.23.4-1.el9.noarc 622 kB/s | 197 kB 00:00 2023-12-16T21:31:25.772 INFO:teuthology.orchestra.run.smithi169.stdout:(11/13): python3-kubernetes-26.1.0-2.el9.noarch 4.2 MB/s | 1.0 MB 00:00 2023-12-16T21:31:25.797 INFO:teuthology.orchestra.run.smithi169.stdout:(12/13): python3-rsa-4.9-2.el9.noarch.rpm 244 kB/s | 59 kB 00:00 2023-12-16T21:31:25.814 INFO:teuthology.orchestra.run.smithi169.stdout:(13/13): python3-websocket-client-1.2.3-2.el9.n 448 kB/s | 90 kB 00:00 2023-12-16T21:31:25.818 INFO:teuthology.orchestra.run.smithi169.stdout:-------------------------------------------------------------------------------- 2023-12-16T21:31:25.819 INFO:teuthology.orchestra.run.smithi169.stdout:Total 2.3 MB/s | 2.0 MB 00:00 2023-12-16T21:31:25.819 INFO:teuthology.orchestra.run.smithi169.stdout:Running transaction check 2023-12-16T21:31:25.831 INFO:teuthology.orchestra.run.smithi169.stdout:Transaction check succeeded. 2023-12-16T21:31:25.831 INFO:teuthology.orchestra.run.smithi169.stdout:Running transaction test 2023-12-16T21:31:25.928 INFO:teuthology.orchestra.run.smithi169.stdout:Transaction test succeeded. 2023-12-16T21:31:25.928 INFO:teuthology.orchestra.run.smithi169.stdout:Running transaction 2023-12-16T21:31:26.046 INFO:teuthology.orchestra.run.smithi045.stdout: Running scriptlet: ceph-mgr-cephadm-2:17.2.7-125.g1f2f5510.el9.noarch 6/6 2023-12-16T21:31:26.046 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : ceph-mgr-cephadm-2:17.2.7-125.g1f2f5510.el9.noarch 1/6 2023-12-16T21:31:26.046 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : python3-babel-2.9.1-2.el9.noarch 2/6 2023-12-16T21:31:26.046 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : python3-jinja2-2.11.3-4.el9.noarch 3/6 2023-12-16T21:31:26.047 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : python3-asyncssh-2.13.2-3.el9.noarch 4/6 2023-12-16T21:31:26.047 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : python3-natsort-7.1.1-5.el9.noarch 5/6 2023-12-16T21:31:26.364 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : python3-typing-extensions-4.4.0-2.el9.noarch 6/6 2023-12-16T21:31:26.364 INFO:teuthology.orchestra.run.smithi045.stdout: 2023-12-16T21:31:26.364 INFO:teuthology.orchestra.run.smithi045.stdout:Installed: 2023-12-16T21:31:26.364 INFO:teuthology.orchestra.run.smithi045.stdout: ceph-mgr-cephadm-2:17.2.7-125.g1f2f5510.el9.noarch 2023-12-16T21:31:26.364 INFO:teuthology.orchestra.run.smithi045.stdout: python3-asyncssh-2.13.2-3.el9.noarch 2023-12-16T21:31:26.364 INFO:teuthology.orchestra.run.smithi045.stdout: python3-babel-2.9.1-2.el9.noarch 2023-12-16T21:31:26.364 INFO:teuthology.orchestra.run.smithi045.stdout: python3-jinja2-2.11.3-4.el9.noarch 2023-12-16T21:31:26.364 INFO:teuthology.orchestra.run.smithi045.stdout: python3-natsort-7.1.1-5.el9.noarch 2023-12-16T21:31:26.364 INFO:teuthology.orchestra.run.smithi045.stdout: python3-typing-extensions-4.4.0-2.el9.noarch 2023-12-16T21:31:26.364 INFO:teuthology.orchestra.run.smithi045.stdout: 2023-12-16T21:31:26.364 INFO:teuthology.orchestra.run.smithi045.stdout:Complete! 2023-12-16T21:31:26.475 INFO:teuthology.orchestra.run.smithi169.stdout: Preparing : 1/1 2023-12-16T21:31:26.617 DEBUG:teuthology.orchestra.run.smithi045:> sudo yum -y install ceph-fuse 2023-12-16T21:31:26.627 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : python3-pyasn1-0.4.8-6.el9.noarch 1/13 2023-12-16T21:31:26.681 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : python3-pyasn1-modules-0.4.8-6.el9.noarch 2/13 2023-12-16T21:31:26.737 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : python3-rsa-4.9-2.el9.noarch 3/13 2023-12-16T21:31:26.774 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : python3-websocket-client-1.2.3-2.el9.noarch 4/13 2023-12-16T21:31:26.815 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : python3-certifi-2021.10.8-2.el9.noarch 5/13 2023-12-16T21:31:26.908 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : python3-cachetools-4.2.4-1.el9.noarch 6/13 2023-12-16T21:31:27.035 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : python3-google-auth-1:2.23.4-1.el9.noarch 7/13 2023-12-16T21:31:27.092 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : python3-oauthlib-3.1.1-5.el9.noarch 8/13 2023-12-16T21:31:27.200 INFO:teuthology.orchestra.run.smithi045.stdout:Last metadata expiration check: 0:01:24 ago on Sat 16 Dec 2023 09:30:03 PM UTC. 2023-12-16T21:31:27.350 INFO:teuthology.orchestra.run.smithi045.stdout:Dependencies resolved. 2023-12-16T21:31:27.351 INFO:teuthology.orchestra.run.smithi045.stdout:================================================================================ 2023-12-16T21:31:27.351 INFO:teuthology.orchestra.run.smithi045.stdout: Package Arch Version Repository Size 2023-12-16T21:31:27.351 INFO:teuthology.orchestra.run.smithi045.stdout:================================================================================ 2023-12-16T21:31:27.351 INFO:teuthology.orchestra.run.smithi045.stdout:Installing: 2023-12-16T21:31:27.351 INFO:teuthology.orchestra.run.smithi045.stdout: ceph-fuse x86_64 2:17.2.7-125.g1f2f5510.el9 ceph 810 k 2023-12-16T21:31:27.351 INFO:teuthology.orchestra.run.smithi045.stdout:Installing dependencies: 2023-12-16T21:31:27.352 INFO:teuthology.orchestra.run.smithi045.stdout: fuse x86_64 2.9.9-15.el9 CentOS-BaseOS 79 k 2023-12-16T21:31:27.352 INFO:teuthology.orchestra.run.smithi045.stdout: 2023-12-16T21:31:27.352 INFO:teuthology.orchestra.run.smithi045.stdout:Transaction Summary 2023-12-16T21:31:27.352 INFO:teuthology.orchestra.run.smithi045.stdout:================================================================================ 2023-12-16T21:31:27.352 INFO:teuthology.orchestra.run.smithi045.stdout:Install 2 Packages 2023-12-16T21:31:27.352 INFO:teuthology.orchestra.run.smithi045.stdout: 2023-12-16T21:31:27.352 INFO:teuthology.orchestra.run.smithi045.stdout:Total download size: 889 k 2023-12-16T21:31:27.353 INFO:teuthology.orchestra.run.smithi045.stdout:Installed size: 2.6 M 2023-12-16T21:31:27.353 INFO:teuthology.orchestra.run.smithi045.stdout:Downloading Packages: 2023-12-16T21:31:27.479 INFO:teuthology.orchestra.run.smithi045.stdout:(1/2): fuse-2.9.9-15.el9.x86_64.rpm 622 kB/s | 79 kB 00:00 2023-12-16T21:31:27.654 INFO:teuthology.orchestra.run.smithi045.stdout:(2/2): ceph-fuse-17.2.7-125.g1f2f5510.el9.x86_6 2.6 MB/s | 810 kB 00:00 2023-12-16T21:31:27.655 INFO:teuthology.orchestra.run.smithi045.stdout:-------------------------------------------------------------------------------- 2023-12-16T21:31:27.655 INFO:teuthology.orchestra.run.smithi045.stdout:Total 2.9 MB/s | 889 kB 00:00 2023-12-16T21:31:27.656 INFO:teuthology.orchestra.run.smithi045.stdout:Running transaction check 2023-12-16T21:31:27.677 INFO:teuthology.orchestra.run.smithi045.stdout:Transaction check succeeded. 2023-12-16T21:31:27.678 INFO:teuthology.orchestra.run.smithi045.stdout:Running transaction test 2023-12-16T21:31:27.757 INFO:teuthology.orchestra.run.smithi045.stdout:Transaction test succeeded. 2023-12-16T21:31:27.757 INFO:teuthology.orchestra.run.smithi045.stdout:Running transaction 2023-12-16T21:31:27.818 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : python3-requests-oauthlib-1.3.0-12.el9.noarch 9/13 2023-12-16T21:31:27.873 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : python3-kubernetes-1:26.1.0-2.el9.noarch 10/13 2023-12-16T21:31:27.906 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : python3-jsonpointer-2.0-4.el9.noarch 11/13 2023-12-16T21:31:27.950 INFO:teuthology.orchestra.run.smithi045.stdout: Preparing : 1/1 2023-12-16T21:31:27.952 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : python3-jsonpatch-1.21-16.el9.noarch 12/13 2023-12-16T21:31:27.968 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : ceph-mgr-rook-2:17.2.7-125.g1f2f5510.el9.noarch 13/13 2023-12-16T21:31:28.087 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : fuse-2.9.9-15.el9.x86_64 1/2 2023-12-16T21:31:28.164 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : ceph-fuse-2:17.2.7-125.g1f2f5510.el9.x86_64 2/2 2023-12-16T21:31:28.788 INFO:teuthology.orchestra.run.smithi169.stdout: Running scriptlet: ceph-mgr-rook-2:17.2.7-125.g1f2f5510.el9.noarch 13/13 2023-12-16T21:31:28.788 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : ceph-mgr-rook-2:17.2.7-125.g1f2f5510.el9.noarch 1/13 2023-12-16T21:31:28.788 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : python3-jsonpatch-1.21-16.el9.noarch 2/13 2023-12-16T21:31:28.788 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : python3-jsonpointer-2.0-4.el9.noarch 3/13 2023-12-16T21:31:28.788 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : python3-oauthlib-3.1.1-5.el9.noarch 4/13 2023-12-16T21:31:28.788 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : python3-pyasn1-0.4.8-6.el9.noarch 5/13 2023-12-16T21:31:28.788 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : python3-pyasn1-modules-0.4.8-6.el9.noarch 6/13 2023-12-16T21:31:28.788 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : python3-requests-oauthlib-1.3.0-12.el9.noarch 7/13 2023-12-16T21:31:28.788 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : python3-cachetools-4.2.4-1.el9.noarch 8/13 2023-12-16T21:31:28.788 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : python3-certifi-2021.10.8-2.el9.noarch 9/13 2023-12-16T21:31:28.788 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : python3-google-auth-1:2.23.4-1.el9.noarch 10/13 2023-12-16T21:31:28.789 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : python3-kubernetes-1:26.1.0-2.el9.noarch 11/13 2023-12-16T21:31:28.789 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : python3-rsa-4.9-2.el9.noarch 12/13 2023-12-16T21:31:28.883 INFO:teuthology.orchestra.run.smithi045.stdout: Running scriptlet: ceph-fuse-2:17.2.7-125.g1f2f5510.el9.x86_64 2/2 2023-12-16T21:31:28.884 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : ceph-fuse-2:17.2.7-125.g1f2f5510.el9.x86_64 1/2 2023-12-16T21:31:29.183 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : python3-websocket-client-1.2.3-2.el9.noarch 13/13 2023-12-16T21:31:29.183 INFO:teuthology.orchestra.run.smithi169.stdout: 2023-12-16T21:31:29.183 INFO:teuthology.orchestra.run.smithi169.stdout:Installed: 2023-12-16T21:31:29.183 INFO:teuthology.orchestra.run.smithi169.stdout: ceph-mgr-rook-2:17.2.7-125.g1f2f5510.el9.noarch 2023-12-16T21:31:29.183 INFO:teuthology.orchestra.run.smithi169.stdout: python3-cachetools-4.2.4-1.el9.noarch 2023-12-16T21:31:29.183 INFO:teuthology.orchestra.run.smithi169.stdout: python3-certifi-2021.10.8-2.el9.noarch 2023-12-16T21:31:29.183 INFO:teuthology.orchestra.run.smithi169.stdout: python3-google-auth-1:2.23.4-1.el9.noarch 2023-12-16T21:31:29.183 INFO:teuthology.orchestra.run.smithi169.stdout: python3-jsonpatch-1.21-16.el9.noarch 2023-12-16T21:31:29.183 INFO:teuthology.orchestra.run.smithi169.stdout: python3-jsonpointer-2.0-4.el9.noarch 2023-12-16T21:31:29.183 INFO:teuthology.orchestra.run.smithi169.stdout: python3-kubernetes-1:26.1.0-2.el9.noarch 2023-12-16T21:31:29.184 INFO:teuthology.orchestra.run.smithi169.stdout: python3-oauthlib-3.1.1-5.el9.noarch 2023-12-16T21:31:29.184 INFO:teuthology.orchestra.run.smithi169.stdout: python3-pyasn1-0.4.8-6.el9.noarch 2023-12-16T21:31:29.184 INFO:teuthology.orchestra.run.smithi169.stdout: python3-pyasn1-modules-0.4.8-6.el9.noarch 2023-12-16T21:31:29.184 INFO:teuthology.orchestra.run.smithi169.stdout: python3-requests-oauthlib-1.3.0-12.el9.noarch 2023-12-16T21:31:29.184 INFO:teuthology.orchestra.run.smithi169.stdout: python3-rsa-4.9-2.el9.noarch 2023-12-16T21:31:29.184 INFO:teuthology.orchestra.run.smithi169.stdout: python3-websocket-client-1.2.3-2.el9.noarch 2023-12-16T21:31:29.184 INFO:teuthology.orchestra.run.smithi169.stdout: 2023-12-16T21:31:29.184 INFO:teuthology.orchestra.run.smithi169.stdout:Complete! 2023-12-16T21:31:29.213 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : fuse-2.9.9-15.el9.x86_64 2/2 2023-12-16T21:31:29.213 INFO:teuthology.orchestra.run.smithi045.stdout: 2023-12-16T21:31:29.213 INFO:teuthology.orchestra.run.smithi045.stdout:Installed: 2023-12-16T21:31:29.213 INFO:teuthology.orchestra.run.smithi045.stdout: ceph-fuse-2:17.2.7-125.g1f2f5510.el9.x86_64 fuse-2.9.9-15.el9.x86_64 2023-12-16T21:31:29.213 INFO:teuthology.orchestra.run.smithi045.stdout: 2023-12-16T21:31:29.213 INFO:teuthology.orchestra.run.smithi045.stdout:Complete! 2023-12-16T21:31:29.453 DEBUG:teuthology.orchestra.run.smithi169:> sudo yum -y install ceph-mgr-cephadm 2023-12-16T21:31:29.493 DEBUG:teuthology.orchestra.run.smithi045:> sudo yum -y install librados-devel 2023-12-16T21:31:30.011 INFO:teuthology.orchestra.run.smithi169.stdout:Last metadata expiration check: 0:01:17 ago on Sat 16 Dec 2023 09:30:13 PM UTC. 2023-12-16T21:31:30.085 INFO:teuthology.orchestra.run.smithi045.stdout:Last metadata expiration check: 0:01:27 ago on Sat 16 Dec 2023 09:30:03 PM UTC. 2023-12-16T21:31:30.156 INFO:teuthology.orchestra.run.smithi169.stdout:Dependencies resolved. 2023-12-16T21:31:30.157 INFO:teuthology.orchestra.run.smithi169.stdout:======================================================================================= 2023-12-16T21:31:30.157 INFO:teuthology.orchestra.run.smithi169.stdout: Package Arch Version Repository Size 2023-12-16T21:31:30.157 INFO:teuthology.orchestra.run.smithi169.stdout:======================================================================================= 2023-12-16T21:31:30.157 INFO:teuthology.orchestra.run.smithi169.stdout:Installing: 2023-12-16T21:31:30.157 INFO:teuthology.orchestra.run.smithi169.stdout: ceph-mgr-cephadm noarch 2:17.2.7-125.g1f2f5510.el9 ceph-noarch 121 k 2023-12-16T21:31:30.157 INFO:teuthology.orchestra.run.smithi169.stdout:Installing dependencies: 2023-12-16T21:31:30.157 INFO:teuthology.orchestra.run.smithi169.stdout: python3-asyncssh noarch 2.13.2-3.el9 epel 548 k 2023-12-16T21:31:30.158 INFO:teuthology.orchestra.run.smithi169.stdout: python3-babel noarch 2.9.1-2.el9 CentOS-AppStream 5.8 M 2023-12-16T21:31:30.158 INFO:teuthology.orchestra.run.smithi169.stdout: python3-jinja2 noarch 2.11.3-4.el9 CentOS-AppStream 228 k 2023-12-16T21:31:30.158 INFO:teuthology.orchestra.run.smithi169.stdout: python3-natsort noarch 7.1.1-5.el9 epel 58 k 2023-12-16T21:31:30.158 INFO:teuthology.orchestra.run.smithi169.stdout: python3-typing-extensions noarch 4.4.0-2.el9 epel 51 k 2023-12-16T21:31:30.158 INFO:teuthology.orchestra.run.smithi169.stdout: 2023-12-16T21:31:30.158 INFO:teuthology.orchestra.run.smithi169.stdout:Transaction Summary 2023-12-16T21:31:30.158 INFO:teuthology.orchestra.run.smithi169.stdout:======================================================================================= 2023-12-16T21:31:30.158 INFO:teuthology.orchestra.run.smithi169.stdout:Install 6 Packages 2023-12-16T21:31:30.158 INFO:teuthology.orchestra.run.smithi169.stdout: 2023-12-16T21:31:30.159 INFO:teuthology.orchestra.run.smithi169.stdout:Total download size: 6.8 M 2023-12-16T21:31:30.159 INFO:teuthology.orchestra.run.smithi169.stdout:Installed size: 33 M 2023-12-16T21:31:30.159 INFO:teuthology.orchestra.run.smithi169.stdout:Downloading Packages: 2023-12-16T21:31:30.236 INFO:teuthology.orchestra.run.smithi045.stdout:Dependencies resolved. 2023-12-16T21:31:30.237 INFO:teuthology.orchestra.run.smithi045.stdout:================================================================================ 2023-12-16T21:31:30.237 INFO:teuthology.orchestra.run.smithi045.stdout: Package Arch Version Repo Size 2023-12-16T21:31:30.237 INFO:teuthology.orchestra.run.smithi045.stdout:================================================================================ 2023-12-16T21:31:30.237 INFO:teuthology.orchestra.run.smithi045.stdout:Installing: 2023-12-16T21:31:30.237 INFO:teuthology.orchestra.run.smithi045.stdout: librados-devel x86_64 2:17.2.7-125.g1f2f5510.el9 ceph 126 k 2023-12-16T21:31:30.237 INFO:teuthology.orchestra.run.smithi045.stdout: 2023-12-16T21:31:30.237 INFO:teuthology.orchestra.run.smithi045.stdout:Transaction Summary 2023-12-16T21:31:30.237 INFO:teuthology.orchestra.run.smithi045.stdout:================================================================================ 2023-12-16T21:31:30.238 INFO:teuthology.orchestra.run.smithi045.stdout:Install 1 Package 2023-12-16T21:31:30.238 INFO:teuthology.orchestra.run.smithi045.stdout: 2023-12-16T21:31:30.238 INFO:teuthology.orchestra.run.smithi045.stdout:Total download size: 126 k 2023-12-16T21:31:30.238 INFO:teuthology.orchestra.run.smithi045.stdout:Installed size: 448 k 2023-12-16T21:31:30.238 INFO:teuthology.orchestra.run.smithi045.stdout:Downloading Packages: 2023-12-16T21:31:30.357 INFO:teuthology.orchestra.run.smithi169.stdout:(1/6): python3-jinja2-2.11.3-4.el9.noarch.rpm 1.5 MB/s | 228 kB 00:00 2023-12-16T21:31:30.451 INFO:teuthology.orchestra.run.smithi045.stdout:librados-devel-17.2.7-125.g1f2f5510.el9.x86_64. 593 kB/s | 126 kB 00:00 2023-12-16T21:31:30.451 INFO:teuthology.orchestra.run.smithi045.stdout:-------------------------------------------------------------------------------- 2023-12-16T21:31:30.451 INFO:teuthology.orchestra.run.smithi045.stdout:Total 589 kB/s | 126 kB 00:00 2023-12-16T21:31:30.452 INFO:teuthology.orchestra.run.smithi045.stdout:Running transaction check 2023-12-16T21:31:30.458 INFO:teuthology.orchestra.run.smithi045.stdout:Transaction check succeeded. 2023-12-16T21:31:30.458 INFO:teuthology.orchestra.run.smithi045.stdout:Running transaction test 2023-12-16T21:31:30.482 INFO:teuthology.orchestra.run.smithi169.stdout:(2/6): python3-babel-2.9.1-2.el9.noarch.rpm 21 MB/s | 5.8 MB 00:00 2023-12-16T21:31:30.515 INFO:teuthology.orchestra.run.smithi045.stdout:Transaction test succeeded. 2023-12-16T21:31:30.516 INFO:teuthology.orchestra.run.smithi045.stdout:Running transaction 2023-12-16T21:31:30.566 INFO:teuthology.orchestra.run.smithi169.stdout:(3/6): python3-natsort-7.1.1-5.el9.noarch.rpm 695 kB/s | 58 kB 00:00 2023-12-16T21:31:30.591 INFO:teuthology.orchestra.run.smithi169.stdout:(4/6): python3-asyncssh-2.13.2-3.el9.noarch.rpm 2.3 MB/s | 548 kB 00:00 2023-12-16T21:31:30.616 INFO:teuthology.orchestra.run.smithi169.stdout:(5/6): ceph-mgr-cephadm-17.2.7-125.g1f2f5510.el 300 kB/s | 121 kB 00:00 2023-12-16T21:31:30.642 INFO:teuthology.orchestra.run.smithi169.stdout:(6/6): python3-typing-extensions-4.4.0-2.el9.no 675 kB/s | 51 kB 00:00 2023-12-16T21:31:30.644 INFO:teuthology.orchestra.run.smithi169.stdout:-------------------------------------------------------------------------------- 2023-12-16T21:31:30.644 INFO:teuthology.orchestra.run.smithi169.stdout:Total 14 MB/s | 6.8 MB 00:00 2023-12-16T21:31:30.644 INFO:teuthology.orchestra.run.smithi169.stdout:Running transaction check 2023-12-16T21:31:30.652 INFO:teuthology.orchestra.run.smithi169.stdout:Transaction check succeeded. 2023-12-16T21:31:30.653 INFO:teuthology.orchestra.run.smithi169.stdout:Running transaction test 2023-12-16T21:31:30.663 INFO:teuthology.orchestra.run.smithi045.stdout: Preparing : 1/1 2023-12-16T21:31:30.740 INFO:teuthology.orchestra.run.smithi169.stdout:Transaction test succeeded. 2023-12-16T21:31:30.740 INFO:teuthology.orchestra.run.smithi169.stdout:Running transaction 2023-12-16T21:31:30.759 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : librados-devel-2:17.2.7-125.g1f2f5510.el9.x86_64 1/1 2023-12-16T21:31:31.036 INFO:teuthology.orchestra.run.smithi169.stdout: Preparing : 1/1 2023-12-16T21:31:31.186 INFO:teuthology.orchestra.run.smithi045.stdout: Running scriptlet: librados-devel-2:17.2.7-125.g1f2f5510.el9.x86_64 1/1 2023-12-16T21:31:31.206 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : python3-typing-extensions-4.4.0-2.el9.noarch 1/6 2023-12-16T21:31:31.259 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : python3-asyncssh-2.13.2-3.el9.noarch 2/6 2023-12-16T21:31:31.454 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : librados-devel-2:17.2.7-125.g1f2f5510.el9.x86_64 1/1 2023-12-16T21:31:31.455 INFO:teuthology.orchestra.run.smithi045.stdout: 2023-12-16T21:31:31.455 INFO:teuthology.orchestra.run.smithi045.stdout:Installed: 2023-12-16T21:31:31.455 INFO:teuthology.orchestra.run.smithi045.stdout: librados-devel-2:17.2.7-125.g1f2f5510.el9.x86_64 2023-12-16T21:31:31.455 INFO:teuthology.orchestra.run.smithi045.stdout: 2023-12-16T21:31:31.455 INFO:teuthology.orchestra.run.smithi045.stdout:Complete! 2023-12-16T21:31:31.678 DEBUG:teuthology.orchestra.run.smithi045:> sudo yum -y install libcephfs2 2023-12-16T21:31:31.781 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : python3-natsort-7.1.1-5.el9.noarch 3/6 2023-12-16T21:31:31.872 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : python3-babel-2.9.1-2.el9.noarch 4/6 2023-12-16T21:31:31.935 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : python3-jinja2-2.11.3-4.el9.noarch 5/6 2023-12-16T21:31:31.964 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : ceph-mgr-cephadm-2:17.2.7-125.g1f2f5510.el9.noarch 6/6 2023-12-16T21:31:32.261 INFO:teuthology.orchestra.run.smithi045.stdout:Last metadata expiration check: 0:01:29 ago on Sat 16 Dec 2023 09:30:03 PM UTC. 2023-12-16T21:31:32.358 INFO:teuthology.orchestra.run.smithi045.stdout:Package libcephfs2-2:17.2.7-125.g1f2f5510.el9.x86_64 is already installed. 2023-12-16T21:31:32.413 INFO:teuthology.orchestra.run.smithi045.stdout:Dependencies resolved. 2023-12-16T21:31:32.413 INFO:teuthology.orchestra.run.smithi045.stdout:Nothing to do. 2023-12-16T21:31:32.414 INFO:teuthology.orchestra.run.smithi045.stdout:Complete! 2023-12-16T21:31:32.486 DEBUG:teuthology.orchestra.run.smithi045:> sudo yum -y install libcephfs-devel 2023-12-16T21:31:32.822 INFO:teuthology.orchestra.run.smithi169.stdout: Running scriptlet: ceph-mgr-cephadm-2:17.2.7-125.g1f2f5510.el9.noarch 6/6 2023-12-16T21:31:32.822 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : ceph-mgr-cephadm-2:17.2.7-125.g1f2f5510.el9.noarch 1/6 2023-12-16T21:31:32.822 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : python3-babel-2.9.1-2.el9.noarch 2/6 2023-12-16T21:31:32.822 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : python3-jinja2-2.11.3-4.el9.noarch 3/6 2023-12-16T21:31:32.823 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : python3-asyncssh-2.13.2-3.el9.noarch 4/6 2023-12-16T21:31:32.823 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : python3-natsort-7.1.1-5.el9.noarch 5/6 2023-12-16T21:31:33.067 INFO:teuthology.orchestra.run.smithi045.stdout:Last metadata expiration check: 0:01:30 ago on Sat 16 Dec 2023 09:30:03 PM UTC. 2023-12-16T21:31:33.130 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : python3-typing-extensions-4.4.0-2.el9.noarch 6/6 2023-12-16T21:31:33.130 INFO:teuthology.orchestra.run.smithi169.stdout: 2023-12-16T21:31:33.130 INFO:teuthology.orchestra.run.smithi169.stdout:Installed: 2023-12-16T21:31:33.130 INFO:teuthology.orchestra.run.smithi169.stdout: ceph-mgr-cephadm-2:17.2.7-125.g1f2f5510.el9.noarch 2023-12-16T21:31:33.130 INFO:teuthology.orchestra.run.smithi169.stdout: python3-asyncssh-2.13.2-3.el9.noarch 2023-12-16T21:31:33.130 INFO:teuthology.orchestra.run.smithi169.stdout: python3-babel-2.9.1-2.el9.noarch 2023-12-16T21:31:33.130 INFO:teuthology.orchestra.run.smithi169.stdout: python3-jinja2-2.11.3-4.el9.noarch 2023-12-16T21:31:33.130 INFO:teuthology.orchestra.run.smithi169.stdout: python3-natsort-7.1.1-5.el9.noarch 2023-12-16T21:31:33.130 INFO:teuthology.orchestra.run.smithi169.stdout: python3-typing-extensions-4.4.0-2.el9.noarch 2023-12-16T21:31:33.130 INFO:teuthology.orchestra.run.smithi169.stdout: 2023-12-16T21:31:33.130 INFO:teuthology.orchestra.run.smithi169.stdout:Complete! 2023-12-16T21:31:33.218 INFO:teuthology.orchestra.run.smithi045.stdout:Dependencies resolved. 2023-12-16T21:31:33.219 INFO:teuthology.orchestra.run.smithi045.stdout:================================================================================ 2023-12-16T21:31:33.219 INFO:teuthology.orchestra.run.smithi045.stdout: Package Arch Version Repo Size 2023-12-16T21:31:33.219 INFO:teuthology.orchestra.run.smithi045.stdout:================================================================================ 2023-12-16T21:31:33.219 INFO:teuthology.orchestra.run.smithi045.stdout:Installing: 2023-12-16T21:31:33.220 INFO:teuthology.orchestra.run.smithi045.stdout: libcephfs-devel x86_64 2:17.2.7-125.g1f2f5510.el9 ceph 25 k 2023-12-16T21:31:33.220 INFO:teuthology.orchestra.run.smithi045.stdout: 2023-12-16T21:31:33.220 INFO:teuthology.orchestra.run.smithi045.stdout:Transaction Summary 2023-12-16T21:31:33.220 INFO:teuthology.orchestra.run.smithi045.stdout:================================================================================ 2023-12-16T21:31:33.220 INFO:teuthology.orchestra.run.smithi045.stdout:Install 1 Package 2023-12-16T21:31:33.220 INFO:teuthology.orchestra.run.smithi045.stdout: 2023-12-16T21:31:33.220 INFO:teuthology.orchestra.run.smithi045.stdout:Total download size: 25 k 2023-12-16T21:31:33.220 INFO:teuthology.orchestra.run.smithi045.stdout:Installed size: 110 k 2023-12-16T21:31:33.220 INFO:teuthology.orchestra.run.smithi045.stdout:Downloading Packages: 2023-12-16T21:31:33.389 DEBUG:teuthology.orchestra.run.smithi169:> sudo yum -y install ceph-fuse 2023-12-16T21:31:33.513 INFO:teuthology.orchestra.run.smithi045.stdout:libcephfs-devel-17.2.7-125.g1f2f5510.el9.x86_64 85 kB/s | 25 kB 00:00 2023-12-16T21:31:33.514 INFO:teuthology.orchestra.run.smithi045.stdout:-------------------------------------------------------------------------------- 2023-12-16T21:31:33.514 INFO:teuthology.orchestra.run.smithi045.stdout:Total 85 kB/s | 25 kB 00:00 2023-12-16T21:31:33.514 INFO:teuthology.orchestra.run.smithi045.stdout:Running transaction check 2023-12-16T21:31:33.518 INFO:teuthology.orchestra.run.smithi045.stdout:Transaction check succeeded. 2023-12-16T21:31:33.518 INFO:teuthology.orchestra.run.smithi045.stdout:Running transaction test 2023-12-16T21:31:33.519 INFO:teuthology.orchestra.run.smithi045.stdout:Transaction test succeeded. 2023-12-16T21:31:33.520 INFO:teuthology.orchestra.run.smithi045.stdout:Running transaction 2023-12-16T21:31:33.605 INFO:teuthology.orchestra.run.smithi045.stdout: Preparing : 1/1 2023-12-16T21:31:33.705 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : libcephfs-devel-2:17.2.7-125.g1f2f5510.el9.x86_64 1/1 2023-12-16T21:31:33.943 INFO:teuthology.orchestra.run.smithi169.stdout:Last metadata expiration check: 0:01:20 ago on Sat 16 Dec 2023 09:30:13 PM UTC. 2023-12-16T21:31:34.075 INFO:teuthology.orchestra.run.smithi045.stdout: Running scriptlet: libcephfs-devel-2:17.2.7-125.g1f2f5510.el9.x86_64 1/1 2023-12-16T21:31:34.086 INFO:teuthology.orchestra.run.smithi169.stdout:Dependencies resolved. 2023-12-16T21:31:34.087 INFO:teuthology.orchestra.run.smithi169.stdout:================================================================================ 2023-12-16T21:31:34.087 INFO:teuthology.orchestra.run.smithi169.stdout: Package Arch Version Repository Size 2023-12-16T21:31:34.088 INFO:teuthology.orchestra.run.smithi169.stdout:================================================================================ 2023-12-16T21:31:34.088 INFO:teuthology.orchestra.run.smithi169.stdout:Installing: 2023-12-16T21:31:34.088 INFO:teuthology.orchestra.run.smithi169.stdout: ceph-fuse x86_64 2:17.2.7-125.g1f2f5510.el9 ceph 810 k 2023-12-16T21:31:34.088 INFO:teuthology.orchestra.run.smithi169.stdout:Installing dependencies: 2023-12-16T21:31:34.088 INFO:teuthology.orchestra.run.smithi169.stdout: fuse x86_64 2.9.9-15.el9 CentOS-BaseOS 79 k 2023-12-16T21:31:34.088 INFO:teuthology.orchestra.run.smithi169.stdout: 2023-12-16T21:31:34.088 INFO:teuthology.orchestra.run.smithi169.stdout:Transaction Summary 2023-12-16T21:31:34.088 INFO:teuthology.orchestra.run.smithi169.stdout:================================================================================ 2023-12-16T21:31:34.088 INFO:teuthology.orchestra.run.smithi169.stdout:Install 2 Packages 2023-12-16T21:31:34.088 INFO:teuthology.orchestra.run.smithi169.stdout: 2023-12-16T21:31:34.089 INFO:teuthology.orchestra.run.smithi169.stdout:Total download size: 889 k 2023-12-16T21:31:34.089 INFO:teuthology.orchestra.run.smithi169.stdout:Installed size: 2.6 M 2023-12-16T21:31:34.089 INFO:teuthology.orchestra.run.smithi169.stdout:Downloading Packages: 2023-12-16T21:31:34.210 INFO:teuthology.orchestra.run.smithi169.stdout:(1/2): fuse-2.9.9-15.el9.x86_64.rpm 647 kB/s | 79 kB 00:00 2023-12-16T21:31:34.353 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : libcephfs-devel-2:17.2.7-125.g1f2f5510.el9.x86_64 1/1 2023-12-16T21:31:34.353 INFO:teuthology.orchestra.run.smithi045.stdout: 2023-12-16T21:31:34.353 INFO:teuthology.orchestra.run.smithi045.stdout:Installed: 2023-12-16T21:31:34.353 INFO:teuthology.orchestra.run.smithi045.stdout: libcephfs-devel-2:17.2.7-125.g1f2f5510.el9.x86_64 2023-12-16T21:31:34.354 INFO:teuthology.orchestra.run.smithi045.stdout: 2023-12-16T21:31:34.354 INFO:teuthology.orchestra.run.smithi045.stdout:Complete! 2023-12-16T21:31:34.444 INFO:teuthology.orchestra.run.smithi169.stdout:(2/2): ceph-fuse-17.2.7-125.g1f2f5510.el9.x86_6 2.2 MB/s | 810 kB 00:00 2023-12-16T21:31:34.445 INFO:teuthology.orchestra.run.smithi169.stdout:-------------------------------------------------------------------------------- 2023-12-16T21:31:34.445 INFO:teuthology.orchestra.run.smithi169.stdout:Total 2.4 MB/s | 889 kB 00:00 2023-12-16T21:31:34.445 INFO:teuthology.orchestra.run.smithi169.stdout:Running transaction check 2023-12-16T21:31:34.466 INFO:teuthology.orchestra.run.smithi169.stdout:Transaction check succeeded. 2023-12-16T21:31:34.466 INFO:teuthology.orchestra.run.smithi169.stdout:Running transaction test 2023-12-16T21:31:34.540 INFO:teuthology.orchestra.run.smithi169.stdout:Transaction test succeeded. 2023-12-16T21:31:34.540 INFO:teuthology.orchestra.run.smithi169.stdout:Running transaction 2023-12-16T21:31:34.563 DEBUG:teuthology.orchestra.run.smithi045:> sudo yum -y install librados2 2023-12-16T21:31:34.727 INFO:teuthology.orchestra.run.smithi169.stdout: Preparing : 1/1 2023-12-16T21:31:34.863 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : fuse-2.9.9-15.el9.x86_64 1/2 2023-12-16T21:31:34.926 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : ceph-fuse-2:17.2.7-125.g1f2f5510.el9.x86_64 2/2 2023-12-16T21:31:35.144 INFO:teuthology.orchestra.run.smithi045.stdout:Last metadata expiration check: 0:01:32 ago on Sat 16 Dec 2023 09:30:03 PM UTC. 2023-12-16T21:31:35.242 INFO:teuthology.orchestra.run.smithi045.stdout:Package librados2-2:17.2.7-125.g1f2f5510.el9.x86_64 is already installed. 2023-12-16T21:31:35.295 INFO:teuthology.orchestra.run.smithi045.stdout:Dependencies resolved. 2023-12-16T21:31:35.296 INFO:teuthology.orchestra.run.smithi045.stdout:Nothing to do. 2023-12-16T21:31:35.296 INFO:teuthology.orchestra.run.smithi045.stdout:Complete! 2023-12-16T21:31:35.376 DEBUG:teuthology.orchestra.run.smithi045:> sudo yum -y install librbd1 2023-12-16T21:31:35.621 INFO:teuthology.orchestra.run.smithi169.stdout: Running scriptlet: ceph-fuse-2:17.2.7-125.g1f2f5510.el9.x86_64 2/2 2023-12-16T21:31:35.622 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : ceph-fuse-2:17.2.7-125.g1f2f5510.el9.x86_64 1/2 2023-12-16T21:31:35.948 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : fuse-2.9.9-15.el9.x86_64 2/2 2023-12-16T21:31:35.948 INFO:teuthology.orchestra.run.smithi169.stdout: 2023-12-16T21:31:35.948 INFO:teuthology.orchestra.run.smithi169.stdout:Installed: 2023-12-16T21:31:35.948 INFO:teuthology.orchestra.run.smithi169.stdout: ceph-fuse-2:17.2.7-125.g1f2f5510.el9.x86_64 fuse-2.9.9-15.el9.x86_64 2023-12-16T21:31:35.948 INFO:teuthology.orchestra.run.smithi169.stdout: 2023-12-16T21:31:35.948 INFO:teuthology.orchestra.run.smithi169.stdout:Complete! 2023-12-16T21:31:35.964 INFO:teuthology.orchestra.run.smithi045.stdout:Last metadata expiration check: 0:01:32 ago on Sat 16 Dec 2023 09:30:03 PM UTC. 2023-12-16T21:31:36.061 INFO:teuthology.orchestra.run.smithi045.stdout:Package librbd1-2:17.2.7-125.g1f2f5510.el9.x86_64 is already installed. 2023-12-16T21:31:36.116 INFO:teuthology.orchestra.run.smithi045.stdout:Dependencies resolved. 2023-12-16T21:31:36.117 INFO:teuthology.orchestra.run.smithi045.stdout:Nothing to do. 2023-12-16T21:31:36.117 INFO:teuthology.orchestra.run.smithi045.stdout:Complete! 2023-12-16T21:31:36.202 DEBUG:teuthology.orchestra.run.smithi045:> sudo yum -y install python3-rados 2023-12-16T21:31:36.213 DEBUG:teuthology.orchestra.run.smithi169:> sudo yum -y install librados-devel 2023-12-16T21:31:36.782 INFO:teuthology.orchestra.run.smithi045.stdout:Last metadata expiration check: 0:01:33 ago on Sat 16 Dec 2023 09:30:03 PM UTC. 2023-12-16T21:31:36.784 INFO:teuthology.orchestra.run.smithi169.stdout:Last metadata expiration check: 0:01:23 ago on Sat 16 Dec 2023 09:30:13 PM UTC. 2023-12-16T21:31:36.878 INFO:teuthology.orchestra.run.smithi045.stdout:Package python3-rados-2:17.2.7-125.g1f2f5510.el9.x86_64 is already installed. 2023-12-16T21:31:36.929 INFO:teuthology.orchestra.run.smithi169.stdout:Dependencies resolved. 2023-12-16T21:31:36.930 INFO:teuthology.orchestra.run.smithi169.stdout:================================================================================ 2023-12-16T21:31:36.930 INFO:teuthology.orchestra.run.smithi169.stdout: Package Arch Version Repo Size 2023-12-16T21:31:36.930 INFO:teuthology.orchestra.run.smithi169.stdout:================================================================================ 2023-12-16T21:31:36.930 INFO:teuthology.orchestra.run.smithi169.stdout:Installing: 2023-12-16T21:31:36.930 INFO:teuthology.orchestra.run.smithi169.stdout: librados-devel x86_64 2:17.2.7-125.g1f2f5510.el9 ceph 126 k 2023-12-16T21:31:36.930 INFO:teuthology.orchestra.run.smithi169.stdout: 2023-12-16T21:31:36.930 INFO:teuthology.orchestra.run.smithi169.stdout:Transaction Summary 2023-12-16T21:31:36.930 INFO:teuthology.orchestra.run.smithi169.stdout:================================================================================ 2023-12-16T21:31:36.930 INFO:teuthology.orchestra.run.smithi169.stdout:Install 1 Package 2023-12-16T21:31:36.931 INFO:teuthology.orchestra.run.smithi169.stdout: 2023-12-16T21:31:36.931 INFO:teuthology.orchestra.run.smithi169.stdout:Total download size: 126 k 2023-12-16T21:31:36.931 INFO:teuthology.orchestra.run.smithi169.stdout:Installed size: 448 k 2023-12-16T21:31:36.932 INFO:teuthology.orchestra.run.smithi169.stdout:Downloading Packages: 2023-12-16T21:31:36.933 INFO:teuthology.orchestra.run.smithi045.stdout:Dependencies resolved. 2023-12-16T21:31:36.933 INFO:teuthology.orchestra.run.smithi045.stdout:Nothing to do. 2023-12-16T21:31:36.933 INFO:teuthology.orchestra.run.smithi045.stdout:Complete! 2023-12-16T21:31:37.009 DEBUG:teuthology.orchestra.run.smithi045:> sudo yum -y install python3-rgw 2023-12-16T21:31:37.165 INFO:teuthology.orchestra.run.smithi169.stdout:librados-devel-17.2.7-125.g1f2f5510.el9.x86_64. 540 kB/s | 126 kB 00:00 2023-12-16T21:31:37.165 INFO:teuthology.orchestra.run.smithi169.stdout:-------------------------------------------------------------------------------- 2023-12-16T21:31:37.165 INFO:teuthology.orchestra.run.smithi169.stdout:Total 536 kB/s | 126 kB 00:00 2023-12-16T21:31:37.166 INFO:teuthology.orchestra.run.smithi169.stdout:Running transaction check 2023-12-16T21:31:37.171 INFO:teuthology.orchestra.run.smithi169.stdout:Transaction check succeeded. 2023-12-16T21:31:37.171 INFO:teuthology.orchestra.run.smithi169.stdout:Running transaction test 2023-12-16T21:31:37.226 INFO:teuthology.orchestra.run.smithi169.stdout:Transaction test succeeded. 2023-12-16T21:31:37.226 INFO:teuthology.orchestra.run.smithi169.stdout:Running transaction 2023-12-16T21:31:37.382 INFO:teuthology.orchestra.run.smithi169.stdout: Preparing : 1/1 2023-12-16T21:31:37.487 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : librados-devel-2:17.2.7-125.g1f2f5510.el9.x86_64 1/1 2023-12-16T21:31:37.592 INFO:teuthology.orchestra.run.smithi045.stdout:Last metadata expiration check: 0:01:34 ago on Sat 16 Dec 2023 09:30:03 PM UTC. 2023-12-16T21:31:37.689 INFO:teuthology.orchestra.run.smithi045.stdout:Package python3-rgw-2:17.2.7-125.g1f2f5510.el9.x86_64 is already installed. 2023-12-16T21:31:37.743 INFO:teuthology.orchestra.run.smithi045.stdout:Dependencies resolved. 2023-12-16T21:31:37.744 INFO:teuthology.orchestra.run.smithi045.stdout:Nothing to do. 2023-12-16T21:31:37.744 INFO:teuthology.orchestra.run.smithi045.stdout:Complete! 2023-12-16T21:31:37.826 DEBUG:teuthology.orchestra.run.smithi045:> sudo yum -y install python3-cephfs 2023-12-16T21:31:37.972 INFO:teuthology.orchestra.run.smithi169.stdout: Running scriptlet: librados-devel-2:17.2.7-125.g1f2f5510.el9.x86_64 1/1 2023-12-16T21:31:38.273 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : librados-devel-2:17.2.7-125.g1f2f5510.el9.x86_64 1/1 2023-12-16T21:31:38.273 INFO:teuthology.orchestra.run.smithi169.stdout: 2023-12-16T21:31:38.273 INFO:teuthology.orchestra.run.smithi169.stdout:Installed: 2023-12-16T21:31:38.273 INFO:teuthology.orchestra.run.smithi169.stdout: librados-devel-2:17.2.7-125.g1f2f5510.el9.x86_64 2023-12-16T21:31:38.273 INFO:teuthology.orchestra.run.smithi169.stdout: 2023-12-16T21:31:38.273 INFO:teuthology.orchestra.run.smithi169.stdout:Complete! 2023-12-16T21:31:38.411 INFO:teuthology.orchestra.run.smithi045.stdout:Last metadata expiration check: 0:01:35 ago on Sat 16 Dec 2023 09:30:03 PM UTC. 2023-12-16T21:31:38.508 INFO:teuthology.orchestra.run.smithi045.stdout:Package python3-cephfs-2:17.2.7-125.g1f2f5510.el9.x86_64 is already installed. 2023-12-16T21:31:38.542 DEBUG:teuthology.orchestra.run.smithi169:> sudo yum -y install libcephfs2 2023-12-16T21:31:38.562 INFO:teuthology.orchestra.run.smithi045.stdout:Dependencies resolved. 2023-12-16T21:31:38.564 INFO:teuthology.orchestra.run.smithi045.stdout:Nothing to do. 2023-12-16T21:31:38.564 INFO:teuthology.orchestra.run.smithi045.stdout:Complete! 2023-12-16T21:31:38.673 DEBUG:teuthology.orchestra.run.smithi045:> sudo yum -y install python3-rbd 2023-12-16T21:31:39.109 INFO:teuthology.orchestra.run.smithi169.stdout:Last metadata expiration check: 0:01:26 ago on Sat 16 Dec 2023 09:30:13 PM UTC. 2023-12-16T21:31:39.205 INFO:teuthology.orchestra.run.smithi169.stdout:Package libcephfs2-2:17.2.7-125.g1f2f5510.el9.x86_64 is already installed. 2023-12-16T21:31:39.251 INFO:teuthology.orchestra.run.smithi045.stdout:Last metadata expiration check: 0:01:36 ago on Sat 16 Dec 2023 09:30:03 PM UTC. 2023-12-16T21:31:39.256 INFO:teuthology.orchestra.run.smithi169.stdout:Dependencies resolved. 2023-12-16T21:31:39.257 INFO:teuthology.orchestra.run.smithi169.stdout:Nothing to do. 2023-12-16T21:31:39.257 INFO:teuthology.orchestra.run.smithi169.stdout:Complete! 2023-12-16T21:31:39.333 DEBUG:teuthology.orchestra.run.smithi169:> sudo yum -y install libcephfs-devel 2023-12-16T21:31:39.349 INFO:teuthology.orchestra.run.smithi045.stdout:Package python3-rbd-2:17.2.7-125.g1f2f5510.el9.x86_64 is already installed. 2023-12-16T21:31:39.403 INFO:teuthology.orchestra.run.smithi045.stdout:Dependencies resolved. 2023-12-16T21:31:39.404 INFO:teuthology.orchestra.run.smithi045.stdout:Nothing to do. 2023-12-16T21:31:39.404 INFO:teuthology.orchestra.run.smithi045.stdout:Complete! 2023-12-16T21:31:39.560 DEBUG:teuthology.orchestra.run.smithi045:> sudo yum -y install rbd-fuse 2023-12-16T21:31:39.886 INFO:teuthology.orchestra.run.smithi169.stdout:Last metadata expiration check: 0:01:26 ago on Sat 16 Dec 2023 09:30:13 PM UTC. 2023-12-16T21:31:40.032 INFO:teuthology.orchestra.run.smithi169.stdout:Dependencies resolved. 2023-12-16T21:31:40.033 INFO:teuthology.orchestra.run.smithi169.stdout:================================================================================ 2023-12-16T21:31:40.033 INFO:teuthology.orchestra.run.smithi169.stdout: Package Arch Version Repo Size 2023-12-16T21:31:40.033 INFO:teuthology.orchestra.run.smithi169.stdout:================================================================================ 2023-12-16T21:31:40.034 INFO:teuthology.orchestra.run.smithi169.stdout:Installing: 2023-12-16T21:31:40.034 INFO:teuthology.orchestra.run.smithi169.stdout: libcephfs-devel x86_64 2:17.2.7-125.g1f2f5510.el9 ceph 25 k 2023-12-16T21:31:40.034 INFO:teuthology.orchestra.run.smithi169.stdout: 2023-12-16T21:31:40.034 INFO:teuthology.orchestra.run.smithi169.stdout:Transaction Summary 2023-12-16T21:31:40.034 INFO:teuthology.orchestra.run.smithi169.stdout:================================================================================ 2023-12-16T21:31:40.034 INFO:teuthology.orchestra.run.smithi169.stdout:Install 1 Package 2023-12-16T21:31:40.034 INFO:teuthology.orchestra.run.smithi169.stdout: 2023-12-16T21:31:40.034 INFO:teuthology.orchestra.run.smithi169.stdout:Total download size: 25 k 2023-12-16T21:31:40.034 INFO:teuthology.orchestra.run.smithi169.stdout:Installed size: 110 k 2023-12-16T21:31:40.034 INFO:teuthology.orchestra.run.smithi169.stdout:Downloading Packages: 2023-12-16T21:31:40.146 INFO:teuthology.orchestra.run.smithi045.stdout:Last metadata expiration check: 0:01:37 ago on Sat 16 Dec 2023 09:30:03 PM UTC. 2023-12-16T21:31:40.292 INFO:teuthology.orchestra.run.smithi169.stdout:libcephfs-devel-17.2.7-125.g1f2f5510.el9.x86_64 97 kB/s | 25 kB 00:00 2023-12-16T21:31:40.293 INFO:teuthology.orchestra.run.smithi169.stdout:-------------------------------------------------------------------------------- 2023-12-16T21:31:40.293 INFO:teuthology.orchestra.run.smithi169.stdout:Total 96 kB/s | 25 kB 00:00 2023-12-16T21:31:40.293 INFO:teuthology.orchestra.run.smithi169.stdout:Running transaction check 2023-12-16T21:31:40.296 INFO:teuthology.orchestra.run.smithi169.stdout:Transaction check succeeded. 2023-12-16T21:31:40.297 INFO:teuthology.orchestra.run.smithi169.stdout:Running transaction test 2023-12-16T21:31:40.297 INFO:teuthology.orchestra.run.smithi045.stdout:Dependencies resolved. 2023-12-16T21:31:40.297 INFO:teuthology.orchestra.run.smithi045.stdout:================================================================================ 2023-12-16T21:31:40.297 INFO:teuthology.orchestra.run.smithi045.stdout: Package Architecture Version Repository Size 2023-12-16T21:31:40.297 INFO:teuthology.orchestra.run.smithi045.stdout:================================================================================ 2023-12-16T21:31:40.297 INFO:teuthology.orchestra.run.smithi045.stdout:Installing: 2023-12-16T21:31:40.298 INFO:teuthology.orchestra.run.smithi045.stdout: rbd-fuse x86_64 2:17.2.7-125.g1f2f5510.el9 ceph 81 k 2023-12-16T21:31:40.298 INFO:teuthology.orchestra.run.smithi045.stdout: 2023-12-16T21:31:40.298 INFO:teuthology.orchestra.run.smithi045.stdout:Transaction Summary 2023-12-16T21:31:40.298 INFO:teuthology.orchestra.run.smithi045.stdout:================================================================================ 2023-12-16T21:31:40.298 INFO:teuthology.orchestra.run.smithi045.stdout:Install 1 Package 2023-12-16T21:31:40.298 INFO:teuthology.orchestra.run.smithi045.stdout: 2023-12-16T21:31:40.299 INFO:teuthology.orchestra.run.smithi169.stdout:Transaction test succeeded. 2023-12-16T21:31:40.299 INFO:teuthology.orchestra.run.smithi169.stdout:Running transaction 2023-12-16T21:31:40.299 INFO:teuthology.orchestra.run.smithi045.stdout:Total download size: 81 k 2023-12-16T21:31:40.299 INFO:teuthology.orchestra.run.smithi045.stdout:Installed size: 222 k 2023-12-16T21:31:40.299 INFO:teuthology.orchestra.run.smithi045.stdout:Downloading Packages: 2023-12-16T21:31:40.385 INFO:teuthology.orchestra.run.smithi169.stdout: Preparing : 1/1 2023-12-16T21:31:40.481 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : libcephfs-devel-2:17.2.7-125.g1f2f5510.el9.x86_64 1/1 2023-12-16T21:31:40.626 INFO:teuthology.orchestra.run.smithi045.stdout:rbd-fuse-17.2.7-125.g1f2f5510.el9.x86_64.rpm 249 kB/s | 81 kB 00:00 2023-12-16T21:31:40.626 INFO:teuthology.orchestra.run.smithi045.stdout:-------------------------------------------------------------------------------- 2023-12-16T21:31:40.626 INFO:teuthology.orchestra.run.smithi045.stdout:Total 248 kB/s | 81 kB 00:00 2023-12-16T21:31:40.626 INFO:teuthology.orchestra.run.smithi045.stdout:Running transaction check 2023-12-16T21:31:40.634 INFO:teuthology.orchestra.run.smithi045.stdout:Transaction check succeeded. 2023-12-16T21:31:40.634 INFO:teuthology.orchestra.run.smithi045.stdout:Running transaction test 2023-12-16T21:31:40.693 INFO:teuthology.orchestra.run.smithi045.stdout:Transaction test succeeded. 2023-12-16T21:31:40.693 INFO:teuthology.orchestra.run.smithi045.stdout:Running transaction 2023-12-16T21:31:40.831 INFO:teuthology.orchestra.run.smithi045.stdout: Preparing : 1/1 2023-12-16T21:31:40.831 INFO:teuthology.orchestra.run.smithi169.stdout: Running scriptlet: libcephfs-devel-2:17.2.7-125.g1f2f5510.el9.x86_64 1/1 2023-12-16T21:31:40.934 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : rbd-fuse-2:17.2.7-125.g1f2f5510.el9.x86_64 1/1 2023-12-16T21:31:41.096 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : libcephfs-devel-2:17.2.7-125.g1f2f5510.el9.x86_64 1/1 2023-12-16T21:31:41.097 INFO:teuthology.orchestra.run.smithi169.stdout: 2023-12-16T21:31:41.097 INFO:teuthology.orchestra.run.smithi169.stdout:Installed: 2023-12-16T21:31:41.097 INFO:teuthology.orchestra.run.smithi169.stdout: libcephfs-devel-2:17.2.7-125.g1f2f5510.el9.x86_64 2023-12-16T21:31:41.097 INFO:teuthology.orchestra.run.smithi169.stdout: 2023-12-16T21:31:41.097 INFO:teuthology.orchestra.run.smithi169.stdout:Complete! 2023-12-16T21:31:41.324 DEBUG:teuthology.orchestra.run.smithi169:> sudo yum -y install librados2 2023-12-16T21:31:41.344 INFO:teuthology.orchestra.run.smithi045.stdout: Running scriptlet: rbd-fuse-2:17.2.7-125.g1f2f5510.el9.x86_64 1/1 2023-12-16T21:31:41.668 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : rbd-fuse-2:17.2.7-125.g1f2f5510.el9.x86_64 1/1 2023-12-16T21:31:41.669 INFO:teuthology.orchestra.run.smithi045.stdout: 2023-12-16T21:31:41.669 INFO:teuthology.orchestra.run.smithi045.stdout:Installed: 2023-12-16T21:31:41.669 INFO:teuthology.orchestra.run.smithi045.stdout: rbd-fuse-2:17.2.7-125.g1f2f5510.el9.x86_64 2023-12-16T21:31:41.669 INFO:teuthology.orchestra.run.smithi045.stdout: 2023-12-16T21:31:41.669 INFO:teuthology.orchestra.run.smithi045.stdout:Complete! 2023-12-16T21:31:41.882 INFO:teuthology.orchestra.run.smithi169.stdout:Last metadata expiration check: 0:01:28 ago on Sat 16 Dec 2023 09:30:13 PM UTC. 2023-12-16T21:31:41.923 DEBUG:teuthology.orchestra.run.smithi045:> sudo yum -y install rbd-mirror 2023-12-16T21:31:41.977 INFO:teuthology.orchestra.run.smithi169.stdout:Package librados2-2:17.2.7-125.g1f2f5510.el9.x86_64 is already installed. 2023-12-16T21:31:42.027 INFO:teuthology.orchestra.run.smithi169.stdout:Dependencies resolved. 2023-12-16T21:31:42.028 INFO:teuthology.orchestra.run.smithi169.stdout:Nothing to do. 2023-12-16T21:31:42.028 INFO:teuthology.orchestra.run.smithi169.stdout:Complete! 2023-12-16T21:31:42.099 DEBUG:teuthology.orchestra.run.smithi169:> sudo yum -y install librbd1 2023-12-16T21:31:42.509 INFO:teuthology.orchestra.run.smithi045.stdout:Last metadata expiration check: 0:01:39 ago on Sat 16 Dec 2023 09:30:03 PM UTC. 2023-12-16T21:31:42.662 INFO:teuthology.orchestra.run.smithi169.stdout:Last metadata expiration check: 0:01:29 ago on Sat 16 Dec 2023 09:30:13 PM UTC. 2023-12-16T21:31:42.667 INFO:teuthology.orchestra.run.smithi045.stdout:Dependencies resolved. 2023-12-16T21:31:42.668 INFO:teuthology.orchestra.run.smithi045.stdout:================================================================================ 2023-12-16T21:31:42.668 INFO:teuthology.orchestra.run.smithi045.stdout: Package Arch Version Repo Size 2023-12-16T21:31:42.668 INFO:teuthology.orchestra.run.smithi045.stdout:================================================================================ 2023-12-16T21:31:42.668 INFO:teuthology.orchestra.run.smithi045.stdout:Installing: 2023-12-16T21:31:42.668 INFO:teuthology.orchestra.run.smithi045.stdout: rbd-mirror x86_64 2:17.2.7-125.g1f2f5510.el9 ceph 2.9 M 2023-12-16T21:31:42.668 INFO:teuthology.orchestra.run.smithi045.stdout: 2023-12-16T21:31:42.668 INFO:teuthology.orchestra.run.smithi045.stdout:Transaction Summary 2023-12-16T21:31:42.668 INFO:teuthology.orchestra.run.smithi045.stdout:================================================================================ 2023-12-16T21:31:42.669 INFO:teuthology.orchestra.run.smithi045.stdout:Install 1 Package 2023-12-16T21:31:42.669 INFO:teuthology.orchestra.run.smithi045.stdout: 2023-12-16T21:31:42.669 INFO:teuthology.orchestra.run.smithi045.stdout:Total download size: 2.9 M 2023-12-16T21:31:42.669 INFO:teuthology.orchestra.run.smithi045.stdout:Installed size: 12 M 2023-12-16T21:31:42.669 INFO:teuthology.orchestra.run.smithi045.stdout:Downloading Packages: 2023-12-16T21:31:42.756 INFO:teuthology.orchestra.run.smithi169.stdout:Package librbd1-2:17.2.7-125.g1f2f5510.el9.x86_64 is already installed. 2023-12-16T21:31:42.806 INFO:teuthology.orchestra.run.smithi169.stdout:Dependencies resolved. 2023-12-16T21:31:42.807 INFO:teuthology.orchestra.run.smithi169.stdout:Nothing to do. 2023-12-16T21:31:42.807 INFO:teuthology.orchestra.run.smithi169.stdout:Complete! 2023-12-16T21:31:42.885 DEBUG:teuthology.orchestra.run.smithi169:> sudo yum -y install python3-rados 2023-12-16T21:31:43.434 INFO:teuthology.orchestra.run.smithi169.stdout:Last metadata expiration check: 0:01:30 ago on Sat 16 Dec 2023 09:30:13 PM UTC. 2023-12-16T21:31:43.527 INFO:teuthology.orchestra.run.smithi169.stdout:Package python3-rados-2:17.2.7-125.g1f2f5510.el9.x86_64 is already installed. 2023-12-16T21:31:43.577 INFO:teuthology.orchestra.run.smithi169.stdout:Dependencies resolved. 2023-12-16T21:31:43.578 INFO:teuthology.orchestra.run.smithi169.stdout:Nothing to do. 2023-12-16T21:31:43.578 INFO:teuthology.orchestra.run.smithi169.stdout:Complete! 2023-12-16T21:31:43.644 DEBUG:teuthology.orchestra.run.smithi169:> sudo yum -y install python3-rgw 2023-12-16T21:31:44.203 INFO:teuthology.orchestra.run.smithi169.stdout:Last metadata expiration check: 0:01:31 ago on Sat 16 Dec 2023 09:30:13 PM UTC. 2023-12-16T21:31:44.480 INFO:teuthology.orchestra.run.smithi045.stdout:rbd-mirror-17.2.7-125.g1f2f5510.el9.x86_64.rpm 1.8 MB/s | 2.9 MB 00:01 2023-12-16T21:31:44.480 INFO:teuthology.orchestra.run.smithi045.stdout:-------------------------------------------------------------------------------- 2023-12-16T21:31:44.480 INFO:teuthology.orchestra.run.smithi045.stdout:Total 1.8 MB/s | 2.9 MB 00:01 2023-12-16T21:31:44.480 INFO:teuthology.orchestra.run.smithi045.stdout:Running transaction check 2023-12-16T21:31:44.480 INFO:teuthology.orchestra.run.smithi045.stdout:Transaction check succeeded. 2023-12-16T21:31:44.480 INFO:teuthology.orchestra.run.smithi045.stdout:Running transaction test 2023-12-16T21:31:44.480 INFO:teuthology.orchestra.run.smithi045.stdout:Transaction test succeeded. 2023-12-16T21:31:44.480 INFO:teuthology.orchestra.run.smithi045.stdout:Running transaction 2023-12-16T21:31:44.481 INFO:teuthology.orchestra.run.smithi169.stdout:Package python3-rgw-2:17.2.7-125.g1f2f5510.el9.x86_64 is already installed. 2023-12-16T21:31:44.481 INFO:teuthology.orchestra.run.smithi169.stdout:Dependencies resolved. 2023-12-16T21:31:44.481 INFO:teuthology.orchestra.run.smithi169.stdout:Nothing to do. 2023-12-16T21:31:44.481 INFO:teuthology.orchestra.run.smithi169.stdout:Complete! 2023-12-16T21:31:44.481 DEBUG:teuthology.orchestra.run.smithi169:> sudo yum -y install python3-cephfs 2023-12-16T21:31:44.799 INFO:teuthology.orchestra.run.smithi045.stdout: Preparing : 1/1 2023-12-16T21:31:44.865 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : rbd-mirror-2:17.2.7-125.g1f2f5510.el9.x86_64 1/1 2023-12-16T21:31:44.890 INFO:teuthology.orchestra.run.smithi045.stdout: Running scriptlet: rbd-mirror-2:17.2.7-125.g1f2f5510.el9.x86_64 1/1 2023-12-16T21:31:44.890 INFO:teuthology.orchestra.run.smithi045.stdout:Glob pattern passed to enable, but globs are not supported for this. 2023-12-16T21:31:44.891 INFO:teuthology.orchestra.run.smithi045.stdout:Invalid unit name "ceph-rbd-mirror@*.service" escaped as "ceph-rbd-mirror@\x2a.service". 2023-12-16T21:31:44.891 INFO:teuthology.orchestra.run.smithi045.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2023-12-16T21:31:44.891 INFO:teuthology.orchestra.run.smithi045.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2023-12-16T21:31:44.891 INFO:teuthology.orchestra.run.smithi045.stdout: 2023-12-16T21:31:45.040 INFO:teuthology.orchestra.run.smithi169.stdout:Last metadata expiration check: 0:01:32 ago on Sat 16 Dec 2023 09:30:13 PM UTC. 2023-12-16T21:31:45.134 INFO:teuthology.orchestra.run.smithi169.stdout:Package python3-cephfs-2:17.2.7-125.g1f2f5510.el9.x86_64 is already installed. 2023-12-16T21:31:45.184 INFO:teuthology.orchestra.run.smithi169.stdout:Dependencies resolved. 2023-12-16T21:31:45.185 INFO:teuthology.orchestra.run.smithi169.stdout:Nothing to do. 2023-12-16T21:31:45.185 INFO:teuthology.orchestra.run.smithi169.stdout:Complete! 2023-12-16T21:31:45.264 DEBUG:teuthology.orchestra.run.smithi169:> sudo yum -y install python3-rbd 2023-12-16T21:31:45.814 INFO:teuthology.orchestra.run.smithi169.stdout:Last metadata expiration check: 0:01:32 ago on Sat 16 Dec 2023 09:30:13 PM UTC. 2023-12-16T21:31:45.907 INFO:teuthology.orchestra.run.smithi169.stdout:Package python3-rbd-2:17.2.7-125.g1f2f5510.el9.x86_64 is already installed. 2023-12-16T21:31:45.958 INFO:teuthology.orchestra.run.smithi169.stdout:Dependencies resolved. 2023-12-16T21:31:45.959 INFO:teuthology.orchestra.run.smithi169.stdout:Nothing to do. 2023-12-16T21:31:45.959 INFO:teuthology.orchestra.run.smithi169.stdout:Complete! 2023-12-16T21:31:46.036 DEBUG:teuthology.orchestra.run.smithi169:> sudo yum -y install rbd-fuse 2023-12-16T21:31:46.043 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : rbd-mirror-2:17.2.7-125.g1f2f5510.el9.x86_64 1/1 2023-12-16T21:31:46.043 INFO:teuthology.orchestra.run.smithi045.stdout: 2023-12-16T21:31:46.043 INFO:teuthology.orchestra.run.smithi045.stdout:Installed: 2023-12-16T21:31:46.043 INFO:teuthology.orchestra.run.smithi045.stdout: rbd-mirror-2:17.2.7-125.g1f2f5510.el9.x86_64 2023-12-16T21:31:46.043 INFO:teuthology.orchestra.run.smithi045.stdout: 2023-12-16T21:31:46.043 INFO:teuthology.orchestra.run.smithi045.stdout:Complete! 2023-12-16T21:31:46.337 DEBUG:teuthology.orchestra.run.smithi045:> sudo yum -y install rbd-nbd 2023-12-16T21:31:46.597 INFO:teuthology.orchestra.run.smithi169.stdout:Last metadata expiration check: 0:01:33 ago on Sat 16 Dec 2023 09:30:13 PM UTC. 2023-12-16T21:31:46.742 INFO:teuthology.orchestra.run.smithi169.stdout:Dependencies resolved. 2023-12-16T21:31:46.742 INFO:teuthology.orchestra.run.smithi169.stdout:================================================================================ 2023-12-16T21:31:46.742 INFO:teuthology.orchestra.run.smithi169.stdout: Package Architecture Version Repository Size 2023-12-16T21:31:46.742 INFO:teuthology.orchestra.run.smithi169.stdout:================================================================================ 2023-12-16T21:31:46.742 INFO:teuthology.orchestra.run.smithi169.stdout:Installing: 2023-12-16T21:31:46.742 INFO:teuthology.orchestra.run.smithi169.stdout: rbd-fuse x86_64 2:17.2.7-125.g1f2f5510.el9 ceph 81 k 2023-12-16T21:31:46.742 INFO:teuthology.orchestra.run.smithi169.stdout: 2023-12-16T21:31:46.742 INFO:teuthology.orchestra.run.smithi169.stdout:Transaction Summary 2023-12-16T21:31:46.743 INFO:teuthology.orchestra.run.smithi169.stdout:================================================================================ 2023-12-16T21:31:46.743 INFO:teuthology.orchestra.run.smithi169.stdout:Install 1 Package 2023-12-16T21:31:46.743 INFO:teuthology.orchestra.run.smithi169.stdout: 2023-12-16T21:31:46.743 INFO:teuthology.orchestra.run.smithi169.stdout:Total download size: 81 k 2023-12-16T21:31:46.743 INFO:teuthology.orchestra.run.smithi169.stdout:Installed size: 222 k 2023-12-16T21:31:46.743 INFO:teuthology.orchestra.run.smithi169.stdout:Downloading Packages: 2023-12-16T21:31:46.928 INFO:teuthology.orchestra.run.smithi045.stdout:Last metadata expiration check: 0:01:43 ago on Sat 16 Dec 2023 09:30:03 PM UTC. 2023-12-16T21:31:46.946 INFO:teuthology.orchestra.run.smithi169.stdout:rbd-fuse-17.2.7-125.g1f2f5510.el9.x86_64.rpm 403 kB/s | 81 kB 00:00 2023-12-16T21:31:46.946 INFO:teuthology.orchestra.run.smithi169.stdout:-------------------------------------------------------------------------------- 2023-12-16T21:31:46.947 INFO:teuthology.orchestra.run.smithi169.stdout:Total 400 kB/s | 81 kB 00:00 2023-12-16T21:31:46.947 INFO:teuthology.orchestra.run.smithi169.stdout:Running transaction check 2023-12-16T21:31:46.953 INFO:teuthology.orchestra.run.smithi169.stdout:Transaction check succeeded. 2023-12-16T21:31:46.954 INFO:teuthology.orchestra.run.smithi169.stdout:Running transaction test 2023-12-16T21:31:47.009 INFO:teuthology.orchestra.run.smithi169.stdout:Transaction test succeeded. 2023-12-16T21:31:47.009 INFO:teuthology.orchestra.run.smithi169.stdout:Running transaction 2023-12-16T21:31:47.078 INFO:teuthology.orchestra.run.smithi045.stdout:Dependencies resolved. 2023-12-16T21:31:47.079 INFO:teuthology.orchestra.run.smithi045.stdout:================================================================================ 2023-12-16T21:31:47.079 INFO:teuthology.orchestra.run.smithi045.stdout: Package Architecture Version Repository Size 2023-12-16T21:31:47.079 INFO:teuthology.orchestra.run.smithi045.stdout:================================================================================ 2023-12-16T21:31:47.079 INFO:teuthology.orchestra.run.smithi045.stdout:Installing: 2023-12-16T21:31:47.079 INFO:teuthology.orchestra.run.smithi045.stdout: rbd-nbd x86_64 2:17.2.7-125.g1f2f5510.el9 ceph 162 k 2023-12-16T21:31:47.079 INFO:teuthology.orchestra.run.smithi045.stdout: 2023-12-16T21:31:47.079 INFO:teuthology.orchestra.run.smithi045.stdout:Transaction Summary 2023-12-16T21:31:47.079 INFO:teuthology.orchestra.run.smithi045.stdout:================================================================================ 2023-12-16T21:31:47.080 INFO:teuthology.orchestra.run.smithi045.stdout:Install 1 Package 2023-12-16T21:31:47.080 INFO:teuthology.orchestra.run.smithi045.stdout: 2023-12-16T21:31:47.080 INFO:teuthology.orchestra.run.smithi045.stdout:Total download size: 162 k 2023-12-16T21:31:47.080 INFO:teuthology.orchestra.run.smithi045.stdout:Installed size: 465 k 2023-12-16T21:31:47.080 INFO:teuthology.orchestra.run.smithi045.stdout:Downloading Packages: 2023-12-16T21:31:47.157 INFO:teuthology.orchestra.run.smithi169.stdout: Preparing : 1/1 2023-12-16T21:31:47.252 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : rbd-fuse-2:17.2.7-125.g1f2f5510.el9.x86_64 1/1 2023-12-16T21:31:47.319 INFO:teuthology.orchestra.run.smithi045.stdout:rbd-nbd-17.2.7-125.g1f2f5510.el9.x86_64.rpm 678 kB/s | 162 kB 00:00 2023-12-16T21:31:47.320 INFO:teuthology.orchestra.run.smithi045.stdout:-------------------------------------------------------------------------------- 2023-12-16T21:31:47.320 INFO:teuthology.orchestra.run.smithi045.stdout:Total 674 kB/s | 162 kB 00:00 2023-12-16T21:31:47.320 INFO:teuthology.orchestra.run.smithi045.stdout:Running transaction check 2023-12-16T21:31:47.329 INFO:teuthology.orchestra.run.smithi045.stdout:Transaction check succeeded. 2023-12-16T21:31:47.329 INFO:teuthology.orchestra.run.smithi045.stdout:Running transaction test 2023-12-16T21:31:47.387 INFO:teuthology.orchestra.run.smithi045.stdout:Transaction test succeeded. 2023-12-16T21:31:47.387 INFO:teuthology.orchestra.run.smithi045.stdout:Running transaction 2023-12-16T21:31:47.545 INFO:teuthology.orchestra.run.smithi045.stdout: Preparing : 1/1 2023-12-16T21:31:47.636 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : rbd-nbd-2:17.2.7-125.g1f2f5510.el9.x86_64 1/1 2023-12-16T21:31:47.655 INFO:teuthology.orchestra.run.smithi169.stdout: Running scriptlet: rbd-fuse-2:17.2.7-125.g1f2f5510.el9.x86_64 1/1 2023-12-16T21:31:47.953 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : rbd-fuse-2:17.2.7-125.g1f2f5510.el9.x86_64 1/1 2023-12-16T21:31:47.954 INFO:teuthology.orchestra.run.smithi169.stdout: 2023-12-16T21:31:47.954 INFO:teuthology.orchestra.run.smithi169.stdout:Installed: 2023-12-16T21:31:47.954 INFO:teuthology.orchestra.run.smithi169.stdout: rbd-fuse-2:17.2.7-125.g1f2f5510.el9.x86_64 2023-12-16T21:31:47.954 INFO:teuthology.orchestra.run.smithi169.stdout: 2023-12-16T21:31:47.954 INFO:teuthology.orchestra.run.smithi169.stdout:Complete! 2023-12-16T21:31:48.114 INFO:teuthology.orchestra.run.smithi045.stdout: Running scriptlet: rbd-nbd-2:17.2.7-125.g1f2f5510.el9.x86_64 1/1 2023-12-16T21:31:48.208 DEBUG:teuthology.orchestra.run.smithi169:> sudo yum -y install rbd-mirror 2023-12-16T21:31:48.443 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : rbd-nbd-2:17.2.7-125.g1f2f5510.el9.x86_64 1/1 2023-12-16T21:31:48.443 INFO:teuthology.orchestra.run.smithi045.stdout: 2023-12-16T21:31:48.444 INFO:teuthology.orchestra.run.smithi045.stdout:Installed: 2023-12-16T21:31:48.444 INFO:teuthology.orchestra.run.smithi045.stdout: rbd-nbd-2:17.2.7-125.g1f2f5510.el9.x86_64 2023-12-16T21:31:48.444 INFO:teuthology.orchestra.run.smithi045.stdout: 2023-12-16T21:31:48.444 INFO:teuthology.orchestra.run.smithi045.stdout:Complete! 2023-12-16T21:31:48.718 DEBUG:teuthology.parallel:result is None 2023-12-16T21:31:48.772 INFO:teuthology.orchestra.run.smithi169.stdout:Last metadata expiration check: 0:01:35 ago on Sat 16 Dec 2023 09:30:13 PM UTC. 2023-12-16T21:31:48.919 INFO:teuthology.orchestra.run.smithi169.stdout:Dependencies resolved. 2023-12-16T21:31:48.919 INFO:teuthology.orchestra.run.smithi169.stdout:================================================================================ 2023-12-16T21:31:48.919 INFO:teuthology.orchestra.run.smithi169.stdout: Package Arch Version Repo Size 2023-12-16T21:31:48.919 INFO:teuthology.orchestra.run.smithi169.stdout:================================================================================ 2023-12-16T21:31:48.919 INFO:teuthology.orchestra.run.smithi169.stdout:Installing: 2023-12-16T21:31:48.919 INFO:teuthology.orchestra.run.smithi169.stdout: rbd-mirror x86_64 2:17.2.7-125.g1f2f5510.el9 ceph 2.9 M 2023-12-16T21:31:48.920 INFO:teuthology.orchestra.run.smithi169.stdout: 2023-12-16T21:31:48.920 INFO:teuthology.orchestra.run.smithi169.stdout:Transaction Summary 2023-12-16T21:31:48.920 INFO:teuthology.orchestra.run.smithi169.stdout:================================================================================ 2023-12-16T21:31:48.920 INFO:teuthology.orchestra.run.smithi169.stdout:Install 1 Package 2023-12-16T21:31:48.920 INFO:teuthology.orchestra.run.smithi169.stdout: 2023-12-16T21:31:48.920 INFO:teuthology.orchestra.run.smithi169.stdout:Total download size: 2.9 M 2023-12-16T21:31:48.920 INFO:teuthology.orchestra.run.smithi169.stdout:Installed size: 12 M 2023-12-16T21:31:48.921 INFO:teuthology.orchestra.run.smithi169.stdout:Downloading Packages: 2023-12-16T21:31:49.499 INFO:teuthology.orchestra.run.smithi169.stdout:rbd-mirror-17.2.7-125.g1f2f5510.el9.x86_64.rpm 5.1 MB/s | 2.9 MB 00:00 2023-12-16T21:31:49.499 INFO:teuthology.orchestra.run.smithi169.stdout:-------------------------------------------------------------------------------- 2023-12-16T21:31:49.500 INFO:teuthology.orchestra.run.smithi169.stdout:Total 5.1 MB/s | 2.9 MB 00:00 2023-12-16T21:31:49.500 INFO:teuthology.orchestra.run.smithi169.stdout:Running transaction check 2023-12-16T21:31:49.509 INFO:teuthology.orchestra.run.smithi169.stdout:Transaction check succeeded. 2023-12-16T21:31:49.509 INFO:teuthology.orchestra.run.smithi169.stdout:Running transaction test 2023-12-16T21:31:49.580 INFO:teuthology.orchestra.run.smithi169.stdout:Transaction test succeeded. 2023-12-16T21:31:49.581 INFO:teuthology.orchestra.run.smithi169.stdout:Running transaction 2023-12-16T21:31:49.990 INFO:teuthology.orchestra.run.smithi169.stdout: Preparing : 1/1 2023-12-16T21:31:50.058 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : rbd-mirror-2:17.2.7-125.g1f2f5510.el9.x86_64 1/1 2023-12-16T21:31:50.082 INFO:teuthology.orchestra.run.smithi169.stdout: Running scriptlet: rbd-mirror-2:17.2.7-125.g1f2f5510.el9.x86_64 1/1 2023-12-16T21:31:50.082 INFO:teuthology.orchestra.run.smithi169.stdout:Glob pattern passed to enable, but globs are not supported for this. 2023-12-16T21:31:50.082 INFO:teuthology.orchestra.run.smithi169.stdout:Invalid unit name "ceph-rbd-mirror@*.service" escaped as "ceph-rbd-mirror@\x2a.service". 2023-12-16T21:31:50.082 INFO:teuthology.orchestra.run.smithi169.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2023-12-16T21:31:50.082 INFO:teuthology.orchestra.run.smithi169.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2023-12-16T21:31:50.082 INFO:teuthology.orchestra.run.smithi169.stdout: 2023-12-16T21:31:51.130 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : rbd-mirror-2:17.2.7-125.g1f2f5510.el9.x86_64 1/1 2023-12-16T21:31:51.130 INFO:teuthology.orchestra.run.smithi169.stdout: 2023-12-16T21:31:51.130 INFO:teuthology.orchestra.run.smithi169.stdout:Installed: 2023-12-16T21:31:51.130 INFO:teuthology.orchestra.run.smithi169.stdout: rbd-mirror-2:17.2.7-125.g1f2f5510.el9.x86_64 2023-12-16T21:31:51.130 INFO:teuthology.orchestra.run.smithi169.stdout: 2023-12-16T21:31:51.130 INFO:teuthology.orchestra.run.smithi169.stdout:Complete! 2023-12-16T21:31:51.396 DEBUG:teuthology.orchestra.run.smithi169:> sudo yum -y install rbd-nbd 2023-12-16T21:31:51.972 INFO:teuthology.orchestra.run.smithi169.stdout:Last metadata expiration check: 0:01:38 ago on Sat 16 Dec 2023 09:30:13 PM UTC. 2023-12-16T21:31:52.116 INFO:teuthology.orchestra.run.smithi169.stdout:Dependencies resolved. 2023-12-16T21:31:52.117 INFO:teuthology.orchestra.run.smithi169.stdout:================================================================================ 2023-12-16T21:31:52.117 INFO:teuthology.orchestra.run.smithi169.stdout: Package Architecture Version Repository Size 2023-12-16T21:31:52.117 INFO:teuthology.orchestra.run.smithi169.stdout:================================================================================ 2023-12-16T21:31:52.117 INFO:teuthology.orchestra.run.smithi169.stdout:Installing: 2023-12-16T21:31:52.118 INFO:teuthology.orchestra.run.smithi169.stdout: rbd-nbd x86_64 2:17.2.7-125.g1f2f5510.el9 ceph 162 k 2023-12-16T21:31:52.118 INFO:teuthology.orchestra.run.smithi169.stdout: 2023-12-16T21:31:52.118 INFO:teuthology.orchestra.run.smithi169.stdout:Transaction Summary 2023-12-16T21:31:52.118 INFO:teuthology.orchestra.run.smithi169.stdout:================================================================================ 2023-12-16T21:31:52.118 INFO:teuthology.orchestra.run.smithi169.stdout:Install 1 Package 2023-12-16T21:31:52.118 INFO:teuthology.orchestra.run.smithi169.stdout: 2023-12-16T21:31:52.118 INFO:teuthology.orchestra.run.smithi169.stdout:Total download size: 162 k 2023-12-16T21:31:52.118 INFO:teuthology.orchestra.run.smithi169.stdout:Installed size: 465 k 2023-12-16T21:31:52.118 INFO:teuthology.orchestra.run.smithi169.stdout:Downloading Packages: 2023-12-16T21:31:52.344 INFO:teuthology.orchestra.run.smithi169.stdout:rbd-nbd-17.2.7-125.g1f2f5510.el9.x86_64.rpm 719 kB/s | 162 kB 00:00 2023-12-16T21:31:52.344 INFO:teuthology.orchestra.run.smithi169.stdout:-------------------------------------------------------------------------------- 2023-12-16T21:31:52.345 INFO:teuthology.orchestra.run.smithi169.stdout:Total 715 kB/s | 162 kB 00:00 2023-12-16T21:31:52.345 INFO:teuthology.orchestra.run.smithi169.stdout:Running transaction check 2023-12-16T21:31:52.353 INFO:teuthology.orchestra.run.smithi169.stdout:Transaction check succeeded. 2023-12-16T21:31:52.353 INFO:teuthology.orchestra.run.smithi169.stdout:Running transaction test 2023-12-16T21:31:52.410 INFO:teuthology.orchestra.run.smithi169.stdout:Transaction test succeeded. 2023-12-16T21:31:52.410 INFO:teuthology.orchestra.run.smithi169.stdout:Running transaction 2023-12-16T21:31:52.569 INFO:teuthology.orchestra.run.smithi169.stdout: Preparing : 1/1 2023-12-16T21:31:52.667 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : rbd-nbd-2:17.2.7-125.g1f2f5510.el9.x86_64 1/1 2023-12-16T21:31:53.111 INFO:teuthology.orchestra.run.smithi169.stdout: Running scriptlet: rbd-nbd-2:17.2.7-125.g1f2f5510.el9.x86_64 1/1 2023-12-16T21:31:53.422 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : rbd-nbd-2:17.2.7-125.g1f2f5510.el9.x86_64 1/1 2023-12-16T21:31:53.422 INFO:teuthology.orchestra.run.smithi169.stdout: 2023-12-16T21:31:53.422 INFO:teuthology.orchestra.run.smithi169.stdout:Installed: 2023-12-16T21:31:53.423 INFO:teuthology.orchestra.run.smithi169.stdout: rbd-nbd-2:17.2.7-125.g1f2f5510.el9.x86_64 2023-12-16T21:31:53.423 INFO:teuthology.orchestra.run.smithi169.stdout: 2023-12-16T21:31:53.423 INFO:teuthology.orchestra.run.smithi169.stdout:Complete! 2023-12-16T21:31:53.677 DEBUG:teuthology.parallel:result is None 2023-12-16T21:31:53.677 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2023-12-16T21:31:53.677 INFO:teuthology.packaging:ref: None 2023-12-16T21:31:53.677 INFO:teuthology.packaging:tag: None 2023-12-16T21:31:53.678 INFO:teuthology.packaging:branch: quincy 2023-12-16T21:31:53.678 INFO:teuthology.packaging:sha1: c7586e95519b7be76f3b6e3a4b333888baf46bc9 2023-12-16T21:31:53.678 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&ref=quincy 2023-12-16T21:31:53.990 DEBUG:teuthology.orchestra.run.smithi045:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2023-12-16T21:31:54.015 INFO:teuthology.orchestra.run.smithi045.stdout:17.2.7-125.g1f2f5510.el9 2023-12-16T21:31:54.016 INFO:teuthology.packaging:The installed version of ceph is 17.2.7-125.g1f2f5510.el9 2023-12-16T21:31:54.016 INFO:teuthology.task.install:The correct ceph version 17.2.7-125.g1f2f5510 is installed. 2023-12-16T21:31:54.018 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2023-12-16T21:31:54.018 INFO:teuthology.packaging:ref: None 2023-12-16T21:31:54.018 INFO:teuthology.packaging:tag: None 2023-12-16T21:31:54.018 INFO:teuthology.packaging:branch: quincy 2023-12-16T21:31:54.018 INFO:teuthology.packaging:sha1: c7586e95519b7be76f3b6e3a4b333888baf46bc9 2023-12-16T21:31:54.018 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&ref=quincy 2023-12-16T21:31:54.328 DEBUG:teuthology.orchestra.run.smithi169:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2023-12-16T21:31:54.350 INFO:teuthology.orchestra.run.smithi169.stdout:17.2.7-125.g1f2f5510.el9 2023-12-16T21:31:54.350 INFO:teuthology.packaging:The installed version of ceph is 17.2.7-125.g1f2f5510.el9 2023-12-16T21:31:54.350 INFO:teuthology.task.install:The correct ceph version 17.2.7-125.g1f2f5510 is installed. 2023-12-16T21:31:54.352 INFO:teuthology.task.install.util:Shipping valgrind.supp... 2023-12-16T21:31:54.352 DEBUG:teuthology.orchestra.run.smithi045:> set -ex 2023-12-16T21:31:54.352 DEBUG:teuthology.orchestra.run.smithi045:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2023-12-16T21:31:54.389 DEBUG:teuthology.orchestra.run.smithi169:> set -ex 2023-12-16T21:31:54.390 DEBUG:teuthology.orchestra.run.smithi169:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2023-12-16T21:31:54.422 INFO:teuthology.task.install.util:Shipping 'daemon-helper'... 2023-12-16T21:31:54.422 DEBUG:teuthology.orchestra.run.smithi045:> set -ex 2023-12-16T21:31:54.423 DEBUG:teuthology.orchestra.run.smithi045:> sudo dd of=/usr/bin/daemon-helper 2023-12-16T21:31:54.470 DEBUG:teuthology.orchestra.run.smithi045:> sudo chmod a=rx -- /usr/bin/daemon-helper 2023-12-16T21:31:54.556 DEBUG:teuthology.orchestra.run.smithi169:> set -ex 2023-12-16T21:31:54.556 DEBUG:teuthology.orchestra.run.smithi169:> sudo dd of=/usr/bin/daemon-helper 2023-12-16T21:31:54.599 DEBUG:teuthology.orchestra.run.smithi169:> sudo chmod a=rx -- /usr/bin/daemon-helper 2023-12-16T21:31:54.686 INFO:teuthology.task.install.util:Shipping 'adjust-ulimits'... 2023-12-16T21:31:54.687 DEBUG:teuthology.orchestra.run.smithi045:> set -ex 2023-12-16T21:31:54.687 DEBUG:teuthology.orchestra.run.smithi045:> sudo dd of=/usr/bin/adjust-ulimits 2023-12-16T21:31:54.730 DEBUG:teuthology.orchestra.run.smithi045:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2023-12-16T21:31:54.815 DEBUG:teuthology.orchestra.run.smithi169:> set -ex 2023-12-16T21:31:54.815 DEBUG:teuthology.orchestra.run.smithi169:> sudo dd of=/usr/bin/adjust-ulimits 2023-12-16T21:31:54.849 DEBUG:teuthology.orchestra.run.smithi169:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2023-12-16T21:31:54.928 INFO:teuthology.task.install.util:Shipping 'stdin-killer'... 2023-12-16T21:31:54.928 DEBUG:teuthology.orchestra.run.smithi045:> set -ex 2023-12-16T21:31:54.928 DEBUG:teuthology.orchestra.run.smithi045:> sudo dd of=/usr/bin/stdin-killer 2023-12-16T21:31:54.968 DEBUG:teuthology.orchestra.run.smithi045:> sudo chmod a=rx -- /usr/bin/stdin-killer 2023-12-16T21:31:55.055 DEBUG:teuthology.orchestra.run.smithi169:> set -ex 2023-12-16T21:31:55.055 DEBUG:teuthology.orchestra.run.smithi169:> sudo dd of=/usr/bin/stdin-killer 2023-12-16T21:31:55.089 DEBUG:teuthology.orchestra.run.smithi169:> sudo chmod a=rx -- /usr/bin/stdin-killer 2023-12-16T21:31:55.167 INFO:teuthology.run_tasks:Running task print... 2023-12-16T21:31:55.275 INFO:teuthology.task.print:**** done install task... 2023-12-16T21:31:55.275 INFO:teuthology.run_tasks:Running task cephadm... 2023-12-16T21:31:55.447 INFO:tasks.cephadm:Config: {'cephadm_branch': 'quincy', 'cephadm_git_url': 'https://github.com/ceph/ceph', 'conf': {'osd': {'osd_class_default_list': '*', 'osd_class_load_list': '*', 'bdev async discard': True, 'bdev enable discard': True, 'bluestore allocator': 'bitmap', 'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'debug bluefs': '1/20', 'debug bluestore': '1/20', 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': '4/10', 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd objectstore': 'bluestore', 'osd op complaint time': 180}, 'client': {'client mount timeout': 600, 'debug client': 20, 'debug ms': 1, 'rados mon op timeout': 900, 'rados osd op timeout': 900}, 'global': {'mon pg warn min per osd': 0}, 'mds': {'debug mds': 20, 'debug mds balancer': 20, 'debug ms': 1, 'mds debug frag': True, 'mds debug scatterstat': True, 'mds op complaint time': 180, 'mds verify scatter': True, 'osd op complaint time': 180, 'rados mon op timeout': 900, 'rados osd op timeout': 900}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon op complaint time': 120}}, 'image': 'quay.ceph.io/ceph-ci/ceph:quincy', 'roleless': True, 'flavor': 'default', 'fs': 'xfs', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', 'overall HEALTH_', '\\(FS_DEGRADED\\)', '\\(MDS_FAILED\\)', '\\(MDS_DEGRADED\\)', '\\(FS_WITH_FAILED_MDS\\)', '\\(MDS_DAMAGE\\)', '\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(FS_INLINE_DATA_DEPRECATED\\)', '\\(POOL_APP_NOT_ENABLED\\)', 'overall HEALTH_', '\\(OSD_DOWN\\)', '\\(OSD_', 'but it is still running', 'is not responding'], 'sha1': 'c7586e95519b7be76f3b6e3a4b333888baf46bc9'} 2023-12-16T21:31:55.447 INFO:tasks.cephadm:Cluster image is quay.ceph.io/ceph-ci/ceph:quincy 2023-12-16T21:31:55.448 INFO:tasks.cephadm:Cluster fsid is 88ff363c-9c5a-11ee-95a5-87774f69a715 2023-12-16T21:31:55.448 INFO:tasks.cephadm:Choosing monitor IPs and ports... 2023-12-16T21:31:55.448 INFO:tasks.cephadm:No mon roles; fabricating mons 2023-12-16T21:31:55.448 INFO:tasks.cephadm:Monitor IPs: {'mon.smithi045': '172.21.15.45', 'mon.smithi169': '172.21.15.169'} 2023-12-16T21:31:55.448 INFO:tasks.cephadm:Normalizing hostnames... 2023-12-16T21:31:55.448 DEBUG:teuthology.orchestra.run.smithi045:> sudo hostname $(hostname -s) 2023-12-16T21:31:55.485 DEBUG:teuthology.orchestra.run.smithi169:> sudo hostname $(hostname -s) 2023-12-16T21:31:55.515 INFO:tasks.cephadm:Downloading cephadm (repo https://github.com/ceph/ceph ref quincy)... 2023-12-16T21:31:55.515 DEBUG:teuthology.orchestra.run.smithi045:> curl --silent https://raw.githubusercontent.com/ceph/ceph/quincy/src/cephadm/cephadm > /home/ubuntu/cephtest/cephadm && ls -l /home/ubuntu/cephtest/cephadm 2023-12-16T21:31:55.759 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-r--r--. 1 ubuntu ubuntu 367151 Dec 16 21:31 /home/ubuntu/cephtest/cephadm 2023-12-16T21:31:55.759 DEBUG:teuthology.orchestra.run.smithi169:> curl --silent https://raw.githubusercontent.com/ceph/ceph/quincy/src/cephadm/cephadm > /home/ubuntu/cephtest/cephadm && ls -l /home/ubuntu/cephtest/cephadm 2023-12-16T21:31:55.927 INFO:teuthology.orchestra.run.smithi169.stdout:-rw-r--r--. 1 ubuntu ubuntu 367151 Dec 16 21:31 /home/ubuntu/cephtest/cephadm 2023-12-16T21:31:55.928 DEBUG:teuthology.orchestra.run.smithi045:> test -s /home/ubuntu/cephtest/cephadm && test $(stat -c%s /home/ubuntu/cephtest/cephadm) -gt 1000 && chmod +x /home/ubuntu/cephtest/cephadm 2023-12-16T21:31:55.949 DEBUG:teuthology.orchestra.run.smithi169:> test -s /home/ubuntu/cephtest/cephadm && test $(stat -c%s /home/ubuntu/cephtest/cephadm) -gt 1000 && chmod +x /home/ubuntu/cephtest/cephadm 2023-12-16T21:31:55.983 INFO:tasks.cephadm:Pulling image quay.ceph.io/ceph-ci/ceph:quincy on all hosts... 2023-12-16T21:31:55.983 DEBUG:teuthology.orchestra.run.smithi045:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy pull 2023-12-16T21:31:55.991 DEBUG:teuthology.orchestra.run.smithi169:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy pull 2023-12-16T21:31:56.761 INFO:teuthology.orchestra.run.smithi045.stderr:Pulling container image quay.ceph.io/ceph-ci/ceph:quincy... 2023-12-16T21:31:56.788 INFO:teuthology.orchestra.run.smithi169.stderr:Pulling container image quay.ceph.io/ceph-ci/ceph:quincy... 2023-12-16T21:32:38.068 INFO:teuthology.orchestra.run.smithi045.stdout:{ 2023-12-16T21:32:38.068 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph_version": "ceph version 17.2.7-125-g1f2f5510 (1f2f55109dc1ed2dc2dda9de1db3c70923ba0dfd) quincy (stable)", 2023-12-16T21:32:38.068 INFO:teuthology.orchestra.run.smithi045.stdout: "image_id": "60cc83a9c8f10fdde41a08d59bce905c1a01fc7389b91726758c2505caa3f0c5", 2023-12-16T21:32:38.068 INFO:teuthology.orchestra.run.smithi045.stdout: "repo_digests": [ 2023-12-16T21:32:38.069 INFO:teuthology.orchestra.run.smithi045.stdout: "quay.ceph.io/ceph-ci/ceph@sha256:7a1644eb327b921af6e6cf30118d696e8368bd632346b48049fba15207e60112" 2023-12-16T21:32:38.069 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2023-12-16T21:32:38.069 INFO:teuthology.orchestra.run.smithi045.stdout:} 2023-12-16T21:32:43.717 INFO:teuthology.orchestra.run.smithi169.stdout:{ 2023-12-16T21:32:43.718 INFO:teuthology.orchestra.run.smithi169.stdout: "ceph_version": "ceph version 17.2.7-125-g1f2f5510 (1f2f55109dc1ed2dc2dda9de1db3c70923ba0dfd) quincy (stable)", 2023-12-16T21:32:43.718 INFO:teuthology.orchestra.run.smithi169.stdout: "image_id": "60cc83a9c8f10fdde41a08d59bce905c1a01fc7389b91726758c2505caa3f0c5", 2023-12-16T21:32:43.718 INFO:teuthology.orchestra.run.smithi169.stdout: "repo_digests": [ 2023-12-16T21:32:43.718 INFO:teuthology.orchestra.run.smithi169.stdout: "quay.ceph.io/ceph-ci/ceph@sha256:7a1644eb327b921af6e6cf30118d696e8368bd632346b48049fba15207e60112" 2023-12-16T21:32:43.718 INFO:teuthology.orchestra.run.smithi169.stdout: ] 2023-12-16T21:32:43.718 INFO:teuthology.orchestra.run.smithi169.stdout:} 2023-12-16T21:32:43.736 DEBUG:teuthology.orchestra.run.smithi045:> sudo mkdir -p /etc/ceph 2023-12-16T21:32:43.775 DEBUG:teuthology.orchestra.run.smithi169:> sudo mkdir -p /etc/ceph 2023-12-16T21:32:43.805 DEBUG:teuthology.orchestra.run.smithi045:> sudo chmod 777 /etc/ceph 2023-12-16T21:32:43.850 DEBUG:teuthology.orchestra.run.smithi169:> sudo chmod 777 /etc/ceph 2023-12-16T21:32:43.877 INFO:tasks.cephadm:Writing seed config... 2023-12-16T21:32:43.878 INFO:tasks.cephadm: override: [osd] osd_class_default_list = * 2023-12-16T21:32:43.878 INFO:tasks.cephadm: override: [osd] osd_class_load_list = * 2023-12-16T21:32:43.878 INFO:tasks.cephadm: override: [osd] bdev async discard = True 2023-12-16T21:32:43.878 INFO:tasks.cephadm: override: [osd] bdev enable discard = True 2023-12-16T21:32:43.878 INFO:tasks.cephadm: override: [osd] bluestore allocator = bitmap 2023-12-16T21:32:43.878 INFO:tasks.cephadm: override: [osd] bluestore block size = 96636764160 2023-12-16T21:32:43.878 INFO:tasks.cephadm: override: [osd] bluestore fsck on mount = True 2023-12-16T21:32:43.878 INFO:tasks.cephadm: override: [osd] debug bluefs = 1/20 2023-12-16T21:32:43.878 INFO:tasks.cephadm: override: [osd] debug bluestore = 1/20 2023-12-16T21:32:43.878 INFO:tasks.cephadm: override: [osd] debug ms = 1 2023-12-16T21:32:43.878 INFO:tasks.cephadm: override: [osd] debug osd = 20 2023-12-16T21:32:43.878 INFO:tasks.cephadm: override: [osd] debug rocksdb = 4/10 2023-12-16T21:32:43.879 INFO:tasks.cephadm: override: [osd] mon osd backfillfull_ratio = 0.85 2023-12-16T21:32:43.879 INFO:tasks.cephadm: override: [osd] mon osd full ratio = 0.9 2023-12-16T21:32:43.879 INFO:tasks.cephadm: override: [osd] mon osd nearfull ratio = 0.8 2023-12-16T21:32:43.879 INFO:tasks.cephadm: override: [osd] osd failsafe full ratio = 0.95 2023-12-16T21:32:43.879 INFO:tasks.cephadm: override: [osd] osd objectstore = bluestore 2023-12-16T21:32:43.879 INFO:tasks.cephadm: override: [osd] osd op complaint time = 180 2023-12-16T21:32:43.879 INFO:tasks.cephadm: override: [client] client mount timeout = 600 2023-12-16T21:32:43.879 INFO:tasks.cephadm: override: [client] debug client = 20 2023-12-16T21:32:43.879 INFO:tasks.cephadm: override: [client] debug ms = 1 2023-12-16T21:32:43.879 INFO:tasks.cephadm: override: [client] rados mon op timeout = 900 2023-12-16T21:32:43.879 INFO:tasks.cephadm: override: [client] rados osd op timeout = 900 2023-12-16T21:32:43.879 INFO:tasks.cephadm: override: [global] mon pg warn min per osd = 0 2023-12-16T21:32:43.879 INFO:tasks.cephadm: override: [mds] debug mds = 20 2023-12-16T21:32:43.879 INFO:tasks.cephadm: override: [mds] debug mds balancer = 20 2023-12-16T21:32:43.880 INFO:tasks.cephadm: override: [mds] debug ms = 1 2023-12-16T21:32:43.880 INFO:tasks.cephadm: override: [mds] mds debug frag = True 2023-12-16T21:32:43.880 INFO:tasks.cephadm: override: [mds] mds debug scatterstat = True 2023-12-16T21:32:43.880 INFO:tasks.cephadm: override: [mds] mds op complaint time = 180 2023-12-16T21:32:43.880 INFO:tasks.cephadm: override: [mds] mds verify scatter = True 2023-12-16T21:32:43.880 INFO:tasks.cephadm: override: [mds] osd op complaint time = 180 2023-12-16T21:32:43.880 INFO:tasks.cephadm: override: [mds] rados mon op timeout = 900 2023-12-16T21:32:43.880 INFO:tasks.cephadm: override: [mds] rados osd op timeout = 900 2023-12-16T21:32:43.880 INFO:tasks.cephadm: override: [mgr] debug mgr = 20 2023-12-16T21:32:43.880 INFO:tasks.cephadm: override: [mgr] debug ms = 1 2023-12-16T21:32:43.880 INFO:tasks.cephadm: override: [mon] debug mon = 20 2023-12-16T21:32:43.880 INFO:tasks.cephadm: override: [mon] debug ms = 1 2023-12-16T21:32:43.880 INFO:tasks.cephadm: override: [mon] debug paxos = 20 2023-12-16T21:32:43.880 INFO:tasks.cephadm: override: [mon] mon op complaint time = 120 2023-12-16T21:32:43.881 DEBUG:teuthology.orchestra.run.smithi045:> set -ex 2023-12-16T21:32:43.881 DEBUG:teuthology.orchestra.run.smithi045:> dd of=/home/ubuntu/cephtest/seed.ceph.conf 2023-12-16T21:32:43.907 DEBUG:tasks.cephadm:Final config: [global] # make logging friendly to teuthology log_to_file = true log_to_stderr = false log to journald = false mon cluster log file level = debug mon clock drift allowed = 1.000 # replicate across OSDs, not hosts osd crush chooseleaf type = 0 #osd pool default size = 2 osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd # enable some debugging auth debug = true ms die on old message = true ms die on bug = true debug asserts on shutdown = true # adjust warnings mon max pg per osd = 10000# >= luminous mon pg warn max object skew = 0 mon osd allow primary affinity = true mon osd allow pg remap = true mon warn on legacy crush tunables = false mon warn on crush straw calc version zero = false mon warn on no sortbitwise = false mon warn on osd down out interval zero = false mon warn on too few osds = false mon_warn_on_pool_pg_num_not_power_of_two = false # disable pg_autoscaler by default for new pools osd_pool_default_pg_autoscale_mode = off # tests delete pools mon allow pool delete = true fsid = 88ff363c-9c5a-11ee-95a5-87774f69a715 mon pg warn min per osd = 0 [osd] osd scrub load threshold = 5.0 osd scrub max interval = 600 osd mclock profile = high_recovery_ops osd recover clone overlap = true osd recovery max chunk = 1048576 osd deep scrub update digest min age = 30 osd map max advance = 10 osd memory target autotune = true # debugging osd debug shutdown = true osd debug op order = true osd debug verify stray on activate = true osd debug pg log writeout = true osd debug verify cached snaps = true osd debug verify missing on start = true osd debug misdirected ops = true osd op queue = debug_random osd op queue cut off = debug_random osd shutdown pgref assert = true bdev debug aio = true osd sloppy crc = true osd_class_default_list = * osd_class_load_list = * bdev async discard = True bdev enable discard = True bluestore allocator = bitmap bluestore block size = 96636764160 bluestore fsck on mount = True debug bluefs = 1/20 debug bluestore = 1/20 debug ms = 1 debug osd = 20 debug rocksdb = 4/10 mon osd backfillfull_ratio = 0.85 mon osd full ratio = 0.9 mon osd nearfull ratio = 0.8 osd failsafe full ratio = 0.95 osd objectstore = bluestore osd op complaint time = 180 [mgr] mon reweight min pgs per osd = 4 mon reweight min bytes per osd = 10 mgr/telemetry/nag = false debug mgr = 20 debug ms = 1 [mon] mon data avail warn = 5 mon mgr mkfs grace = 240 mon reweight min pgs per osd = 4 mon osd reporter subtree level = osd mon osd prime pg temp = true mon reweight min bytes per osd = 10 # rotate auth tickets quickly to exercise renewal paths auth mon ticket ttl = 660# 11m auth service ticket ttl = 240# 4m # don't complain about global id reclaim mon_warn_on_insecure_global_id_reclaim = false mon_warn_on_insecure_global_id_reclaim_allowed = false debug mon = 20 debug ms = 1 debug paxos = 20 mon op complaint time = 120 [client.rgw] rgw cache enabled = true rgw enable ops log = true rgw enable usage log = true [client] client mount timeout = 600 debug client = 20 debug ms = 1 rados mon op timeout = 900 rados osd op timeout = 900 [mds] debug mds = 20 debug mds balancer = 20 debug ms = 1 mds debug frag = True mds debug scatterstat = True mds op complaint time = 180 mds verify scatter = True osd op complaint time = 180 rados mon op timeout = 900 rados osd op timeout = 900 2023-12-16T21:32:43.908 DEBUG:teuthology.orchestra.run.smithi045:mon.smithi045> sudo journalctl -f -n 0 -u ceph-88ff363c-9c5a-11ee-95a5-87774f69a715@mon.smithi045.service 2023-12-16T21:32:43.950 INFO:tasks.cephadm:Bootstrapping... 2023-12-16T21:32:43.951 DEBUG:teuthology.orchestra.run.smithi045:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy -v bootstrap --fsid 88ff363c-9c5a-11ee-95a5-87774f69a715 --config /home/ubuntu/cephtest/seed.ceph.conf --output-config /etc/ceph/ceph.conf --output-keyring /etc/ceph/ceph.client.admin.keyring --output-pub-ssh-key /home/ubuntu/cephtest/ceph.pub --mon-ip 172.21.15.45 --skip-admin-label && sudo chmod +r /etc/ceph/ceph.client.admin.keyring 2023-12-16T21:32:44.171 INFO:teuthology.orchestra.run.smithi045.stdout:-------------------------------------------------------------------------------- 2023-12-16T21:32:44.172 INFO:teuthology.orchestra.run.smithi045.stdout:cephadm ['--image', 'quay.ceph.io/ceph-ci/ceph:quincy', '-v', 'bootstrap', '--fsid', '88ff363c-9c5a-11ee-95a5-87774f69a715', '--config', '/home/ubuntu/cephtest/seed.ceph.conf', '--output-config', '/etc/ceph/ceph.conf', '--output-keyring', '/etc/ceph/ceph.client.admin.keyring', '--output-pub-ssh-key', '/home/ubuntu/cephtest/ceph.pub', '--mon-ip', '172.21.15.45', '--skip-admin-label'] 2023-12-16T21:32:44.198 INFO:teuthology.orchestra.run.smithi045.stdout:/bin/podman: stdout 4.7.2 2023-12-16T21:32:44.199 INFO:teuthology.orchestra.run.smithi045.stderr:Specifying an fsid for your cluster offers no advantages and may increase the likelihood of fsid conflicts. 2023-12-16T21:32:44.199 INFO:teuthology.orchestra.run.smithi045.stdout:Verifying podman|docker is present... 2023-12-16T21:32:44.223 INFO:teuthology.orchestra.run.smithi045.stdout:/bin/podman: stdout 4.7.2 2023-12-16T21:32:44.223 INFO:teuthology.orchestra.run.smithi045.stdout:Verifying lvm2 is present... 2023-12-16T21:32:44.223 INFO:teuthology.orchestra.run.smithi045.stdout:Verifying time synchronization is in place... 2023-12-16T21:32:44.231 INFO:teuthology.orchestra.run.smithi045.stdout:Non-zero exit code 1 from systemctl is-enabled chrony.service 2023-12-16T21:32:44.232 INFO:teuthology.orchestra.run.smithi045.stdout:systemctl: stderr Failed to get unit file state for chrony.service: No such file or directory 2023-12-16T21:32:44.238 INFO:teuthology.orchestra.run.smithi045.stdout:Non-zero exit code 3 from systemctl is-active chrony.service 2023-12-16T21:32:44.238 INFO:teuthology.orchestra.run.smithi045.stdout:systemctl: stdout inactive 2023-12-16T21:32:44.246 INFO:teuthology.orchestra.run.smithi045.stdout:systemctl: stdout enabled 2023-12-16T21:32:44.253 INFO:teuthology.orchestra.run.smithi045.stdout:systemctl: stdout active 2023-12-16T21:32:44.253 INFO:teuthology.orchestra.run.smithi045.stdout:Unit chronyd.service is enabled and running 2023-12-16T21:32:44.253 INFO:teuthology.orchestra.run.smithi045.stdout:Repeating the final host check... 2023-12-16T21:32:44.279 INFO:teuthology.orchestra.run.smithi045.stdout:/bin/podman: stdout 4.7.2 2023-12-16T21:32:44.279 INFO:teuthology.orchestra.run.smithi045.stdout:podman (/bin/podman) version 4.7.2 is present 2023-12-16T21:32:44.279 INFO:teuthology.orchestra.run.smithi045.stdout:systemctl is present 2023-12-16T21:32:44.279 INFO:teuthology.orchestra.run.smithi045.stdout:lvcreate is present 2023-12-16T21:32:44.285 INFO:teuthology.orchestra.run.smithi045.stdout:Non-zero exit code 1 from systemctl is-enabled chrony.service 2023-12-16T21:32:44.286 INFO:teuthology.orchestra.run.smithi045.stdout:systemctl: stderr Failed to get unit file state for chrony.service: No such file or directory 2023-12-16T21:32:44.293 INFO:teuthology.orchestra.run.smithi045.stdout:Non-zero exit code 3 from systemctl is-active chrony.service 2023-12-16T21:32:44.293 INFO:teuthology.orchestra.run.smithi045.stdout:systemctl: stdout inactive 2023-12-16T21:32:44.300 INFO:teuthology.orchestra.run.smithi045.stdout:systemctl: stdout enabled 2023-12-16T21:32:44.309 INFO:teuthology.orchestra.run.smithi045.stdout:systemctl: stdout active 2023-12-16T21:32:44.309 INFO:teuthology.orchestra.run.smithi045.stdout:Unit chronyd.service is enabled and running 2023-12-16T21:32:44.309 INFO:teuthology.orchestra.run.smithi045.stdout:Host looks OK 2023-12-16T21:32:44.310 INFO:teuthology.orchestra.run.smithi045.stdout:Cluster fsid: 88ff363c-9c5a-11ee-95a5-87774f69a715 2023-12-16T21:32:44.310 INFO:teuthology.orchestra.run.smithi045.stdout:Acquiring lock 139861480403008 on /run/cephadm/88ff363c-9c5a-11ee-95a5-87774f69a715.lock 2023-12-16T21:32:44.310 INFO:teuthology.orchestra.run.smithi045.stdout:Lock 139861480403008 acquired on /run/cephadm/88ff363c-9c5a-11ee-95a5-87774f69a715.lock 2023-12-16T21:32:44.310 INFO:teuthology.orchestra.run.smithi045.stdout:Verifying IP 172.21.15.45 port 3300 ... 2023-12-16T21:32:44.310 INFO:teuthology.orchestra.run.smithi045.stdout:Verifying IP 172.21.15.45 port 6789 ... 2023-12-16T21:32:44.310 INFO:teuthology.orchestra.run.smithi045.stdout:Base mon IP(s) is [172.21.15.45:3300, 172.21.15.45:6789], mon addrv is [v2:172.21.15.45:3300,v1:172.21.15.45:6789] 2023-12-16T21:32:44.312 INFO:teuthology.orchestra.run.smithi045.stdout:/sbin/ip: stdout default via 172.21.15.254 dev ens1f0 proto dhcp src 172.21.15.45 metric 100 2023-12-16T21:32:44.312 INFO:teuthology.orchestra.run.smithi045.stdout:/sbin/ip: stdout 172.21.0.0/20 dev ens1f0 proto kernel scope link src 172.21.15.45 metric 100 2023-12-16T21:32:44.315 INFO:teuthology.orchestra.run.smithi045.stdout:/sbin/ip: stdout ::1 dev lo proto kernel metric 256 pref medium 2023-12-16T21:32:44.315 INFO:teuthology.orchestra.run.smithi045.stdout:/sbin/ip: stdout fe80::/64 dev ens1f0 proto kernel metric 1024 pref medium 2023-12-16T21:32:44.315 INFO:teuthology.orchestra.run.smithi045.stdout:/sbin/ip: stdout default via fe80::327c:5e00:6487:71e0 dev ens1f0 proto ra metric 100 pref medium 2023-12-16T21:32:44.318 INFO:teuthology.orchestra.run.smithi045.stdout:/sbin/ip: stdout 1: lo: mtu 65536 state UNKNOWN qlen 1000 2023-12-16T21:32:44.318 INFO:teuthology.orchestra.run.smithi045.stdout:/sbin/ip: stdout inet6 ::1/128 scope host 2023-12-16T21:32:44.318 INFO:teuthology.orchestra.run.smithi045.stdout:/sbin/ip: stdout valid_lft forever preferred_lft forever 2023-12-16T21:32:44.318 INFO:teuthology.orchestra.run.smithi045.stdout:/sbin/ip: stdout 4: ens1f0: mtu 1500 state UP qlen 1000 2023-12-16T21:32:44.319 INFO:teuthology.orchestra.run.smithi045.stdout:/sbin/ip: stdout inet6 fe80::ec4:7aff:febd:12be/64 scope link noprefixroute 2023-12-16T21:32:44.319 INFO:teuthology.orchestra.run.smithi045.stdout:/sbin/ip: stdout valid_lft forever preferred_lft forever 2023-12-16T21:32:44.319 INFO:teuthology.orchestra.run.smithi045.stdout:Mon IP `172.21.15.45` is in CIDR network `172.21.0.0/20` 2023-12-16T21:32:44.319 INFO:teuthology.orchestra.run.smithi045.stdout:Mon IP `172.21.15.45` is in CIDR network `172.21.0.0/20` 2023-12-16T21:32:44.319 INFO:teuthology.orchestra.run.smithi045.stdout:Inferred mon public CIDR from local network configuration ['172.21.0.0/20', '172.21.0.0/20'] 2023-12-16T21:32:44.320 INFO:teuthology.orchestra.run.smithi045.stdout:Internal network (--cluster-network) has not been provided, OSD replication will default to the public_network 2023-12-16T21:32:44.320 INFO:teuthology.orchestra.run.smithi045.stdout:Pulling container image quay.ceph.io/ceph-ci/ceph:quincy... 2023-12-16T21:32:44.746 INFO:teuthology.orchestra.run.smithi045.stdout:/bin/podman: stdout 60cc83a9c8f10fdde41a08d59bce905c1a01fc7389b91726758c2505caa3f0c5 2023-12-16T21:32:44.747 INFO:teuthology.orchestra.run.smithi045.stdout:/bin/podman: stderr Trying to pull quay.ceph.io/ceph-ci/ceph:quincy... 2023-12-16T21:32:44.747 INFO:teuthology.orchestra.run.smithi045.stdout:/bin/podman: stderr Getting image source signatures 2023-12-16T21:32:44.747 INFO:teuthology.orchestra.run.smithi045.stdout:/bin/podman: stderr Copying blob sha256:46af8f5390d4e94fc57efb422ccb97bb53dfe5b948546bfc191b46557eb2dbd9 2023-12-16T21:32:44.747 INFO:teuthology.orchestra.run.smithi045.stdout:/bin/podman: stderr Copying blob sha256:3c21fa6304a567e69d15522c48775b11468df75d53cc353263c55c753e1a9b4e 2023-12-16T21:32:44.747 INFO:teuthology.orchestra.run.smithi045.stdout:/bin/podman: stderr Copying config sha256:60cc83a9c8f10fdde41a08d59bce905c1a01fc7389b91726758c2505caa3f0c5 2023-12-16T21:32:44.747 INFO:teuthology.orchestra.run.smithi045.stdout:/bin/podman: stderr Writing manifest to image destination 2023-12-16T21:32:45.180 INFO:teuthology.orchestra.run.smithi045.stdout:ceph: stdout ceph version 17.2.7-125-g1f2f5510 (1f2f55109dc1ed2dc2dda9de1db3c70923ba0dfd) quincy (stable) 2023-12-16T21:32:45.180 INFO:teuthology.orchestra.run.smithi045.stdout:Ceph version: ceph version 17.2.7-125-g1f2f5510 (1f2f55109dc1ed2dc2dda9de1db3c70923ba0dfd) quincy (stable) 2023-12-16T21:32:45.180 INFO:teuthology.orchestra.run.smithi045.stdout:Extracting ceph user uid/gid from container image... 2023-12-16T21:32:45.504 INFO:teuthology.orchestra.run.smithi045.stdout:stat: stdout 167 167 2023-12-16T21:32:45.504 INFO:teuthology.orchestra.run.smithi045.stdout:Creating initial keys... 2023-12-16T21:32:45.937 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph-authtool: stdout AQB9F35lNDohJRAAIR7zoc6J0gd4vI1e4KD9/w== 2023-12-16T21:32:46.280 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph-authtool: stdout AQB+F35lvQHTBBAAFRHInBfqxtnZPp0VYzLZNA== 2023-12-16T21:32:46.612 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph-authtool: stdout AQB+F35l2mqnFxAA+nOLz6XIL21qwfghAKPa8w== 2023-12-16T21:32:46.613 INFO:teuthology.orchestra.run.smithi045.stdout:Creating initial monmap... 2023-12-16T21:32:46.954 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/monmaptool: stdout /usr/bin/monmaptool: monmap file /tmp/monmap 2023-12-16T21:32:46.954 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/monmaptool: stdout setting min_mon_release = octopus 2023-12-16T21:32:46.954 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/monmaptool: stdout /usr/bin/monmaptool: set fsid to 88ff363c-9c5a-11ee-95a5-87774f69a715 2023-12-16T21:32:46.954 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/monmaptool: stdout /usr/bin/monmaptool: writing epoch 0 to /tmp/monmap (1 monitors) 2023-12-16T21:32:46.954 INFO:teuthology.orchestra.run.smithi045.stdout:monmaptool for smithi045 [v2:172.21.15.45:3300,v1:172.21.15.45:6789] on /usr/bin/monmaptool: monmap file /tmp/monmap 2023-12-16T21:32:46.955 INFO:teuthology.orchestra.run.smithi045.stdout:setting min_mon_release = octopus 2023-12-16T21:32:46.955 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/monmaptool: set fsid to 88ff363c-9c5a-11ee-95a5-87774f69a715 2023-12-16T21:32:46.955 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/monmaptool: writing epoch 0 to /tmp/monmap (1 monitors) 2023-12-16T21:32:46.955 INFO:teuthology.orchestra.run.smithi045.stdout: 2023-12-16T21:32:46.955 INFO:teuthology.orchestra.run.smithi045.stdout:Creating mon... 2023-12-16T21:32:47.369 INFO:teuthology.orchestra.run.smithi045.stdout:create mon.smithi045 on 2023-12-16T21:32:47.544 INFO:teuthology.orchestra.run.smithi045.stdout:systemctl: stderr Removed "/etc/systemd/system/multi-user.target.wants/ceph.target". 2023-12-16T21:32:47.694 INFO:teuthology.orchestra.run.smithi045.stdout:systemctl: stderr Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /etc/systemd/system/ceph.target. 2023-12-16T21:32:47.849 INFO:teuthology.orchestra.run.smithi045.stdout:systemctl: stderr Created symlink /etc/systemd/system/multi-user.target.wants/ceph-88ff363c-9c5a-11ee-95a5-87774f69a715.target → /etc/systemd/system/ceph-88ff363c-9c5a-11ee-95a5-87774f69a715.target. 2023-12-16T21:32:47.849 INFO:teuthology.orchestra.run.smithi045.stdout:systemctl: stderr Created symlink /etc/systemd/system/ceph.target.wants/ceph-88ff363c-9c5a-11ee-95a5-87774f69a715.target → /etc/systemd/system/ceph-88ff363c-9c5a-11ee-95a5-87774f69a715.target. 2023-12-16T21:32:48.022 INFO:teuthology.orchestra.run.smithi045.stdout:Non-zero exit code 1 from systemctl reset-failed ceph-88ff363c-9c5a-11ee-95a5-87774f69a715@mon.smithi045 2023-12-16T21:32:48.022 INFO:teuthology.orchestra.run.smithi045.stdout:systemctl: stderr Failed to reset failed state of unit ceph-88ff363c-9c5a-11ee-95a5-87774f69a715@mon.smithi045.service: Unit ceph-88ff363c-9c5a-11ee-95a5-87774f69a715@mon.smithi045.service not loaded. 2023-12-16T21:32:48.185 INFO:teuthology.orchestra.run.smithi045.stdout:systemctl: stderr Created symlink /etc/systemd/system/ceph-88ff363c-9c5a-11ee-95a5-87774f69a715.target.wants/ceph-88ff363c-9c5a-11ee-95a5-87774f69a715@mon.smithi045.service → /etc/systemd/system/ceph-88ff363c-9c5a-11ee-95a5-87774f69a715@.service. 2023-12-16T21:32:48.450 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Dec 16 21:32:48 smithi045 podman[80371]: 2023-12-16 21:32:48.429156534 +0000 UTC m=+0.130849924 container start aad8ef6c9b6482c13517d7e92937ffff644131111bf65441c47f574d05049177 (image=quay.ceph.io/ceph-ci/ceph:quincy, name=ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-mon-smithi045, ceph=True, maintainer=Guillaume Abrioux , GIT_COMMIT=0396eef90bef641b676c164ec7a3876f45010308, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, GIT_CLEAN=True, CEPH_POINT_RELEASE=, RELEASE=quincy-1f2f551, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.31.3, org.label-schema.build-date=20231212) 2023-12-16T21:32:48.450 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Dec 16 21:32:48 smithi045 bash[80371]: aad8ef6c9b6482c13517d7e92937ffff644131111bf65441c47f574d05049177 2023-12-16T21:32:48.464 INFO:teuthology.orchestra.run.smithi045.stdout:Non-zero exit code 1 from systemctl is-enabled firewalld.service 2023-12-16T21:32:48.464 INFO:teuthology.orchestra.run.smithi045.stdout:systemctl: stdout disabled 2023-12-16T21:32:48.471 INFO:teuthology.orchestra.run.smithi045.stdout:Non-zero exit code 3 from systemctl is-active firewalld.service 2023-12-16T21:32:48.471 INFO:teuthology.orchestra.run.smithi045.stdout:systemctl: stdout inactive 2023-12-16T21:32:48.471 INFO:teuthology.orchestra.run.smithi045.stdout:firewalld.service is not enabled 2023-12-16T21:32:48.471 INFO:teuthology.orchestra.run.smithi045.stdout:Not possible to enable service . firewalld.service is not available 2023-12-16T21:32:48.472 INFO:teuthology.orchestra.run.smithi045.stdout:Waiting for mon to start... 2023-12-16T21:32:48.472 INFO:teuthology.orchestra.run.smithi045.stdout:Waiting for mon... 2023-12-16T21:32:48.751 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Dec 16 21:32:48 smithi045 systemd[1]: Started Ceph mon.smithi045 for 88ff363c-9c5a-11ee-95a5-87774f69a715. 2023-12-16T21:32:48.752 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Dec 16 21:32:48 smithi045 ceph-mon[80390]: mkfs 88ff363c-9c5a-11ee-95a5-87774f69a715 2023-12-16T21:32:48.752 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Dec 16 21:32:48 smithi045 ceph-mon[80390]: mon.smithi045 is new leader, mons smithi045 in quorum (ranks 0) 2023-12-16T21:32:49.054 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stdout cluster: 2023-12-16T21:32:49.060 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stdout id: 88ff363c-9c5a-11ee-95a5-87774f69a715 2023-12-16T21:32:49.060 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stdout health: HEALTH_OK 2023-12-16T21:32:49.060 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stdout 2023-12-16T21:32:49.060 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stdout services: 2023-12-16T21:32:49.060 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stdout mon: 1 daemons, quorum smithi045 (age 0.340681s) 2023-12-16T21:32:49.060 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stdout mgr: no daemons active 2023-12-16T21:32:49.060 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stdout osd: 0 osds: 0 up, 0 in 2023-12-16T21:32:49.060 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stdout 2023-12-16T21:32:49.060 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stdout data: 2023-12-16T21:32:49.060 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stdout pools: 0 pools, 0 pgs 2023-12-16T21:32:49.060 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stdout objects: 0 objects, 0 B 2023-12-16T21:32:49.060 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stdout usage: 0 B used, 0 B / 0 B avail 2023-12-16T21:32:49.061 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stdout pgs: 2023-12-16T21:32:49.061 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stdout 2023-12-16T21:32:49.061 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2023-12-16T21:32:48.748+0000 7f232717e700 1 Processor -- start 2023-12-16T21:32:49.061 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2023-12-16T21:32:48.749+0000 7f232717e700 1 -- start start 2023-12-16T21:32:49.061 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2023-12-16T21:32:48.749+0000 7f232717e700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f2320078dc0 0x7f23200771d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:32:49.061 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2023-12-16T21:32:48.749+0000 7f232717e700 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2320077710 con 0x7f2320078dc0 2023-12-16T21:32:49.061 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2023-12-16T21:32:48.749+0000 7f2324f1a700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f2320078dc0 0x7f23200771d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:32:49.061 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2023-12-16T21:32:48.749+0000 7f2324f1a700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f2320078dc0 0x7f23200771d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:53048/0 (socket says 172.21.15.45:53048) 2023-12-16T21:32:49.061 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2023-12-16T21:32:48.749+0000 7f2324f1a700 1 -- 172.21.15.45:0/2498487873 learned_addr learned my addr 172.21.15.45:0/2498487873 (peer_addr_for_me v2:172.21.15.45:0/0) 2023-12-16T21:32:49.061 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2023-12-16T21:32:48.750+0000 7f2324f1a700 1 -- 172.21.15.45:0/2498487873 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2320077850 con 0x7f2320078dc0 2023-12-16T21:32:49.061 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2023-12-16T21:32:48.750+0000 7f2324f1a700 1 --2- 172.21.15.45:0/2498487873 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f2320078dc0 0x7f23200771d0 secure :-1 s=READY pgs=1 cs=0 l=1 rev1=1 crypto rx=0x7f2318009e50 tx=0x7f2318009840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=104c59bcc7fdfa01 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:32:49.061 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2023-12-16T21:32:48.750+0000 7f231f7fe700 1 -- 172.21.15.45:0/2498487873 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f23180092e0 con 0x7f2320078dc0 2023-12-16T21:32:49.061 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2023-12-16T21:32:48.750+0000 7f231f7fe700 1 -- 172.21.15.45:0/2498487873 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f231800ba20 con 0x7f2320078dc0 2023-12-16T21:32:49.061 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2023-12-16T21:32:48.750+0000 7f231f7fe700 1 -- 172.21.15.45:0/2498487873 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f231800bd30 con 0x7f2320078dc0 2023-12-16T21:32:49.062 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2023-12-16T21:32:48.750+0000 7f232717e700 1 -- 172.21.15.45:0/2498487873 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f2320078dc0 msgr2=0x7f23200771d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:32:49.062 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2023-12-16T21:32:48.750+0000 7f232717e700 1 --2- 172.21.15.45:0/2498487873 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f2320078dc0 0x7f23200771d0 secure :-1 s=READY pgs=1 cs=0 l=1 rev1=1 crypto rx=0x7f2318009e50 tx=0x7f2318009840 comp rx=0 tx=0).stop 2023-12-16T21:32:49.062 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2023-12-16T21:32:48.751+0000 7f232717e700 1 -- 172.21.15.45:0/2498487873 shutdown_connections 2023-12-16T21:32:49.062 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2023-12-16T21:32:48.751+0000 7f232717e700 1 --2- 172.21.15.45:0/2498487873 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f2320078dc0 0x7f23200771d0 unknown :-1 s=CLOSED pgs=1 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:32:49.062 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2023-12-16T21:32:48.751+0000 7f232717e700 1 -- 172.21.15.45:0/2498487873 >> 172.21.15.45:0/2498487873 conn(0x7f23200fda40 msgr2=0x7f23200ffe70 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-16T21:32:49.062 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2023-12-16T21:32:48.751+0000 7f232717e700 1 -- 172.21.15.45:0/2498487873 shutdown_connections 2023-12-16T21:32:49.062 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2023-12-16T21:32:48.751+0000 7f232717e700 1 -- 172.21.15.45:0/2498487873 wait complete. 2023-12-16T21:32:49.062 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2023-12-16T21:32:48.751+0000 7f232717e700 1 Processor -- start 2023-12-16T21:32:49.062 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2023-12-16T21:32:48.751+0000 7f232717e700 1 -- start start 2023-12-16T21:32:49.062 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2023-12-16T21:32:48.752+0000 7f232717e700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f2320078dc0 0x7f2320112c10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:32:49.062 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2023-12-16T21:32:48.752+0000 7f232717e700 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2320113150 con 0x7f2320078dc0 2023-12-16T21:32:49.062 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2023-12-16T21:32:48.752+0000 7f2324f1a700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f2320078dc0 0x7f2320112c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:32:49.062 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2023-12-16T21:32:48.752+0000 7f2324f1a700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f2320078dc0 0x7f2320112c10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:53060/0 (socket says 172.21.15.45:53060) 2023-12-16T21:32:49.062 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2023-12-16T21:32:48.752+0000 7f2324f1a700 1 -- 172.21.15.45:0/2028230515 learned_addr learned my addr 172.21.15.45:0/2028230515 (peer_addr_for_me v2:172.21.15.45:0/0) 2023-12-16T21:32:49.062 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2023-12-16T21:32:48.752+0000 7f2324f1a700 1 -- 172.21.15.45:0/2028230515 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2318009ae0 con 0x7f2320078dc0 2023-12-16T21:32:49.063 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2023-12-16T21:32:48.752+0000 7f2324f1a700 1 --2- 172.21.15.45:0/2028230515 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f2320078dc0 0x7f2320112c10 secure :-1 s=READY pgs=2 cs=0 l=1 rev1=1 crypto rx=0x7f2318003000 tx=0x7f2318017000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:32:49.063 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2023-12-16T21:32:48.752+0000 7f231d7fa700 1 -- 172.21.15.45:0/2028230515 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f2318012b90 con 0x7f2320078dc0 2023-12-16T21:32:49.063 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2023-12-16T21:32:48.752+0000 7f231d7fa700 1 -- 172.21.15.45:0/2028230515 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2318012d10 con 0x7f2320078dc0 2023-12-16T21:32:49.063 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2023-12-16T21:32:48.752+0000 7f232717e700 1 -- 172.21.15.45:0/2028230515 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2320113350 con 0x7f2320078dc0 2023-12-16T21:32:49.063 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2023-12-16T21:32:48.752+0000 7f231d7fa700 1 -- 172.21.15.45:0/2028230515 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f231801e400 con 0x7f2320078dc0 2023-12-16T21:32:49.063 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2023-12-16T21:32:48.752+0000 7f232717e700 1 -- 172.21.15.45:0/2028230515 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2320113ff0 con 0x7f2320078dc0 2023-12-16T21:32:49.063 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2023-12-16T21:32:48.753+0000 7f231d7fa700 1 -- 172.21.15.45:0/2028230515 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 1) v1 ==== 521+0+0 (secure 0 0 0) 0x7f2318019030 con 0x7f2320078dc0 2023-12-16T21:32:49.063 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2023-12-16T21:32:48.753+0000 7f231d7fa700 1 -- 172.21.15.45:0/2028230515 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 721+0+0 (secure 0 0 0) 0x7f231801ed50 con 0x7f2320078dc0 2023-12-16T21:32:49.063 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2023-12-16T21:32:48.753+0000 7f232717e700 1 -- 172.21.15.45:0/2028230515 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f23201137c0 con 0x7f2320078dc0 2023-12-16T21:32:49.063 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2023-12-16T21:32:48.756+0000 7f231d7fa700 1 -- 172.21.15.45:0/2028230515 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+71349 (secure 0 0 0) 0x7f2318017070 con 0x7f2320078dc0 2023-12-16T21:32:49.063 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2023-12-16T21:32:48.829+0000 7f232717e700 1 -- 172.21.15.45:0/2028230515 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "status"} v 0) v1 -- 0x7f2320077bc0 con 0x7f2320078dc0 2023-12-16T21:32:49.063 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2023-12-16T21:32:48.829+0000 7f231d7fa700 1 -- 172.21.15.45:0/2028230515 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mon_command_ack([{"prefix": "status"}]=0 v0) v1 ==== 54+0+325 (secure 0 0 0) 0x7f231801e620 con 0x7f2320078dc0 2023-12-16T21:32:49.063 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2023-12-16T21:32:48.830+0000 7f232717e700 1 -- 172.21.15.45:0/2028230515 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f2320078dc0 msgr2=0x7f2320112c10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:32:49.064 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2023-12-16T21:32:48.830+0000 7f232717e700 1 --2- 172.21.15.45:0/2028230515 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f2320078dc0 0x7f2320112c10 secure :-1 s=READY pgs=2 cs=0 l=1 rev1=1 crypto rx=0x7f2318003000 tx=0x7f2318017000 comp rx=0 tx=0).stop 2023-12-16T21:32:49.064 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2023-12-16T21:32:48.830+0000 7f232717e700 1 -- 172.21.15.45:0/2028230515 shutdown_connections 2023-12-16T21:32:49.064 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2023-12-16T21:32:48.830+0000 7f232717e700 1 --2- 172.21.15.45:0/2028230515 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f2320078dc0 0x7f2320112c10 unknown :-1 s=CLOSED pgs=2 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:32:49.064 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2023-12-16T21:32:48.830+0000 7f232717e700 1 -- 172.21.15.45:0/2028230515 >> 172.21.15.45:0/2028230515 conn(0x7f23200fda40 msgr2=0x7f23201070a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-16T21:32:49.064 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2023-12-16T21:32:48.830+0000 7f232717e700 1 -- 172.21.15.45:0/2028230515 shutdown_connections 2023-12-16T21:32:49.064 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2023-12-16T21:32:48.830+0000 7f232717e700 1 -- 172.21.15.45:0/2028230515 wait complete. 2023-12-16T21:32:49.064 INFO:teuthology.orchestra.run.smithi045.stdout:mon is available 2023-12-16T21:32:49.064 INFO:teuthology.orchestra.run.smithi045.stdout:Assimilating anything we can from ceph.conf... 2023-12-16T21:32:49.625 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Dec 16 21:32:49 smithi045 ceph-mon[80390]: mon.smithi045 is new leader, mons smithi045 in quorum (ranks 0) 2023-12-16T21:32:49.625 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Dec 16 21:32:49 smithi045 ceph-mon[80390]: monmap e1: 1 mons at {smithi045=[v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0]} removed_ranks: {} disallowed_leaders: {} 2023-12-16T21:32:49.625 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Dec 16 21:32:49 smithi045 ceph-mon[80390]: fsmap 2023-12-16T21:32:49.625 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Dec 16 21:32:49 smithi045 ceph-mon[80390]: osdmap e1: 0 total, 0 up, 0 in 2023-12-16T21:32:49.625 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Dec 16 21:32:49 smithi045 ceph-mon[80390]: mgrmap e1: no daemons active 2023-12-16T21:32:49.625 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Dec 16 21:32:49 smithi045 ceph-mon[80390]: from='client.? 172.21.15.45:0/2028230515' entity='client.admin' cmd=[{"prefix": "status"}]: dispatch 2023-12-16T21:32:49.625 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Dec 16 21:32:49 smithi045 ceph-mon[80390]: from='client.? 172.21.15.45:0/1400334285' entity='client.admin' cmd=[{"prefix": "config assimilate-conf"}]: dispatch 2023-12-16T21:32:49.625 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Dec 16 21:32:49 smithi045 ceph-mon[80390]: from='client.? 172.21.15.45:0/1400334285' entity='client.admin' cmd='[{"prefix": "config assimilate-conf"}]': finished 2023-12-16T21:32:49.636 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stdout 2023-12-16T21:32:49.636 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stdout [global] 2023-12-16T21:32:49.636 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stdout fsid = 88ff363c-9c5a-11ee-95a5-87774f69a715 2023-12-16T21:32:49.636 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stdout mon_host = [v2:172.21.15.45:3300,v1:172.21.15.45:6789] 2023-12-16T21:32:49.636 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stdout mon_osd_allow_pg_remap = true 2023-12-16T21:32:49.636 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stdout mon_osd_allow_primary_affinity = true 2023-12-16T21:32:49.636 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stdout mon_warn_on_no_sortbitwise = false 2023-12-16T21:32:49.636 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stdout osd_crush_chooseleaf_type = 0 2023-12-16T21:32:49.636 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stdout 2023-12-16T21:32:49.637 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stdout [mgr] 2023-12-16T21:32:49.637 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stdout mgr/telemetry/nag = false 2023-12-16T21:32:49.637 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stdout 2023-12-16T21:32:49.637 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stdout [osd] 2023-12-16T21:32:49.637 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stdout mon_osd_backfillfull_ratio = 0.85 2023-12-16T21:32:49.637 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stdout mon_osd_full_ratio = 0.9 2023-12-16T21:32:49.637 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stdout mon_osd_nearfull_ratio = 0.8 2023-12-16T21:32:49.637 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stdout osd_map_max_advance = 10 2023-12-16T21:32:49.637 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stdout osd_sloppy_crc = true 2023-12-16T21:32:49.637 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2023-12-16T21:32:49.331+0000 7f4c5d934700 1 Processor -- start 2023-12-16T21:32:49.637 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2023-12-16T21:32:49.331+0000 7f4c5d934700 1 -- start start 2023-12-16T21:32:49.637 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2023-12-16T21:32:49.331+0000 7f4c5d934700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f4c58076c00 0x7f4c58107300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:32:49.637 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2023-12-16T21:32:49.331+0000 7f4c5d934700 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4c58102cf0 con 0x7f4c58076c00 2023-12-16T21:32:49.638 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2023-12-16T21:32:49.331+0000 7f4c56ffd700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f4c58076c00 0x7f4c58107300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:32:49.638 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2023-12-16T21:32:49.331+0000 7f4c56ffd700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f4c58076c00 0x7f4c58107300 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:53070/0 (socket says 172.21.15.45:53070) 2023-12-16T21:32:49.638 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2023-12-16T21:32:49.331+0000 7f4c56ffd700 1 -- 172.21.15.45:0/149202291 learned_addr learned my addr 172.21.15.45:0/149202291 (peer_addr_for_me v2:172.21.15.45:0/0) 2023-12-16T21:32:49.638 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2023-12-16T21:32:49.331+0000 7f4c56ffd700 1 -- 172.21.15.45:0/149202291 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4c58107840 con 0x7f4c58076c00 2023-12-16T21:32:49.638 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2023-12-16T21:32:49.332+0000 7f4c56ffd700 1 --2- 172.21.15.45:0/149202291 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f4c58076c00 0x7f4c58107300 secure :-1 s=READY pgs=3 cs=0 l=1 rev1=1 crypto rx=0x7f4c4c0098e0 tx=0x7f4c4c009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=dc2f92607dab23bd server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:32:49.638 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2023-12-16T21:32:49.332+0000 7f4c55ffb700 1 -- 172.21.15.45:0/149202291 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f4c4c00b860 con 0x7f4c58076c00 2023-12-16T21:32:49.638 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2023-12-16T21:32:49.332+0000 7f4c55ffb700 1 -- 172.21.15.45:0/149202291 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4c4c00b9c0 con 0x7f4c58076c00 2023-12-16T21:32:49.638 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2023-12-16T21:32:49.332+0000 7f4c55ffb700 1 -- 172.21.15.45:0/149202291 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f4c4c00bc90 con 0x7f4c58076c00 2023-12-16T21:32:49.638 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2023-12-16T21:32:49.332+0000 7f4c5d934700 1 -- 172.21.15.45:0/149202291 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f4c58076c00 msgr2=0x7f4c58107300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:32:49.638 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2023-12-16T21:32:49.332+0000 7f4c5d934700 1 --2- 172.21.15.45:0/149202291 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f4c58076c00 0x7f4c58107300 secure :-1 s=READY pgs=3 cs=0 l=1 rev1=1 crypto rx=0x7f4c4c0098e0 tx=0x7f4c4c009250 comp rx=0 tx=0).stop 2023-12-16T21:32:49.638 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2023-12-16T21:32:49.332+0000 7f4c5d934700 1 -- 172.21.15.45:0/149202291 shutdown_connections 2023-12-16T21:32:49.639 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2023-12-16T21:32:49.332+0000 7f4c5d934700 1 --2- 172.21.15.45:0/149202291 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f4c58076c00 0x7f4c58107300 unknown :-1 s=CLOSED pgs=3 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:32:49.639 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2023-12-16T21:32:49.332+0000 7f4c5d934700 1 -- 172.21.15.45:0/149202291 >> 172.21.15.45:0/149202291 conn(0x7f4c580fc950 msgr2=0x7f4c580fed60 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-16T21:32:49.639 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2023-12-16T21:32:49.332+0000 7f4c5d934700 1 -- 172.21.15.45:0/149202291 shutdown_connections 2023-12-16T21:32:49.639 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2023-12-16T21:32:49.332+0000 7f4c5d934700 1 -- 172.21.15.45:0/149202291 wait complete. 2023-12-16T21:32:49.639 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2023-12-16T21:32:49.333+0000 7f4c5d934700 1 Processor -- start 2023-12-16T21:32:49.639 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2023-12-16T21:32:49.333+0000 7f4c5d934700 1 -- start start 2023-12-16T21:32:49.639 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2023-12-16T21:32:49.333+0000 7f4c5d934700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f4c58076c00 0x7f4c58112b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:32:49.639 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2023-12-16T21:32:49.333+0000 7f4c5d934700 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4c581130b0 con 0x7f4c58076c00 2023-12-16T21:32:49.639 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2023-12-16T21:32:49.334+0000 7f4c56ffd700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f4c58076c00 0x7f4c58112b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:32:49.639 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2023-12-16T21:32:49.334+0000 7f4c56ffd700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f4c58076c00 0x7f4c58112b70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:53080/0 (socket says 172.21.15.45:53080) 2023-12-16T21:32:49.639 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2023-12-16T21:32:49.334+0000 7f4c56ffd700 1 -- 172.21.15.45:0/1400334285 learned_addr learned my addr 172.21.15.45:0/1400334285 (peer_addr_for_me v2:172.21.15.45:0/0) 2023-12-16T21:32:49.639 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2023-12-16T21:32:49.334+0000 7f4c56ffd700 1 -- 172.21.15.45:0/1400334285 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4c4c0094f0 con 0x7f4c58076c00 2023-12-16T21:32:49.640 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2023-12-16T21:32:49.334+0000 7f4c56ffd700 1 --2- 172.21.15.45:0/1400334285 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f4c58076c00 0x7f4c58112b70 secure :-1 s=READY pgs=4 cs=0 l=1 rev1=1 crypto rx=0x7f4c4c009290 tx=0x7f4c4c016000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:32:49.640 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2023-12-16T21:32:49.334+0000 7f4c3f7fe700 1 -- 172.21.15.45:0/1400334285 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f4c4c01c5c0 con 0x7f4c58076c00 2023-12-16T21:32:49.640 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2023-12-16T21:32:49.334+0000 7f4c3f7fe700 1 -- 172.21.15.45:0/1400334285 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4c4c01c740 con 0x7f4c58076c00 2023-12-16T21:32:49.640 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2023-12-16T21:32:49.334+0000 7f4c5d934700 1 -- 172.21.15.45:0/1400334285 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4c581132b0 con 0x7f4c58076c00 2023-12-16T21:32:49.640 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2023-12-16T21:32:49.334+0000 7f4c3f7fe700 1 -- 172.21.15.45:0/1400334285 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f4c4c01ca50 con 0x7f4c58076c00 2023-12-16T21:32:49.640 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2023-12-16T21:32:49.334+0000 7f4c5d934700 1 -- 172.21.15.45:0/1400334285 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4c581136c0 con 0x7f4c58076c00 2023-12-16T21:32:49.640 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2023-12-16T21:32:49.334+0000 7f4c3f7fe700 1 -- 172.21.15.45:0/1400334285 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 1) v1 ==== 521+0+0 (secure 0 0 0) 0x7f4c4c01a450 con 0x7f4c58076c00 2023-12-16T21:32:49.640 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2023-12-16T21:32:49.334+0000 7f4c3f7fe700 1 -- 172.21.15.45:0/1400334285 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 721+0+0 (secure 0 0 0) 0x7f4c4c02e790 con 0x7f4c58076c00 2023-12-16T21:32:49.640 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2023-12-16T21:32:49.335+0000 7f4c5d934700 1 -- 172.21.15.45:0/1400334285 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4c5804bb80 con 0x7f4c58076c00 2023-12-16T21:32:49.641 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2023-12-16T21:32:49.338+0000 7f4c3f7fe700 1 -- 172.21.15.45:0/1400334285 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+71349 (secure 0 0 0) 0x7f4c4c01ce30 con 0x7f4c58076c00 2023-12-16T21:32:49.641 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2023-12-16T21:32:49.409+0000 7f4c5d934700 1 -- 172.21.15.45:0/1400334285 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "config assimilate-conf"} v 0) v1 -- 0x7f4c5810c2c0 con 0x7f4c58076c00 2023-12-16T21:32:49.641 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2023-12-16T21:32:49.413+0000 7f4c3f7fe700 1 -- 172.21.15.45:0/1400334285 <== mon.0 v2:172.21.15.45:3300/0 7 ==== config(26 keys) v1 ==== 1009+0+0 (secure 0 0 0) 0x7f4c4c0243e0 con 0x7f4c58076c00 2023-12-16T21:32:49.641 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2023-12-16T21:32:49.413+0000 7f4c3f7fe700 1 -- 172.21.15.45:0/1400334285 <== mon.0 v2:172.21.15.45:3300/0 8 ==== mon_command_ack([{"prefix": "config assimilate-conf"}]=0 v2) v1 ==== 70+0+429 (secure 0 0 0) 0x7f4c4c02e9f0 con 0x7f4c58076c00 2023-12-16T21:32:49.641 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2023-12-16T21:32:49.414+0000 7f4c5d934700 1 -- 172.21.15.45:0/1400334285 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f4c58076c00 msgr2=0x7f4c58112b70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:32:49.641 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2023-12-16T21:32:49.414+0000 7f4c5d934700 1 --2- 172.21.15.45:0/1400334285 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f4c58076c00 0x7f4c58112b70 secure :-1 s=READY pgs=4 cs=0 l=1 rev1=1 crypto rx=0x7f4c4c009290 tx=0x7f4c4c016000 comp rx=0 tx=0).stop 2023-12-16T21:32:49.641 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2023-12-16T21:32:49.414+0000 7f4c5d934700 1 -- 172.21.15.45:0/1400334285 shutdown_connections 2023-12-16T21:32:49.641 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2023-12-16T21:32:49.414+0000 7f4c5d934700 1 --2- 172.21.15.45:0/1400334285 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f4c58076c00 0x7f4c58112b70 unknown :-1 s=CLOSED pgs=4 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:32:49.641 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2023-12-16T21:32:49.414+0000 7f4c5d934700 1 -- 172.21.15.45:0/1400334285 >> 172.21.15.45:0/1400334285 conn(0x7f4c580fc950 msgr2=0x7f4c580fe3f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-16T21:32:49.641 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2023-12-16T21:32:49.414+0000 7f4c5d934700 1 -- 172.21.15.45:0/1400334285 shutdown_connections 2023-12-16T21:32:49.641 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2023-12-16T21:32:49.414+0000 7f4c5d934700 1 -- 172.21.15.45:0/1400334285 wait complete. 2023-12-16T21:32:49.641 INFO:teuthology.orchestra.run.smithi045.stdout:Generating new minimal ceph.conf... 2023-12-16T21:32:50.149 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2023-12-16T21:32:49.854+0000 7fb47f7e4700 1 Processor -- start 2023-12-16T21:32:50.149 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2023-12-16T21:32:49.854+0000 7fb47f7e4700 1 -- start start 2023-12-16T21:32:50.149 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2023-12-16T21:32:49.855+0000 7fb47f7e4700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fb4781011d0 0x7fb4781015e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:32:50.149 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2023-12-16T21:32:49.855+0000 7fb47f7e4700 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb478101b20 con 0x7fb4781011d0 2023-12-16T21:32:50.149 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2023-12-16T21:32:49.855+0000 7fb47d580700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fb4781011d0 0x7fb4781015e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:32:50.149 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2023-12-16T21:32:49.855+0000 7fb47d580700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fb4781011d0 0x7fb4781015e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:53092/0 (socket says 172.21.15.45:53092) 2023-12-16T21:32:50.149 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2023-12-16T21:32:49.855+0000 7fb47d580700 1 -- 172.21.15.45:0/2192299118 learned_addr learned my addr 172.21.15.45:0/2192299118 (peer_addr_for_me v2:172.21.15.45:0/0) 2023-12-16T21:32:50.149 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2023-12-16T21:32:49.855+0000 7fb47d580700 1 -- 172.21.15.45:0/2192299118 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb478101c60 con 0x7fb4781011d0 2023-12-16T21:32:50.149 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2023-12-16T21:32:49.855+0000 7fb47d580700 1 --2- 172.21.15.45:0/2192299118 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fb4781011d0 0x7fb4781015e0 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 crypto rx=0x7fb46800d0d0 tx=0x7fb46800ba50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b12e4225e33e67c server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:32:50.149 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2023-12-16T21:32:49.855+0000 7fb477fff700 1 -- 172.21.15.45:0/2192299118 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fb46800ddf0 con 0x7fb4781011d0 2023-12-16T21:32:50.150 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2023-12-16T21:32:49.855+0000 7fb477fff700 1 -- 172.21.15.45:0/2192299118 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(26 keys) v1 ==== 1009+0+0 (secure 0 0 0) 0x7fb468009610 con 0x7fb4781011d0 2023-12-16T21:32:50.150 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2023-12-16T21:32:49.855+0000 7fb477fff700 1 -- 172.21.15.45:0/2192299118 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fb4680145b0 con 0x7fb4781011d0 2023-12-16T21:32:50.150 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2023-12-16T21:32:49.856+0000 7fb47f7e4700 1 -- 172.21.15.45:0/2192299118 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fb4781011d0 msgr2=0x7fb4781015e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:32:50.150 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2023-12-16T21:32:49.856+0000 7fb47f7e4700 1 --2- 172.21.15.45:0/2192299118 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fb4781011d0 0x7fb4781015e0 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 crypto rx=0x7fb46800d0d0 tx=0x7fb46800ba50 comp rx=0 tx=0).stop 2023-12-16T21:32:50.150 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2023-12-16T21:32:49.856+0000 7fb47f7e4700 1 -- 172.21.15.45:0/2192299118 shutdown_connections 2023-12-16T21:32:50.150 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2023-12-16T21:32:49.856+0000 7fb47f7e4700 1 --2- 172.21.15.45:0/2192299118 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fb4781011d0 0x7fb4781015e0 unknown :-1 s=CLOSED pgs=5 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:32:50.150 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2023-12-16T21:32:49.856+0000 7fb47f7e4700 1 -- 172.21.15.45:0/2192299118 >> 172.21.15.45:0/2192299118 conn(0x7fb4780fc970 msgr2=0x7fb4780fed80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-16T21:32:50.150 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2023-12-16T21:32:49.856+0000 7fb47f7e4700 1 -- 172.21.15.45:0/2192299118 shutdown_connections 2023-12-16T21:32:50.150 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2023-12-16T21:32:49.856+0000 7fb47f7e4700 1 -- 172.21.15.45:0/2192299118 wait complete. 2023-12-16T21:32:50.150 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2023-12-16T21:32:49.856+0000 7fb47f7e4700 1 Processor -- start 2023-12-16T21:32:50.150 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2023-12-16T21:32:49.857+0000 7fb47f7e4700 1 -- start start 2023-12-16T21:32:50.150 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2023-12-16T21:32:49.857+0000 7fb47f7e4700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fb4781011d0 0x7fb47810efa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:32:50.150 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2023-12-16T21:32:49.857+0000 7fb47f7e4700 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb47810f4e0 con 0x7fb4781011d0 2023-12-16T21:32:50.150 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2023-12-16T21:32:49.857+0000 7fb47d580700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fb4781011d0 0x7fb47810efa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:32:50.151 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2023-12-16T21:32:49.857+0000 7fb47d580700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fb4781011d0 0x7fb47810efa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:53102/0 (socket says 172.21.15.45:53102) 2023-12-16T21:32:50.151 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2023-12-16T21:32:49.857+0000 7fb47d580700 1 -- 172.21.15.45:0/198896985 learned_addr learned my addr 172.21.15.45:0/198896985 (peer_addr_for_me v2:172.21.15.45:0/0) 2023-12-16T21:32:50.151 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2023-12-16T21:32:49.857+0000 7fb47d580700 1 -- 172.21.15.45:0/198896985 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb46800bcf0 con 0x7fb4781011d0 2023-12-16T21:32:50.151 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2023-12-16T21:32:49.857+0000 7fb47d580700 1 --2- 172.21.15.45:0/198896985 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fb4781011d0 0x7fb47810efa0 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7fb468009e00 tx=0x7fb46800ba50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:32:50.151 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2023-12-16T21:32:49.857+0000 7fb475ffb700 1 -- 172.21.15.45:0/198896985 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fb468014430 con 0x7fb4781011d0 2023-12-16T21:32:50.151 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2023-12-16T21:32:49.857+0000 7fb475ffb700 1 -- 172.21.15.45:0/198896985 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(26 keys) v1 ==== 1009+0+0 (secure 0 0 0) 0x7fb468014a00 con 0x7fb4781011d0 2023-12-16T21:32:50.151 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2023-12-16T21:32:49.857+0000 7fb47f7e4700 1 -- 172.21.15.45:0/198896985 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb47810f6e0 con 0x7fb4781011d0 2023-12-16T21:32:50.151 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2023-12-16T21:32:49.857+0000 7fb475ffb700 1 -- 172.21.15.45:0/198896985 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fb46801d910 con 0x7fb4781011d0 2023-12-16T21:32:50.151 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2023-12-16T21:32:49.858+0000 7fb47f7e4700 1 -- 172.21.15.45:0/198896985 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb478110380 con 0x7fb4781011d0 2023-12-16T21:32:50.151 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2023-12-16T21:32:49.858+0000 7fb475ffb700 1 -- 172.21.15.45:0/198896985 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 1) v1 ==== 521+0+0 (secure 0 0 0) 0x7fb46801b070 con 0x7fb4781011d0 2023-12-16T21:32:50.151 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2023-12-16T21:32:49.858+0000 7fb475ffb700 1 -- 172.21.15.45:0/198896985 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 721+0+0 (secure 0 0 0) 0x7fb4680267a0 con 0x7fb4781011d0 2023-12-16T21:32:50.151 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2023-12-16T21:32:49.859+0000 7fb47f7e4700 1 -- 172.21.15.45:0/198896985 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb47805f270 con 0x7fb4781011d0 2023-12-16T21:32:50.151 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2023-12-16T21:32:49.861+0000 7fb475ffb700 1 -- 172.21.15.45:0/198896985 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+71349 (secure 0 0 0) 0x7fb468021020 con 0x7fb4781011d0 2023-12-16T21:32:50.151 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2023-12-16T21:32:49.931+0000 7fb47f7e4700 1 -- 172.21.15.45:0/198896985 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "config generate-minimal-conf"} v 0) v1 -- 0x7fb478102fe0 con 0x7fb4781011d0 2023-12-16T21:32:50.152 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2023-12-16T21:32:49.931+0000 7fb475ffb700 1 -- 172.21.15.45:0/198896985 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mon_command_ack([{"prefix": "config generate-minimal-conf"}]=0 v2) v1 ==== 76+0+175 (secure 0 0 0) 0x7fb468034020 con 0x7fb4781011d0 2023-12-16T21:32:50.152 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2023-12-16T21:32:49.932+0000 7fb47f7e4700 1 -- 172.21.15.45:0/198896985 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fb4781011d0 msgr2=0x7fb47810efa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:32:50.152 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2023-12-16T21:32:49.932+0000 7fb47f7e4700 1 --2- 172.21.15.45:0/198896985 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fb4781011d0 0x7fb47810efa0 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7fb468009e00 tx=0x7fb46800ba50 comp rx=0 tx=0).stop 2023-12-16T21:32:50.152 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2023-12-16T21:32:49.932+0000 7fb47f7e4700 1 -- 172.21.15.45:0/198896985 shutdown_connections 2023-12-16T21:32:50.152 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2023-12-16T21:32:49.932+0000 7fb47f7e4700 1 --2- 172.21.15.45:0/198896985 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fb4781011d0 0x7fb47810efa0 unknown :-1 s=CLOSED pgs=6 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:32:50.152 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2023-12-16T21:32:49.932+0000 7fb47f7e4700 1 -- 172.21.15.45:0/198896985 >> 172.21.15.45:0/198896985 conn(0x7fb4780fc970 msgr2=0x7fb478105f40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-16T21:32:50.152 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2023-12-16T21:32:49.932+0000 7fb47f7e4700 1 -- 172.21.15.45:0/198896985 shutdown_connections 2023-12-16T21:32:50.152 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2023-12-16T21:32:49.932+0000 7fb47f7e4700 1 -- 172.21.15.45:0/198896985 wait complete. 2023-12-16T21:32:50.152 INFO:teuthology.orchestra.run.smithi045.stdout:Restarting the monitor... 2023-12-16T21:32:50.315 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Dec 16 21:32:50 smithi045 systemd[1]: Stopping Ceph mon.smithi045 for 88ff363c-9c5a-11ee-95a5-87774f69a715... 2023-12-16T21:32:50.601 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Dec 16 21:32:50 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-mon-smithi045[80386]: 2023-12-16T21:32:50.342+0000 7f4443a64700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mon -n mon.smithi045 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-journald=true --default-mon-cluster-log-to-stderr=false (PID: 1) UID: 0 2023-12-16T21:32:50.601 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Dec 16 21:32:50 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-mon-smithi045[80386]: 2023-12-16T21:32:50.342+0000 7f4443a64700 -1 mon.smithi045@0(leader) e1 *** Got Signal Terminated *** 2023-12-16T21:32:50.947 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Dec 16 21:32:50 smithi045 podman[80635]: 2023-12-16 21:32:50.59955104 +0000 UTC m=+0.303793749 container died aad8ef6c9b6482c13517d7e92937ffff644131111bf65441c47f574d05049177 (image=quay.ceph.io/ceph-ci/ceph:quincy, name=ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-mon-smithi045, GIT_COMMIT=0396eef90bef641b676c164ec7a3876f45010308, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, ceph=True, CEPH_POINT_RELEASE=, RELEASE=quincy-1f2f551, org.label-schema.build-date=20231212, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , io.buildah.version=1.31.3) 2023-12-16T21:32:50.948 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Dec 16 21:32:50 smithi045 podman[80635]: 2023-12-16 21:32:50.681910796 +0000 UTC m=+0.386153449 container cleanup aad8ef6c9b6482c13517d7e92937ffff644131111bf65441c47f574d05049177 (image=quay.ceph.io/ceph-ci/ceph:quincy, name=ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-mon-smithi045, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=quincy-1f2f551, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=0396eef90bef641b676c164ec7a3876f45010308, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, io.buildah.version=1.31.3, org.label-schema.build-date=20231212, GIT_BRANCH=HEAD, GIT_CLEAN=True) 2023-12-16T21:32:50.948 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Dec 16 21:32:50 smithi045 bash[80635]: ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-mon-smithi045 2023-12-16T21:32:50.948 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Dec 16 21:32:50 smithi045 podman[80649]: 2023-12-16 21:32:50.790212691 +0000 UTC m=+0.185268326 container remove aad8ef6c9b6482c13517d7e92937ffff644131111bf65441c47f574d05049177 (image=quay.ceph.io/ceph-ci/ceph:quincy, name=ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-mon-smithi045, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20231212, GIT_CLEAN=True, RELEASE=quincy-1f2f551, org.label-schema.schema-version=1.0, io.buildah.version=1.31.3, ceph=True, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, GIT_COMMIT=0396eef90bef641b676c164ec7a3876f45010308) 2023-12-16T21:32:51.219 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Dec 16 21:32:50 smithi045 systemd[1]: ceph-88ff363c-9c5a-11ee-95a5-87774f69a715@mon.smithi045.service: Deactivated successfully. 2023-12-16T21:32:51.219 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Dec 16 21:32:50 smithi045 systemd[1]: Stopped Ceph mon.smithi045 for 88ff363c-9c5a-11ee-95a5-87774f69a715. 2023-12-16T21:32:51.220 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Dec 16 21:32:50 smithi045 systemd[1]: Starting Ceph mon.smithi045 for 88ff363c-9c5a-11ee-95a5-87774f69a715... 2023-12-16T21:32:51.220 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Dec 16 21:32:51 smithi045 podman[80744]: 2023-12-16 21:32:51.098430423 +0000 UTC m=+0.050058628 container create 1cddbf5059f8aabb81ecf60d6ebf743c70edae445f2da2f20f0280c5672e87b1 (image=quay.ceph.io/ceph-ci/ceph:quincy, name=ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-mon-smithi045, ceph=True, io.buildah.version=1.31.3, maintainer=Guillaume Abrioux , RELEASE=quincy-1f2f551, GIT_COMMIT=0396eef90bef641b676c164ec7a3876f45010308, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20231212, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2) 2023-12-16T21:32:51.220 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Dec 16 21:32:51 smithi045 podman[80744]: 2023-12-16 21:32:51.066350142 +0000 UTC m=+0.017978355 image pull 60cc83a9c8f10fdde41a08d59bce905c1a01fc7389b91726758c2505caa3f0c5 quay.ceph.io/ceph-ci/ceph:quincy 2023-12-16T21:32:51.220 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Dec 16 21:32:51 smithi045 podman[80744]: 2023-12-16 21:32:51.176319072 +0000 UTC m=+0.127947285 container init 1cddbf5059f8aabb81ecf60d6ebf743c70edae445f2da2f20f0280c5672e87b1 (image=quay.ceph.io/ceph-ci/ceph:quincy, name=ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-mon-smithi045, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=quincy-1f2f551, GIT_COMMIT=0396eef90bef641b676c164ec7a3876f45010308, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20231212, org.label-schema.license=GPLv2, io.buildah.version=1.31.3, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, ceph=True, org.label-schema.schema-version=1.0, GIT_CLEAN=True) 2023-12-16T21:32:51.220 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Dec 16 21:32:51 smithi045 podman[80744]: 2023-12-16 21:32:51.197452573 +0000 UTC m=+0.149080779 container start 1cddbf5059f8aabb81ecf60d6ebf743c70edae445f2da2f20f0280c5672e87b1 (image=quay.ceph.io/ceph-ci/ceph:quincy, name=ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-mon-smithi045, ceph=True, maintainer=Guillaume Abrioux , RELEASE=quincy-1f2f551, GIT_COMMIT=0396eef90bef641b676c164ec7a3876f45010308, org.label-schema.build-date=20231212, GIT_CLEAN=True, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.31.3, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD) 2023-12-16T21:32:51.220 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Dec 16 21:32:51 smithi045 ceph-mon[80763]: set uid:gid to 167:167 (ceph:ceph) 2023-12-16T21:32:51.222 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Dec 16 21:32:51 smithi045 ceph-mon[80763]: ceph version 17.2.7-125-g1f2f5510 (1f2f55109dc1ed2dc2dda9de1db3c70923ba0dfd) quincy (stable), process ceph-mon, pid 2 2023-12-16T21:32:51.236 INFO:teuthology.orchestra.run.smithi045.stdout:Setting mon public_network to 172.21.0.0/20 2023-12-16T21:32:51.478 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Dec 16 21:32:51 smithi045 ceph-mon[80763]: pidfile_write: ignore empty --pid-file 2023-12-16T21:32:51.478 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Dec 16 21:32:51 smithi045 ceph-mon[80763]: load: jerasure load: lrc 2023-12-16T21:32:51.478 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Dec 16 21:32:51 smithi045 ceph-mon[80763]: rocksdb: RocksDB version: 6.15.5 2023-12-16T21:32:51.479 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Dec 16 21:32:51 smithi045 ceph-mon[80763]: rocksdb: Git sha rocksdb_build_git_sha:@0@ 2023-12-16T21:32:51.479 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Dec 16 21:32:51 smithi045 ceph-mon[80763]: rocksdb: Compile date Dec 13 2023 2023-12-16T21:32:51.479 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Dec 16 21:32:51 smithi045 ceph-mon[80763]: rocksdb: DB SUMMARY 2023-12-16T21:32:51.479 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Dec 16 21:32:51 smithi045 ceph-mon[80763]: rocksdb: DB Session ID: 11B99WTFFUGGUGF4CYCV 2023-12-16T21:32:51.479 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Dec 16 21:32:51 smithi045 ceph-mon[80763]: rocksdb: CURRENT file: CURRENT 2023-12-16T21:32:51.479 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Dec 16 21:32:51 smithi045 ceph-mon[80763]: rocksdb: IDENTITY file: IDENTITY 2023-12-16T21:32:51.479 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Dec 16 21:32:51 smithi045 ceph-mon[80763]: rocksdb: MANIFEST file: MANIFEST-000009 size: 131 Bytes 2023-12-16T21:32:51.479 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Dec 16 21:32:51 smithi045 ceph-mon[80763]: rocksdb: SST files in /var/lib/ceph/mon/ceph-smithi045/store.db dir, Total Num: 1, files: 000008.sst 2023-12-16T21:32:51.479 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Dec 16 21:32:51 smithi045 bash[80744]: 1cddbf5059f8aabb81ecf60d6ebf743c70edae445f2da2f20f0280c5672e87b1 2023-12-16T21:32:51.479 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Dec 16 21:32:51 smithi045 ceph-mon[80763]: rocksdb: Write Ahead Log file in /var/lib/ceph/mon/ceph-smithi045/store.db: 000010.log size: 95072 ; 2023-12-16T21:32:51.479 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Dec 16 21:32:51 smithi045 ceph-mon[80763]: rocksdb: Options.error_if_exists: 0 2023-12-16T21:32:51.479 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Dec 16 21:32:51 smithi045 ceph-mon[80763]: rocksdb: Options.create_if_missing: 0 2023-12-16T21:32:51.479 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Dec 16 21:32:51 smithi045 ceph-mon[80763]: rocksdb: Options.paranoid_checks: 1 2023-12-16T21:32:51.479 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Dec 16 21:32:51 smithi045 ceph-mon[80763]: rocksdb: Options.track_and_verify_wals_in_manifest: 0 2023-12-16T21:32:51.480 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Dec 16 21:32:51 smithi045 ceph-mon[80763]: rocksdb: Options.env: 0x5620210d0a80 2023-12-16T21:32:51.480 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Dec 16 21:32:51 smithi045 ceph-mon[80763]: rocksdb: Options.fs: Posix File System 2023-12-16T21:32:51.480 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Dec 16 21:32:51 smithi045 ceph-mon[80763]: rocksdb: Options.info_log: 0x562022a01000 2023-12-16T21:32:51.480 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Dec 16 21:32:51 smithi045 ceph-mon[80763]: rocksdb: Options.max_file_opening_threads: 16 2023-12-16T21:32:51.480 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Dec 16 21:32:51 smithi045 ceph-mon[80763]: rocksdb: Options.statistics: (nil) 2023-12-16T21:32:51.480 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Dec 16 21:32:51 smithi045 ceph-mon[80763]: rocksdb: Options.use_fsync: 0 2023-12-16T21:32:51.480 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Dec 16 21:32:51 smithi045 ceph-mon[80763]: rocksdb: Options.max_log_file_size: 0 2023-12-16T21:32:51.480 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Dec 16 21:32:51 smithi045 ceph-mon[80763]: rocksdb: Options.max_manifest_file_size: 1073741824 2023-12-16T21:32:51.480 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Dec 16 21:32:51 smithi045 ceph-mon[80763]: rocksdb: Options.log_file_time_to_roll: 0 2023-12-16T21:32:51.480 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Dec 16 21:32:51 smithi045 ceph-mon[80763]: rocksdb: Options.keep_log_file_num: 1000 2023-12-16T21:32:51.480 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Dec 16 21:32:51 smithi045 ceph-mon[80763]: rocksdb: Options.recycle_log_file_num: 0 2023-12-16T21:32:51.480 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Dec 16 21:32:51 smithi045 ceph-mon[80763]: rocksdb: Options.allow_fallocate: 1 2023-12-16T21:32:51.480 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Dec 16 21:32:51 smithi045 ceph-mon[80763]: rocksdb: Options.allow_mmap_reads: 0 2023-12-16T21:32:51.480 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Dec 16 21:32:51 smithi045 ceph-mon[80763]: rocksdb: Options.allow_mmap_writes: 0 2023-12-16T21:32:51.481 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Dec 16 21:32:51 smithi045 ceph-mon[80763]: rocksdb: Options.use_direct_reads: 0 2023-12-16T21:32:51.481 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Dec 16 21:32:51 smithi045 ceph-mon[80763]: rocksdb: Options.use_direct_io_for_flush_and_compaction: 0 2023-12-16T21:32:51.481 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Dec 16 21:32:51 smithi045 ceph-mon[80763]: rocksdb: Options.create_missing_column_families: 0 2023-12-16T21:32:51.481 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Dec 16 21:32:51 smithi045 ceph-mon[80763]: rocksdb: Options.db_log_dir: 2023-12-16T21:32:51.481 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Dec 16 21:32:51 smithi045 ceph-mon[80763]: rocksdb: Options.wal_dir: /var/lib/ceph/mon/ceph-smithi045/store.db 2023-12-16T21:32:51.481 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Dec 16 21:32:51 smithi045 ceph-mon[80763]: rocksdb: Options.table_cache_numshardbits: 6 2023-12-16T21:32:51.481 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Dec 16 21:32:51 smithi045 ceph-mon[80763]: rocksdb: Options.WAL_ttl_seconds: 0 2023-12-16T21:32:51.481 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Dec 16 21:32:51 smithi045 ceph-mon[80763]: rocksdb: Options.WAL_size_limit_MB: 0 2023-12-16T21:32:51.482 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Dec 16 21:32:51 smithi045 ceph-mon[80763]: rocksdb: Options.max_write_batch_group_size_bytes: 1048576 2023-12-16T21:32:51.482 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Dec 16 21:32:51 smithi045 ceph-mon[80763]: rocksdb: Options.manifest_preallocation_size: 4194304 2023-12-16T21:32:51.482 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Dec 16 21:32:51 smithi045 ceph-mon[80763]: rocksdb: Options.is_fd_close_on_exec: 1 2023-12-16T21:32:51.483 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Dec 16 21:32:51 smithi045 ceph-mon[80763]: rocksdb: Options.advise_random_on_open: 1 2023-12-16T21:32:51.483 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Dec 16 21:32:51 smithi045 ceph-mon[80763]: rocksdb: Options.db_write_buffer_size: 0 2023-12-16T21:32:51.483 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Dec 16 21:32:51 smithi045 ceph-mon[80763]: rocksdb: Options.write_buffer_manager: 0x5620237769f0 2023-12-16T21:32:51.483 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Dec 16 21:32:51 smithi045 ceph-mon[80763]: rocksdb: Options.access_hint_on_compaction_start: 1 2023-12-16T21:32:51.483 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Dec 16 21:32:51 smithi045 ceph-mon[80763]: rocksdb: Options.new_table_reader_for_compaction_inputs: 0 2023-12-16T21:32:51.483 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Dec 16 21:32:51 smithi045 ceph-mon[80763]: rocksdb: Options.random_access_max_buffer_size: 1048576 2023-12-16T21:32:51.483 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Dec 16 21:32:51 smithi045 ceph-mon[80763]: rocksdb: Options.use_adaptive_mutex: 0 2023-12-16T21:32:51.483 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Dec 16 21:32:51 smithi045 ceph-mon[80763]: rocksdb: Options.rate_limiter: (nil) 2023-12-16T21:32:51.483 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Dec 16 21:32:51 smithi045 ceph-mon[80763]: rocksdb: Options.sst_file_manager.rate_bytes_per_sec: 0 2023-12-16T21:32:51.483 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Dec 16 21:32:51 smithi045 ceph-mon[80763]: rocksdb: Options.wal_recovery_mode: 2 2023-12-16T21:32:51.483 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Dec 16 21:32:51 smithi045 ceph-mon[80763]: rocksdb: Options.enable_thread_tracking: 0 2023-12-16T21:32:51.483 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Dec 16 21:32:51 smithi045 ceph-mon[80763]: rocksdb: Options.enable_pipelined_write: 0 2023-12-16T21:32:51.483 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Dec 16 21:32:51 smithi045 ceph-mon[80763]: rocksdb: Options.unordered_write: 0 2023-12-16T21:32:51.483 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Dec 16 21:32:51 smithi045 ceph-mon[80763]: rocksdb: Options.allow_concurrent_memtable_write: 1 2023-12-16T21:32:51.484 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Dec 16 21:32:51 smithi045 ceph-mon[80763]: rocksdb: Options.enable_write_thread_adaptive_yield: 1 2023-12-16T21:32:51.484 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Dec 16 21:32:51 smithi045 ceph-mon[80763]: rocksdb: Options.write_thread_max_yield_usec: 100 2023-12-16T21:32:51.484 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Dec 16 21:32:51 smithi045 ceph-mon[80763]: rocksdb: Options.write_thread_slow_yield_usec: 3 2023-12-16T21:32:51.484 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Dec 16 21:32:51 smithi045 ceph-mon[80763]: rocksdb: Options.row_cache: None 2023-12-16T21:32:51.484 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Dec 16 21:32:51 smithi045 ceph-mon[80763]: rocksdb: Options.wal_filter: None 2023-12-16T21:32:51.484 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Dec 16 21:32:51 smithi045 ceph-mon[80763]: rocksdb: Options.avoid_flush_during_recovery: 0 2023-12-16T21:32:51.484 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Dec 16 21:32:51 smithi045 ceph-mon[80763]: rocksdb: Options.allow_ingest_behind: 0 2023-12-16T21:32:51.484 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Dec 16 21:32:51 smithi045 ceph-mon[80763]: rocksdb: Options.preserve_deletes: 0 2023-12-16T21:32:51.484 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Dec 16 21:32:51 smithi045 ceph-mon[80763]: rocksdb: Options.two_write_queues: 0 2023-12-16T21:32:51.484 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Dec 16 21:32:51 smithi045 ceph-mon[80763]: rocksdb: Options.manual_wal_flush: 0 2023-12-16T21:32:51.484 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Dec 16 21:32:51 smithi045 ceph-mon[80763]: rocksdb: Options.atomic_flush: 0 2023-12-16T21:32:51.484 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Dec 16 21:32:51 smithi045 ceph-mon[80763]: rocksdb: Options.avoid_unnecessary_blocking_io: 0 2023-12-16T21:32:51.484 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Dec 16 21:32:51 smithi045 ceph-mon[80763]: rocksdb: Options.persist_stats_to_disk: 0 2023-12-16T21:32:51.484 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Dec 16 21:32:51 smithi045 ceph-mon[80763]: rocksdb: Options.write_dbid_to_manifest: 0 2023-12-16T21:32:51.484 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Dec 16 21:32:51 smithi045 ceph-mon[80763]: rocksdb: Options.log_readahead_size: 0 2023-12-16T21:32:51.484 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Dec 16 21:32:51 smithi045 ceph-mon[80763]: rocksdb: Options.file_checksum_gen_factory: Unknown 2023-12-16T21:32:51.485 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Dec 16 21:32:51 smithi045 ceph-mon[80763]: rocksdb: Options.best_efforts_recovery: 0 2023-12-16T21:32:51.485 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Dec 16 21:32:51 smithi045 ceph-mon[80763]: rocksdb: Options.max_bgerror_resume_count: 2147483647 2023-12-16T21:32:51.485 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Dec 16 21:32:51 smithi045 ceph-mon[80763]: rocksdb: Options.bgerror_resume_retry_interval: 1000000 2023-12-16T21:32:51.485 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Dec 16 21:32:51 smithi045 ceph-mon[80763]: rocksdb: Options.allow_data_in_errors: 0 2023-12-16T21:32:51.485 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Dec 16 21:32:51 smithi045 ceph-mon[80763]: rocksdb: Options.db_host_id: __hostname__ 2023-12-16T21:32:51.485 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Dec 16 21:32:51 smithi045 ceph-mon[80763]: rocksdb: Options.max_background_jobs: 2 2023-12-16T21:32:51.485 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Dec 16 21:32:51 smithi045 ceph-mon[80763]: rocksdb: Options.max_background_compactions: -1 2023-12-16T21:32:51.485 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Dec 16 21:32:51 smithi045 ceph-mon[80763]: rocksdb: Options.max_subcompactions: 1 2023-12-16T21:32:51.485 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Dec 16 21:32:51 smithi045 ceph-mon[80763]: rocksdb: Options.avoid_flush_during_shutdown: 0 2023-12-16T21:32:51.485 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Dec 16 21:32:51 smithi045 ceph-mon[80763]: rocksdb: Options.writable_file_max_buffer_size: 1048576 2023-12-16T21:32:51.485 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Dec 16 21:32:51 smithi045 ceph-mon[80763]: rocksdb: Options.delayed_write_rate : 16777216 2023-12-16T21:32:51.485 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Dec 16 21:32:51 smithi045 ceph-mon[80763]: rocksdb: Options.max_total_wal_size: 0 2023-12-16T21:32:51.486 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Dec 16 21:32:51 smithi045 ceph-mon[80763]: rocksdb: Options.delete_obsolete_files_period_micros: 21600000000 2023-12-16T21:32:51.486 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Dec 16 21:32:51 smithi045 ceph-mon[80763]: rocksdb: Options.stats_dump_period_sec: 600 2023-12-16T21:32:51.486 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Dec 16 21:32:51 smithi045 ceph-mon[80763]: rocksdb: Options.stats_persist_period_sec: 600 2023-12-16T21:32:51.486 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Dec 16 21:32:51 smithi045 ceph-mon[80763]: rocksdb: Options.stats_history_buffer_size: 1048576 2023-12-16T21:32:51.486 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Dec 16 21:32:51 smithi045 ceph-mon[80763]: rocksdb: Options.max_open_files: -1 2023-12-16T21:32:51.486 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Dec 16 21:32:51 smithi045 ceph-mon[80763]: rocksdb: Options.bytes_per_sync: 0 2023-12-16T21:32:51.486 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Dec 16 21:32:51 smithi045 ceph-mon[80763]: rocksdb: Options.wal_bytes_per_sync: 0 2023-12-16T21:32:51.486 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Dec 16 21:32:51 smithi045 ceph-mon[80763]: rocksdb: Options.strict_bytes_per_sync: 0 2023-12-16T21:32:51.486 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Dec 16 21:32:51 smithi045 ceph-mon[80763]: rocksdb: Options.compaction_readahead_size: 0 2023-12-16T21:32:51.486 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Dec 16 21:32:51 smithi045 ceph-mon[80763]: rocksdb: Options.max_background_flushes: -1 2023-12-16T21:32:51.486 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Dec 16 21:32:51 smithi045 ceph-mon[80763]: rocksdb: Compression algorithms supported: 2023-12-16T21:32:51.486 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Dec 16 21:32:51 smithi045 ceph-mon[80763]: rocksdb: kZSTDNotFinalCompression supported: 0 2023-12-16T21:32:51.486 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Dec 16 21:32:51 smithi045 ceph-mon[80763]: rocksdb: kZSTD supported: 0 2023-12-16T21:32:51.486 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Dec 16 21:32:51 smithi045 ceph-mon[80763]: rocksdb: kXpressCompression supported: 0 2023-12-16T21:32:51.486 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Dec 16 21:32:51 smithi045 ceph-mon[80763]: rocksdb: kLZ4HCCompression supported: 1 2023-12-16T21:32:51.487 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Dec 16 21:32:51 smithi045 ceph-mon[80763]: rocksdb: kZlibCompression supported: 1 2023-12-16T21:32:51.487 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Dec 16 21:32:51 smithi045 ceph-mon[80763]: rocksdb: kSnappyCompression supported: 1 2023-12-16T21:32:51.487 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Dec 16 21:32:51 smithi045 ceph-mon[80763]: rocksdb: kLZ4Compression supported: 1 2023-12-16T21:32:51.487 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Dec 16 21:32:51 smithi045 ceph-mon[80763]: rocksdb: kBZip2Compression supported: 0 2023-12-16T21:32:51.487 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Dec 16 21:32:51 smithi045 ceph-mon[80763]: rocksdb: Fast CRC32 supported: Supported on x86 2023-12-16T21:32:51.487 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Dec 16 21:32:51 smithi045 ceph-mon[80763]: rocksdb: [db/version_set.cc:4724] Recovering from manifest file: /var/lib/ceph/mon/ceph-smithi045/store.db/MANIFEST-000009 2023-12-16T21:32:51.487 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Dec 16 21:32:51 smithi045 ceph-mon[80763]: rocksdb: [db/column_family.cc:595] --------------- Options for column family [default]: 2023-12-16T21:32:51.487 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Dec 16 21:32:51 smithi045 ceph-mon[80763]: rocksdb: Options.comparator: leveldb.BytewiseComparator 2023-12-16T21:32:51.487 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Dec 16 21:32:51 smithi045 ceph-mon[80763]: rocksdb: Options.merge_operator: 2023-12-16T21:32:51.487 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Dec 16 21:32:51 smithi045 ceph-mon[80763]: rocksdb: Options.compaction_filter: None 2023-12-16T21:32:51.487 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Dec 16 21:32:51 smithi045 ceph-mon[80763]: rocksdb: Options.compaction_filter_factory: None 2023-12-16T21:32:51.487 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Dec 16 21:32:51 smithi045 ceph-mon[80763]: rocksdb: Options.sst_partitioner_factory: None 2023-12-16T21:32:51.487 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Dec 16 21:32:51 smithi045 ceph-mon[80763]: rocksdb: Options.memtable_factory: SkipListFactory 2023-12-16T21:32:51.487 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Dec 16 21:32:51 smithi045 ceph-mon[80763]: rocksdb: Options.table_factory: BlockBasedTable 2023-12-16T21:32:51.487 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Dec 16 21:32:51 smithi045 ceph-mon[80763]: rocksdb: table_factory options: flush_block_policy_factory: FlushBlockBySizePolicyFactory (0x562022a56f90) 2023-12-16T21:32:51.487 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout: cache_index_and_filter_blocks: 1 2023-12-16T21:32:51.488 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout: cache_index_and_filter_blocks_with_high_priority: 0 2023-12-16T21:32:51.488 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout: pin_l0_filter_and_index_blocks_in_cache: 0 2023-12-16T21:32:51.488 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout: pin_top_level_index_and_filter: 1 2023-12-16T21:32:51.488 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout: index_type: 0 2023-12-16T21:32:51.488 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout: data_block_index_type: 0 2023-12-16T21:32:51.488 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout: index_shortening: 1 2023-12-16T21:32:51.488 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout: data_block_hash_table_util_ratio: 0.750000 2023-12-16T21:32:51.488 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout: hash_index_allow_collision: 1 2023-12-16T21:32:51.488 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout: checksum: 1 2023-12-16T21:32:51.488 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout: no_block_cache: 0 2023-12-16T21:32:51.488 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout: block_cache: 0x562022a89090 2023-12-16T21:32:51.488 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout: block_cache_name: BinnedLRUCache 2023-12-16T21:32:51.489 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout: block_cache_options: 2023-12-16T21:32:51.489 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout: capacity : 536870912 2023-12-16T21:32:51.489 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout: num_shard_bits : 4 2023-12-16T21:32:51.489 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout: strict_capacity_limit : 0 2023-12-16T21:32:51.489 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout: high_pri_pool_ratio: 0.000 2023-12-16T21:32:51.489 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout: block_cache_compressed: (nil) 2023-12-16T21:32:51.489 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout: persistent_cache: (nil) 2023-12-16T21:32:51.489 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout: block_size: 4096 2023-12-16T21:32:51.489 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout: block_size_deviation: 10 2023-12-16T21:32:51.489 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout: block_restart_interval: 16 2023-12-16T21:32:51.489 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout: index_block_restart_interval: 1 2023-12-16T21:32:51.489 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout: metadata_block_size: 4096 2023-12-16T21:32:51.489 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout: partition_filters: 0 2023-12-16T21:32:51.490 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout: use_delta_encoding: 1 2023-12-16T21:32:51.490 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout: filter_policy: rocksdb.BuiltinBloomFilter 2023-12-16T21:32:51.490 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout: whole_key_filtering: 1 2023-12-16T21:32:51.490 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout: verify_compression: 0 2023-12-16T21:32:51.490 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout: read_amp_bytes_per_bit: 0 2023-12-16T21:32:51.490 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout: format_version: 4 2023-12-16T21:32:51.490 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout: enable_index_compression: 1 2023-12-16T21:32:51.490 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout: block_align: 0 2023-12-16T21:32:51.490 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Dec 16 21:32:51 smithi045 ceph-mon[80763]: rocksdb: Options.write_buffer_size: 33554432 2023-12-16T21:32:51.490 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Dec 16 21:32:51 smithi045 ceph-mon[80763]: rocksdb: Options.max_write_buffer_number: 2 2023-12-16T21:32:51.490 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Dec 16 21:32:51 smithi045 ceph-mon[80763]: rocksdb: Options.compression: NoCompression 2023-12-16T21:32:51.490 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Dec 16 21:32:51 smithi045 ceph-mon[80763]: rocksdb: Options.bottommost_compression: Disabled 2023-12-16T21:32:51.490 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Dec 16 21:32:51 smithi045 ceph-mon[80763]: rocksdb: Options.prefix_extractor: nullptr 2023-12-16T21:32:51.490 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Dec 16 21:32:51 smithi045 ceph-mon[80763]: rocksdb: Options.memtable_insert_with_hint_prefix_extractor: nullptr 2023-12-16T21:32:51.491 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Dec 16 21:32:51 smithi045 ceph-mon[80763]: rocksdb: Options.num_levels: 7 2023-12-16T21:32:51.491 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Dec 16 21:32:51 smithi045 ceph-mon[80763]: rocksdb: Options.min_write_buffer_number_to_merge: 1 2023-12-16T21:32:51.491 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Dec 16 21:32:51 smithi045 ceph-mon[80763]: rocksdb: Options.max_write_buffer_number_to_maintain: 0 2023-12-16T21:32:51.491 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Dec 16 21:32:51 smithi045 ceph-mon[80763]: rocksdb: Options.max_write_buffer_size_to_maintain: 0 2023-12-16T21:32:51.491 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Dec 16 21:32:51 smithi045 ceph-mon[80763]: rocksdb: Options.bottommost_compression_opts.window_bits: -14 2023-12-16T21:32:51.491 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Dec 16 21:32:51 smithi045 ceph-mon[80763]: rocksdb: Options.bottommost_compression_opts.level: 32767 2023-12-16T21:32:51.491 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Dec 16 21:32:51 smithi045 ceph-mon[80763]: rocksdb: Options.bottommost_compression_opts.strategy: 0 2023-12-16T21:32:51.491 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Dec 16 21:32:51 smithi045 ceph-mon[80763]: rocksdb: Options.bottommost_compression_opts.max_dict_bytes: 0 2023-12-16T21:32:51.491 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Dec 16 21:32:51 smithi045 ceph-mon[80763]: rocksdb: Options.bottommost_compression_opts.zstd_max_train_bytes: 0 2023-12-16T21:32:51.491 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Dec 16 21:32:51 smithi045 ceph-mon[80763]: rocksdb: Options.bottommost_compression_opts.parallel_threads: 1 2023-12-16T21:32:51.491 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Dec 16 21:32:51 smithi045 ceph-mon[80763]: rocksdb: Options.bottommost_compression_opts.enabled: false 2023-12-16T21:32:51.491 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Dec 16 21:32:51 smithi045 ceph-mon[80763]: rocksdb: Options.compression_opts.window_bits: -14 2023-12-16T21:32:51.491 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Dec 16 21:32:51 smithi045 ceph-mon[80763]: rocksdb: Options.compression_opts.level: 32767 2023-12-16T21:32:51.492 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Dec 16 21:32:51 smithi045 ceph-mon[80763]: rocksdb: Options.compression_opts.strategy: 0 2023-12-16T21:32:51.492 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Dec 16 21:32:51 smithi045 ceph-mon[80763]: rocksdb: Options.compression_opts.max_dict_bytes: 0 2023-12-16T21:32:51.492 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Dec 16 21:32:51 smithi045 ceph-mon[80763]: rocksdb: Options.compression_opts.zstd_max_train_bytes: 0 2023-12-16T21:32:51.492 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Dec 16 21:32:51 smithi045 ceph-mon[80763]: rocksdb: Options.compression_opts.parallel_threads: 1 2023-12-16T21:32:51.492 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Dec 16 21:32:51 smithi045 ceph-mon[80763]: rocksdb: Options.compression_opts.enabled: false 2023-12-16T21:32:51.492 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Dec 16 21:32:51 smithi045 ceph-mon[80763]: rocksdb: Options.level0_file_num_compaction_trigger: 4 2023-12-16T21:32:51.492 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Dec 16 21:32:51 smithi045 ceph-mon[80763]: rocksdb: Options.level0_slowdown_writes_trigger: 20 2023-12-16T21:32:51.492 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Dec 16 21:32:51 smithi045 ceph-mon[80763]: rocksdb: Options.level0_stop_writes_trigger: 36 2023-12-16T21:32:51.492 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Dec 16 21:32:51 smithi045 ceph-mon[80763]: rocksdb: Options.target_file_size_base: 67108864 2023-12-16T21:32:51.492 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Dec 16 21:32:51 smithi045 ceph-mon[80763]: rocksdb: Options.target_file_size_multiplier: 1 2023-12-16T21:32:51.492 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Dec 16 21:32:51 smithi045 ceph-mon[80763]: rocksdb: Options.max_bytes_for_level_base: 268435456 2023-12-16T21:32:51.493 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Dec 16 21:32:51 smithi045 ceph-mon[80763]: rocksdb: Options.level_compaction_dynamic_level_bytes: 1 2023-12-16T21:32:51.493 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Dec 16 21:32:51 smithi045 ceph-mon[80763]: rocksdb: Options.max_bytes_for_level_multiplier: 10.000000 2023-12-16T21:32:51.493 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Dec 16 21:32:51 smithi045 ceph-mon[80763]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[0]: 1 2023-12-16T21:32:51.493 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Dec 16 21:32:51 smithi045 ceph-mon[80763]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[1]: 1 2023-12-16T21:32:51.493 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Dec 16 21:32:51 smithi045 ceph-mon[80763]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[2]: 1 2023-12-16T21:32:51.493 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Dec 16 21:32:51 smithi045 ceph-mon[80763]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[3]: 1 2023-12-16T21:32:51.493 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Dec 16 21:32:51 smithi045 ceph-mon[80763]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[4]: 1 2023-12-16T21:32:51.493 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Dec 16 21:32:51 smithi045 ceph-mon[80763]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[5]: 1 2023-12-16T21:32:51.493 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Dec 16 21:32:51 smithi045 ceph-mon[80763]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[6]: 1 2023-12-16T21:32:51.493 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Dec 16 21:32:51 smithi045 ceph-mon[80763]: rocksdb: Options.max_sequential_skip_in_iterations: 8 2023-12-16T21:32:51.493 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Dec 16 21:32:51 smithi045 ceph-mon[80763]: rocksdb: Options.max_compaction_bytes: 1677721600 2023-12-16T21:32:51.493 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Dec 16 21:32:51 smithi045 ceph-mon[80763]: rocksdb: Options.arena_block_size: 4194304 2023-12-16T21:32:51.493 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Dec 16 21:32:51 smithi045 ceph-mon[80763]: rocksdb: Options.soft_pending_compaction_bytes_limit: 68719476736 2023-12-16T21:32:51.494 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Dec 16 21:32:51 smithi045 ceph-mon[80763]: rocksdb: Options.hard_pending_compaction_bytes_limit: 274877906944 2023-12-16T21:32:51.494 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Dec 16 21:32:51 smithi045 ceph-mon[80763]: rocksdb: Options.rate_limit_delay_max_milliseconds: 100 2023-12-16T21:32:51.494 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Dec 16 21:32:51 smithi045 ceph-mon[80763]: rocksdb: Options.disable_auto_compactions: 0 2023-12-16T21:32:51.494 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Dec 16 21:32:51 smithi045 ceph-mon[80763]: rocksdb: Options.compaction_style: kCompactionStyleLevel 2023-12-16T21:32:51.494 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Dec 16 21:32:51 smithi045 ceph-mon[80763]: rocksdb: Options.compaction_pri: kMinOverlappingRatio 2023-12-16T21:32:51.494 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Dec 16 21:32:51 smithi045 ceph-mon[80763]: rocksdb: Options.compaction_options_universal.size_ratio: 1 2023-12-16T21:32:51.494 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Dec 16 21:32:51 smithi045 ceph-mon[80763]: rocksdb: Options.compaction_options_universal.min_merge_width: 2 2023-12-16T21:32:51.494 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Dec 16 21:32:51 smithi045 ceph-mon[80763]: rocksdb: Options.compaction_options_universal.max_merge_width: 4294967295 2023-12-16T21:32:51.494 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Dec 16 21:32:51 smithi045 ceph-mon[80763]: rocksdb: Options.compaction_options_universal.max_size_amplification_percent: 200 2023-12-16T21:32:51.494 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Dec 16 21:32:51 smithi045 ceph-mon[80763]: rocksdb: Options.compaction_options_universal.compression_size_percent: -1 2023-12-16T21:32:51.494 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Dec 16 21:32:51 smithi045 ceph-mon[80763]: rocksdb: Options.compaction_options_universal.stop_style: kCompactionStopStyleTotalSize 2023-12-16T21:32:51.494 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Dec 16 21:32:51 smithi045 ceph-mon[80763]: rocksdb: Options.compaction_options_fifo.max_table_files_size: 1073741824 2023-12-16T21:32:51.494 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Dec 16 21:32:51 smithi045 ceph-mon[80763]: rocksdb: Options.compaction_options_fifo.allow_compaction: 0 2023-12-16T21:32:51.494 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Dec 16 21:32:51 smithi045 ceph-mon[80763]: rocksdb: Options.table_properties_collectors: 2023-12-16T21:32:51.494 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Dec 16 21:32:51 smithi045 ceph-mon[80763]: rocksdb: Options.inplace_update_support: 0 2023-12-16T21:32:51.495 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Dec 16 21:32:51 smithi045 ceph-mon[80763]: rocksdb: Options.inplace_update_num_locks: 10000 2023-12-16T21:32:51.495 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Dec 16 21:32:51 smithi045 ceph-mon[80763]: rocksdb: Options.memtable_prefix_bloom_size_ratio: 0.000000 2023-12-16T21:32:51.495 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Dec 16 21:32:51 smithi045 ceph-mon[80763]: rocksdb: Options.memtable_whole_key_filtering: 0 2023-12-16T21:32:51.495 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Dec 16 21:32:51 smithi045 ceph-mon[80763]: rocksdb: Options.memtable_huge_page_size: 0 2023-12-16T21:32:51.495 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Dec 16 21:32:51 smithi045 ceph-mon[80763]: rocksdb: Options.bloom_locality: 0 2023-12-16T21:32:51.495 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Dec 16 21:32:51 smithi045 ceph-mon[80763]: rocksdb: Options.max_successive_merges: 0 2023-12-16T21:32:51.495 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Dec 16 21:32:51 smithi045 ceph-mon[80763]: rocksdb: Options.optimize_filters_for_hits: 0 2023-12-16T21:32:51.495 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Dec 16 21:32:51 smithi045 ceph-mon[80763]: rocksdb: Options.paranoid_file_checks: 0 2023-12-16T21:32:51.495 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Dec 16 21:32:51 smithi045 ceph-mon[80763]: rocksdb: Options.force_consistency_checks: 1 2023-12-16T21:32:51.495 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Dec 16 21:32:51 smithi045 ceph-mon[80763]: rocksdb: Options.report_bg_io_stats: 0 2023-12-16T21:32:51.495 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Dec 16 21:32:51 smithi045 ceph-mon[80763]: rocksdb: Options.ttl: 2592000 2023-12-16T21:32:51.495 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Dec 16 21:32:51 smithi045 ceph-mon[80763]: rocksdb: Options.periodic_compaction_seconds: 0 2023-12-16T21:32:51.496 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Dec 16 21:32:51 smithi045 ceph-mon[80763]: rocksdb: Options.enable_blob_files: false 2023-12-16T21:32:51.496 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Dec 16 21:32:51 smithi045 ceph-mon[80763]: rocksdb: Options.min_blob_size: 0 2023-12-16T21:32:51.496 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Dec 16 21:32:51 smithi045 ceph-mon[80763]: rocksdb: Options.blob_file_size: 268435456 2023-12-16T21:32:51.496 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Dec 16 21:32:51 smithi045 ceph-mon[80763]: rocksdb: Options.blob_compression_type: NoCompression 2023-12-16T21:32:51.496 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Dec 16 21:32:51 smithi045 ceph-mon[80763]: rocksdb: Options.enable_blob_garbage_collection: false 2023-12-16T21:32:51.496 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Dec 16 21:32:51 smithi045 ceph-mon[80763]: rocksdb: Options.blob_garbage_collection_age_cutoff: 0.250000 2023-12-16T21:32:51.496 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Dec 16 21:32:51 smithi045 systemd[1]: Started Ceph mon.smithi045 for 88ff363c-9c5a-11ee-95a5-87774f69a715. 2023-12-16T21:32:51.496 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Dec 16 21:32:51 smithi045 ceph-mon[80763]: rocksdb: [db/version_set.cc:4764] Recovered from manifest file:/var/lib/ceph/mon/ceph-smithi045/store.db/MANIFEST-000009 succeeded,manifest_file_number is 9, next_file_number is 11, last_sequence is 5, log_number is 5,prev_log_number is 0,max_column_family is 0,min_log_number_to_keep is 0 2023-12-16T21:32:51.496 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Dec 16 21:32:51 smithi045 ceph-mon[80763]: rocksdb: [db/version_set.cc:4779] Column family [default] (ID 0), log number is 5 2023-12-16T21:32:51.496 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Dec 16 21:32:51 smithi045 ceph-mon[80763]: rocksdb: [db/version_set.cc:4082] Creating manifest 13 2023-12-16T21:32:51.496 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Dec 16 21:32:51 smithi045 ceph-mon[80763]: rocksdb: EVENT_LOG_v1 {"time_micros": 1702762371236371, "job": 1, "event": "recovery_started", "wal_files": [10]} 2023-12-16T21:32:51.496 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Dec 16 21:32:51 smithi045 ceph-mon[80763]: rocksdb: [db/db_impl/db_impl_open.cc:845] Recovering log #10 mode 2 2023-12-16T21:32:51.496 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Dec 16 21:32:51 smithi045 ceph-mon[80763]: rocksdb: [table/block_based/filter_policy.cc:991] Using legacy Bloom filter with high (20) bits/key. Dramatic filter space and/or accuracy improvement is available with format_version>=5. 2023-12-16T21:32:51.496 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Dec 16 21:32:51 smithi045 ceph-mon[80763]: rocksdb: EVENT_LOG_v1 {"time_micros": 1702762371237600, "cf_name": "default", "job": 1, "event": "table_file_creation", "file_number": 14, "file_size": 90708, "file_checksum": "", "file_checksum_func_name": "Unknown", "table_properties": {"data_size": 88821, "index_size": 247, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 1, "index_value_is_delta_encoded": 1, "filter_size": 709, "raw_key_size": 13140, "raw_average_key_size": 50, "raw_value_size": 81644, "raw_average_value_size": 311, "num_data_blocks": 10, "num_entries": 262, "num_deletions": 3, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 0, "filter_policy": "rocksdb.BuiltinBloomFilter", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "", "prefix_extractor_name": "nullptr", "property_collectors": "[]", "compression": "NoCompression", "compression_options": "window_bits=-14; level=32767; strategy=0; max_dict_bytes=0; zstd_max_train_bytes=0; enabled=0; ", "creation_time": 1702762371, "oldest_key_time": 0, "file_creation_time": 0, "db_id": "a896b77e-8bb6-40f8-a390-21c5c667a97d", "db_session_id": "11B99WTFFUGGUGF4CYCV"}} 2023-12-16T21:32:51.497 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Dec 16 21:32:51 smithi045 ceph-mon[80763]: rocksdb: [db/version_set.cc:4082] Creating manifest 15 2023-12-16T21:32:51.497 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Dec 16 21:32:51 smithi045 ceph-mon[80763]: rocksdb: EVENT_LOG_v1 {"time_micros": 1702762371238013, "job": 1, "event": "recovery_finished"} 2023-12-16T21:32:51.497 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Dec 16 21:32:51 smithi045 ceph-mon[80763]: rocksdb: [file/delete_scheduler.cc:69] Deleted file /var/lib/ceph/mon/ceph-smithi045/store.db/000010.log immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2023-12-16T21:32:51.497 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Dec 16 21:32:51 smithi045 ceph-mon[80763]: rocksdb: [db/db_impl/db_impl_open.cc:1700] SstFileManager instance 0x562022a1d180 2023-12-16T21:32:51.497 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Dec 16 21:32:51 smithi045 ceph-mon[80763]: rocksdb: DB pointer 0x562022af2000 2023-12-16T21:32:51.497 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Dec 16 21:32:51 smithi045 ceph-mon[80763]: rocksdb: [db/db_impl/db_impl.cc:901] ------- DUMPING STATS ------- 2023-12-16T21:32:51.497 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Dec 16 21:32:51 smithi045 ceph-mon[80763]: rocksdb: [db/db_impl/db_impl.cc:903] 2023-12-16T21:32:51.497 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout: ** DB Stats ** 2023-12-16T21:32:51.497 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout: Uptime(secs): 0.0 total, 0.0 interval 2023-12-16T21:32:51.497 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout: Cumulative writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 GB, 0.00 MB/s 2023-12-16T21:32:51.497 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout: Cumulative WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2023-12-16T21:32:51.497 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout: Cumulative stall: 00:00:0.000 H:M:S, 0.0 percent 2023-12-16T21:32:51.497 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout: Interval writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 MB, 0.00 MB/s 2023-12-16T21:32:51.498 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout: Interval WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 MB, 0.00 MB/s 2023-12-16T21:32:51.498 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout: Interval stall: 00:00:0.000 H:M:S, 0.0 percent 2023-12-16T21:32:51.498 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout: 2023-12-16T21:32:51.498 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout: ** Compaction Stats [default] ** 2023-12-16T21:32:51.498 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout: Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop 2023-12-16T21:32:51.498 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout: ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2023-12-16T21:32:51.498 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout: L0 2/0 90.38 KB 0.5 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 106.1 0.00 0.00 1 0.001 0 0 2023-12-16T21:32:51.498 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout: Sum 2/0 90.38 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 106.1 0.00 0.00 1 0.001 0 0 2023-12-16T21:32:51.498 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout: Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 106.1 0.00 0.00 1 0.001 0 0 2023-12-16T21:32:51.498 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout: 2023-12-16T21:32:51.498 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout: ** Compaction Stats [default] ** 2023-12-16T21:32:51.498 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout: Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop 2023-12-16T21:32:51.498 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout: ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2023-12-16T21:32:51.498 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout: User 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 106.1 0.00 0.00 1 0.001 0 0 2023-12-16T21:32:51.498 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout: Uptime(secs): 0.0 total, 0.0 interval 2023-12-16T21:32:51.499 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout: Flush(GB): cumulative 0.000, interval 0.000 2023-12-16T21:32:51.499 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout: AddFile(GB): cumulative 0.000, interval 0.000 2023-12-16T21:32:51.499 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout: AddFile(Total Files): cumulative 0, interval 0 2023-12-16T21:32:51.499 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout: AddFile(L0 Files): cumulative 0, interval 0 2023-12-16T21:32:51.499 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout: AddFile(Keys): cumulative 0, interval 0 2023-12-16T21:32:51.499 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout: Cumulative compaction: 0.00 GB write, 5.51 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2023-12-16T21:32:51.499 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout: Interval compaction: 0.00 GB write, 5.51 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2023-12-16T21:32:51.499 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout: Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count 2023-12-16T21:32:51.499 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout: 2023-12-16T21:32:51.499 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout: ** File Read Latency Histogram By Level [default] ** 2023-12-16T21:32:51.499 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout: 2023-12-16T21:32:51.499 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout: ** Compaction Stats [default] ** 2023-12-16T21:32:51.499 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout: Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop 2023-12-16T21:32:51.500 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout: ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2023-12-16T21:32:51.500 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout: L0 2/0 90.38 KB 0.5 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 106.1 0.00 0.00 1 0.001 0 0 2023-12-16T21:32:51.500 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout: Sum 2/0 90.38 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 106.1 0.00 0.00 1 0.001 0 0 2023-12-16T21:32:51.500 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout: Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.00 0.00 0 0.000 0 0 2023-12-16T21:32:51.500 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout: 2023-12-16T21:32:51.500 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout: ** Compaction Stats [default] ** 2023-12-16T21:32:51.500 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout: Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop 2023-12-16T21:32:51.500 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout: ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2023-12-16T21:32:51.500 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout: User 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 106.1 0.00 0.00 1 0.001 0 0 2023-12-16T21:32:51.500 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout: Uptime(secs): 0.0 total, 0.0 interval 2023-12-16T21:32:51.500 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout: Flush(GB): cumulative 0.000, interval 0.000 2023-12-16T21:32:51.500 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout: AddFile(GB): cumulative 0.000, interval 0.000 2023-12-16T21:32:51.500 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout: AddFile(Total Files): cumulative 0, interval 0 2023-12-16T21:32:51.500 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout: AddFile(L0 Files): cumulative 0, interval 0 2023-12-16T21:32:51.500 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout: AddFile(Keys): cumulative 0, interval 0 2023-12-16T21:32:51.501 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout: Cumulative compaction: 0.00 GB write, 5.50 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2023-12-16T21:32:51.501 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout: Interval compaction: 0.00 GB write, 0.00 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2023-12-16T21:32:51.501 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout: Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count 2023-12-16T21:32:51.501 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout: 2023-12-16T21:32:51.501 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout: ** File Read Latency Histogram By Level [default] ** 2023-12-16T21:32:51.501 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Dec 16 21:32:51 smithi045 ceph-mon[80763]: starting mon.smithi045 rank 0 at public addrs [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] at bind addrs [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon_data /var/lib/ceph/mon/ceph-smithi045 fsid 88ff363c-9c5a-11ee-95a5-87774f69a715 2023-12-16T21:32:51.501 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Dec 16 21:32:51 smithi045 ceph-mon[80763]: mon.smithi045@-1(???) e1 preinit fsid 88ff363c-9c5a-11ee-95a5-87774f69a715 2023-12-16T21:32:51.501 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Dec 16 21:32:51 smithi045 ceph-mon[80763]: mon.smithi045@-1(???).mds e1 new map 2023-12-16T21:32:51.501 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Dec 16 21:32:51 smithi045 ceph-mon[80763]: mon.smithi045@-1(???).mds e1 print_map 2023-12-16T21:32:51.501 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout: e1 2023-12-16T21:32:51.501 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout: enable_multiple, ever_enabled_multiple: 1,1 2023-12-16T21:32:51.501 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout: default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2023-12-16T21:32:51.501 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout: legacy client fscid: -1 2023-12-16T21:32:51.501 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout: 2023-12-16T21:32:51.501 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout: No filesystems configured 2023-12-16T21:32:51.502 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Dec 16 21:32:51 smithi045 ceph-mon[80763]: mon.smithi045@-1(???).osd e1 crush map has features 3314932999778484224, adjusting msgr requires 2023-12-16T21:32:51.502 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Dec 16 21:32:51 smithi045 ceph-mon[80763]: mon.smithi045@-1(???).osd e1 crush map has features 288514050185494528, adjusting msgr requires 2023-12-16T21:32:51.502 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Dec 16 21:32:51 smithi045 ceph-mon[80763]: mon.smithi045@-1(???).osd e1 crush map has features 288514050185494528, adjusting msgr requires 2023-12-16T21:32:51.502 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Dec 16 21:32:51 smithi045 ceph-mon[80763]: mon.smithi045@-1(???).osd e1 crush map has features 288514050185494528, adjusting msgr requires 2023-12-16T21:32:51.502 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Dec 16 21:32:51 smithi045 ceph-mon[80763]: mon.smithi045@-1(???).paxosservice(auth 1..2) refresh upgraded, format 0 -> 3 2023-12-16T21:32:51.502 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Dec 16 21:32:51 smithi045 ceph-mon[80763]: mon.smithi045 is new leader, mons smithi045 in quorum (ranks 0) 2023-12-16T21:32:51.502 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Dec 16 21:32:51 smithi045 ceph-mon[80763]: monmap e1: 1 mons at {smithi045=[v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0]} removed_ranks: {} disallowed_leaders: {} 2023-12-16T21:32:51.502 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Dec 16 21:32:51 smithi045 ceph-mon[80763]: fsmap 2023-12-16T21:32:51.502 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Dec 16 21:32:51 smithi045 ceph-mon[80763]: osdmap e1: 0 total, 0 up, 0 in 2023-12-16T21:32:51.502 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Dec 16 21:32:51 smithi045 ceph-mon[80763]: mgrmap e1: no daemons active 2023-12-16T21:32:52.816 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Dec 16 21:32:52 smithi045 ceph-mon[80763]: from='client.? 172.21.15.45:0/1792906839' entity='client.admin' 2023-12-16T21:32:53.675 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2023-12-16T21:32:51.475+0000 7f1783706700 1 Processor -- start 2023-12-16T21:32:53.675 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2023-12-16T21:32:51.475+0000 7f1783706700 1 -- start start 2023-12-16T21:32:53.675 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2023-12-16T21:32:51.475+0000 7f1783706700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f177c076c60 0x7f177c104ea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:32:53.675 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2023-12-16T21:32:51.475+0000 7f1783706700 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f177c1053e0 con 0x7f177c076c60 2023-12-16T21:32:53.675 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2023-12-16T21:32:51.475+0000 7f17814a2700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f177c076c60 0x7f177c104ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:32:53.675 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2023-12-16T21:32:51.476+0000 7f17814a2700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f177c076c60 0x7f177c104ea0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:53104/0 (socket says 172.21.15.45:53104) 2023-12-16T21:32:53.675 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2023-12-16T21:32:51.476+0000 7f17814a2700 1 -- 172.21.15.45:0/2236211805 learned_addr learned my addr 172.21.15.45:0/2236211805 (peer_addr_for_me v2:172.21.15.45:0/0) 2023-12-16T21:32:53.675 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2023-12-16T21:32:51.476+0000 7f17814a2700 1 -- 172.21.15.45:0/2236211805 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f177c105520 con 0x7f177c076c60 2023-12-16T21:32:53.675 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2023-12-16T21:32:51.476+0000 7f17814a2700 1 --2- 172.21.15.45:0/2236211805 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f177c076c60 0x7f177c104ea0 secure :-1 s=READY pgs=1 cs=0 l=1 rev1=1 crypto rx=0x7f17780098e0 tx=0x7f1778009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=85f5a1c7eacc70c4 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:32:53.676 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2023-12-16T21:32:51.476+0000 7f1773fff700 1 -- 172.21.15.45:0/2236211805 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f177800b860 con 0x7f177c076c60 2023-12-16T21:32:53.676 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2023-12-16T21:32:51.476+0000 7f1773fff700 1 -- 172.21.15.45:0/2236211805 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(26 keys) v1 ==== 1009+0+0 (secure 0 0 0) 0x7f177800be30 con 0x7f177c076c60 2023-12-16T21:32:53.676 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2023-12-16T21:32:51.476+0000 7f1773fff700 1 -- 172.21.15.45:0/2236211805 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f1778010380 con 0x7f177c076c60 2023-12-16T21:32:53.676 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2023-12-16T21:32:51.477+0000 7f1783706700 1 -- 172.21.15.45:0/2236211805 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f177c076c60 msgr2=0x7f177c104ea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:32:53.676 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2023-12-16T21:32:51.477+0000 7f1783706700 1 --2- 172.21.15.45:0/2236211805 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f177c076c60 0x7f177c104ea0 secure :-1 s=READY pgs=1 cs=0 l=1 rev1=1 crypto rx=0x7f17780098e0 tx=0x7f1778009250 comp rx=0 tx=0).stop 2023-12-16T21:32:53.676 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2023-12-16T21:32:51.477+0000 7f1783706700 1 -- 172.21.15.45:0/2236211805 shutdown_connections 2023-12-16T21:32:53.676 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2023-12-16T21:32:51.477+0000 7f1783706700 1 --2- 172.21.15.45:0/2236211805 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f177c076c60 0x7f177c104ea0 unknown :-1 s=CLOSED pgs=1 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:32:53.676 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2023-12-16T21:32:51.477+0000 7f1783706700 1 -- 172.21.15.45:0/2236211805 >> 172.21.15.45:0/2236211805 conn(0x7f177c0fc950 msgr2=0x7f177c0fed60 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-16T21:32:53.676 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2023-12-16T21:32:51.477+0000 7f1783706700 1 -- 172.21.15.45:0/2236211805 shutdown_connections 2023-12-16T21:32:53.676 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2023-12-16T21:32:51.477+0000 7f1783706700 1 -- 172.21.15.45:0/2236211805 wait complete. 2023-12-16T21:32:53.676 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2023-12-16T21:32:51.478+0000 7f1783706700 1 Processor -- start 2023-12-16T21:32:53.676 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2023-12-16T21:32:51.478+0000 7f1783706700 1 -- start start 2023-12-16T21:32:53.676 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2023-12-16T21:32:51.478+0000 7f1783706700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f177c076c60 0x7f177c111160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:32:53.676 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2023-12-16T21:32:51.478+0000 7f1783706700 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f177c1116a0 con 0x7f177c076c60 2023-12-16T21:32:53.676 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2023-12-16T21:32:51.478+0000 7f17814a2700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f177c076c60 0x7f177c111160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:32:53.677 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2023-12-16T21:32:51.478+0000 7f17814a2700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f177c076c60 0x7f177c111160 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:53118/0 (socket says 172.21.15.45:53118) 2023-12-16T21:32:53.677 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2023-12-16T21:32:51.478+0000 7f17814a2700 1 -- 172.21.15.45:0/1792906839 learned_addr learned my addr 172.21.15.45:0/1792906839 (peer_addr_for_me v2:172.21.15.45:0/0) 2023-12-16T21:32:53.677 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2023-12-16T21:32:51.479+0000 7f17814a2700 1 -- 172.21.15.45:0/1792906839 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f17780094f0 con 0x7f177c076c60 2023-12-16T21:32:53.677 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2023-12-16T21:32:51.479+0000 7f17814a2700 1 --2- 172.21.15.45:0/1792906839 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f177c076c60 0x7f177c111160 secure :-1 s=READY pgs=2 cs=0 l=1 rev1=1 crypto rx=0x7f1778006b00 tx=0x7f1778009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:32:53.677 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2023-12-16T21:32:51.479+0000 7f1771ffb700 1 -- 172.21.15.45:0/1792906839 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f1778010a80 con 0x7f177c076c60 2023-12-16T21:32:53.677 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2023-12-16T21:32:51.479+0000 7f1771ffb700 1 -- 172.21.15.45:0/1792906839 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(26 keys) v1 ==== 1009+0+0 (secure 0 0 0) 0x7f177801a430 con 0x7f177c076c60 2023-12-16T21:32:53.677 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2023-12-16T21:32:51.479+0000 7f1783706700 1 -- 172.21.15.45:0/1792906839 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f177c1118a0 con 0x7f177c076c60 2023-12-16T21:32:53.677 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2023-12-16T21:32:51.479+0000 7f1771ffb700 1 -- 172.21.15.45:0/1792906839 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f17780245b0 con 0x7f177c076c60 2023-12-16T21:32:53.677 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2023-12-16T21:32:51.479+0000 7f1783706700 1 -- 172.21.15.45:0/1792906839 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f177c111cb0 con 0x7f177c076c60 2023-12-16T21:32:53.677 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2023-12-16T21:32:51.479+0000 7f1771ffb700 1 -- 172.21.15.45:0/1792906839 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 1) v1 ==== 521+0+0 (secure 0 0 0) 0x7f1778024a60 con 0x7f177c076c60 2023-12-16T21:32:53.677 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2023-12-16T21:32:51.479+0000 7f1771ffb700 1 -- 172.21.15.45:0/1792906839 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 721+0+0 (secure 0 0 0) 0x7f177802eaa0 con 0x7f177c076c60 2023-12-16T21:32:53.677 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2023-12-16T21:32:51.480+0000 7f1783706700 1 -- 172.21.15.45:0/1792906839 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f177c04bb80 con 0x7f177c076c60 2023-12-16T21:32:53.677 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2023-12-16T21:32:51.483+0000 7f1771ffb700 1 -- 172.21.15.45:0/1792906839 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+71349 (secure 0 0 0) 0x7f177802ec60 con 0x7f177c076c60 2023-12-16T21:32:53.678 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2023-12-16T21:32:51.553+0000 7f1783706700 1 -- 172.21.15.45:0/1792906839 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command([{prefix=config set, name=public_network}] v 0) v1 -- 0x7f177c10ab60 con 0x7f177c076c60 2023-12-16T21:32:53.678 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2023-12-16T21:32:51.555+0000 7f1771ffb700 1 -- 172.21.15.45:0/1792906839 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mon_command_ack([{prefix=config set, name=public_network}]=0 v3)=0 v3) v1 ==== 124+0+0 (secure 0 0 0) 0x7f177802ee00 con 0x7f177c076c60 2023-12-16T21:32:53.678 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2023-12-16T21:32:51.556+0000 7f1783706700 1 -- 172.21.15.45:0/1792906839 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f177c076c60 msgr2=0x7f177c111160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:32:53.678 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2023-12-16T21:32:51.556+0000 7f1783706700 1 --2- 172.21.15.45:0/1792906839 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f177c076c60 0x7f177c111160 secure :-1 s=READY pgs=2 cs=0 l=1 rev1=1 crypto rx=0x7f1778006b00 tx=0x7f1778009250 comp rx=0 tx=0).stop 2023-12-16T21:32:53.678 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2023-12-16T21:32:51.556+0000 7f1783706700 1 -- 172.21.15.45:0/1792906839 shutdown_connections 2023-12-16T21:32:53.678 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2023-12-16T21:32:51.556+0000 7f1783706700 1 --2- 172.21.15.45:0/1792906839 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f177c076c60 0x7f177c111160 unknown :-1 s=CLOSED pgs=2 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:32:53.678 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2023-12-16T21:32:51.556+0000 7f1783706700 1 -- 172.21.15.45:0/1792906839 >> 172.21.15.45:0/1792906839 conn(0x7f177c0fc950 msgr2=0x7f177c0fed10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-16T21:32:53.678 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2023-12-16T21:32:51.556+0000 7f1783706700 1 -- 172.21.15.45:0/1792906839 shutdown_connections 2023-12-16T21:32:53.678 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2023-12-16T21:32:51.556+0000 7f1783706700 1 -- 172.21.15.45:0/1792906839 wait complete. 2023-12-16T21:32:53.678 INFO:teuthology.orchestra.run.smithi045.stdout:Wrote config to /etc/ceph/ceph.conf 2023-12-16T21:32:53.678 INFO:teuthology.orchestra.run.smithi045.stdout:Wrote keyring to /etc/ceph/ceph.client.admin.keyring 2023-12-16T21:32:53.678 INFO:teuthology.orchestra.run.smithi045.stdout:Creating mgr... 2023-12-16T21:32:53.678 INFO:teuthology.orchestra.run.smithi045.stdout:Verifying port 9283 ... 2023-12-16T21:32:53.969 INFO:teuthology.orchestra.run.smithi045.stdout:Non-zero exit code 1 from systemctl reset-failed ceph-88ff363c-9c5a-11ee-95a5-87774f69a715@mgr.smithi045.nhtmwc 2023-12-16T21:32:53.969 INFO:teuthology.orchestra.run.smithi045.stdout:systemctl: stderr Failed to reset failed state of unit ceph-88ff363c-9c5a-11ee-95a5-87774f69a715@mgr.smithi045.nhtmwc.service: Unit ceph-88ff363c-9c5a-11ee-95a5-87774f69a715@mgr.smithi045.nhtmwc.service not loaded. 2023-12-16T21:32:54.373 INFO:teuthology.orchestra.run.smithi045.stdout:systemctl: stderr Created symlink /etc/systemd/system/ceph-88ff363c-9c5a-11ee-95a5-87774f69a715.target.wants/ceph-88ff363c-9c5a-11ee-95a5-87774f69a715@mgr.smithi045.nhtmwc.service → /etc/systemd/system/ceph-88ff363c-9c5a-11ee-95a5-87774f69a715@.service. 2023-12-16T21:32:55.124 INFO:teuthology.orchestra.run.smithi045.stdout:Non-zero exit code 1 from systemctl is-enabled firewalld.service 2023-12-16T21:32:55.124 INFO:teuthology.orchestra.run.smithi045.stdout:systemctl: stdout disabled 2023-12-16T21:32:55.132 INFO:teuthology.orchestra.run.smithi045.stdout:Non-zero exit code 3 from systemctl is-active firewalld.service 2023-12-16T21:32:55.132 INFO:teuthology.orchestra.run.smithi045.stdout:systemctl: stdout inactive 2023-12-16T21:32:55.132 INFO:teuthology.orchestra.run.smithi045.stdout:firewalld.service is not enabled 2023-12-16T21:32:55.132 INFO:teuthology.orchestra.run.smithi045.stdout:Not possible to enable service . firewalld.service is not available 2023-12-16T21:32:55.143 INFO:teuthology.orchestra.run.smithi045.stdout:Non-zero exit code 1 from systemctl is-enabled firewalld.service 2023-12-16T21:32:55.143 INFO:teuthology.orchestra.run.smithi045.stdout:systemctl: stdout disabled 2023-12-16T21:32:55.152 INFO:teuthology.orchestra.run.smithi045.stdout:Non-zero exit code 3 from systemctl is-active firewalld.service 2023-12-16T21:32:55.153 INFO:teuthology.orchestra.run.smithi045.stdout:systemctl: stdout inactive 2023-12-16T21:32:55.153 INFO:teuthology.orchestra.run.smithi045.stdout:firewalld.service is not enabled 2023-12-16T21:32:55.153 INFO:teuthology.orchestra.run.smithi045.stdout:Not possible to open ports <[9283]>. firewalld.service is not available 2023-12-16T21:32:55.153 INFO:teuthology.orchestra.run.smithi045.stdout:Waiting for mgr to start... 2023-12-16T21:32:55.153 INFO:teuthology.orchestra.run.smithi045.stdout:Waiting for mgr... 2023-12-16T21:32:55.652 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Dec 16 21:32:55 smithi045 ceph-mon[80763]: from='client.? 172.21.15.45:0/1659092037' entity='client.admin' cmd=[{"prefix": "status", "format": "json-pretty"}]: dispatch 2023-12-16T21:32:55.747 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stdout 2023-12-16T21:32:55.747 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stdout { 2023-12-16T21:32:55.748 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stdout "fsid": "88ff363c-9c5a-11ee-95a5-87774f69a715", 2023-12-16T21:32:55.748 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stdout "health": { 2023-12-16T21:32:55.748 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stdout "status": "HEALTH_OK", 2023-12-16T21:32:55.748 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stdout "checks": {}, 2023-12-16T21:32:55.748 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stdout "mutes": [] 2023-12-16T21:32:55.748 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stdout }, 2023-12-16T21:32:55.748 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stdout "election_epoch": 5, 2023-12-16T21:32:55.748 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stdout "quorum": [ 2023-12-16T21:32:55.748 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stdout 0 2023-12-16T21:32:55.748 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stdout ], 2023-12-16T21:32:55.748 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stdout "quorum_names": [ 2023-12-16T21:32:55.748 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stdout "smithi045" 2023-12-16T21:32:55.748 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stdout ], 2023-12-16T21:32:55.749 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stdout "quorum_age": 4, 2023-12-16T21:32:55.749 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stdout "monmap": { 2023-12-16T21:32:55.749 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stdout "epoch": 1, 2023-12-16T21:32:55.749 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stdout "min_mon_release_name": "quincy", 2023-12-16T21:32:55.749 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stdout "num_mons": 1 2023-12-16T21:32:55.749 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stdout }, 2023-12-16T21:32:55.749 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stdout "osdmap": { 2023-12-16T21:32:55.749 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stdout "epoch": 1, 2023-12-16T21:32:55.749 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stdout "num_osds": 0, 2023-12-16T21:32:55.749 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stdout "num_up_osds": 0, 2023-12-16T21:32:55.749 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stdout "osd_up_since": 0, 2023-12-16T21:32:55.757 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stdout "num_in_osds": 0, 2023-12-16T21:32:55.757 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stdout "osd_in_since": 0, 2023-12-16T21:32:55.757 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stdout "num_remapped_pgs": 0 2023-12-16T21:32:55.757 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stdout }, 2023-12-16T21:32:55.757 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stdout "pgmap": { 2023-12-16T21:32:55.757 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stdout "pgs_by_state": [], 2023-12-16T21:32:55.757 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stdout "num_pgs": 0, 2023-12-16T21:32:55.757 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stdout "num_pools": 0, 2023-12-16T21:32:55.757 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stdout "num_objects": 0, 2023-12-16T21:32:55.757 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stdout "data_bytes": 0, 2023-12-16T21:32:55.757 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stdout "bytes_used": 0, 2023-12-16T21:32:55.758 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stdout "bytes_avail": 0, 2023-12-16T21:32:55.758 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stdout "bytes_total": 0 2023-12-16T21:32:55.758 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stdout }, 2023-12-16T21:32:55.758 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stdout "fsmap": { 2023-12-16T21:32:55.758 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stdout "epoch": 1, 2023-12-16T21:32:55.758 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stdout "by_rank": [], 2023-12-16T21:32:55.758 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stdout "up:standby": 0 2023-12-16T21:32:55.758 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stdout }, 2023-12-16T21:32:55.758 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stdout "mgrmap": { 2023-12-16T21:32:55.758 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stdout "available": false, 2023-12-16T21:32:55.758 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stdout "num_standbys": 0, 2023-12-16T21:32:55.758 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stdout "modules": [ 2023-12-16T21:32:55.758 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stdout "iostat", 2023-12-16T21:32:55.758 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stdout "nfs", 2023-12-16T21:32:55.759 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stdout "restful" 2023-12-16T21:32:55.759 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stdout ], 2023-12-16T21:32:55.759 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stdout "services": {} 2023-12-16T21:32:55.759 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stdout }, 2023-12-16T21:32:55.759 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stdout "servicemap": { 2023-12-16T21:32:55.759 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stdout "epoch": 1, 2023-12-16T21:32:55.759 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stdout "modified": "2023-12-16T21:32:48.503304+0000", 2023-12-16T21:32:55.759 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stdout "services": {} 2023-12-16T21:32:55.759 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stdout }, 2023-12-16T21:32:55.759 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stdout "progress_events": {} 2023-12-16T21:32:55.759 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stdout } 2023-12-16T21:32:55.759 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2023-12-16T21:32:55.410+0000 7f79eccc7700 1 Processor -- start 2023-12-16T21:32:55.759 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2023-12-16T21:32:55.410+0000 7f79eccc7700 1 -- start start 2023-12-16T21:32:55.759 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2023-12-16T21:32:55.410+0000 7f79eccc7700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f79e8070f60 0x7f79e8107340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:32:55.760 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2023-12-16T21:32:55.410+0000 7f79eccc7700 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f79e8102d30 con 0x7f79e8070f60 2023-12-16T21:32:55.760 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2023-12-16T21:32:55.411+0000 7f79e659c700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f79e8070f60 0x7f79e8107340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:32:55.760 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2023-12-16T21:32:55.411+0000 7f79e659c700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f79e8070f60 0x7f79e8107340 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:48072/0 (socket says 172.21.15.45:48072) 2023-12-16T21:32:55.760 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2023-12-16T21:32:55.411+0000 7f79e659c700 1 -- 172.21.15.45:0/3610257293 learned_addr learned my addr 172.21.15.45:0/3610257293 (peer_addr_for_me v2:172.21.15.45:0/0) 2023-12-16T21:32:55.760 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2023-12-16T21:32:55.411+0000 7f79e659c700 1 -- 172.21.15.45:0/3610257293 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f79e8107880 con 0x7f79e8070f60 2023-12-16T21:32:55.760 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2023-12-16T21:32:55.411+0000 7f79e659c700 1 --2- 172.21.15.45:0/3610257293 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f79e8070f60 0x7f79e8107340 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 crypto rx=0x7f79dc0098e0 tx=0x7f79dc009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cd877c3bbdbfbf02 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:32:55.760 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2023-12-16T21:32:55.411+0000 7f79e559a700 1 -- 172.21.15.45:0/3610257293 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f79dc00b860 con 0x7f79e8070f60 2023-12-16T21:32:55.760 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2023-12-16T21:32:55.411+0000 7f79e559a700 1 -- 172.21.15.45:0/3610257293 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(26 keys) v1 ==== 1009+0+0 (secure 0 0 0) 0x7f79dc00be30 con 0x7f79e8070f60 2023-12-16T21:32:55.760 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2023-12-16T21:32:55.411+0000 7f79e559a700 1 -- 172.21.15.45:0/3610257293 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f79dc010380 con 0x7f79e8070f60 2023-12-16T21:32:55.760 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2023-12-16T21:32:55.412+0000 7f79eccc7700 1 -- 172.21.15.45:0/3610257293 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f79e8070f60 msgr2=0x7f79e8107340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:32:55.760 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2023-12-16T21:32:55.412+0000 7f79eccc7700 1 --2- 172.21.15.45:0/3610257293 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f79e8070f60 0x7f79e8107340 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 crypto rx=0x7f79dc0098e0 tx=0x7f79dc009250 comp rx=0 tx=0).stop 2023-12-16T21:32:55.760 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2023-12-16T21:32:55.412+0000 7f79eccc7700 1 -- 172.21.15.45:0/3610257293 shutdown_connections 2023-12-16T21:32:55.760 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2023-12-16T21:32:55.412+0000 7f79eccc7700 1 --2- 172.21.15.45:0/3610257293 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f79e8070f60 0x7f79e8107340 unknown :-1 s=CLOSED pgs=5 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:32:55.760 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2023-12-16T21:32:55.412+0000 7f79eccc7700 1 -- 172.21.15.45:0/3610257293 >> 172.21.15.45:0/3610257293 conn(0x7f79e80fc970 msgr2=0x7f79e80fed80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-16T21:32:55.761 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2023-12-16T21:32:55.412+0000 7f79eccc7700 1 -- 172.21.15.45:0/3610257293 shutdown_connections 2023-12-16T21:32:55.761 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2023-12-16T21:32:55.412+0000 7f79eccc7700 1 -- 172.21.15.45:0/3610257293 wait complete. 2023-12-16T21:32:55.761 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2023-12-16T21:32:55.413+0000 7f79eccc7700 1 Processor -- start 2023-12-16T21:32:55.761 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2023-12-16T21:32:55.413+0000 7f79eccc7700 1 -- start start 2023-12-16T21:32:55.761 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2023-12-16T21:32:55.413+0000 7f79eccc7700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f79e8070f60 0x7f79e8113390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:32:55.761 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2023-12-16T21:32:55.413+0000 7f79eccc7700 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f79e81138d0 con 0x7f79e8070f60 2023-12-16T21:32:55.761 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2023-12-16T21:32:55.413+0000 7f79e659c700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f79e8070f60 0x7f79e8113390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:32:55.761 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2023-12-16T21:32:55.413+0000 7f79e659c700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f79e8070f60 0x7f79e8113390 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:48088/0 (socket says 172.21.15.45:48088) 2023-12-16T21:32:55.761 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2023-12-16T21:32:55.413+0000 7f79e659c700 1 -- 172.21.15.45:0/1659092037 learned_addr learned my addr 172.21.15.45:0/1659092037 (peer_addr_for_me v2:172.21.15.45:0/0) 2023-12-16T21:32:55.761 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2023-12-16T21:32:55.413+0000 7f79e659c700 1 -- 172.21.15.45:0/1659092037 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f79dc0094f0 con 0x7f79e8070f60 2023-12-16T21:32:55.761 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2023-12-16T21:32:55.413+0000 7f79e659c700 1 --2- 172.21.15.45:0/1659092037 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f79e8070f60 0x7f79e8113390 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7f79dc006d40 tx=0x7f79dc009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:32:55.761 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2023-12-16T21:32:55.414+0000 7f79d6ffd700 1 -- 172.21.15.45:0/1659092037 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f79dc0109b0 con 0x7f79e8070f60 2023-12-16T21:32:55.761 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2023-12-16T21:32:55.414+0000 7f79d6ffd700 1 -- 172.21.15.45:0/1659092037 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(26 keys) v1 ==== 1009+0+0 (secure 0 0 0) 0x7f79dc024440 con 0x7f79e8070f60 2023-12-16T21:32:55.761 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2023-12-16T21:32:55.414+0000 7f79eccc7700 1 -- 172.21.15.45:0/1659092037 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f79e8113ad0 con 0x7f79e8070f60 2023-12-16T21:32:55.762 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2023-12-16T21:32:55.414+0000 7f79d6ffd700 1 -- 172.21.15.45:0/1659092037 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f79dc01a5f0 con 0x7f79e8070f60 2023-12-16T21:32:55.762 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2023-12-16T21:32:55.414+0000 7f79eccc7700 1 -- 172.21.15.45:0/1659092037 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f79e8113ee0 con 0x7f79e8070f60 2023-12-16T21:32:55.762 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2023-12-16T21:32:55.414+0000 7f79d6ffd700 1 -- 172.21.15.45:0/1659092037 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 1) v1 ==== 521+0+0 (secure 0 0 0) 0x7f79dc01aaf0 con 0x7f79e8070f60 2023-12-16T21:32:55.762 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2023-12-16T21:32:55.414+0000 7f79d6ffd700 1 -- 172.21.15.45:0/1659092037 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 721+0+0 (secure 0 0 0) 0x7f79dc02da60 con 0x7f79e8070f60 2023-12-16T21:32:55.762 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2023-12-16T21:32:55.415+0000 7f79eccc7700 1 -- 172.21.15.45:0/1659092037 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f79e804bb80 con 0x7f79e8070f60 2023-12-16T21:32:55.762 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2023-12-16T21:32:55.419+0000 7f79d6ffd700 1 -- 172.21.15.45:0/1659092037 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+71349 (secure 0 0 0) 0x7f79dc024970 con 0x7f79e8070f60 2023-12-16T21:32:55.762 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2023-12-16T21:32:55.492+0000 7f79eccc7700 1 -- 172.21.15.45:0/1659092037 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "status", "format": "json-pretty"} v 0) v1 -- 0x7f79e8108510 con 0x7f79e8070f60 2023-12-16T21:32:55.762 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2023-12-16T21:32:55.493+0000 7f79d6ffd700 1 -- 172.21.15.45:0/1659092037 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mon_command_ack([{"prefix": "status", "format": "json-pretty"}]=0 v0) v1 ==== 79+0+1248 (secure 0 0 0) 0x7f79dc0245b0 con 0x7f79e8070f60 2023-12-16T21:32:55.762 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2023-12-16T21:32:55.494+0000 7f79eccc7700 1 -- 172.21.15.45:0/1659092037 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f79e8070f60 msgr2=0x7f79e8113390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:32:55.762 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2023-12-16T21:32:55.494+0000 7f79eccc7700 1 --2- 172.21.15.45:0/1659092037 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f79e8070f60 0x7f79e8113390 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7f79dc006d40 tx=0x7f79dc009250 comp rx=0 tx=0).stop 2023-12-16T21:32:55.762 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2023-12-16T21:32:55.494+0000 7f79eccc7700 1 -- 172.21.15.45:0/1659092037 shutdown_connections 2023-12-16T21:32:55.762 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2023-12-16T21:32:55.494+0000 7f79eccc7700 1 --2- 172.21.15.45:0/1659092037 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f79e8070f60 0x7f79e8113390 unknown :-1 s=CLOSED pgs=6 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:32:55.762 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2023-12-16T21:32:55.494+0000 7f79eccc7700 1 -- 172.21.15.45:0/1659092037 >> 172.21.15.45:0/1659092037 conn(0x7f79e80fc970 msgr2=0x7f79e80fe190 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-16T21:32:55.762 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2023-12-16T21:32:55.494+0000 7f79eccc7700 1 -- 172.21.15.45:0/1659092037 shutdown_connections 2023-12-16T21:32:55.763 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2023-12-16T21:32:55.494+0000 7f79eccc7700 1 -- 172.21.15.45:0/1659092037 wait complete. 2023-12-16T21:32:55.763 INFO:teuthology.orchestra.run.smithi045.stdout:mgr not available, waiting (1/15)... 2023-12-16T21:32:58.347 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stdout 2023-12-16T21:32:58.347 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stdout { 2023-12-16T21:32:58.347 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stdout "fsid": "88ff363c-9c5a-11ee-95a5-87774f69a715", 2023-12-16T21:32:58.347 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stdout "health": { 2023-12-16T21:32:58.347 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stdout "status": "HEALTH_OK", 2023-12-16T21:32:58.347 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stdout "checks": {}, 2023-12-16T21:32:58.347 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stdout "mutes": [] 2023-12-16T21:32:58.347 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stdout }, 2023-12-16T21:32:58.347 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stdout "election_epoch": 5, 2023-12-16T21:32:58.347 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stdout "quorum": [ 2023-12-16T21:32:58.347 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stdout 0 2023-12-16T21:32:58.348 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stdout ], 2023-12-16T21:32:58.348 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stdout "quorum_names": [ 2023-12-16T21:32:58.348 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stdout "smithi045" 2023-12-16T21:32:58.348 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stdout ], 2023-12-16T21:32:58.348 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stdout "quorum_age": 6, 2023-12-16T21:32:58.348 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stdout "monmap": { 2023-12-16T21:32:58.348 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stdout "epoch": 1, 2023-12-16T21:32:58.348 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stdout "min_mon_release_name": "quincy", 2023-12-16T21:32:58.348 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stdout "num_mons": 1 2023-12-16T21:32:58.348 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stdout }, 2023-12-16T21:32:58.348 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stdout "osdmap": { 2023-12-16T21:32:58.348 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stdout "epoch": 1, 2023-12-16T21:32:58.348 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stdout "num_osds": 0, 2023-12-16T21:32:58.348 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stdout "num_up_osds": 0, 2023-12-16T21:32:58.348 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stdout "osd_up_since": 0, 2023-12-16T21:32:58.356 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stdout "num_in_osds": 0, 2023-12-16T21:32:58.356 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stdout "osd_in_since": 0, 2023-12-16T21:32:58.356 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stdout "num_remapped_pgs": 0 2023-12-16T21:32:58.356 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stdout }, 2023-12-16T21:32:58.356 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stdout "pgmap": { 2023-12-16T21:32:58.356 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stdout "pgs_by_state": [], 2023-12-16T21:32:58.356 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stdout "num_pgs": 0, 2023-12-16T21:32:58.356 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stdout "num_pools": 0, 2023-12-16T21:32:58.356 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stdout "num_objects": 0, 2023-12-16T21:32:58.356 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stdout "data_bytes": 0, 2023-12-16T21:32:58.356 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stdout "bytes_used": 0, 2023-12-16T21:32:58.357 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stdout "bytes_avail": 0, 2023-12-16T21:32:58.357 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stdout "bytes_total": 0 2023-12-16T21:32:58.357 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stdout }, 2023-12-16T21:32:58.357 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stdout "fsmap": { 2023-12-16T21:32:58.357 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stdout "epoch": 1, 2023-12-16T21:32:58.357 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stdout "by_rank": [], 2023-12-16T21:32:58.357 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stdout "up:standby": 0 2023-12-16T21:32:58.357 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stdout }, 2023-12-16T21:32:58.357 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stdout "mgrmap": { 2023-12-16T21:32:58.357 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stdout "available": false, 2023-12-16T21:32:58.357 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stdout "num_standbys": 0, 2023-12-16T21:32:58.357 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stdout "modules": [ 2023-12-16T21:32:58.357 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stdout "iostat", 2023-12-16T21:32:58.357 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stdout "nfs", 2023-12-16T21:32:58.357 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stdout "restful" 2023-12-16T21:32:58.358 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stdout ], 2023-12-16T21:32:58.358 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stdout "services": {} 2023-12-16T21:32:58.358 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stdout }, 2023-12-16T21:32:58.358 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stdout "servicemap": { 2023-12-16T21:32:58.358 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stdout "epoch": 1, 2023-12-16T21:32:58.358 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stdout "modified": "2023-12-16T21:32:48.503304+0000", 2023-12-16T21:32:58.358 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stdout "services": {} 2023-12-16T21:32:58.358 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stdout }, 2023-12-16T21:32:58.358 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stdout "progress_events": {} 2023-12-16T21:32:58.358 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stdout } 2023-12-16T21:32:58.358 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2023-12-16T21:32:58.022+0000 7fcb594ba700 1 Processor -- start 2023-12-16T21:32:58.358 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2023-12-16T21:32:58.022+0000 7fcb594ba700 1 -- start start 2023-12-16T21:32:58.358 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2023-12-16T21:32:58.022+0000 7fcb594ba700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fcb541038f0 0x7fcb54103d00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:32:58.358 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2023-12-16T21:32:58.022+0000 7fcb594ba700 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcb54104240 con 0x7fcb541038f0 2023-12-16T21:32:58.359 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2023-12-16T21:32:58.023+0000 7fcb52ffd700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fcb541038f0 0x7fcb54103d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:32:58.359 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2023-12-16T21:32:58.023+0000 7fcb52ffd700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fcb541038f0 0x7fcb54103d00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:48090/0 (socket says 172.21.15.45:48090) 2023-12-16T21:32:58.359 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2023-12-16T21:32:58.023+0000 7fcb52ffd700 1 -- 172.21.15.45:0/3247136860 learned_addr learned my addr 172.21.15.45:0/3247136860 (peer_addr_for_me v2:172.21.15.45:0/0) 2023-12-16T21:32:58.359 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2023-12-16T21:32:58.023+0000 7fcb52ffd700 1 -- 172.21.15.45:0/3247136860 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcb54104380 con 0x7fcb541038f0 2023-12-16T21:32:58.359 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2023-12-16T21:32:58.023+0000 7fcb52ffd700 1 --2- 172.21.15.45:0/3247136860 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fcb541038f0 0x7fcb54103d00 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7fcb3c0098e0 tx=0x7fcb3c009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=acdd18251b46a833 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:32:58.359 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2023-12-16T21:32:58.023+0000 7fcb51ffb700 1 -- 172.21.15.45:0/3247136860 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fcb3c00b860 con 0x7fcb541038f0 2023-12-16T21:32:58.359 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2023-12-16T21:32:58.023+0000 7fcb51ffb700 1 -- 172.21.15.45:0/3247136860 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(26 keys) v1 ==== 1009+0+0 (secure 0 0 0) 0x7fcb3c00be30 con 0x7fcb541038f0 2023-12-16T21:32:58.359 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2023-12-16T21:32:58.023+0000 7fcb51ffb700 1 -- 172.21.15.45:0/3247136860 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fcb3c010380 con 0x7fcb541038f0 2023-12-16T21:32:58.359 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2023-12-16T21:32:58.024+0000 7fcb594ba700 1 -- 172.21.15.45:0/3247136860 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fcb541038f0 msgr2=0x7fcb54103d00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:32:58.359 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2023-12-16T21:32:58.024+0000 7fcb594ba700 1 --2- 172.21.15.45:0/3247136860 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fcb541038f0 0x7fcb54103d00 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7fcb3c0098e0 tx=0x7fcb3c009250 comp rx=0 tx=0).stop 2023-12-16T21:32:58.359 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2023-12-16T21:32:58.024+0000 7fcb594ba700 1 -- 172.21.15.45:0/3247136860 shutdown_connections 2023-12-16T21:32:58.359 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2023-12-16T21:32:58.024+0000 7fcb594ba700 1 --2- 172.21.15.45:0/3247136860 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fcb541038f0 0x7fcb54103d00 unknown :-1 s=CLOSED pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:32:58.359 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2023-12-16T21:32:58.024+0000 7fcb594ba700 1 -- 172.21.15.45:0/3247136860 >> 172.21.15.45:0/3247136860 conn(0x7fcb540fc0b0 msgr2=0x7fcb540fe4e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-16T21:32:58.359 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2023-12-16T21:32:58.024+0000 7fcb594ba700 1 -- 172.21.15.45:0/3247136860 shutdown_connections 2023-12-16T21:32:58.359 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2023-12-16T21:32:58.024+0000 7fcb594ba700 1 -- 172.21.15.45:0/3247136860 wait complete. 2023-12-16T21:32:58.360 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2023-12-16T21:32:58.025+0000 7fcb594ba700 1 Processor -- start 2023-12-16T21:32:58.360 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2023-12-16T21:32:58.025+0000 7fcb594ba700 1 -- start start 2023-12-16T21:32:58.360 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2023-12-16T21:32:58.025+0000 7fcb594ba700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fcb541038f0 0x7fcb54103130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:32:58.360 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2023-12-16T21:32:58.025+0000 7fcb594ba700 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcb541015b0 con 0x7fcb541038f0 2023-12-16T21:32:58.360 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2023-12-16T21:32:58.025+0000 7fcb52ffd700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fcb541038f0 0x7fcb54103130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:32:58.360 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2023-12-16T21:32:58.025+0000 7fcb52ffd700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fcb541038f0 0x7fcb54103130 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:48096/0 (socket says 172.21.15.45:48096) 2023-12-16T21:32:58.360 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2023-12-16T21:32:58.025+0000 7fcb52ffd700 1 -- 172.21.15.45:0/48170738 learned_addr learned my addr 172.21.15.45:0/48170738 (peer_addr_for_me v2:172.21.15.45:0/0) 2023-12-16T21:32:58.360 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2023-12-16T21:32:58.025+0000 7fcb52ffd700 1 -- 172.21.15.45:0/48170738 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcb3c0094f0 con 0x7fcb541038f0 2023-12-16T21:32:58.360 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2023-12-16T21:32:58.026+0000 7fcb52ffd700 1 --2- 172.21.15.45:0/48170738 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fcb541038f0 0x7fcb54103130 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7fcb3c000c00 tx=0x7fcb3c009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:32:58.360 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2023-12-16T21:32:58.026+0000 7fcb4b7fe700 1 -- 172.21.15.45:0/48170738 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fcb3c010a80 con 0x7fcb541038f0 2023-12-16T21:32:58.360 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2023-12-16T21:32:58.026+0000 7fcb4b7fe700 1 -- 172.21.15.45:0/48170738 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(26 keys) v1 ==== 1009+0+0 (secure 0 0 0) 0x7fcb3c01a430 con 0x7fcb541038f0 2023-12-16T21:32:58.360 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2023-12-16T21:32:58.026+0000 7fcb594ba700 1 -- 172.21.15.45:0/48170738 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcb541017b0 con 0x7fcb541038f0 2023-12-16T21:32:58.360 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2023-12-16T21:32:58.026+0000 7fcb594ba700 1 -- 172.21.15.45:0/48170738 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcb54101bc0 con 0x7fcb541038f0 2023-12-16T21:32:58.360 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2023-12-16T21:32:58.026+0000 7fcb4b7fe700 1 -- 172.21.15.45:0/48170738 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fcb3c0245b0 con 0x7fcb541038f0 2023-12-16T21:32:58.361 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2023-12-16T21:32:58.026+0000 7fcb4b7fe700 1 -- 172.21.15.45:0/48170738 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 1) v1 ==== 521+0+0 (secure 0 0 0) 0x7fcb3c024a20 con 0x7fcb541038f0 2023-12-16T21:32:58.361 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2023-12-16T21:32:58.026+0000 7fcb4b7fe700 1 -- 172.21.15.45:0/48170738 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 721+0+0 (secure 0 0 0) 0x7fcb3c02eb50 con 0x7fcb541038f0 2023-12-16T21:32:58.361 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2023-12-16T21:32:58.027+0000 7fcb594ba700 1 -- 172.21.15.45:0/48170738 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcb340052e0 con 0x7fcb541038f0 2023-12-16T21:32:58.361 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2023-12-16T21:32:58.029+0000 7fcb4b7fe700 1 -- 172.21.15.45:0/48170738 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+71349 (secure 0 0 0) 0x7fcb3c02edb0 con 0x7fcb541038f0 2023-12-16T21:32:58.361 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2023-12-16T21:32:58.105+0000 7fcb594ba700 1 -- 172.21.15.45:0/48170738 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "status", "format": "json-pretty"} v 0) v1 -- 0x7fcb34005860 con 0x7fcb541038f0 2023-12-16T21:32:58.361 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2023-12-16T21:32:58.105+0000 7fcb4b7fe700 1 -- 172.21.15.45:0/48170738 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mon_command_ack([{"prefix": "status", "format": "json-pretty"}]=0 v0) v1 ==== 79+0+1248 (secure 0 0 0) 0x7fcb3c010be0 con 0x7fcb541038f0 2023-12-16T21:32:58.361 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2023-12-16T21:32:58.106+0000 7fcb594ba700 1 -- 172.21.15.45:0/48170738 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fcb541038f0 msgr2=0x7fcb54103130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:32:58.361 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2023-12-16T21:32:58.106+0000 7fcb594ba700 1 --2- 172.21.15.45:0/48170738 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fcb541038f0 0x7fcb54103130 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7fcb3c000c00 tx=0x7fcb3c009250 comp rx=0 tx=0).stop 2023-12-16T21:32:58.361 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2023-12-16T21:32:58.106+0000 7fcb594ba700 1 -- 172.21.15.45:0/48170738 shutdown_connections 2023-12-16T21:32:58.361 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2023-12-16T21:32:58.106+0000 7fcb594ba700 1 --2- 172.21.15.45:0/48170738 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fcb541038f0 0x7fcb54103130 unknown :-1 s=CLOSED pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:32:58.361 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2023-12-16T21:32:58.106+0000 7fcb594ba700 1 -- 172.21.15.45:0/48170738 >> 172.21.15.45:0/48170738 conn(0x7fcb540fc0b0 msgr2=0x7fcb540fd810 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-16T21:32:58.361 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2023-12-16T21:32:58.106+0000 7fcb594ba700 1 -- 172.21.15.45:0/48170738 shutdown_connections 2023-12-16T21:32:58.361 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2023-12-16T21:32:58.106+0000 7fcb594ba700 1 -- 172.21.15.45:0/48170738 wait complete. 2023-12-16T21:32:58.361 INFO:teuthology.orchestra.run.smithi045.stdout:mgr not available, waiting (2/15)... 2023-12-16T21:32:58.565 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Dec 16 21:32:58 smithi045 ceph-mon[80763]: from='client.? 172.21.15.45:0/48170738' entity='client.admin' cmd=[{"prefix": "status", "format": "json-pretty"}]: dispatch 2023-12-16T21:33:02.566 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Dec 16 21:33:02 smithi045 ceph-mon[80763]: from='client.? 172.21.15.45:0/926428800' entity='client.admin' cmd=[{"prefix": "status", "format": "json-pretty"}]: dispatch 2023-12-16T21:33:02.567 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Dec 16 21:33:02 smithi045 ceph-mon[80763]: Activating manager daemon smithi045.nhtmwc 2023-12-16T21:33:02.567 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Dec 16 21:33:02 smithi045 ceph-mon[80763]: mgrmap e2: smithi045.nhtmwc(active, starting, since 0.0028775s) 2023-12-16T21:33:02.567 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Dec 16 21:33:02 smithi045 ceph-mon[80763]: from='mgr.14100 172.21.15.45:0/689229981' entity='mgr.smithi045.nhtmwc' cmd=[{"prefix": "mds metadata"}]: dispatch 2023-12-16T21:33:02.567 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Dec 16 21:33:02 smithi045 ceph-mon[80763]: from='mgr.14100 172.21.15.45:0/689229981' entity='mgr.smithi045.nhtmwc' cmd=[{"prefix": "osd metadata"}]: dispatch 2023-12-16T21:33:02.567 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Dec 16 21:33:02 smithi045 ceph-mon[80763]: from='mgr.14100 172.21.15.45:0/689229981' entity='mgr.smithi045.nhtmwc' cmd=[{"prefix": "mon metadata"}]: dispatch 2023-12-16T21:33:02.567 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Dec 16 21:33:02 smithi045 ceph-mon[80763]: from='mgr.14100 172.21.15.45:0/689229981' entity='mgr.smithi045.nhtmwc' cmd=[{"prefix": "mon metadata", "id": "smithi045"}]: dispatch 2023-12-16T21:33:02.567 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Dec 16 21:33:02 smithi045 ceph-mon[80763]: from='mgr.14100 172.21.15.45:0/689229981' entity='mgr.smithi045.nhtmwc' cmd=[{"prefix": "mgr metadata", "who": "smithi045.nhtmwc", "id": "smithi045.nhtmwc"}]: dispatch 2023-12-16T21:33:02.568 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Dec 16 21:33:02 smithi045 ceph-mon[80763]: Manager daemon smithi045.nhtmwc is now available 2023-12-16T21:33:02.568 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Dec 16 21:33:02 smithi045 ceph-mon[80763]: from='mgr.14100 172.21.15.45:0/689229981' entity='mgr.smithi045.nhtmwc' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi045.nhtmwc/mirror_snapshot_schedule"}]: dispatch 2023-12-16T21:33:02.568 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Dec 16 21:33:02 smithi045 ceph-mon[80763]: from='mgr.14100 172.21.15.45:0/689229981' entity='mgr.smithi045.nhtmwc' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi045.nhtmwc/trash_purge_schedule"}]: dispatch 2023-12-16T21:33:02.568 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Dec 16 21:33:02 smithi045 ceph-mon[80763]: from='mgr.14100 172.21.15.45:0/689229981' entity='mgr.smithi045.nhtmwc' 2023-12-16T21:33:02.568 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Dec 16 21:33:02 smithi045 ceph-mon[80763]: from='mgr.14100 172.21.15.45:0/689229981' entity='mgr.smithi045.nhtmwc' 2023-12-16T21:33:02.568 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Dec 16 21:33:02 smithi045 ceph-mon[80763]: from='mgr.14100 172.21.15.45:0/689229981' entity='mgr.smithi045.nhtmwc' 2023-12-16T21:33:03.332 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stdout 2023-12-16T21:33:03.332 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stdout { 2023-12-16T21:33:03.332 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stdout "fsid": "88ff363c-9c5a-11ee-95a5-87774f69a715", 2023-12-16T21:33:03.332 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stdout "health": { 2023-12-16T21:33:03.332 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stdout "status": "HEALTH_OK", 2023-12-16T21:33:03.332 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stdout "checks": {}, 2023-12-16T21:33:03.333 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stdout "mutes": [] 2023-12-16T21:33:03.333 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stdout }, 2023-12-16T21:33:03.333 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stdout "election_epoch": 5, 2023-12-16T21:33:03.333 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stdout "quorum": [ 2023-12-16T21:33:03.333 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stdout 0 2023-12-16T21:33:03.333 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stdout ], 2023-12-16T21:33:03.333 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stdout "quorum_names": [ 2023-12-16T21:33:03.333 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stdout "smithi045" 2023-12-16T21:33:03.333 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stdout ], 2023-12-16T21:33:03.333 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stdout "quorum_age": 10, 2023-12-16T21:33:03.333 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stdout "monmap": { 2023-12-16T21:33:03.333 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stdout "epoch": 1, 2023-12-16T21:33:03.334 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stdout "min_mon_release_name": "quincy", 2023-12-16T21:33:03.334 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stdout "num_mons": 1 2023-12-16T21:33:03.334 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stdout }, 2023-12-16T21:33:03.334 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stdout "osdmap": { 2023-12-16T21:33:03.334 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stdout "epoch": 1, 2023-12-16T21:33:03.334 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stdout "num_osds": 0, 2023-12-16T21:33:03.334 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stdout "num_up_osds": 0, 2023-12-16T21:33:03.334 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stdout "osd_up_since": 0, 2023-12-16T21:33:03.334 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stdout "num_in_osds": 0, 2023-12-16T21:33:03.334 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stdout "osd_in_since": 0, 2023-12-16T21:33:03.343 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stdout "num_remapped_pgs": 0 2023-12-16T21:33:03.343 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stdout }, 2023-12-16T21:33:03.343 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stdout "pgmap": { 2023-12-16T21:33:03.343 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stdout "pgs_by_state": [], 2023-12-16T21:33:03.343 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stdout "num_pgs": 0, 2023-12-16T21:33:03.343 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stdout "num_pools": 0, 2023-12-16T21:33:03.343 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stdout "num_objects": 0, 2023-12-16T21:33:03.343 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stdout "data_bytes": 0, 2023-12-16T21:33:03.343 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stdout "bytes_used": 0, 2023-12-16T21:33:03.343 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stdout "bytes_avail": 0, 2023-12-16T21:33:03.343 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stdout "bytes_total": 0 2023-12-16T21:33:03.343 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stdout }, 2023-12-16T21:33:03.343 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stdout "fsmap": { 2023-12-16T21:33:03.344 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stdout "epoch": 1, 2023-12-16T21:33:03.344 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stdout "by_rank": [], 2023-12-16T21:33:03.344 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stdout "up:standby": 0 2023-12-16T21:33:03.344 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stdout }, 2023-12-16T21:33:03.344 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stdout "mgrmap": { 2023-12-16T21:33:03.344 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stdout "available": false, 2023-12-16T21:33:03.344 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stdout "num_standbys": 0, 2023-12-16T21:33:03.344 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stdout "modules": [ 2023-12-16T21:33:03.344 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stdout "iostat", 2023-12-16T21:33:03.344 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stdout "nfs", 2023-12-16T21:33:03.344 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stdout "restful" 2023-12-16T21:33:03.344 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stdout ], 2023-12-16T21:33:03.344 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stdout "services": {} 2023-12-16T21:33:03.344 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stdout }, 2023-12-16T21:33:03.345 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stdout "servicemap": { 2023-12-16T21:33:03.345 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stdout "epoch": 1, 2023-12-16T21:33:03.345 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stdout "modified": "2023-12-16T21:32:48.503304+0000", 2023-12-16T21:33:03.345 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stdout "services": {} 2023-12-16T21:33:03.345 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stdout }, 2023-12-16T21:33:03.345 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stdout "progress_events": {} 2023-12-16T21:33:03.345 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stdout } 2023-12-16T21:33:03.345 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2023-12-16T21:33:01.371+0000 7fdd6c835700 1 Processor -- start 2023-12-16T21:33:03.345 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2023-12-16T21:33:01.371+0000 7fdd6c835700 1 -- start start 2023-12-16T21:33:03.345 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2023-12-16T21:33:01.371+0000 7fdd6c835700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fdd681022d0 0x7fdd681026e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:33:03.345 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2023-12-16T21:33:01.371+0000 7fdd6c835700 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdd68102c20 con 0x7fdd681022d0 2023-12-16T21:33:03.345 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2023-12-16T21:33:01.372+0000 7fdd6659c700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fdd681022d0 0x7fdd681026e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:33:03.345 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2023-12-16T21:33:01.372+0000 7fdd6659c700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fdd681022d0 0x7fdd681026e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:48112/0 (socket says 172.21.15.45:48112) 2023-12-16T21:33:03.345 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2023-12-16T21:33:01.372+0000 7fdd6659c700 1 -- 172.21.15.45:0/3784515316 learned_addr learned my addr 172.21.15.45:0/3784515316 (peer_addr_for_me v2:172.21.15.45:0/0) 2023-12-16T21:33:03.346 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2023-12-16T21:33:01.372+0000 7fdd6659c700 1 -- 172.21.15.45:0/3784515316 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdd68102d60 con 0x7fdd681022d0 2023-12-16T21:33:03.346 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2023-12-16T21:33:01.372+0000 7fdd6659c700 1 --2- 172.21.15.45:0/3784515316 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fdd681022d0 0x7fdd681026e0 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7fdd500098e0 tx=0x7fdd50009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cda4be40befd6392 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:33:03.346 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2023-12-16T21:33:01.372+0000 7fdd6559a700 1 -- 172.21.15.45:0/3784515316 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fdd5000b860 con 0x7fdd681022d0 2023-12-16T21:33:03.346 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2023-12-16T21:33:01.372+0000 7fdd6559a700 1 -- 172.21.15.45:0/3784515316 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(26 keys) v1 ==== 1009+0+0 (secure 0 0 0) 0x7fdd5000be30 con 0x7fdd681022d0 2023-12-16T21:33:03.346 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2023-12-16T21:33:01.372+0000 7fdd6559a700 1 -- 172.21.15.45:0/3784515316 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fdd50010380 con 0x7fdd681022d0 2023-12-16T21:33:03.346 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2023-12-16T21:33:01.373+0000 7fdd6c835700 1 -- 172.21.15.45:0/3784515316 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fdd681022d0 msgr2=0x7fdd681026e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:33:03.346 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2023-12-16T21:33:01.373+0000 7fdd6c835700 1 --2- 172.21.15.45:0/3784515316 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fdd681022d0 0x7fdd681026e0 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7fdd500098e0 tx=0x7fdd50009250 comp rx=0 tx=0).stop 2023-12-16T21:33:03.346 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2023-12-16T21:33:01.373+0000 7fdd6c835700 1 -- 172.21.15.45:0/3784515316 shutdown_connections 2023-12-16T21:33:03.346 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2023-12-16T21:33:01.373+0000 7fdd6c835700 1 --2- 172.21.15.45:0/3784515316 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fdd681022d0 0x7fdd681026e0 unknown :-1 s=CLOSED pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:33:03.346 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2023-12-16T21:33:01.373+0000 7fdd6c835700 1 -- 172.21.15.45:0/3784515316 >> 172.21.15.45:0/3784515316 conn(0x7fdd680fda50 msgr2=0x7fdd680ffe60 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-16T21:33:03.346 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2023-12-16T21:33:01.373+0000 7fdd6c835700 1 -- 172.21.15.45:0/3784515316 shutdown_connections 2023-12-16T21:33:03.346 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2023-12-16T21:33:01.373+0000 7fdd6c835700 1 -- 172.21.15.45:0/3784515316 wait complete. 2023-12-16T21:33:03.346 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2023-12-16T21:33:01.373+0000 7fdd6c835700 1 Processor -- start 2023-12-16T21:33:03.347 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2023-12-16T21:33:01.373+0000 7fdd6c835700 1 -- start start 2023-12-16T21:33:03.347 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2023-12-16T21:33:01.374+0000 7fdd6c835700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fdd681022d0 0x7fdd68111140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:33:03.347 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2023-12-16T21:33:01.374+0000 7fdd6c835700 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdd68111680 con 0x7fdd681022d0 2023-12-16T21:33:03.347 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2023-12-16T21:33:01.374+0000 7fdd6659c700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fdd681022d0 0x7fdd68111140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:33:03.347 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2023-12-16T21:33:01.374+0000 7fdd6659c700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fdd681022d0 0x7fdd68111140 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:48120/0 (socket says 172.21.15.45:48120) 2023-12-16T21:33:03.347 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2023-12-16T21:33:01.374+0000 7fdd6659c700 1 -- 172.21.15.45:0/926428800 learned_addr learned my addr 172.21.15.45:0/926428800 (peer_addr_for_me v2:172.21.15.45:0/0) 2023-12-16T21:33:03.347 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2023-12-16T21:33:01.374+0000 7fdd6659c700 1 -- 172.21.15.45:0/926428800 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdd500094f0 con 0x7fdd681022d0 2023-12-16T21:33:03.347 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2023-12-16T21:33:01.374+0000 7fdd6659c700 1 --2- 172.21.15.45:0/926428800 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fdd681022d0 0x7fdd68111140 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7fdd50006b00 tx=0x7fdd50009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:33:03.347 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2023-12-16T21:33:01.374+0000 7fdd5effd700 1 -- 172.21.15.45:0/926428800 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fdd500109b0 con 0x7fdd681022d0 2023-12-16T21:33:03.347 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2023-12-16T21:33:01.374+0000 7fdd5effd700 1 -- 172.21.15.45:0/926428800 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(26 keys) v1 ==== 1009+0+0 (secure 0 0 0) 0x7fdd50024440 con 0x7fdd681022d0 2023-12-16T21:33:03.347 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2023-12-16T21:33:01.374+0000 7fdd6c835700 1 -- 172.21.15.45:0/926428800 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdd68111880 con 0x7fdd681022d0 2023-12-16T21:33:03.348 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2023-12-16T21:33:01.374+0000 7fdd5effd700 1 -- 172.21.15.45:0/926428800 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fdd5001a5f0 con 0x7fdd681022d0 2023-12-16T21:33:03.348 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2023-12-16T21:33:01.374+0000 7fdd6c835700 1 -- 172.21.15.45:0/926428800 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdd68111c90 con 0x7fdd681022d0 2023-12-16T21:33:03.348 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2023-12-16T21:33:01.375+0000 7fdd5effd700 1 -- 172.21.15.45:0/926428800 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 1) v1 ==== 521+0+0 (secure 0 0 0) 0x7fdd5001ab30 con 0x7fdd681022d0 2023-12-16T21:33:03.348 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2023-12-16T21:33:01.375+0000 7fdd5effd700 1 -- 172.21.15.45:0/926428800 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 721+0+0 (secure 0 0 0) 0x7fdd5002db50 con 0x7fdd681022d0 2023-12-16T21:33:03.348 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2023-12-16T21:33:01.375+0000 7fdd6c835700 1 -- 172.21.15.45:0/926428800 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdd6804bb80 con 0x7fdd681022d0 2023-12-16T21:33:03.348 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2023-12-16T21:33:01.378+0000 7fdd5effd700 1 -- 172.21.15.45:0/926428800 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+71349 (secure 0 0 0) 0x7fdd50006d70 con 0x7fdd681022d0 2023-12-16T21:33:03.348 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2023-12-16T21:33:01.452+0000 7fdd6c835700 1 -- 172.21.15.45:0/926428800 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "status", "format": "json-pretty"} v 0) v1 -- 0x7fdd681040e0 con 0x7fdd681022d0 2023-12-16T21:33:03.348 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2023-12-16T21:33:01.452+0000 7fdd5effd700 1 -- 172.21.15.45:0/926428800 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mon_command_ack([{"prefix": "status", "format": "json-pretty"}]=0 v0) v1 ==== 79+0+1249 (secure 0 0 0) 0x7fdd50032030 con 0x7fdd681022d0 2023-12-16T21:33:03.348 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2023-12-16T21:33:01.453+0000 7fdd6c835700 1 -- 172.21.15.45:0/926428800 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fdd681022d0 msgr2=0x7fdd68111140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:33:03.348 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2023-12-16T21:33:01.453+0000 7fdd6c835700 1 --2- 172.21.15.45:0/926428800 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fdd681022d0 0x7fdd68111140 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7fdd50006b00 tx=0x7fdd50009250 comp rx=0 tx=0).stop 2023-12-16T21:33:03.348 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2023-12-16T21:33:01.454+0000 7fdd6c835700 1 -- 172.21.15.45:0/926428800 shutdown_connections 2023-12-16T21:33:03.348 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2023-12-16T21:33:01.454+0000 7fdd6c835700 1 --2- 172.21.15.45:0/926428800 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fdd681022d0 0x7fdd68111140 unknown :-1 s=CLOSED pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:33:03.348 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2023-12-16T21:33:01.454+0000 7fdd6c835700 1 -- 172.21.15.45:0/926428800 >> 172.21.15.45:0/926428800 conn(0x7fdd680fda50 msgr2=0x7fdd680ffe50 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-16T21:33:03.348 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2023-12-16T21:33:01.454+0000 7fdd6c835700 1 -- 172.21.15.45:0/926428800 shutdown_connections 2023-12-16T21:33:03.349 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2023-12-16T21:33:01.454+0000 7fdd6c835700 1 -- 172.21.15.45:0/926428800 wait complete. 2023-12-16T21:33:03.349 INFO:teuthology.orchestra.run.smithi045.stdout:mgr not available, waiting (3/15)... 2023-12-16T21:33:04.316 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Dec 16 21:33:03 smithi045 ceph-mon[80763]: mgrmap e3: smithi045.nhtmwc(active, since 1.00688s) 2023-12-16T21:33:06.316 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Dec 16 21:33:05 smithi045 ceph-mon[80763]: mgrmap e4: smithi045.nhtmwc(active, since 3s) 2023-12-16T21:33:07.136 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stdout 2023-12-16T21:33:07.136 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stdout { 2023-12-16T21:33:07.137 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stdout "fsid": "88ff363c-9c5a-11ee-95a5-87774f69a715", 2023-12-16T21:33:07.137 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stdout "health": { 2023-12-16T21:33:07.137 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stdout "status": "HEALTH_OK", 2023-12-16T21:33:07.137 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stdout "checks": {}, 2023-12-16T21:33:07.137 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stdout "mutes": [] 2023-12-16T21:33:07.137 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stdout }, 2023-12-16T21:33:07.137 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stdout "election_epoch": 5, 2023-12-16T21:33:07.137 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stdout "quorum": [ 2023-12-16T21:33:07.137 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stdout 0 2023-12-16T21:33:07.137 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stdout ], 2023-12-16T21:33:07.137 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stdout "quorum_names": [ 2023-12-16T21:33:07.137 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stdout "smithi045" 2023-12-16T21:33:07.137 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stdout ], 2023-12-16T21:33:07.137 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stdout "quorum_age": 15, 2023-12-16T21:33:07.138 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stdout "monmap": { 2023-12-16T21:33:07.138 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stdout "epoch": 1, 2023-12-16T21:33:07.138 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stdout "min_mon_release_name": "quincy", 2023-12-16T21:33:07.138 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stdout "num_mons": 1 2023-12-16T21:33:07.138 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stdout }, 2023-12-16T21:33:07.138 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stdout "osdmap": { 2023-12-16T21:33:07.138 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stdout "epoch": 1, 2023-12-16T21:33:07.138 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stdout "num_osds": 0, 2023-12-16T21:33:07.138 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stdout "num_up_osds": 0, 2023-12-16T21:33:07.138 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stdout "osd_up_since": 0, 2023-12-16T21:33:07.146 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stdout "num_in_osds": 0, 2023-12-16T21:33:07.146 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stdout "osd_in_since": 0, 2023-12-16T21:33:07.146 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stdout "num_remapped_pgs": 0 2023-12-16T21:33:07.146 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stdout }, 2023-12-16T21:33:07.146 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stdout "pgmap": { 2023-12-16T21:33:07.146 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stdout "pgs_by_state": [], 2023-12-16T21:33:07.146 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stdout "num_pgs": 0, 2023-12-16T21:33:07.146 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stdout "num_pools": 0, 2023-12-16T21:33:07.147 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stdout "num_objects": 0, 2023-12-16T21:33:07.147 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stdout "data_bytes": 0, 2023-12-16T21:33:07.147 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stdout "bytes_used": 0, 2023-12-16T21:33:07.147 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stdout "bytes_avail": 0, 2023-12-16T21:33:07.147 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stdout "bytes_total": 0 2023-12-16T21:33:07.147 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stdout }, 2023-12-16T21:33:07.147 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stdout "fsmap": { 2023-12-16T21:33:07.147 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stdout "epoch": 1, 2023-12-16T21:33:07.147 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stdout "by_rank": [], 2023-12-16T21:33:07.147 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stdout "up:standby": 0 2023-12-16T21:33:07.147 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stdout }, 2023-12-16T21:33:07.147 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stdout "mgrmap": { 2023-12-16T21:33:07.147 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stdout "available": true, 2023-12-16T21:33:07.147 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stdout "num_standbys": 0, 2023-12-16T21:33:07.148 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stdout "modules": [ 2023-12-16T21:33:07.148 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stdout "iostat", 2023-12-16T21:33:07.148 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stdout "nfs", 2023-12-16T21:33:07.148 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stdout "restful" 2023-12-16T21:33:07.148 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stdout ], 2023-12-16T21:33:07.148 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stdout "services": {} 2023-12-16T21:33:07.148 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stdout }, 2023-12-16T21:33:07.148 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stdout "servicemap": { 2023-12-16T21:33:07.148 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stdout "epoch": 1, 2023-12-16T21:33:07.148 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stdout "modified": "2023-12-16T21:32:48.503304+0000", 2023-12-16T21:33:07.148 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stdout "services": {} 2023-12-16T21:33:07.148 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stdout }, 2023-12-16T21:33:07.148 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stdout "progress_events": {} 2023-12-16T21:33:07.148 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stdout } 2023-12-16T21:33:07.148 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2023-12-16T21:33:06.616+0000 7f0b59998700 1 Processor -- start 2023-12-16T21:33:07.149 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2023-12-16T21:33:06.616+0000 7f0b59998700 1 -- start start 2023-12-16T21:33:07.149 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2023-12-16T21:33:06.616+0000 7f0b59998700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f0b54078da0 0x7f0b540771b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:33:07.149 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2023-12-16T21:33:06.616+0000 7f0b59998700 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0b540776f0 con 0x7f0b54078da0 2023-12-16T21:33:07.149 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2023-12-16T21:33:06.616+0000 7f0b52ffd700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f0b54078da0 0x7f0b540771b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:33:07.149 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2023-12-16T21:33:06.616+0000 7f0b52ffd700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f0b54078da0 0x7f0b540771b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:38058/0 (socket says 172.21.15.45:38058) 2023-12-16T21:33:07.149 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2023-12-16T21:33:06.616+0000 7f0b52ffd700 1 -- 172.21.15.45:0/1183027811 learned_addr learned my addr 172.21.15.45:0/1183027811 (peer_addr_for_me v2:172.21.15.45:0/0) 2023-12-16T21:33:07.149 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2023-12-16T21:33:06.616+0000 7f0b52ffd700 1 -- 172.21.15.45:0/1183027811 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0b54077830 con 0x7f0b54078da0 2023-12-16T21:33:07.149 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2023-12-16T21:33:06.617+0000 7f0b52ffd700 1 --2- 172.21.15.45:0/1183027811 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f0b54078da0 0x7f0b540771b0 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7f0b440098e0 tx=0x7f0b44009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=28a8dc74a9a0042e server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:33:07.149 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2023-12-16T21:33:06.617+0000 7f0b51ffb700 1 -- 172.21.15.45:0/1183027811 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f0b4400b860 con 0x7f0b54078da0 2023-12-16T21:33:07.149 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2023-12-16T21:33:06.617+0000 7f0b51ffb700 1 -- 172.21.15.45:0/1183027811 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(26 keys) v1 ==== 1009+0+0 (secure 0 0 0) 0x7f0b4400be30 con 0x7f0b54078da0 2023-12-16T21:33:07.149 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2023-12-16T21:33:06.617+0000 7f0b51ffb700 1 -- 172.21.15.45:0/1183027811 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f0b44010380 con 0x7f0b54078da0 2023-12-16T21:33:07.149 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2023-12-16T21:33:06.617+0000 7f0b59998700 1 -- 172.21.15.45:0/1183027811 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f0b54078da0 msgr2=0x7f0b540771b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:33:07.149 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2023-12-16T21:33:06.617+0000 7f0b59998700 1 --2- 172.21.15.45:0/1183027811 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f0b54078da0 0x7f0b540771b0 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7f0b440098e0 tx=0x7f0b44009250 comp rx=0 tx=0).stop 2023-12-16T21:33:07.149 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2023-12-16T21:33:06.617+0000 7f0b59998700 1 -- 172.21.15.45:0/1183027811 shutdown_connections 2023-12-16T21:33:07.150 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2023-12-16T21:33:06.617+0000 7f0b59998700 1 --2- 172.21.15.45:0/1183027811 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f0b54078da0 0x7f0b540771b0 unknown :-1 s=CLOSED pgs=18 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:33:07.150 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2023-12-16T21:33:06.617+0000 7f0b59998700 1 -- 172.21.15.45:0/1183027811 >> 172.21.15.45:0/1183027811 conn(0x7f0b540fda40 msgr2=0x7f0b540ffe70 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-16T21:33:07.150 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2023-12-16T21:33:06.618+0000 7f0b59998700 1 -- 172.21.15.45:0/1183027811 shutdown_connections 2023-12-16T21:33:07.150 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2023-12-16T21:33:06.618+0000 7f0b59998700 1 -- 172.21.15.45:0/1183027811 wait complete. 2023-12-16T21:33:07.150 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2023-12-16T21:33:06.618+0000 7f0b59998700 1 Processor -- start 2023-12-16T21:33:07.150 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2023-12-16T21:33:06.618+0000 7f0b59998700 1 -- start start 2023-12-16T21:33:07.150 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2023-12-16T21:33:06.618+0000 7f0b59998700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f0b54078da0 0x7f0b54113340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:33:07.150 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2023-12-16T21:33:06.619+0000 7f0b59998700 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0b54113880 con 0x7f0b54078da0 2023-12-16T21:33:07.150 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2023-12-16T21:33:06.619+0000 7f0b52ffd700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f0b54078da0 0x7f0b54113340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:33:07.150 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2023-12-16T21:33:06.619+0000 7f0b52ffd700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f0b54078da0 0x7f0b54113340 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:38072/0 (socket says 172.21.15.45:38072) 2023-12-16T21:33:07.150 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2023-12-16T21:33:06.619+0000 7f0b52ffd700 1 -- 172.21.15.45:0/247966975 learned_addr learned my addr 172.21.15.45:0/247966975 (peer_addr_for_me v2:172.21.15.45:0/0) 2023-12-16T21:33:07.150 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2023-12-16T21:33:06.619+0000 7f0b52ffd700 1 -- 172.21.15.45:0/247966975 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0b440094f0 con 0x7f0b54078da0 2023-12-16T21:33:07.150 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2023-12-16T21:33:06.619+0000 7f0b52ffd700 1 --2- 172.21.15.45:0/247966975 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f0b54078da0 0x7f0b54113340 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7f0b44006d20 tx=0x7f0b44009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:33:07.151 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2023-12-16T21:33:06.619+0000 7f0b3b7fe700 1 -- 172.21.15.45:0/247966975 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f0b440109b0 con 0x7f0b54078da0 2023-12-16T21:33:07.151 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2023-12-16T21:33:06.619+0000 7f0b3b7fe700 1 -- 172.21.15.45:0/247966975 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(26 keys) v1 ==== 1009+0+0 (secure 0 0 0) 0x7f0b44024440 con 0x7f0b54078da0 2023-12-16T21:33:07.151 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2023-12-16T21:33:06.619+0000 7f0b3b7fe700 1 -- 172.21.15.45:0/247966975 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f0b4401a5f0 con 0x7f0b54078da0 2023-12-16T21:33:07.151 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2023-12-16T21:33:06.619+0000 7f0b59998700 1 -- 172.21.15.45:0/247966975 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0b54113a80 con 0x7f0b54078da0 2023-12-16T21:33:07.151 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2023-12-16T21:33:06.619+0000 7f0b59998700 1 -- 172.21.15.45:0/247966975 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0b54113e90 con 0x7f0b54078da0 2023-12-16T21:33:07.151 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2023-12-16T21:33:06.620+0000 7f0b3b7fe700 1 -- 172.21.15.45:0/247966975 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 4) v1 ==== 43270+0+0 (secure 0 0 0) 0x7f0b4401a8b0 con 0x7f0b54078da0 2023-12-16T21:33:07.151 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2023-12-16T21:33:06.620+0000 7f0b59998700 1 -- 172.21.15.45:0/247966975 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0b54108640 con 0x7f0b54078da0 2023-12-16T21:33:07.151 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2023-12-16T21:33:06.620+0000 7f0b3b7fe700 1 --2- 172.21.15.45:0/247966975 >> [v2:172.21.15.45:6800/1656764601,v1:172.21.15.45:6801/1656764601] conn(0x7f0b3c0365a0 0x7f0b3c038a50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:33:07.151 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2023-12-16T21:33:06.620+0000 7f0b3b7fe700 1 -- 172.21.15.45:0/247966975 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 721+0+0 (secure 0 0 0) 0x7f0b4404a5c0 con 0x7f0b54078da0 2023-12-16T21:33:07.151 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2023-12-16T21:33:06.621+0000 7f0b527fc700 1 --2- 172.21.15.45:0/247966975 >> [v2:172.21.15.45:6800/1656764601,v1:172.21.15.45:6801/1656764601] conn(0x7f0b3c0365a0 0x7f0b3c038a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:33:07.151 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2023-12-16T21:33:06.621+0000 7f0b527fc700 1 --2- 172.21.15.45:0/247966975 >> [v2:172.21.15.45:6800/1656764601,v1:172.21.15.45:6801/1656764601] conn(0x7f0b3c0365a0 0x7f0b3c038a50 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7f0b48009990 tx=0x7f0b48006e40 comp rx=0 tx=0).ready entity=mgr.14100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:33:07.152 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2023-12-16T21:33:06.625+0000 7f0b3b7fe700 1 -- 172.21.15.45:0/247966975 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+174570 (secure 0 0 0) 0x7f0b4401ab60 con 0x7f0b54078da0 2023-12-16T21:33:07.152 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2023-12-16T21:33:06.898+0000 7f0b59998700 1 -- 172.21.15.45:0/247966975 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "status", "format": "json-pretty"} v 0) v1 -- 0x7f0b5410cf70 con 0x7f0b54078da0 2023-12-16T21:33:07.152 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2023-12-16T21:33:06.899+0000 7f0b3b7fe700 1 -- 172.21.15.45:0/247966975 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mon_command_ack([{"prefix": "status", "format": "json-pretty"}]=0 v0) v1 ==== 79+0+1248 (secure 0 0 0) 0x7f0b44018350 con 0x7f0b54078da0 2023-12-16T21:33:07.152 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2023-12-16T21:33:06.901+0000 7f0b59998700 1 -- 172.21.15.45:0/247966975 >> [v2:172.21.15.45:6800/1656764601,v1:172.21.15.45:6801/1656764601] conn(0x7f0b3c0365a0 msgr2=0x7f0b3c038a50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:33:07.152 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2023-12-16T21:33:06.901+0000 7f0b59998700 1 --2- 172.21.15.45:0/247966975 >> [v2:172.21.15.45:6800/1656764601,v1:172.21.15.45:6801/1656764601] conn(0x7f0b3c0365a0 0x7f0b3c038a50 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7f0b48009990 tx=0x7f0b48006e40 comp rx=0 tx=0).stop 2023-12-16T21:33:07.152 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2023-12-16T21:33:06.901+0000 7f0b59998700 1 -- 172.21.15.45:0/247966975 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f0b54078da0 msgr2=0x7f0b54113340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:33:07.152 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2023-12-16T21:33:06.901+0000 7f0b59998700 1 --2- 172.21.15.45:0/247966975 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f0b54078da0 0x7f0b54113340 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7f0b44006d20 tx=0x7f0b44009250 comp rx=0 tx=0).stop 2023-12-16T21:33:07.152 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2023-12-16T21:33:06.901+0000 7f0b59998700 1 -- 172.21.15.45:0/247966975 shutdown_connections 2023-12-16T21:33:07.152 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2023-12-16T21:33:06.901+0000 7f0b59998700 1 --2- 172.21.15.45:0/247966975 >> [v2:172.21.15.45:6800/1656764601,v1:172.21.15.45:6801/1656764601] conn(0x7f0b3c0365a0 0x7f0b3c038a50 unknown :-1 s=CLOSED pgs=6 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:33:07.152 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2023-12-16T21:33:06.901+0000 7f0b59998700 1 --2- 172.21.15.45:0/247966975 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f0b54078da0 0x7f0b54113340 unknown :-1 s=CLOSED pgs=19 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:33:07.152 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2023-12-16T21:33:06.901+0000 7f0b59998700 1 -- 172.21.15.45:0/247966975 >> 172.21.15.45:0/247966975 conn(0x7f0b540fda40 msgr2=0x7f0b540ffe40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-16T21:33:07.152 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2023-12-16T21:33:06.901+0000 7f0b59998700 1 -- 172.21.15.45:0/247966975 shutdown_connections 2023-12-16T21:33:07.152 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2023-12-16T21:33:06.901+0000 7f0b59998700 1 -- 172.21.15.45:0/247966975 wait complete. 2023-12-16T21:33:07.152 INFO:teuthology.orchestra.run.smithi045.stdout:mgr is available 2023-12-16T21:33:07.565 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Dec 16 21:33:07 smithi045 ceph-mon[80763]: from='client.? 172.21.15.45:0/247966975' entity='client.admin' cmd=[{"prefix": "status", "format": "json-pretty"}]: dispatch 2023-12-16T21:33:07.849 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stdout 2023-12-16T21:33:07.849 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stdout [global] 2023-12-16T21:33:07.849 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stdout fsid = 88ff363c-9c5a-11ee-95a5-87774f69a715 2023-12-16T21:33:07.849 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stdout mon_osd_allow_pg_remap = true 2023-12-16T21:33:07.849 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stdout mon_osd_allow_primary_affinity = true 2023-12-16T21:33:07.849 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stdout mon_warn_on_no_sortbitwise = false 2023-12-16T21:33:07.850 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stdout osd_crush_chooseleaf_type = 0 2023-12-16T21:33:07.850 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stdout 2023-12-16T21:33:07.850 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stdout [mgr] 2023-12-16T21:33:07.850 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stdout mgr/telemetry/nag = false 2023-12-16T21:33:07.850 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stdout 2023-12-16T21:33:07.850 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stdout [osd] 2023-12-16T21:33:07.850 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stdout mon_osd_backfillfull_ratio = 0.85 2023-12-16T21:33:07.850 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stdout mon_osd_full_ratio = 0.9 2023-12-16T21:33:07.850 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stdout mon_osd_nearfull_ratio = 0.8 2023-12-16T21:33:07.850 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stdout osd_map_max_advance = 10 2023-12-16T21:33:07.850 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stdout osd_sloppy_crc = true 2023-12-16T21:33:07.850 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2023-12-16T21:33:07.401+0000 7fdcba386700 1 Processor -- start 2023-12-16T21:33:07.850 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2023-12-16T21:33:07.401+0000 7fdcba386700 1 -- start start 2023-12-16T21:33:07.850 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2023-12-16T21:33:07.401+0000 7fdcba386700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fdcb41039c0 0x7fdcb4103dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:33:07.851 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2023-12-16T21:33:07.401+0000 7fdcba386700 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdcb4104310 con 0x7fdcb41039c0 2023-12-16T21:33:07.851 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2023-12-16T21:33:07.401+0000 7fdcb3fff700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fdcb41039c0 0x7fdcb4103dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:33:07.851 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2023-12-16T21:33:07.401+0000 7fdcb3fff700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fdcb41039c0 0x7fdcb4103dd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:38086/0 (socket says 172.21.15.45:38086) 2023-12-16T21:33:07.851 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2023-12-16T21:33:07.401+0000 7fdcb3fff700 1 -- 172.21.15.45:0/1106691117 learned_addr learned my addr 172.21.15.45:0/1106691117 (peer_addr_for_me v2:172.21.15.45:0/0) 2023-12-16T21:33:07.851 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2023-12-16T21:33:07.401+0000 7fdcb3fff700 1 -- 172.21.15.45:0/1106691117 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdcb4101fb0 con 0x7fdcb41039c0 2023-12-16T21:33:07.851 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2023-12-16T21:33:07.402+0000 7fdcb3fff700 1 --2- 172.21.15.45:0/1106691117 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fdcb41039c0 0x7fdcb4103dd0 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7fdca40098e0 tx=0x7fdca4009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2619a7de2451ca8d server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:33:07.851 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2023-12-16T21:33:07.402+0000 7fdcb2ffd700 1 -- 172.21.15.45:0/1106691117 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fdca400b860 con 0x7fdcb41039c0 2023-12-16T21:33:07.851 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2023-12-16T21:33:07.402+0000 7fdcb2ffd700 1 -- 172.21.15.45:0/1106691117 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(26 keys) v1 ==== 1009+0+0 (secure 0 0 0) 0x7fdca400be30 con 0x7fdcb41039c0 2023-12-16T21:33:07.851 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2023-12-16T21:33:07.402+0000 7fdcb2ffd700 1 -- 172.21.15.45:0/1106691117 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fdca4010380 con 0x7fdcb41039c0 2023-12-16T21:33:07.851 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2023-12-16T21:33:07.402+0000 7fdcba386700 1 -- 172.21.15.45:0/1106691117 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fdcb41039c0 msgr2=0x7fdcb4103dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:33:07.851 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2023-12-16T21:33:07.402+0000 7fdcba386700 1 --2- 172.21.15.45:0/1106691117 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fdcb41039c0 0x7fdcb4103dd0 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7fdca40098e0 tx=0x7fdca4009250 comp rx=0 tx=0).stop 2023-12-16T21:33:07.851 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2023-12-16T21:33:07.402+0000 7fdcba386700 1 -- 172.21.15.45:0/1106691117 shutdown_connections 2023-12-16T21:33:07.851 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2023-12-16T21:33:07.402+0000 7fdcba386700 1 --2- 172.21.15.45:0/1106691117 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fdcb41039c0 0x7fdcb4103dd0 unknown :-1 s=CLOSED pgs=20 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:33:07.851 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2023-12-16T21:33:07.402+0000 7fdcba386700 1 -- 172.21.15.45:0/1106691117 >> 172.21.15.45:0/1106691117 conn(0x7fdcb40fda40 msgr2=0x7fdcb40ffe70 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-16T21:33:07.851 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2023-12-16T21:33:07.403+0000 7fdcba386700 1 -- 172.21.15.45:0/1106691117 shutdown_connections 2023-12-16T21:33:07.852 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2023-12-16T21:33:07.403+0000 7fdcba386700 1 -- 172.21.15.45:0/1106691117 wait complete. 2023-12-16T21:33:07.852 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2023-12-16T21:33:07.403+0000 7fdcba386700 1 Processor -- start 2023-12-16T21:33:07.852 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2023-12-16T21:33:07.403+0000 7fdcba386700 1 -- start start 2023-12-16T21:33:07.852 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2023-12-16T21:33:07.403+0000 7fdcba386700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fdcb41039c0 0x7fdcb4078d10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:33:07.852 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2023-12-16T21:33:07.403+0000 7fdcba386700 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdcb4077190 con 0x7fdcb41039c0 2023-12-16T21:33:07.852 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2023-12-16T21:33:07.403+0000 7fdcb3fff700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fdcb41039c0 0x7fdcb4078d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:33:07.852 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2023-12-16T21:33:07.403+0000 7fdcb3fff700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fdcb41039c0 0x7fdcb4078d10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:38100/0 (socket says 172.21.15.45:38100) 2023-12-16T21:33:07.852 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2023-12-16T21:33:07.403+0000 7fdcb3fff700 1 -- 172.21.15.45:0/972148044 learned_addr learned my addr 172.21.15.45:0/972148044 (peer_addr_for_me v2:172.21.15.45:0/0) 2023-12-16T21:33:07.852 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2023-12-16T21:33:07.404+0000 7fdcb3fff700 1 -- 172.21.15.45:0/972148044 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdca40094f0 con 0x7fdcb41039c0 2023-12-16T21:33:07.852 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2023-12-16T21:33:07.404+0000 7fdcb3fff700 1 --2- 172.21.15.45:0/972148044 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fdcb41039c0 0x7fdcb4078d10 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7fdca4006b00 tx=0x7fdca4009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:33:07.852 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2023-12-16T21:33:07.404+0000 7fdcb0ff9700 1 -- 172.21.15.45:0/972148044 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fdca40109b0 con 0x7fdcb41039c0 2023-12-16T21:33:07.852 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2023-12-16T21:33:07.404+0000 7fdcba386700 1 -- 172.21.15.45:0/972148044 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdcb4077390 con 0x7fdcb41039c0 2023-12-16T21:33:07.852 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2023-12-16T21:33:07.404+0000 7fdcb0ff9700 1 -- 172.21.15.45:0/972148044 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(26 keys) v1 ==== 1009+0+0 (secure 0 0 0) 0x7fdca401a430 con 0x7fdcb41039c0 2023-12-16T21:33:07.852 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2023-12-16T21:33:07.404+0000 7fdcb0ff9700 1 -- 172.21.15.45:0/972148044 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fdca40245b0 con 0x7fdcb41039c0 2023-12-16T21:33:07.852 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2023-12-16T21:33:07.404+0000 7fdcba386700 1 -- 172.21.15.45:0/972148044 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdcb40777a0 con 0x7fdcb41039c0 2023-12-16T21:33:07.853 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2023-12-16T21:33:07.405+0000 7fdcb0ff9700 1 -- 172.21.15.45:0/972148044 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 4) v1 ==== 43270+0+0 (secure 0 0 0) 0x7fdca4024830 con 0x7fdcb41039c0 2023-12-16T21:33:07.853 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2023-12-16T21:33:07.405+0000 7fdcba386700 1 -- 172.21.15.45:0/972148044 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdcb410b150 con 0x7fdcb41039c0 2023-12-16T21:33:07.853 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2023-12-16T21:33:07.405+0000 7fdcb0ff9700 1 --2- 172.21.15.45:0/972148044 >> [v2:172.21.15.45:6800/1656764601,v1:172.21.15.45:6801/1656764601] conn(0x7fdc9c0365a0 0x7fdc9c038a50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:33:07.853 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2023-12-16T21:33:07.405+0000 7fdcb0ff9700 1 -- 172.21.15.45:0/972148044 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 721+0+0 (secure 0 0 0) 0x7fdca404acd0 con 0x7fdcb41039c0 2023-12-16T21:33:07.853 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2023-12-16T21:33:07.406+0000 7fdcb37fe700 1 --2- 172.21.15.45:0/972148044 >> [v2:172.21.15.45:6800/1656764601,v1:172.21.15.45:6801/1656764601] conn(0x7fdc9c0365a0 0x7fdc9c038a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:33:07.853 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2023-12-16T21:33:07.406+0000 7fdcb37fe700 1 --2- 172.21.15.45:0/972148044 >> [v2:172.21.15.45:6800/1656764601,v1:172.21.15.45:6801/1656764601] conn(0x7fdc9c0365a0 0x7fdc9c038a50 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7fdca8009990 tx=0x7fdca8006e40 comp rx=0 tx=0).ready entity=mgr.14100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:33:07.853 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2023-12-16T21:33:07.410+0000 7fdcb0ff9700 1 -- 172.21.15.45:0/972148044 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+174570 (secure 0 0 0) 0x7fdca4018350 con 0x7fdcb41039c0 2023-12-16T21:33:07.853 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2023-12-16T21:33:07.603+0000 7fdcba386700 1 -- 172.21.15.45:0/972148044 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "config assimilate-conf"} v 0) v1 -- 0x7fdcb405f270 con 0x7fdcb41039c0 2023-12-16T21:33:07.853 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2023-12-16T21:33:07.604+0000 7fdcb0ff9700 1 -- 172.21.15.45:0/972148044 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mon_command_ack([{"prefix": "config assimilate-conf"}]=0 v3) v1 ==== 70+0+373 (secure 0 0 0) 0x7fdca402cb30 con 0x7fdcb41039c0 2023-12-16T21:33:07.853 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2023-12-16T21:33:07.606+0000 7fdcba386700 1 -- 172.21.15.45:0/972148044 >> [v2:172.21.15.45:6800/1656764601,v1:172.21.15.45:6801/1656764601] conn(0x7fdc9c0365a0 msgr2=0x7fdc9c038a50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:33:07.853 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2023-12-16T21:33:07.606+0000 7fdcba386700 1 --2- 172.21.15.45:0/972148044 >> [v2:172.21.15.45:6800/1656764601,v1:172.21.15.45:6801/1656764601] conn(0x7fdc9c0365a0 0x7fdc9c038a50 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7fdca8009990 tx=0x7fdca8006e40 comp rx=0 tx=0).stop 2023-12-16T21:33:07.853 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2023-12-16T21:33:07.606+0000 7fdcba386700 1 -- 172.21.15.45:0/972148044 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fdcb41039c0 msgr2=0x7fdcb4078d10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:33:07.854 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2023-12-16T21:33:07.606+0000 7fdcba386700 1 --2- 172.21.15.45:0/972148044 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fdcb41039c0 0x7fdcb4078d10 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7fdca4006b00 tx=0x7fdca4009250 comp rx=0 tx=0).stop 2023-12-16T21:33:07.854 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2023-12-16T21:33:07.606+0000 7fdcba386700 1 -- 172.21.15.45:0/972148044 shutdown_connections 2023-12-16T21:33:07.854 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2023-12-16T21:33:07.606+0000 7fdcba386700 1 --2- 172.21.15.45:0/972148044 >> [v2:172.21.15.45:6800/1656764601,v1:172.21.15.45:6801/1656764601] conn(0x7fdc9c0365a0 0x7fdc9c038a50 unknown :-1 s=CLOSED pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:33:07.854 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2023-12-16T21:33:07.606+0000 7fdcba386700 1 --2- 172.21.15.45:0/972148044 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fdcb41039c0 0x7fdcb4078d10 unknown :-1 s=CLOSED pgs=21 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:33:07.854 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2023-12-16T21:33:07.606+0000 7fdcba386700 1 -- 172.21.15.45:0/972148044 >> 172.21.15.45:0/972148044 conn(0x7fdcb40fda40 msgr2=0x7fdcb40ff1c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-16T21:33:07.854 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2023-12-16T21:33:07.607+0000 7fdcba386700 1 -- 172.21.15.45:0/972148044 shutdown_connections 2023-12-16T21:33:07.854 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2023-12-16T21:33:07.607+0000 7fdcba386700 1 -- 172.21.15.45:0/972148044 wait complete. 2023-12-16T21:33:07.854 INFO:teuthology.orchestra.run.smithi045.stdout:Enabling cephadm module... 2023-12-16T21:33:08.565 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Dec 16 21:33:08 smithi045 ceph-mon[80763]: from='client.? 172.21.15.45:0/972148044' entity='client.admin' cmd=[{"prefix": "config assimilate-conf"}]: dispatch 2023-12-16T21:33:09.505 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Dec 16 21:33:09 smithi045 ceph-mon[80763]: from='client.? 172.21.15.45:0/1604199040' entity='client.admin' cmd=[{"prefix": "mgr module enable", "module": "cephadm"}]: dispatch 2023-12-16T21:33:09.513 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2023-12-16T21:33:08.103+0000 7f4b480d3700 1 Processor -- start 2023-12-16T21:33:09.514 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2023-12-16T21:33:08.103+0000 7f4b480d3700 1 -- start start 2023-12-16T21:33:09.514 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2023-12-16T21:33:08.103+0000 7f4b480d3700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f4b401038f0 0x7f4b40103d00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:33:09.514 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2023-12-16T21:33:08.103+0000 7f4b480d3700 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4b40104240 con 0x7f4b401038f0 2023-12-16T21:33:09.514 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2023-12-16T21:33:08.103+0000 7f4b45e6f700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f4b401038f0 0x7f4b40103d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:33:09.514 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2023-12-16T21:33:08.103+0000 7f4b45e6f700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f4b401038f0 0x7f4b40103d00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:38108/0 (socket says 172.21.15.45:38108) 2023-12-16T21:33:09.514 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2023-12-16T21:33:08.103+0000 7f4b45e6f700 1 -- 172.21.15.45:0/2137743810 learned_addr learned my addr 172.21.15.45:0/2137743810 (peer_addr_for_me v2:172.21.15.45:0/0) 2023-12-16T21:33:09.514 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2023-12-16T21:33:08.104+0000 7f4b45e6f700 1 -- 172.21.15.45:0/2137743810 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4b40104380 con 0x7f4b401038f0 2023-12-16T21:33:09.514 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2023-12-16T21:33:08.104+0000 7f4b45e6f700 1 --2- 172.21.15.45:0/2137743810 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f4b401038f0 0x7f4b40103d00 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7f4b3c0098e0 tx=0x7f4b3c009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ff6bd96a85727211 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:33:09.514 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2023-12-16T21:33:08.104+0000 7f4b44e6d700 1 -- 172.21.15.45:0/2137743810 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f4b3c00b860 con 0x7f4b401038f0 2023-12-16T21:33:09.514 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2023-12-16T21:33:08.104+0000 7f4b44e6d700 1 -- 172.21.15.45:0/2137743810 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(26 keys) v1 ==== 1009+0+0 (secure 0 0 0) 0x7f4b3c00be30 con 0x7f4b401038f0 2023-12-16T21:33:09.514 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2023-12-16T21:33:08.104+0000 7f4b44e6d700 1 -- 172.21.15.45:0/2137743810 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f4b3c010380 con 0x7f4b401038f0 2023-12-16T21:33:09.514 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2023-12-16T21:33:08.104+0000 7f4b480d3700 1 -- 172.21.15.45:0/2137743810 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f4b401038f0 msgr2=0x7f4b40103d00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:33:09.514 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2023-12-16T21:33:08.104+0000 7f4b480d3700 1 --2- 172.21.15.45:0/2137743810 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f4b401038f0 0x7f4b40103d00 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7f4b3c0098e0 tx=0x7f4b3c009250 comp rx=0 tx=0).stop 2023-12-16T21:33:09.515 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2023-12-16T21:33:08.105+0000 7f4b480d3700 1 -- 172.21.15.45:0/2137743810 shutdown_connections 2023-12-16T21:33:09.515 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2023-12-16T21:33:08.105+0000 7f4b480d3700 1 --2- 172.21.15.45:0/2137743810 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f4b401038f0 0x7f4b40103d00 unknown :-1 s=CLOSED pgs=22 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:33:09.515 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2023-12-16T21:33:08.105+0000 7f4b480d3700 1 -- 172.21.15.45:0/2137743810 >> 172.21.15.45:0/2137743810 conn(0x7f4b400fc0b0 msgr2=0x7f4b400fe4e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-16T21:33:09.515 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2023-12-16T21:33:08.105+0000 7f4b480d3700 1 -- 172.21.15.45:0/2137743810 shutdown_connections 2023-12-16T21:33:09.515 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2023-12-16T21:33:08.105+0000 7f4b480d3700 1 -- 172.21.15.45:0/2137743810 wait complete. 2023-12-16T21:33:09.515 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2023-12-16T21:33:08.105+0000 7f4b480d3700 1 Processor -- start 2023-12-16T21:33:09.515 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2023-12-16T21:33:08.105+0000 7f4b480d3700 1 -- start start 2023-12-16T21:33:09.515 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2023-12-16T21:33:08.105+0000 7f4b480d3700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f4b401038f0 0x7f4b40103130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:33:09.515 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2023-12-16T21:33:08.105+0000 7f4b480d3700 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4b401015b0 con 0x7f4b401038f0 2023-12-16T21:33:09.515 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2023-12-16T21:33:08.105+0000 7f4b45e6f700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f4b401038f0 0x7f4b40103130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:33:09.515 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2023-12-16T21:33:08.105+0000 7f4b45e6f700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f4b401038f0 0x7f4b40103130 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:38112/0 (socket says 172.21.15.45:38112) 2023-12-16T21:33:09.515 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2023-12-16T21:33:08.105+0000 7f4b45e6f700 1 -- 172.21.15.45:0/1604199040 learned_addr learned my addr 172.21.15.45:0/1604199040 (peer_addr_for_me v2:172.21.15.45:0/0) 2023-12-16T21:33:09.515 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2023-12-16T21:33:08.106+0000 7f4b45e6f700 1 -- 172.21.15.45:0/1604199040 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4b3c0094f0 con 0x7f4b401038f0 2023-12-16T21:33:09.515 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2023-12-16T21:33:08.106+0000 7f4b45e6f700 1 --2- 172.21.15.45:0/1604199040 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f4b401038f0 0x7f4b40103130 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7f4b3c006b00 tx=0x7f4b3c009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:33:09.516 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2023-12-16T21:33:08.106+0000 7f4b367fc700 1 -- 172.21.15.45:0/1604199040 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f4b3c0109b0 con 0x7f4b401038f0 2023-12-16T21:33:09.516 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2023-12-16T21:33:08.106+0000 7f4b367fc700 1 -- 172.21.15.45:0/1604199040 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(26 keys) v1 ==== 1009+0+0 (secure 0 0 0) 0x7f4b3c024440 con 0x7f4b401038f0 2023-12-16T21:33:09.516 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2023-12-16T21:33:08.106+0000 7f4b480d3700 1 -- 172.21.15.45:0/1604199040 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4b401017b0 con 0x7f4b401038f0 2023-12-16T21:33:09.516 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2023-12-16T21:33:08.106+0000 7f4b367fc700 1 -- 172.21.15.45:0/1604199040 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f4b3c01a5f0 con 0x7f4b401038f0 2023-12-16T21:33:09.516 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2023-12-16T21:33:08.106+0000 7f4b480d3700 1 -- 172.21.15.45:0/1604199040 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4b40101bc0 con 0x7f4b401038f0 2023-12-16T21:33:09.516 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2023-12-16T21:33:08.107+0000 7f4b367fc700 1 -- 172.21.15.45:0/1604199040 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 4) v1 ==== 43270+0+0 (secure 0 0 0) 0x7f4b3c01a8b0 con 0x7f4b401038f0 2023-12-16T21:33:09.516 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2023-12-16T21:33:08.107+0000 7f4b480d3700 1 -- 172.21.15.45:0/1604199040 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4b4004bb80 con 0x7f4b401038f0 2023-12-16T21:33:09.516 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2023-12-16T21:33:08.107+0000 7f4b367fc700 1 --2- 172.21.15.45:0/1604199040 >> [v2:172.21.15.45:6800/1656764601,v1:172.21.15.45:6801/1656764601] conn(0x7f4b2c0365a0 0x7f4b2c038a50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:33:09.516 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2023-12-16T21:33:08.107+0000 7f4b367fc700 1 -- 172.21.15.45:0/1604199040 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 721+0+0 (secure 0 0 0) 0x7f4b3c04a800 con 0x7f4b401038f0 2023-12-16T21:33:09.516 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2023-12-16T21:33:08.107+0000 7f4b4566e700 1 --2- 172.21.15.45:0/1604199040 >> [v2:172.21.15.45:6800/1656764601,v1:172.21.15.45:6801/1656764601] conn(0x7f4b2c0365a0 0x7f4b2c038a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:33:09.516 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2023-12-16T21:33:08.108+0000 7f4b4566e700 1 --2- 172.21.15.45:0/1604199040 >> [v2:172.21.15.45:6800/1656764601,v1:172.21.15.45:6801/1656764601] conn(0x7f4b2c0365a0 0x7f4b2c038a50 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f4b30009990 tx=0x7f4b30006e40 comp rx=0 tx=0).ready entity=mgr.14100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:33:09.516 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2023-12-16T21:33:08.112+0000 7f4b367fc700 1 -- 172.21.15.45:0/1604199040 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+174570 (secure 0 0 0) 0x7f4b3c01ab60 con 0x7f4b401038f0 2023-12-16T21:33:09.516 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2023-12-16T21:33:08.361+0000 7f4b480d3700 1 -- 172.21.15.45:0/1604199040 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "mgr module enable", "module": "cephadm"} v 0) v1 -- 0x7f4b40100dd0 con 0x7f4b401038f0 2023-12-16T21:33:09.516 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2023-12-16T21:33:09.253+0000 7f4b367fc700 1 -- 172.21.15.45:0/1604199040 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mgrmap(e 5) v1 ==== 43281+0+0 (secure 0 0 0) 0x7f4b3c00fdc0 con 0x7f4b401038f0 2023-12-16T21:33:09.517 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2023-12-16T21:33:09.254+0000 7f4b367fc700 1 -- 172.21.15.45:0/1604199040 <== mon.0 v2:172.21.15.45:3300/0 8 ==== mon_command_ack([{"prefix": "mgr module enable", "module": "cephadm"}]=0 v5) v1 ==== 86+0+0 (secure 0 0 0) 0x7f4b3c04c0c0 con 0x7f4b401038f0 2023-12-16T21:33:09.517 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2023-12-16T21:33:09.256+0000 7f4b480d3700 1 -- 172.21.15.45:0/1604199040 >> [v2:172.21.15.45:6800/1656764601,v1:172.21.15.45:6801/1656764601] conn(0x7f4b2c0365a0 msgr2=0x7f4b2c038a50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:33:09.517 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2023-12-16T21:33:09.256+0000 7f4b480d3700 1 --2- 172.21.15.45:0/1604199040 >> [v2:172.21.15.45:6800/1656764601,v1:172.21.15.45:6801/1656764601] conn(0x7f4b2c0365a0 0x7f4b2c038a50 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f4b30009990 tx=0x7f4b30006e40 comp rx=0 tx=0).stop 2023-12-16T21:33:09.517 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2023-12-16T21:33:09.256+0000 7f4b480d3700 1 -- 172.21.15.45:0/1604199040 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f4b401038f0 msgr2=0x7f4b40103130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:33:09.517 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2023-12-16T21:33:09.256+0000 7f4b480d3700 1 --2- 172.21.15.45:0/1604199040 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f4b401038f0 0x7f4b40103130 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7f4b3c006b00 tx=0x7f4b3c009250 comp rx=0 tx=0).stop 2023-12-16T21:33:09.517 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2023-12-16T21:33:09.257+0000 7f4b480d3700 1 -- 172.21.15.45:0/1604199040 shutdown_connections 2023-12-16T21:33:09.517 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2023-12-16T21:33:09.257+0000 7f4b480d3700 1 --2- 172.21.15.45:0/1604199040 >> [v2:172.21.15.45:6800/1656764601,v1:172.21.15.45:6801/1656764601] conn(0x7f4b2c0365a0 0x7f4b2c038a50 unknown :-1 s=CLOSED pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:33:09.517 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2023-12-16T21:33:09.257+0000 7f4b480d3700 1 --2- 172.21.15.45:0/1604199040 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f4b401038f0 0x7f4b40103130 unknown :-1 s=CLOSED pgs=23 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:33:09.517 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2023-12-16T21:33:09.257+0000 7f4b480d3700 1 -- 172.21.15.45:0/1604199040 >> 172.21.15.45:0/1604199040 conn(0x7f4b400fc0b0 msgr2=0x7f4b400fe4a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-16T21:33:09.517 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2023-12-16T21:33:09.257+0000 7f4b480d3700 1 -- 172.21.15.45:0/1604199040 shutdown_connections 2023-12-16T21:33:09.517 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2023-12-16T21:33:09.257+0000 7f4b480d3700 1 -- 172.21.15.45:0/1604199040 wait complete. 2023-12-16T21:33:10.298 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stdout { 2023-12-16T21:33:10.298 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stdout "epoch": 5, 2023-12-16T21:33:10.298 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stdout "available": true, 2023-12-16T21:33:10.298 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stdout "active_name": "smithi045.nhtmwc", 2023-12-16T21:33:10.298 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stdout "num_standby": 0 2023-12-16T21:33:10.298 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stdout } 2023-12-16T21:33:10.298 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2023-12-16T21:33:09.783+0000 7f054ae1b700 1 Processor -- start 2023-12-16T21:33:10.298 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2023-12-16T21:33:09.783+0000 7f054ae1b700 1 -- start start 2023-12-16T21:33:10.298 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2023-12-16T21:33:09.783+0000 7f054ae1b700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f0544078cc0 0x7f05440771d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:33:10.298 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2023-12-16T21:33:09.783+0000 7f054ae1b700 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f05440790d0 con 0x7f0544078cc0 2023-12-16T21:33:10.298 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2023-12-16T21:33:09.784+0000 7f0548bb7700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f0544078cc0 0x7f05440771d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:33:10.298 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2023-12-16T21:33:09.784+0000 7f0548bb7700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f0544078cc0 0x7f05440771d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:38132/0 (socket says 172.21.15.45:38132) 2023-12-16T21:33:10.298 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2023-12-16T21:33:09.784+0000 7f0548bb7700 1 -- 172.21.15.45:0/152216675 learned_addr learned my addr 172.21.15.45:0/152216675 (peer_addr_for_me v2:172.21.15.45:0/0) 2023-12-16T21:33:10.299 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2023-12-16T21:33:09.784+0000 7f0548bb7700 1 -- 172.21.15.45:0/152216675 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0544077710 con 0x7f0544078cc0 2023-12-16T21:33:10.299 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2023-12-16T21:33:09.784+0000 7f0548bb7700 1 --2- 172.21.15.45:0/152216675 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f0544078cc0 0x7f05440771d0 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f05380098e0 tx=0x7f0538009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=297079be88d05a5f server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:33:10.299 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2023-12-16T21:33:09.784+0000 7f05437fe700 1 -- 172.21.15.45:0/152216675 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f053800b860 con 0x7f0544078cc0 2023-12-16T21:33:10.299 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2023-12-16T21:33:09.784+0000 7f05437fe700 1 -- 172.21.15.45:0/152216675 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(26 keys) v1 ==== 1009+0+0 (secure 0 0 0) 0x7f053800be30 con 0x7f0544078cc0 2023-12-16T21:33:10.299 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2023-12-16T21:33:09.784+0000 7f05437fe700 1 -- 172.21.15.45:0/152216675 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f0538010380 con 0x7f0544078cc0 2023-12-16T21:33:10.299 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2023-12-16T21:33:09.785+0000 7f054ae1b700 1 -- 172.21.15.45:0/152216675 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f0544078cc0 msgr2=0x7f05440771d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:33:10.299 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2023-12-16T21:33:09.785+0000 7f054ae1b700 1 --2- 172.21.15.45:0/152216675 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f0544078cc0 0x7f05440771d0 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f05380098e0 tx=0x7f0538009250 comp rx=0 tx=0).stop 2023-12-16T21:33:10.299 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2023-12-16T21:33:09.785+0000 7f054ae1b700 1 -- 172.21.15.45:0/152216675 shutdown_connections 2023-12-16T21:33:10.299 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2023-12-16T21:33:09.785+0000 7f054ae1b700 1 --2- 172.21.15.45:0/152216675 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f0544078cc0 0x7f05440771d0 unknown :-1 s=CLOSED pgs=26 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:33:10.299 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2023-12-16T21:33:09.785+0000 7f054ae1b700 1 -- 172.21.15.45:0/152216675 >> 172.21.15.45:0/152216675 conn(0x7f05440fda10 msgr2=0x7f05440ffe40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-16T21:33:10.299 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2023-12-16T21:33:09.785+0000 7f054ae1b700 1 -- 172.21.15.45:0/152216675 shutdown_connections 2023-12-16T21:33:10.299 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2023-12-16T21:33:09.785+0000 7f054ae1b700 1 -- 172.21.15.45:0/152216675 wait complete. 2023-12-16T21:33:10.299 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2023-12-16T21:33:09.786+0000 7f054ae1b700 1 Processor -- start 2023-12-16T21:33:10.299 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2023-12-16T21:33:09.786+0000 7f054ae1b700 1 -- start start 2023-12-16T21:33:10.299 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2023-12-16T21:33:09.786+0000 7f054ae1b700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f0544078cc0 0x7f0544113330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:33:10.300 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2023-12-16T21:33:09.786+0000 7f054ae1b700 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0544113870 con 0x7f0544078cc0 2023-12-16T21:33:10.300 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2023-12-16T21:33:09.786+0000 7f0548bb7700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f0544078cc0 0x7f0544113330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:33:10.300 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2023-12-16T21:33:09.787+0000 7f0548bb7700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f0544078cc0 0x7f0544113330 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:38140/0 (socket says 172.21.15.45:38140) 2023-12-16T21:33:10.300 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2023-12-16T21:33:09.787+0000 7f0548bb7700 1 -- 172.21.15.45:0/2081072033 learned_addr learned my addr 172.21.15.45:0/2081072033 (peer_addr_for_me v2:172.21.15.45:0/0) 2023-12-16T21:33:10.300 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2023-12-16T21:33:09.787+0000 7f0548bb7700 1 -- 172.21.15.45:0/2081072033 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f05380094f0 con 0x7f0544078cc0 2023-12-16T21:33:10.300 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2023-12-16T21:33:09.787+0000 7f0548bb7700 1 --2- 172.21.15.45:0/2081072033 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f0544078cc0 0x7f0544113330 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f0538006d40 tx=0x7f0538009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:33:10.300 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2023-12-16T21:33:09.787+0000 7f05417fa700 1 -- 172.21.15.45:0/2081072033 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f05380107a0 con 0x7f0544078cc0 2023-12-16T21:33:10.300 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2023-12-16T21:33:09.787+0000 7f05417fa700 1 -- 172.21.15.45:0/2081072033 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(26 keys) v1 ==== 1009+0+0 (secure 0 0 0) 0x7f0538010d70 con 0x7f0544078cc0 2023-12-16T21:33:10.300 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2023-12-16T21:33:09.787+0000 7f05417fa700 1 -- 172.21.15.45:0/2081072033 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f053801ad60 con 0x7f0544078cc0 2023-12-16T21:33:10.300 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2023-12-16T21:33:09.787+0000 7f054ae1b700 1 -- 172.21.15.45:0/2081072033 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0544113a70 con 0x7f0544078cc0 2023-12-16T21:33:10.300 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2023-12-16T21:33:09.787+0000 7f054ae1b700 1 -- 172.21.15.45:0/2081072033 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0544113e80 con 0x7f0544078cc0 2023-12-16T21:33:10.300 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2023-12-16T21:33:09.788+0000 7f05417fa700 1 -- 172.21.15.45:0/2081072033 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 5) v1 ==== 43281+0+0 (secure 0 0 0) 0x7f0538022630 con 0x7f0544078cc0 2023-12-16T21:33:10.300 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2023-12-16T21:33:09.788+0000 7f054ae1b700 1 -- 172.21.15.45:0/2081072033 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f054404bb80 con 0x7f0544078cc0 2023-12-16T21:33:10.300 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2023-12-16T21:33:09.788+0000 7f05417fa700 1 --2- 172.21.15.45:0/2081072033 >> [v2:172.21.15.45:6800/1656764601,v1:172.21.15.45:6801/1656764601] conn(0x7f0534036640 0x7f0534038af0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:33:10.301 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2023-12-16T21:33:09.788+0000 7f05417fa700 1 -- 172.21.15.45:0/2081072033 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 721+0+0 (secure 0 0 0) 0x7f053801e070 con 0x7f0544078cc0 2023-12-16T21:33:10.301 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2023-12-16T21:33:09.788+0000 7f0543fff700 1 -- 172.21.15.45:0/2081072033 >> [v2:172.21.15.45:6800/1656764601,v1:172.21.15.45:6801/1656764601] conn(0x7f0534036640 msgr2=0x7f0534038af0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.45:6800/1656764601 2023-12-16T21:33:10.301 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2023-12-16T21:33:09.788+0000 7f0543fff700 1 --2- 172.21.15.45:0/2081072033 >> [v2:172.21.15.45:6800/1656764601,v1:172.21.15.45:6801/1656764601] conn(0x7f0534036640 0x7f0534038af0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-12-16T21:33:10.301 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2023-12-16T21:33:09.793+0000 7f05417fa700 1 -- 172.21.15.45:0/2081072033 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+174570 (secure 0 0 0) 0x7f0538020370 con 0x7f0544078cc0 2023-12-16T21:33:10.301 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2023-12-16T21:33:09.989+0000 7f0543fff700 1 -- 172.21.15.45:0/2081072033 >> [v2:172.21.15.45:6800/1656764601,v1:172.21.15.45:6801/1656764601] conn(0x7f0534036640 msgr2=0x7f0534038af0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.45:6800/1656764601 2023-12-16T21:33:10.301 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2023-12-16T21:33:09.989+0000 7f0543fff700 1 --2- 172.21.15.45:0/2081072033 >> [v2:172.21.15.45:6800/1656764601,v1:172.21.15.45:6801/1656764601] conn(0x7f0534036640 0x7f0534038af0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-12-16T21:33:10.301 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2023-12-16T21:33:10.044+0000 7f054ae1b700 1 -- 172.21.15.45:0/2081072033 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "mgr stat"} v 0) v1 -- 0x7f054410d170 con 0x7f0544078cc0 2023-12-16T21:33:10.301 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2023-12-16T21:33:10.044+0000 7f05417fa700 1 -- 172.21.15.45:0/2081072033 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mon_command_ack([{"prefix": "mgr stat"}]=0 v5) v1 ==== 56+0+103 (secure 0 0 0) 0x7f053802ab80 con 0x7f0544078cc0 2023-12-16T21:33:10.301 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2023-12-16T21:33:10.047+0000 7f054ae1b700 1 -- 172.21.15.45:0/2081072033 >> [v2:172.21.15.45:6800/1656764601,v1:172.21.15.45:6801/1656764601] conn(0x7f0534036640 msgr2=0x7f0534038af0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-12-16T21:33:10.301 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2023-12-16T21:33:10.047+0000 7f054ae1b700 1 --2- 172.21.15.45:0/2081072033 >> [v2:172.21.15.45:6800/1656764601,v1:172.21.15.45:6801/1656764601] conn(0x7f0534036640 0x7f0534038af0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:33:10.301 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2023-12-16T21:33:10.047+0000 7f054ae1b700 1 -- 172.21.15.45:0/2081072033 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f0544078cc0 msgr2=0x7f0544113330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:33:10.301 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2023-12-16T21:33:10.047+0000 7f054ae1b700 1 --2- 172.21.15.45:0/2081072033 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f0544078cc0 0x7f0544113330 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f0538006d40 tx=0x7f0538009250 comp rx=0 tx=0).stop 2023-12-16T21:33:10.301 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2023-12-16T21:33:10.048+0000 7f054ae1b700 1 -- 172.21.15.45:0/2081072033 shutdown_connections 2023-12-16T21:33:10.301 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2023-12-16T21:33:10.048+0000 7f054ae1b700 1 --2- 172.21.15.45:0/2081072033 >> [v2:172.21.15.45:6800/1656764601,v1:172.21.15.45:6801/1656764601] conn(0x7f0534036640 0x7f0534038af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:33:10.301 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2023-12-16T21:33:10.048+0000 7f054ae1b700 1 --2- 172.21.15.45:0/2081072033 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f0544078cc0 0x7f0544113330 unknown :-1 s=CLOSED pgs=27 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:33:10.302 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2023-12-16T21:33:10.048+0000 7f054ae1b700 1 -- 172.21.15.45:0/2081072033 >> 172.21.15.45:0/2081072033 conn(0x7f05440fda10 msgr2=0x7f05440ff1b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-16T21:33:10.302 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2023-12-16T21:33:10.048+0000 7f054ae1b700 1 -- 172.21.15.45:0/2081072033 shutdown_connections 2023-12-16T21:33:10.302 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2023-12-16T21:33:10.048+0000 7f054ae1b700 1 -- 172.21.15.45:0/2081072033 wait complete. 2023-12-16T21:33:10.302 INFO:teuthology.orchestra.run.smithi045.stdout:Waiting for the mgr to restart... 2023-12-16T21:33:10.302 INFO:teuthology.orchestra.run.smithi045.stdout:Waiting for mgr epoch 5... 2023-12-16T21:33:10.315 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Dec 16 21:33:10 smithi045 ceph-mon[80763]: from='client.? 172.21.15.45:0/1604199040' entity='client.admin' cmd='[{"prefix": "mgr module enable", "module": "cephadm"}]': finished 2023-12-16T21:33:10.316 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Dec 16 21:33:10 smithi045 ceph-mon[80763]: mgrmap e5: smithi045.nhtmwc(active, since 7s) 2023-12-16T21:33:10.316 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Dec 16 21:33:10 smithi045 ceph-mon[80763]: from='client.? 172.21.15.45:0/2081072033' entity='client.admin' cmd=[{"prefix": "mgr stat"}]: dispatch 2023-12-16T21:33:16.566 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Dec 16 21:33:16 smithi045 ceph-mon[80763]: Active manager daemon smithi045.nhtmwc restarted 2023-12-16T21:33:16.566 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Dec 16 21:33:16 smithi045 ceph-mon[80763]: Activating manager daemon smithi045.nhtmwc 2023-12-16T21:33:16.566 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Dec 16 21:33:16 smithi045 ceph-mon[80763]: osdmap e2: 0 total, 0 up, 0 in 2023-12-16T21:33:16.566 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Dec 16 21:33:16 smithi045 ceph-mon[80763]: mgrmap e6: smithi045.nhtmwc(active, starting, since 0.00341061s) 2023-12-16T21:33:16.566 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Dec 16 21:33:16 smithi045 ceph-mon[80763]: from='mgr.14120 172.21.15.45:0/902863156' entity='mgr.smithi045.nhtmwc' cmd=[{"prefix": "mon metadata", "id": "smithi045"}]: dispatch 2023-12-16T21:33:16.566 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Dec 16 21:33:16 smithi045 ceph-mon[80763]: from='mgr.14120 172.21.15.45:0/902863156' entity='mgr.smithi045.nhtmwc' cmd=[{"prefix": "mgr metadata", "who": "smithi045.nhtmwc", "id": "smithi045.nhtmwc"}]: dispatch 2023-12-16T21:33:16.566 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Dec 16 21:33:16 smithi045 ceph-mon[80763]: from='mgr.14120 172.21.15.45:0/902863156' entity='mgr.smithi045.nhtmwc' cmd=[{"prefix": "mds metadata"}]: dispatch 2023-12-16T21:33:16.566 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Dec 16 21:33:16 smithi045 ceph-mon[80763]: from='mgr.14120 172.21.15.45:0/902863156' entity='mgr.smithi045.nhtmwc' cmd=[{"prefix": "osd metadata"}]: dispatch 2023-12-16T21:33:16.566 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Dec 16 21:33:16 smithi045 ceph-mon[80763]: from='mgr.14120 172.21.15.45:0/902863156' entity='mgr.smithi045.nhtmwc' cmd=[{"prefix": "mon metadata"}]: dispatch 2023-12-16T21:33:16.567 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Dec 16 21:33:16 smithi045 ceph-mon[80763]: Manager daemon smithi045.nhtmwc is now available 2023-12-16T21:33:16.567 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Dec 16 21:33:16 smithi045 ceph-mon[80763]: from='mgr.14120 172.21.15.45:0/902863156' entity='mgr.smithi045.nhtmwc' 2023-12-16T21:33:16.567 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Dec 16 21:33:16 smithi045 ceph-mon[80763]: from='mgr.14120 172.21.15.45:0/902863156' entity='mgr.smithi045.nhtmwc' 2023-12-16T21:33:16.567 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Dec 16 21:33:16 smithi045 ceph-mon[80763]: from='mgr.14120 172.21.15.45:0/902863156' entity='mgr.smithi045.nhtmwc' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-12-16T21:33:16.567 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Dec 16 21:33:16 smithi045 ceph-mon[80763]: from='mgr.14120 172.21.15.45:0/902863156' entity='mgr.smithi045.nhtmwc' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-12-16T21:33:16.567 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Dec 16 21:33:16 smithi045 ceph-mon[80763]: from='mgr.14120 172.21.15.45:0/902863156' entity='mgr.smithi045.nhtmwc' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-12-16T21:33:17.404 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Dec 16 21:33:17 smithi045 ceph-mon[80763]: Found migration_current of "None". Setting to last migration. 2023-12-16T21:33:17.405 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Dec 16 21:33:17 smithi045 ceph-mon[80763]: from='mgr.14120 172.21.15.45:0/902863156' entity='mgr.smithi045.nhtmwc' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi045.nhtmwc/mirror_snapshot_schedule"}]: dispatch 2023-12-16T21:33:17.405 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Dec 16 21:33:17 smithi045 ceph-mon[80763]: from='mgr.14120 172.21.15.45:0/902863156' entity='mgr.smithi045.nhtmwc' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi045.nhtmwc/trash_purge_schedule"}]: dispatch 2023-12-16T21:33:17.405 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Dec 16 21:33:17 smithi045 ceph-mon[80763]: mgrmap e7: smithi045.nhtmwc(active, since 1.00513s) 2023-12-16T21:33:17.407 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stdout { 2023-12-16T21:33:17.414 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stdout "mgrmap_epoch": 7, 2023-12-16T21:33:17.415 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stdout "initialized": true 2023-12-16T21:33:17.415 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stdout } 2023-12-16T21:33:17.415 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2023-12-16T21:33:10.545+0000 7f411fb91700 1 Processor -- start 2023-12-16T21:33:17.415 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2023-12-16T21:33:10.545+0000 7f411fb91700 1 -- start start 2023-12-16T21:33:17.415 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2023-12-16T21:33:10.545+0000 7f411fb91700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f4118070f20 0x7f4118104f80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:33:17.415 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2023-12-16T21:33:10.545+0000 7f411fb91700 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f41181054c0 con 0x7f4118070f20 2023-12-16T21:33:17.415 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2023-12-16T21:33:10.545+0000 7f411d92d700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f4118070f20 0x7f4118104f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:33:17.415 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2023-12-16T21:33:10.545+0000 7f411d92d700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f4118070f20 0x7f4118104f80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:38154/0 (socket says 172.21.15.45:38154) 2023-12-16T21:33:17.415 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2023-12-16T21:33:10.545+0000 7f411d92d700 1 -- 172.21.15.45:0/1582381012 learned_addr learned my addr 172.21.15.45:0/1582381012 (peer_addr_for_me v2:172.21.15.45:0/0) 2023-12-16T21:33:17.415 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2023-12-16T21:33:10.545+0000 7f411d92d700 1 -- 172.21.15.45:0/1582381012 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4118105600 con 0x7f4118070f20 2023-12-16T21:33:17.415 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2023-12-16T21:33:10.546+0000 7f411d92d700 1 --2- 172.21.15.45:0/1582381012 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f4118070f20 0x7f4118104f80 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7f41140098e0 tx=0x7f4114009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9b8bdcbb0863ea26 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:33:17.415 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2023-12-16T21:33:10.546+0000 7f411c92b700 1 -- 172.21.15.45:0/1582381012 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f411400b860 con 0x7f4118070f20 2023-12-16T21:33:17.416 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2023-12-16T21:33:10.546+0000 7f411c92b700 1 -- 172.21.15.45:0/1582381012 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(26 keys) v1 ==== 1009+0+0 (secure 0 0 0) 0x7f411400be30 con 0x7f4118070f20 2023-12-16T21:33:17.416 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2023-12-16T21:33:10.546+0000 7f411c92b700 1 -- 172.21.15.45:0/1582381012 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f4114010380 con 0x7f4118070f20 2023-12-16T21:33:17.416 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2023-12-16T21:33:10.546+0000 7f411fb91700 1 -- 172.21.15.45:0/1582381012 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f4118070f20 msgr2=0x7f4118104f80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:33:17.416 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2023-12-16T21:33:10.546+0000 7f411fb91700 1 --2- 172.21.15.45:0/1582381012 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f4118070f20 0x7f4118104f80 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7f41140098e0 tx=0x7f4114009250 comp rx=0 tx=0).stop 2023-12-16T21:33:17.416 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2023-12-16T21:33:10.547+0000 7f411fb91700 1 -- 172.21.15.45:0/1582381012 shutdown_connections 2023-12-16T21:33:17.416 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2023-12-16T21:33:10.547+0000 7f411fb91700 1 --2- 172.21.15.45:0/1582381012 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f4118070f20 0x7f4118104f80 unknown :-1 s=CLOSED pgs=28 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:33:17.416 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2023-12-16T21:33:10.547+0000 7f411fb91700 1 -- 172.21.15.45:0/1582381012 >> 172.21.15.45:0/1582381012 conn(0x7f41180fc970 msgr2=0x7f41180fed80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-16T21:33:17.416 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2023-12-16T21:33:10.547+0000 7f411fb91700 1 -- 172.21.15.45:0/1582381012 shutdown_connections 2023-12-16T21:33:17.416 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2023-12-16T21:33:10.547+0000 7f411fb91700 1 -- 172.21.15.45:0/1582381012 wait complete. 2023-12-16T21:33:17.416 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2023-12-16T21:33:10.547+0000 7f411fb91700 1 Processor -- start 2023-12-16T21:33:17.416 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2023-12-16T21:33:10.547+0000 7f411fb91700 1 -- start start 2023-12-16T21:33:17.416 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2023-12-16T21:33:10.547+0000 7f411fb91700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f4118070f20 0x7f4118102a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:33:17.416 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2023-12-16T21:33:10.547+0000 7f411fb91700 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4118100e80 con 0x7f4118070f20 2023-12-16T21:33:17.416 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2023-12-16T21:33:10.548+0000 7f411d92d700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f4118070f20 0x7f4118102a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:33:17.416 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2023-12-16T21:33:10.548+0000 7f411d92d700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f4118070f20 0x7f4118102a00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:38170/0 (socket says 172.21.15.45:38170) 2023-12-16T21:33:17.417 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2023-12-16T21:33:10.548+0000 7f411d92d700 1 -- 172.21.15.45:0/1303769192 learned_addr learned my addr 172.21.15.45:0/1303769192 (peer_addr_for_me v2:172.21.15.45:0/0) 2023-12-16T21:33:17.417 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2023-12-16T21:33:10.548+0000 7f411d92d700 1 -- 172.21.15.45:0/1303769192 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f41140094f0 con 0x7f4118070f20 2023-12-16T21:33:17.417 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2023-12-16T21:33:10.548+0000 7f411d92d700 1 --2- 172.21.15.45:0/1303769192 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f4118070f20 0x7f4118102a00 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7f4114006d40 tx=0x7f4114009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:33:17.417 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2023-12-16T21:33:10.548+0000 7f410e7fc700 1 -- 172.21.15.45:0/1303769192 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f41140109b0 con 0x7f4118070f20 2023-12-16T21:33:17.417 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2023-12-16T21:33:10.548+0000 7f411fb91700 1 -- 172.21.15.45:0/1303769192 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4118101080 con 0x7f4118070f20 2023-12-16T21:33:17.417 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2023-12-16T21:33:10.548+0000 7f410e7fc700 1 -- 172.21.15.45:0/1303769192 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(26 keys) v1 ==== 1009+0+0 (secure 0 0 0) 0x7f4114024440 con 0x7f4118070f20 2023-12-16T21:33:17.417 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2023-12-16T21:33:10.548+0000 7f410e7fc700 1 -- 172.21.15.45:0/1303769192 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f411401a5f0 con 0x7f4118070f20 2023-12-16T21:33:17.417 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2023-12-16T21:33:10.548+0000 7f411fb91700 1 -- 172.21.15.45:0/1303769192 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4118101490 con 0x7f4118070f20 2023-12-16T21:33:17.417 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2023-12-16T21:33:10.549+0000 7f410e7fc700 1 -- 172.21.15.45:0/1303769192 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 5) v1 ==== 43281+0+0 (secure 0 0 0) 0x7f411401a870 con 0x7f4118070f20 2023-12-16T21:33:17.417 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2023-12-16T21:33:10.549+0000 7f410e7fc700 1 --2- 172.21.15.45:0/1303769192 >> [v2:172.21.15.45:6800/1656764601,v1:172.21.15.45:6801/1656764601] conn(0x7f41040365f0 0x7f4104038aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:33:17.417 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2023-12-16T21:33:10.549+0000 7f410e7fc700 1 -- 172.21.15.45:0/1303769192 --> [v2:172.21.15.45:6800/1656764601,v1:172.21.15.45:6801/1656764601] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x7f41040391b0 con 0x7f41040365f0 2023-12-16T21:33:17.417 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2023-12-16T21:33:10.549+0000 7f410e7fc700 1 -- 172.21.15.45:0/1303769192 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 721+0+0 (secure 0 0 0) 0x7f411404a4b0 con 0x7f4118070f20 2023-12-16T21:33:17.417 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2023-12-16T21:33:10.549+0000 7f411d12c700 1 -- 172.21.15.45:0/1303769192 >> [v2:172.21.15.45:6800/1656764601,v1:172.21.15.45:6801/1656764601] conn(0x7f41040365f0 msgr2=0x7f4104038aa0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.45:6800/1656764601 2023-12-16T21:33:17.418 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2023-12-16T21:33:10.549+0000 7f411d12c700 1 --2- 172.21.15.45:0/1303769192 >> [v2:172.21.15.45:6800/1656764601,v1:172.21.15.45:6801/1656764601] conn(0x7f41040365f0 0x7f4104038aa0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-12-16T21:33:17.418 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2023-12-16T21:33:10.750+0000 7f411d12c700 1 -- 172.21.15.45:0/1303769192 >> [v2:172.21.15.45:6800/1656764601,v1:172.21.15.45:6801/1656764601] conn(0x7f41040365f0 msgr2=0x7f4104038aa0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.45:6800/1656764601 2023-12-16T21:33:17.418 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2023-12-16T21:33:10.750+0000 7f411d12c700 1 --2- 172.21.15.45:0/1303769192 >> [v2:172.21.15.45:6800/1656764601,v1:172.21.15.45:6801/1656764601] conn(0x7f41040365f0 0x7f4104038aa0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-12-16T21:33:17.418 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2023-12-16T21:33:11.151+0000 7f411d12c700 1 -- 172.21.15.45:0/1303769192 >> [v2:172.21.15.45:6800/1656764601,v1:172.21.15.45:6801/1656764601] conn(0x7f41040365f0 msgr2=0x7f4104038aa0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.45:6800/1656764601 2023-12-16T21:33:17.418 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2023-12-16T21:33:11.151+0000 7f411d12c700 1 --2- 172.21.15.45:0/1303769192 >> [v2:172.21.15.45:6800/1656764601,v1:172.21.15.45:6801/1656764601] conn(0x7f41040365f0 0x7f4104038aa0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2023-12-16T21:33:17.418 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2023-12-16T21:33:11.952+0000 7f411d12c700 1 -- 172.21.15.45:0/1303769192 >> [v2:172.21.15.45:6800/1656764601,v1:172.21.15.45:6801/1656764601] conn(0x7f41040365f0 msgr2=0x7f4104038aa0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.45:6800/1656764601 2023-12-16T21:33:17.418 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2023-12-16T21:33:11.952+0000 7f411d12c700 1 --2- 172.21.15.45:0/1303769192 >> [v2:172.21.15.45:6800/1656764601,v1:172.21.15.45:6801/1656764601] conn(0x7f41040365f0 0x7f4104038aa0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2023-12-16T21:33:17.418 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2023-12-16T21:33:13.553+0000 7f411d12c700 1 -- 172.21.15.45:0/1303769192 >> [v2:172.21.15.45:6800/1656764601,v1:172.21.15.45:6801/1656764601] conn(0x7f41040365f0 msgr2=0x7f4104038aa0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.45:6800/1656764601 2023-12-16T21:33:17.418 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2023-12-16T21:33:13.553+0000 7f411d12c700 1 --2- 172.21.15.45:0/1303769192 >> [v2:172.21.15.45:6800/1656764601,v1:172.21.15.45:6801/1656764601] conn(0x7f41040365f0 0x7f4104038aa0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 3.200000 2023-12-16T21:33:17.418 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2023-12-16T21:33:16.107+0000 7f410e7fc700 1 -- 172.21.15.45:0/1303769192 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mgrmap(e 6) v1 ==== 43048+0+0 (secure 0 0 0) 0x7f411402ad20 con 0x7f4118070f20 2023-12-16T21:33:17.418 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2023-12-16T21:33:16.107+0000 7f410e7fc700 1 -- 172.21.15.45:0/1303769192 >> [v2:172.21.15.45:6800/1656764601,v1:172.21.15.45:6801/1656764601] conn(0x7f41040365f0 msgr2=0x7f4104038aa0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-12-16T21:33:17.418 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2023-12-16T21:33:16.107+0000 7f410e7fc700 1 --2- 172.21.15.45:0/1303769192 >> [v2:172.21.15.45:6800/1656764601,v1:172.21.15.45:6801/1656764601] conn(0x7f41040365f0 0x7f4104038aa0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:33:17.418 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2023-12-16T21:33:17.108+0000 7f410e7fc700 1 -- 172.21.15.45:0/1303769192 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mgrmap(e 7) v1 ==== 43175+0+0 (secure 0 0 0) 0x7f4114024970 con 0x7f4118070f20 2023-12-16T21:33:17.419 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2023-12-16T21:33:17.108+0000 7f410e7fc700 1 --2- 172.21.15.45:0/1303769192 >> [v2:172.21.15.45:6800/2502560408,v1:172.21.15.45:6801/2502560408] conn(0x7f4104039f60 0x7f410403c340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:33:17.419 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2023-12-16T21:33:17.108+0000 7f410e7fc700 1 -- 172.21.15.45:0/1303769192 --> [v2:172.21.15.45:6800/2502560408,v1:172.21.15.45:6801/2502560408] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x7f41040391b0 con 0x7f4104039f60 2023-12-16T21:33:17.419 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2023-12-16T21:33:17.109+0000 7f411d12c700 1 --2- 172.21.15.45:0/1303769192 >> [v2:172.21.15.45:6800/2502560408,v1:172.21.15.45:6801/2502560408] conn(0x7f4104039f60 0x7f410403c340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:33:17.419 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2023-12-16T21:33:17.109+0000 7f411d12c700 1 --2- 172.21.15.45:0/1303769192 >> [v2:172.21.15.45:6800/2502560408,v1:172.21.15.45:6801/2502560408] conn(0x7f4104039f60 0x7f410403c340 secure :-1 s=READY pgs=1 cs=0 l=1 rev1=1 crypto rx=0x7f41080039d0 tx=0x7f4108009350 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:33:17.419 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2023-12-16T21:33:17.110+0000 7f410e7fc700 1 -- 172.21.15.45:0/1303769192 <== mgr.14120 v2:172.21.15.45:6800/2502560408 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+6549 (secure 0 0 0) 0x7f41040391b0 con 0x7f4104039f60 2023-12-16T21:33:17.419 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2023-12-16T21:33:17.116+0000 7f411fb91700 1 -- 172.21.15.45:0/1303769192 --> [v2:172.21.15.45:6800/2502560408,v1:172.21.15.45:6801/2502560408] -- command(tid 1: {"prefix": "mgr_status"}) v1 -- 0x7f411805f270 con 0x7f4104039f60 2023-12-16T21:33:17.419 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2023-12-16T21:33:17.116+0000 7f410e7fc700 1 -- 172.21.15.45:0/1303769192 <== mgr.14120 v2:172.21.15.45:6800/2502560408 2 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+51 (secure 0 0 0) 0x7f411805f270 con 0x7f4104039f60 2023-12-16T21:33:17.419 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2023-12-16T21:33:17.116+0000 7f411fb91700 1 -- 172.21.15.45:0/1303769192 >> [v2:172.21.15.45:6800/2502560408,v1:172.21.15.45:6801/2502560408] conn(0x7f4104039f60 msgr2=0x7f410403c340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:33:17.419 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2023-12-16T21:33:17.116+0000 7f411fb91700 1 --2- 172.21.15.45:0/1303769192 >> [v2:172.21.15.45:6800/2502560408,v1:172.21.15.45:6801/2502560408] conn(0x7f4104039f60 0x7f410403c340 secure :-1 s=READY pgs=1 cs=0 l=1 rev1=1 crypto rx=0x7f41080039d0 tx=0x7f4108009350 comp rx=0 tx=0).stop 2023-12-16T21:33:17.419 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2023-12-16T21:33:17.116+0000 7f411fb91700 1 -- 172.21.15.45:0/1303769192 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f4118070f20 msgr2=0x7f4118102a00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:33:17.419 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2023-12-16T21:33:17.116+0000 7f411fb91700 1 --2- 172.21.15.45:0/1303769192 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f4118070f20 0x7f4118102a00 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7f4114006d40 tx=0x7f4114009250 comp rx=0 tx=0).stop 2023-12-16T21:33:17.419 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2023-12-16T21:33:17.116+0000 7f411fb91700 1 -- 172.21.15.45:0/1303769192 shutdown_connections 2023-12-16T21:33:17.419 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2023-12-16T21:33:17.116+0000 7f411fb91700 1 --2- 172.21.15.45:0/1303769192 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f4118070f20 0x7f4118102a00 unknown :-1 s=CLOSED pgs=29 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:33:17.419 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2023-12-16T21:33:17.116+0000 7f411fb91700 1 --2- 172.21.15.45:0/1303769192 >> [v2:172.21.15.45:6800/2502560408,v1:172.21.15.45:6801/2502560408] conn(0x7f4104039f60 0x7f410403c340 unknown :-1 s=CLOSED pgs=1 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:33:17.419 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2023-12-16T21:33:17.116+0000 7f411fb91700 1 --2- 172.21.15.45:0/1303769192 >> [v2:172.21.15.45:6800/1656764601,v1:172.21.15.45:6801/1656764601] conn(0x7f41040365f0 0x7f4104038aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:33:17.420 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2023-12-16T21:33:17.116+0000 7f411fb91700 1 -- 172.21.15.45:0/1303769192 >> 172.21.15.45:0/1303769192 conn(0x7f41180fc970 msgr2=0x7f41180fe170 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-16T21:33:17.420 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2023-12-16T21:33:17.116+0000 7f411fb91700 1 -- 172.21.15.45:0/1303769192 shutdown_connections 2023-12-16T21:33:17.420 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2023-12-16T21:33:17.116+0000 7f411fb91700 1 -- 172.21.15.45:0/1303769192 wait complete. 2023-12-16T21:33:17.420 INFO:teuthology.orchestra.run.smithi045.stdout:mgr epoch 5 is available 2023-12-16T21:33:17.420 INFO:teuthology.orchestra.run.smithi045.stdout:Setting orchestrator backend to cephadm... 2023-12-16T21:33:18.138 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2023-12-16T21:33:17.644+0000 7ffb8599b700 1 Processor -- start 2023-12-16T21:33:18.138 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2023-12-16T21:33:17.644+0000 7ffb8599b700 1 -- start start 2023-12-16T21:33:18.139 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2023-12-16T21:33:17.644+0000 7ffb8599b700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ffb801039c0 0x7ffb80103dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:33:18.139 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2023-12-16T21:33:17.644+0000 7ffb8599b700 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffb80104310 con 0x7ffb801039c0 2023-12-16T21:33:18.139 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2023-12-16T21:33:17.644+0000 7ffb7effd700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ffb801039c0 0x7ffb80103dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:33:18.139 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2023-12-16T21:33:17.644+0000 7ffb7effd700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ffb801039c0 0x7ffb80103dd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:59084/0 (socket says 172.21.15.45:59084) 2023-12-16T21:33:18.139 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2023-12-16T21:33:17.644+0000 7ffb7effd700 1 -- 172.21.15.45:0/4279874400 learned_addr learned my addr 172.21.15.45:0/4279874400 (peer_addr_for_me v2:172.21.15.45:0/0) 2023-12-16T21:33:18.139 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2023-12-16T21:33:17.644+0000 7ffb7effd700 1 -- 172.21.15.45:0/4279874400 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffb80101fb0 con 0x7ffb801039c0 2023-12-16T21:33:18.139 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2023-12-16T21:33:17.645+0000 7ffb7effd700 1 --2- 172.21.15.45:0/4279874400 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ffb801039c0 0x7ffb80103dd0 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7ffb700098e0 tx=0x7ffb70009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ca590ff190ea905b server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:33:18.139 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2023-12-16T21:33:17.645+0000 7ffb7dffb700 1 -- 172.21.15.45:0/4279874400 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7ffb7000b860 con 0x7ffb801039c0 2023-12-16T21:33:18.139 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2023-12-16T21:33:17.645+0000 7ffb7dffb700 1 -- 172.21.15.45:0/4279874400 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(26 keys) v1 ==== 1009+0+0 (secure 0 0 0) 0x7ffb7000be30 con 0x7ffb801039c0 2023-12-16T21:33:18.139 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2023-12-16T21:33:17.645+0000 7ffb7dffb700 1 -- 172.21.15.45:0/4279874400 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7ffb70010380 con 0x7ffb801039c0 2023-12-16T21:33:18.139 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2023-12-16T21:33:17.645+0000 7ffb8599b700 1 -- 172.21.15.45:0/4279874400 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ffb801039c0 msgr2=0x7ffb80103dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:33:18.140 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2023-12-16T21:33:17.645+0000 7ffb8599b700 1 --2- 172.21.15.45:0/4279874400 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ffb801039c0 0x7ffb80103dd0 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7ffb700098e0 tx=0x7ffb70009250 comp rx=0 tx=0).stop 2023-12-16T21:33:18.140 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2023-12-16T21:33:17.645+0000 7ffb8599b700 1 -- 172.21.15.45:0/4279874400 shutdown_connections 2023-12-16T21:33:18.140 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2023-12-16T21:33:17.645+0000 7ffb8599b700 1 --2- 172.21.15.45:0/4279874400 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ffb801039c0 0x7ffb80103dd0 unknown :-1 s=CLOSED pgs=37 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:33:18.140 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2023-12-16T21:33:17.645+0000 7ffb8599b700 1 -- 172.21.15.45:0/4279874400 >> 172.21.15.45:0/4279874400 conn(0x7ffb800fda40 msgr2=0x7ffb800ffe70 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-16T21:33:18.140 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2023-12-16T21:33:17.645+0000 7ffb8599b700 1 -- 172.21.15.45:0/4279874400 shutdown_connections 2023-12-16T21:33:18.140 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2023-12-16T21:33:17.645+0000 7ffb8599b700 1 -- 172.21.15.45:0/4279874400 wait complete. 2023-12-16T21:33:18.140 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2023-12-16T21:33:17.646+0000 7ffb8599b700 1 Processor -- start 2023-12-16T21:33:18.140 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2023-12-16T21:33:17.646+0000 7ffb8599b700 1 -- start start 2023-12-16T21:33:18.140 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2023-12-16T21:33:17.646+0000 7ffb8599b700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ffb801039c0 0x7ffb801110f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:33:18.140 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2023-12-16T21:33:17.646+0000 7ffb8599b700 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffb80111630 con 0x7ffb801039c0 2023-12-16T21:33:18.140 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2023-12-16T21:33:17.646+0000 7ffb7effd700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ffb801039c0 0x7ffb801110f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:33:18.140 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2023-12-16T21:33:17.646+0000 7ffb7effd700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ffb801039c0 0x7ffb801110f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:59086/0 (socket says 172.21.15.45:59086) 2023-12-16T21:33:18.141 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2023-12-16T21:33:17.646+0000 7ffb7effd700 1 -- 172.21.15.45:0/3505111401 learned_addr learned my addr 172.21.15.45:0/3505111401 (peer_addr_for_me v2:172.21.15.45:0/0) 2023-12-16T21:33:18.141 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2023-12-16T21:33:17.647+0000 7ffb7effd700 1 -- 172.21.15.45:0/3505111401 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffb700094f0 con 0x7ffb801039c0 2023-12-16T21:33:18.141 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2023-12-16T21:33:17.647+0000 7ffb7effd700 1 --2- 172.21.15.45:0/3505111401 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ffb801039c0 0x7ffb801110f0 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7ffb70006b00 tx=0x7ffb70009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:33:18.141 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2023-12-16T21:33:17.647+0000 7ffb677fe700 1 -- 172.21.15.45:0/3505111401 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7ffb700109b0 con 0x7ffb801039c0 2023-12-16T21:33:18.141 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2023-12-16T21:33:17.647+0000 7ffb677fe700 1 -- 172.21.15.45:0/3505111401 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(26 keys) v1 ==== 1009+0+0 (secure 0 0 0) 0x7ffb7001a430 con 0x7ffb801039c0 2023-12-16T21:33:18.141 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2023-12-16T21:33:17.647+0000 7ffb8599b700 1 -- 172.21.15.45:0/3505111401 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ffb80111830 con 0x7ffb801039c0 2023-12-16T21:33:18.141 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2023-12-16T21:33:17.647+0000 7ffb677fe700 1 -- 172.21.15.45:0/3505111401 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7ffb700245b0 con 0x7ffb801039c0 2023-12-16T21:33:18.141 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2023-12-16T21:33:17.647+0000 7ffb8599b700 1 -- 172.21.15.45:0/3505111401 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ffb80111c40 con 0x7ffb801039c0 2023-12-16T21:33:18.141 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2023-12-16T21:33:17.648+0000 7ffb677fe700 1 -- 172.21.15.45:0/3505111401 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 7) v1 ==== 43175+0+0 (secure 0 0 0) 0x7ffb70024830 con 0x7ffb801039c0 2023-12-16T21:33:18.141 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2023-12-16T21:33:17.648+0000 7ffb677fe700 1 --2- 172.21.15.45:0/3505111401 >> [v2:172.21.15.45:6800/2502560408,v1:172.21.15.45:6801/2502560408] conn(0x7ffb680364c0 0x7ffb68038970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:33:18.141 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2023-12-16T21:33:17.648+0000 7ffb8599b700 1 -- 172.21.15.45:0/3505111401 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ffb80102d60 con 0x7ffb801039c0 2023-12-16T21:33:18.141 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2023-12-16T21:33:17.648+0000 7ffb677fe700 1 -- 172.21.15.45:0/3505111401 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 936+0+0 (secure 0 0 0) 0x7ffb7004aa10 con 0x7ffb801039c0 2023-12-16T21:33:18.142 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2023-12-16T21:33:17.648+0000 7ffb7e7fc700 1 --2- 172.21.15.45:0/3505111401 >> [v2:172.21.15.45:6800/2502560408,v1:172.21.15.45:6801/2502560408] conn(0x7ffb680364c0 0x7ffb68038970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:33:18.142 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2023-12-16T21:33:17.649+0000 7ffb7e7fc700 1 --2- 172.21.15.45:0/3505111401 >> [v2:172.21.15.45:6800/2502560408,v1:172.21.15.45:6801/2502560408] conn(0x7ffb680364c0 0x7ffb68038970 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7ffb74009990 tx=0x7ffb74006e40 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:33:18.142 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2023-12-16T21:33:17.653+0000 7ffb677fe700 1 -- 172.21.15.45:0/3505111401 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+174570 (secure 0 0 0) 0x7ffb70018350 con 0x7ffb801039c0 2023-12-16T21:33:18.142 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2023-12-16T21:33:17.862+0000 7ffb8599b700 1 -- 172.21.15.45:0/3505111401 --> [v2:172.21.15.45:6800/2502560408,v1:172.21.15.45:6801/2502560408] -- mgr_command(tid 0: {"prefix": "orch set backend", "module_name": "cephadm", "target": ["mon-mgr", ""]}) v1 -- 0x7ffb800ffdb0 con 0x7ffb680364c0 2023-12-16T21:33:18.142 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2023-12-16T21:33:17.869+0000 7ffb677fe700 1 -- 172.21.15.45:0/3505111401 <== mgr.14120 v2:172.21.15.45:6800/2502560408 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+0 (secure 0 0 0) 0x7ffb800ffdb0 con 0x7ffb680364c0 2023-12-16T21:33:18.142 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2023-12-16T21:33:17.872+0000 7ffb8599b700 1 -- 172.21.15.45:0/3505111401 >> [v2:172.21.15.45:6800/2502560408,v1:172.21.15.45:6801/2502560408] conn(0x7ffb680364c0 msgr2=0x7ffb68038970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:33:18.142 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2023-12-16T21:33:17.872+0000 7ffb8599b700 1 --2- 172.21.15.45:0/3505111401 >> [v2:172.21.15.45:6800/2502560408,v1:172.21.15.45:6801/2502560408] conn(0x7ffb680364c0 0x7ffb68038970 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7ffb74009990 tx=0x7ffb74006e40 comp rx=0 tx=0).stop 2023-12-16T21:33:18.142 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2023-12-16T21:33:17.872+0000 7ffb8599b700 1 -- 172.21.15.45:0/3505111401 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ffb801039c0 msgr2=0x7ffb801110f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:33:18.142 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2023-12-16T21:33:17.872+0000 7ffb8599b700 1 --2- 172.21.15.45:0/3505111401 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ffb801039c0 0x7ffb801110f0 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7ffb70006b00 tx=0x7ffb70009250 comp rx=0 tx=0).stop 2023-12-16T21:33:18.142 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2023-12-16T21:33:17.872+0000 7ffb8599b700 1 -- 172.21.15.45:0/3505111401 shutdown_connections 2023-12-16T21:33:18.142 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2023-12-16T21:33:17.872+0000 7ffb8599b700 1 --2- 172.21.15.45:0/3505111401 >> [v2:172.21.15.45:6800/2502560408,v1:172.21.15.45:6801/2502560408] conn(0x7ffb680364c0 0x7ffb68038970 unknown :-1 s=CLOSED pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:33:18.143 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2023-12-16T21:33:17.872+0000 7ffb8599b700 1 --2- 172.21.15.45:0/3505111401 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ffb801039c0 0x7ffb801110f0 unknown :-1 s=CLOSED pgs=38 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:33:18.143 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2023-12-16T21:33:17.872+0000 7ffb8599b700 1 -- 172.21.15.45:0/3505111401 >> 172.21.15.45:0/3505111401 conn(0x7ffb800fda40 msgr2=0x7ffb800ff190 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-16T21:33:18.143 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2023-12-16T21:33:17.872+0000 7ffb8599b700 1 -- 172.21.15.45:0/3505111401 shutdown_connections 2023-12-16T21:33:18.143 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2023-12-16T21:33:17.872+0000 7ffb8599b700 1 -- 172.21.15.45:0/3505111401 wait complete. 2023-12-16T21:33:18.379 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Dec 16 21:33:18 smithi045 ceph-mon[80763]: from='client.14124 -' entity='client.admin' cmd=[{"prefix": "get_command_descriptions"}]: dispatch 2023-12-16T21:33:18.379 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Dec 16 21:33:18 smithi045 ceph-mon[80763]: from='client.14124 -' entity='client.admin' cmd=[{"prefix": "mgr_status"}]: dispatch 2023-12-16T21:33:18.379 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Dec 16 21:33:18 smithi045 ceph-mon[80763]: [16/Dec/2023:21:33:17] ENGINE Bus STARTING 2023-12-16T21:33:18.379 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Dec 16 21:33:18 smithi045 ceph-mon[80763]: from='mgr.14120 172.21.15.45:0/902863156' entity='mgr.smithi045.nhtmwc' 2023-12-16T21:33:18.379 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Dec 16 21:33:18 smithi045 ceph-mon[80763]: from='mgr.14120 172.21.15.45:0/902863156' entity='mgr.smithi045.nhtmwc' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-12-16T21:33:18.379 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Dec 16 21:33:18 smithi045 ceph-mon[80763]: from='mgr.14120 172.21.15.45:0/902863156' entity='mgr.smithi045.nhtmwc' 2023-12-16T21:33:18.379 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Dec 16 21:33:18 smithi045 ceph-mon[80763]: from='mgr.14120 172.21.15.45:0/902863156' entity='mgr.smithi045.nhtmwc' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-12-16T21:33:18.829 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stdout value unchanged 2023-12-16T21:33:18.829 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2023-12-16T21:33:18.376+0000 7fef4bda3700 1 Processor -- start 2023-12-16T21:33:18.829 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2023-12-16T21:33:18.376+0000 7fef4bda3700 1 -- start start 2023-12-16T21:33:18.829 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2023-12-16T21:33:18.376+0000 7fef4bda3700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fef44076c20 0x7fef44104ee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:33:18.829 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2023-12-16T21:33:18.376+0000 7fef4bda3700 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fef44105420 con 0x7fef44076c20 2023-12-16T21:33:18.829 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2023-12-16T21:33:18.376+0000 7fef49b3f700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fef44076c20 0x7fef44104ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:33:18.829 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2023-12-16T21:33:18.376+0000 7fef49b3f700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fef44076c20 0x7fef44104ee0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:59092/0 (socket says 172.21.15.45:59092) 2023-12-16T21:33:18.829 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2023-12-16T21:33:18.376+0000 7fef49b3f700 1 -- 172.21.15.45:0/890159556 learned_addr learned my addr 172.21.15.45:0/890159556 (peer_addr_for_me v2:172.21.15.45:0/0) 2023-12-16T21:33:18.829 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2023-12-16T21:33:18.377+0000 7fef49b3f700 1 -- 172.21.15.45:0/890159556 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fef44105560 con 0x7fef44076c20 2023-12-16T21:33:18.829 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2023-12-16T21:33:18.377+0000 7fef49b3f700 1 --2- 172.21.15.45:0/890159556 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fef44076c20 0x7fef44104ee0 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7fef400098e0 tx=0x7fef40009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bb4b6bd5bc018f0 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:33:18.829 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2023-12-16T21:33:18.377+0000 7fef48b3d700 1 -- 172.21.15.45:0/890159556 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fef4000b860 con 0x7fef44076c20 2023-12-16T21:33:18.829 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2023-12-16T21:33:18.377+0000 7fef48b3d700 1 -- 172.21.15.45:0/890159556 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(26 keys) v1 ==== 1009+0+0 (secure 0 0 0) 0x7fef4000be30 con 0x7fef44076c20 2023-12-16T21:33:18.830 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2023-12-16T21:33:18.377+0000 7fef48b3d700 1 -- 172.21.15.45:0/890159556 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fef40010380 con 0x7fef44076c20 2023-12-16T21:33:18.830 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2023-12-16T21:33:18.378+0000 7fef4bda3700 1 -- 172.21.15.45:0/890159556 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fef44076c20 msgr2=0x7fef44104ee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:33:18.830 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2023-12-16T21:33:18.378+0000 7fef4bda3700 1 --2- 172.21.15.45:0/890159556 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fef44076c20 0x7fef44104ee0 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7fef400098e0 tx=0x7fef40009250 comp rx=0 tx=0).stop 2023-12-16T21:33:18.830 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2023-12-16T21:33:18.378+0000 7fef4bda3700 1 -- 172.21.15.45:0/890159556 shutdown_connections 2023-12-16T21:33:18.830 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2023-12-16T21:33:18.378+0000 7fef4bda3700 1 --2- 172.21.15.45:0/890159556 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fef44076c20 0x7fef44104ee0 unknown :-1 s=CLOSED pgs=39 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:33:18.830 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2023-12-16T21:33:18.378+0000 7fef4bda3700 1 -- 172.21.15.45:0/890159556 >> 172.21.15.45:0/890159556 conn(0x7fef440fc950 msgr2=0x7fef440fed60 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-16T21:33:18.830 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2023-12-16T21:33:18.378+0000 7fef4bda3700 1 -- 172.21.15.45:0/890159556 shutdown_connections 2023-12-16T21:33:18.830 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2023-12-16T21:33:18.378+0000 7fef4bda3700 1 -- 172.21.15.45:0/890159556 wait complete. 2023-12-16T21:33:18.830 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2023-12-16T21:33:18.378+0000 7fef4bda3700 1 Processor -- start 2023-12-16T21:33:18.830 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2023-12-16T21:33:18.379+0000 7fef4bda3700 1 -- start start 2023-12-16T21:33:18.830 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2023-12-16T21:33:18.379+0000 7fef4bda3700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fef44076c20 0x7fef44102960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:33:18.830 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2023-12-16T21:33:18.379+0000 7fef4bda3700 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fef44100de0 con 0x7fef44076c20 2023-12-16T21:33:18.830 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2023-12-16T21:33:18.379+0000 7fef49b3f700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fef44076c20 0x7fef44102960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:33:18.830 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2023-12-16T21:33:18.379+0000 7fef49b3f700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fef44076c20 0x7fef44102960 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:59104/0 (socket says 172.21.15.45:59104) 2023-12-16T21:33:18.831 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2023-12-16T21:33:18.379+0000 7fef49b3f700 1 -- 172.21.15.45:0/3350749887 learned_addr learned my addr 172.21.15.45:0/3350749887 (peer_addr_for_me v2:172.21.15.45:0/0) 2023-12-16T21:33:18.831 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2023-12-16T21:33:18.379+0000 7fef49b3f700 1 -- 172.21.15.45:0/3350749887 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fef400094f0 con 0x7fef44076c20 2023-12-16T21:33:18.831 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2023-12-16T21:33:18.379+0000 7fef49b3f700 1 --2- 172.21.15.45:0/3350749887 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fef44076c20 0x7fef44102960 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7fef40006b00 tx=0x7fef40009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:33:18.831 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2023-12-16T21:33:18.379+0000 7fef3a7fc700 1 -- 172.21.15.45:0/3350749887 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fef40010a80 con 0x7fef44076c20 2023-12-16T21:33:18.831 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2023-12-16T21:33:18.379+0000 7fef3a7fc700 1 -- 172.21.15.45:0/3350749887 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(26 keys) v1 ==== 1009+0+0 (secure 0 0 0) 0x7fef4001a430 con 0x7fef44076c20 2023-12-16T21:33:18.831 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2023-12-16T21:33:18.379+0000 7fef4bda3700 1 -- 172.21.15.45:0/3350749887 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fef44100fe0 con 0x7fef44076c20 2023-12-16T21:33:18.831 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2023-12-16T21:33:18.380+0000 7fef3a7fc700 1 -- 172.21.15.45:0/3350749887 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fef400245b0 con 0x7fef44076c20 2023-12-16T21:33:18.831 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2023-12-16T21:33:18.380+0000 7fef4bda3700 1 -- 172.21.15.45:0/3350749887 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fef441013f0 con 0x7fef44076c20 2023-12-16T21:33:18.831 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2023-12-16T21:33:18.380+0000 7fef3a7fc700 1 -- 172.21.15.45:0/3350749887 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 7) v1 ==== 43175+0+0 (secure 0 0 0) 0x7fef40010be0 con 0x7fef44076c20 2023-12-16T21:33:18.831 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2023-12-16T21:33:18.381+0000 7fef3a7fc700 1 --2- 172.21.15.45:0/3350749887 >> [v2:172.21.15.45:6800/2502560408,v1:172.21.15.45:6801/2502560408] conn(0x7fef300364c0 0x7fef30038970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:33:18.831 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2023-12-16T21:33:18.381+0000 7fef3a7fc700 1 -- 172.21.15.45:0/3350749887 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 936+0+0 (secure 0 0 0) 0x7fef4004a940 con 0x7fef44076c20 2023-12-16T21:33:18.831 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2023-12-16T21:33:18.381+0000 7fef4933e700 1 --2- 172.21.15.45:0/3350749887 >> [v2:172.21.15.45:6800/2502560408,v1:172.21.15.45:6801/2502560408] conn(0x7fef300364c0 0x7fef30038970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:33:18.831 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2023-12-16T21:33:18.381+0000 7fef4bda3700 1 -- 172.21.15.45:0/3350749887 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fef4404bb80 con 0x7fef44076c20 2023-12-16T21:33:18.831 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2023-12-16T21:33:18.381+0000 7fef4933e700 1 --2- 172.21.15.45:0/3350749887 >> [v2:172.21.15.45:6800/2502560408,v1:172.21.15.45:6801/2502560408] conn(0x7fef300364c0 0x7fef30038970 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7fef34009990 tx=0x7fef34006e40 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:33:18.832 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2023-12-16T21:33:18.385+0000 7fef3a7fc700 1 -- 172.21.15.45:0/3350749887 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+174570 (secure 0 0 0) 0x7fef400239e0 con 0x7fef44076c20 2023-12-16T21:33:18.832 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2023-12-16T21:33:18.582+0000 7fef4bda3700 1 -- 172.21.15.45:0/3350749887 --> [v2:172.21.15.45:6800/2502560408,v1:172.21.15.45:6801/2502560408] -- mgr_command(tid 0: {"prefix": "cephadm set-user", "user": "root", "target": ["mon-mgr", ""]}) v1 -- 0x7fef4410aca0 con 0x7fef300364c0 2023-12-16T21:33:18.832 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2023-12-16T21:33:18.583+0000 7fef3a7fc700 1 -- 172.21.15.45:0/3350749887 <== mgr.14120 v2:172.21.15.45:6800/2502560408 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+16 (secure 0 0 0) 0x7fef4410aca0 con 0x7fef300364c0 2023-12-16T21:33:18.832 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2023-12-16T21:33:18.585+0000 7fef4bda3700 1 -- 172.21.15.45:0/3350749887 >> [v2:172.21.15.45:6800/2502560408,v1:172.21.15.45:6801/2502560408] conn(0x7fef300364c0 msgr2=0x7fef30038970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:33:18.832 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2023-12-16T21:33:18.585+0000 7fef4bda3700 1 --2- 172.21.15.45:0/3350749887 >> [v2:172.21.15.45:6800/2502560408,v1:172.21.15.45:6801/2502560408] conn(0x7fef300364c0 0x7fef30038970 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7fef34009990 tx=0x7fef34006e40 comp rx=0 tx=0).stop 2023-12-16T21:33:18.832 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2023-12-16T21:33:18.585+0000 7fef4bda3700 1 -- 172.21.15.45:0/3350749887 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fef44076c20 msgr2=0x7fef44102960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:33:18.832 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2023-12-16T21:33:18.585+0000 7fef4bda3700 1 --2- 172.21.15.45:0/3350749887 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fef44076c20 0x7fef44102960 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7fef40006b00 tx=0x7fef40009250 comp rx=0 tx=0).stop 2023-12-16T21:33:18.832 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2023-12-16T21:33:18.585+0000 7fef4bda3700 1 -- 172.21.15.45:0/3350749887 shutdown_connections 2023-12-16T21:33:18.832 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2023-12-16T21:33:18.585+0000 7fef4bda3700 1 --2- 172.21.15.45:0/3350749887 >> [v2:172.21.15.45:6800/2502560408,v1:172.21.15.45:6801/2502560408] conn(0x7fef300364c0 0x7fef30038970 unknown :-1 s=CLOSED pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:33:18.832 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2023-12-16T21:33:18.585+0000 7fef4bda3700 1 --2- 172.21.15.45:0/3350749887 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fef44076c20 0x7fef44102960 unknown :-1 s=CLOSED pgs=40 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:33:18.832 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2023-12-16T21:33:18.585+0000 7fef4bda3700 1 -- 172.21.15.45:0/3350749887 >> 172.21.15.45:0/3350749887 conn(0x7fef440fc950 msgr2=0x7fef440fe150 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-16T21:33:18.832 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2023-12-16T21:33:18.585+0000 7fef4bda3700 1 -- 172.21.15.45:0/3350749887 shutdown_connections 2023-12-16T21:33:18.832 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2023-12-16T21:33:18.585+0000 7fef4bda3700 1 -- 172.21.15.45:0/3350749887 wait complete. 2023-12-16T21:33:18.832 INFO:teuthology.orchestra.run.smithi045.stdout:Generating ssh key... 2023-12-16T21:33:19.256 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Dec 16 21:33:19 smithi045 ceph-mon[80763]: [16/Dec/2023:21:33:17] ENGINE Serving on https://172.21.15.45:7150 2023-12-16T21:33:19.256 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Dec 16 21:33:19 smithi045 ceph-mon[80763]: [16/Dec/2023:21:33:17] ENGINE Bus STARTED 2023-12-16T21:33:19.256 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Dec 16 21:33:19 smithi045 ceph-mon[80763]: from='client.14132 -' entity='client.admin' cmd=[{"prefix": "orch set backend", "module_name": "cephadm", "target": ["mon-mgr", ""]}]: dispatch 2023-12-16T21:33:19.257 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Dec 16 21:33:19 smithi045 ceph-mon[80763]: mgrmap e8: smithi045.nhtmwc(active, since 2s) 2023-12-16T21:33:19.679 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2023-12-16T21:33:19.050+0000 7fe494842700 1 Processor -- start 2023-12-16T21:33:19.679 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2023-12-16T21:33:19.050+0000 7fe494842700 1 -- start start 2023-12-16T21:33:19.680 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2023-12-16T21:33:19.050+0000 7fe494842700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fe4901039c0 0x7fe490103dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:33:19.680 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2023-12-16T21:33:19.050+0000 7fe494842700 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe490104310 con 0x7fe4901039c0 2023-12-16T21:33:19.680 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2023-12-16T21:33:19.050+0000 7fe48e59c700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fe4901039c0 0x7fe490103dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:33:19.680 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2023-12-16T21:33:19.050+0000 7fe48e59c700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fe4901039c0 0x7fe490103dd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:59118/0 (socket says 172.21.15.45:59118) 2023-12-16T21:33:19.680 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2023-12-16T21:33:19.050+0000 7fe48e59c700 1 -- 172.21.15.45:0/1994827757 learned_addr learned my addr 172.21.15.45:0/1994827757 (peer_addr_for_me v2:172.21.15.45:0/0) 2023-12-16T21:33:19.680 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2023-12-16T21:33:19.050+0000 7fe48e59c700 1 -- 172.21.15.45:0/1994827757 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe490101fb0 con 0x7fe4901039c0 2023-12-16T21:33:19.680 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2023-12-16T21:33:19.051+0000 7fe48e59c700 1 --2- 172.21.15.45:0/1994827757 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fe4901039c0 0x7fe490103dd0 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7fe4800098e0 tx=0x7fe480009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=597ad4b9385dcc81 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:33:19.680 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2023-12-16T21:33:19.051+0000 7fe48d59a700 1 -- 172.21.15.45:0/1994827757 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fe48000b860 con 0x7fe4901039c0 2023-12-16T21:33:19.680 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2023-12-16T21:33:19.051+0000 7fe48d59a700 1 -- 172.21.15.45:0/1994827757 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(26 keys) v1 ==== 1009+0+0 (secure 0 0 0) 0x7fe48000be30 con 0x7fe4901039c0 2023-12-16T21:33:19.680 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2023-12-16T21:33:19.051+0000 7fe48d59a700 1 -- 172.21.15.45:0/1994827757 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fe480010380 con 0x7fe4901039c0 2023-12-16T21:33:19.680 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2023-12-16T21:33:19.051+0000 7fe494842700 1 -- 172.21.15.45:0/1994827757 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fe4901039c0 msgr2=0x7fe490103dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:33:19.680 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2023-12-16T21:33:19.051+0000 7fe494842700 1 --2- 172.21.15.45:0/1994827757 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fe4901039c0 0x7fe490103dd0 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7fe4800098e0 tx=0x7fe480009250 comp rx=0 tx=0).stop 2023-12-16T21:33:19.681 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2023-12-16T21:33:19.051+0000 7fe494842700 1 -- 172.21.15.45:0/1994827757 shutdown_connections 2023-12-16T21:33:19.681 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2023-12-16T21:33:19.051+0000 7fe494842700 1 --2- 172.21.15.45:0/1994827757 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fe4901039c0 0x7fe490103dd0 unknown :-1 s=CLOSED pgs=41 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:33:19.681 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2023-12-16T21:33:19.051+0000 7fe494842700 1 -- 172.21.15.45:0/1994827757 >> 172.21.15.45:0/1994827757 conn(0x7fe4900fda40 msgr2=0x7fe4900ffe70 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-16T21:33:19.681 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2023-12-16T21:33:19.051+0000 7fe494842700 1 -- 172.21.15.45:0/1994827757 shutdown_connections 2023-12-16T21:33:19.681 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2023-12-16T21:33:19.051+0000 7fe494842700 1 -- 172.21.15.45:0/1994827757 wait complete. 2023-12-16T21:33:19.681 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2023-12-16T21:33:19.052+0000 7fe494842700 1 Processor -- start 2023-12-16T21:33:19.681 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2023-12-16T21:33:19.052+0000 7fe494842700 1 -- start start 2023-12-16T21:33:19.681 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2023-12-16T21:33:19.052+0000 7fe494842700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fe4901039c0 0x7fe4901110f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:33:19.681 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2023-12-16T21:33:19.052+0000 7fe494842700 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe490111630 con 0x7fe4901039c0 2023-12-16T21:33:19.681 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2023-12-16T21:33:19.052+0000 7fe48e59c700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fe4901039c0 0x7fe4901110f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:33:19.682 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2023-12-16T21:33:19.052+0000 7fe48e59c700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fe4901039c0 0x7fe4901110f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:59134/0 (socket says 172.21.15.45:59134) 2023-12-16T21:33:19.682 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2023-12-16T21:33:19.052+0000 7fe48e59c700 1 -- 172.21.15.45:0/1112285982 learned_addr learned my addr 172.21.15.45:0/1112285982 (peer_addr_for_me v2:172.21.15.45:0/0) 2023-12-16T21:33:19.682 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2023-12-16T21:33:19.053+0000 7fe48e59c700 1 -- 172.21.15.45:0/1112285982 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe4800094f0 con 0x7fe4901039c0 2023-12-16T21:33:19.682 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2023-12-16T21:33:19.053+0000 7fe48e59c700 1 --2- 172.21.15.45:0/1112285982 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fe4901039c0 0x7fe4901110f0 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7fe480000c00 tx=0x7fe480009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:33:19.682 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2023-12-16T21:33:19.053+0000 7fe47effd700 1 -- 172.21.15.45:0/1112285982 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fe480010a80 con 0x7fe4901039c0 2023-12-16T21:33:19.682 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2023-12-16T21:33:19.053+0000 7fe47effd700 1 -- 172.21.15.45:0/1112285982 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(26 keys) v1 ==== 1009+0+0 (secure 0 0 0) 0x7fe48001a430 con 0x7fe4901039c0 2023-12-16T21:33:19.682 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2023-12-16T21:33:19.053+0000 7fe494842700 1 -- 172.21.15.45:0/1112285982 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe490111830 con 0x7fe4901039c0 2023-12-16T21:33:19.682 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2023-12-16T21:33:19.053+0000 7fe47effd700 1 -- 172.21.15.45:0/1112285982 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fe4800245b0 con 0x7fe4901039c0 2023-12-16T21:33:19.682 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2023-12-16T21:33:19.053+0000 7fe494842700 1 -- 172.21.15.45:0/1112285982 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe490111c40 con 0x7fe4901039c0 2023-12-16T21:33:19.682 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2023-12-16T21:33:19.054+0000 7fe47effd700 1 -- 172.21.15.45:0/1112285982 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 8) v1 ==== 43281+0+0 (secure 0 0 0) 0x7fe480010be0 con 0x7fe4901039c0 2023-12-16T21:33:19.682 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2023-12-16T21:33:19.054+0000 7fe47effd700 1 --2- 172.21.15.45:0/1112285982 >> [v2:172.21.15.45:6800/2502560408,v1:172.21.15.45:6801/2502560408] conn(0x7fe478036640 0x7fe478038af0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:33:19.682 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2023-12-16T21:33:19.054+0000 7fe494842700 1 -- 172.21.15.45:0/1112285982 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe490102d60 con 0x7fe4901039c0 2023-12-16T21:33:19.683 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2023-12-16T21:33:19.054+0000 7fe47effd700 1 -- 172.21.15.45:0/1112285982 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 936+0+0 (secure 0 0 0) 0x7fe48004a650 con 0x7fe4901039c0 2023-12-16T21:33:19.683 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2023-12-16T21:33:19.054+0000 7fe48dd9b700 1 --2- 172.21.15.45:0/1112285982 >> [v2:172.21.15.45:6800/2502560408,v1:172.21.15.45:6801/2502560408] conn(0x7fe478036640 0x7fe478038af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:33:19.683 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2023-12-16T21:33:19.055+0000 7fe48dd9b700 1 --2- 172.21.15.45:0/1112285982 >> [v2:172.21.15.45:6800/2502560408,v1:172.21.15.45:6801/2502560408] conn(0x7fe478036640 0x7fe478038af0 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7fe484009990 tx=0x7fe484006e40 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:33:19.683 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2023-12-16T21:33:19.059+0000 7fe47effd700 1 -- 172.21.15.45:0/1112285982 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+174570 (secure 0 0 0) 0x7fe48000f980 con 0x7fe4901039c0 2023-12-16T21:33:19.683 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2023-12-16T21:33:19.254+0000 7fe494842700 1 -- 172.21.15.45:0/1112285982 --> [v2:172.21.15.45:6800/2502560408,v1:172.21.15.45:6801/2502560408] -- mgr_command(tid 0: {"prefix": "cephadm generate-key", "target": ["mon-mgr", ""]}) v1 -- 0x7fe49010b070 con 0x7fe478036640 2023-12-16T21:33:19.683 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2023-12-16T21:33:19.428+0000 7fe47effd700 1 -- 172.21.15.45:0/1112285982 <== mgr.14120 v2:172.21.15.45:6800/2502560408 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+0 (secure 0 0 0) 0x7fe49010b070 con 0x7fe478036640 2023-12-16T21:33:19.683 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2023-12-16T21:33:19.430+0000 7fe494842700 1 -- 172.21.15.45:0/1112285982 >> [v2:172.21.15.45:6800/2502560408,v1:172.21.15.45:6801/2502560408] conn(0x7fe478036640 msgr2=0x7fe478038af0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:33:19.683 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2023-12-16T21:33:19.430+0000 7fe494842700 1 --2- 172.21.15.45:0/1112285982 >> [v2:172.21.15.45:6800/2502560408,v1:172.21.15.45:6801/2502560408] conn(0x7fe478036640 0x7fe478038af0 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7fe484009990 tx=0x7fe484006e40 comp rx=0 tx=0).stop 2023-12-16T21:33:19.683 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2023-12-16T21:33:19.430+0000 7fe494842700 1 -- 172.21.15.45:0/1112285982 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fe4901039c0 msgr2=0x7fe4901110f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:33:19.683 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2023-12-16T21:33:19.430+0000 7fe494842700 1 --2- 172.21.15.45:0/1112285982 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fe4901039c0 0x7fe4901110f0 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7fe480000c00 tx=0x7fe480009250 comp rx=0 tx=0).stop 2023-12-16T21:33:19.684 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2023-12-16T21:33:19.430+0000 7fe494842700 1 -- 172.21.15.45:0/1112285982 shutdown_connections 2023-12-16T21:33:19.684 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2023-12-16T21:33:19.430+0000 7fe494842700 1 --2- 172.21.15.45:0/1112285982 >> [v2:172.21.15.45:6800/2502560408,v1:172.21.15.45:6801/2502560408] conn(0x7fe478036640 0x7fe478038af0 unknown :-1 s=CLOSED pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:33:19.684 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2023-12-16T21:33:19.430+0000 7fe494842700 1 --2- 172.21.15.45:0/1112285982 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fe4901039c0 0x7fe4901110f0 unknown :-1 s=CLOSED pgs=42 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:33:19.684 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2023-12-16T21:33:19.430+0000 7fe494842700 1 -- 172.21.15.45:0/1112285982 >> 172.21.15.45:0/1112285982 conn(0x7fe4900fda40 msgr2=0x7fe4900ff1c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-16T21:33:19.684 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2023-12-16T21:33:19.430+0000 7fe494842700 1 -- 172.21.15.45:0/1112285982 shutdown_connections 2023-12-16T21:33:19.684 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2023-12-16T21:33:19.430+0000 7fe494842700 1 -- 172.21.15.45:0/1112285982 wait complete. 2023-12-16T21:33:20.363 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stdout ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDY/F0Hv6LE/uJ1jo40L0WUNThVihgYwfPy+mLt2OQbJWonIqKHi+B4LiwYorryFM9Fy0fuR5Rwyf230hg1kl1sZmhdvsmFa2huLbc9543VlR8SlXiLiQJXCvjE9Gm2Ba3h2W7LhS1+S8ZOTJOhms6VTb7IO2WF8OofHyymCM8a7zO0N2x8f+Khkh7/IoBI8cDQzHXB73NqS4+ONXOhoGPAwyvgp3EJXcy23eKD1Ss9lml8XU3Ek5oRsfhSeyIXJHicj6EFwHnZNuL8dooEwEdiaJ00Oeunrp5EksuK7TblA1lfDpZT3gMWvY+PP92eKruWbC4Ai7ZAAg/L+WKpj5/za4+cQqSCtwmaT6cWsavGVY0SwVQmyl5z/E3JLDqyPAMH0YnHE+vcn/0VFIUbLMt//8c3LOTwBnTGFSFz5uThRxxOdNAKXZouRU4nt7+L/gl+TWhovSyoBdNPS0lHmIo7jDc1QaJVpt5cmAvN0zvh4qRdsNEUV13xQrq6wPqDgyM= ceph-88ff363c-9c5a-11ee-95a5-87774f69a715 2023-12-16T21:33:20.363 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2023-12-16T21:33:19.908+0000 7f77efc85700 1 Processor -- start 2023-12-16T21:33:20.363 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2023-12-16T21:33:19.908+0000 7f77efc85700 1 -- start start 2023-12-16T21:33:20.363 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2023-12-16T21:33:19.908+0000 7f77efc85700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f77e81038f0 0x7f77e8103d00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:33:20.363 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2023-12-16T21:33:19.908+0000 7f77efc85700 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f77e8104240 con 0x7f77e81038f0 2023-12-16T21:33:20.363 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2023-12-16T21:33:19.908+0000 7f77eda21700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f77e81038f0 0x7f77e8103d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:33:20.363 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2023-12-16T21:33:19.908+0000 7f77eda21700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f77e81038f0 0x7f77e8103d00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:59142/0 (socket says 172.21.15.45:59142) 2023-12-16T21:33:20.363 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2023-12-16T21:33:19.909+0000 7f77eda21700 1 -- 172.21.15.45:0/626175178 learned_addr learned my addr 172.21.15.45:0/626175178 (peer_addr_for_me v2:172.21.15.45:0/0) 2023-12-16T21:33:20.364 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2023-12-16T21:33:19.909+0000 7f77eda21700 1 -- 172.21.15.45:0/626175178 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f77e8104380 con 0x7f77e81038f0 2023-12-16T21:33:20.364 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2023-12-16T21:33:19.909+0000 7f77eda21700 1 --2- 172.21.15.45:0/626175178 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f77e81038f0 0x7f77e8103d00 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7f77d8009e50 tx=0x7f77d8009840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=16dec1dc99983859 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:33:20.364 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2023-12-16T21:33:19.909+0000 7f77eca1f700 1 -- 172.21.15.45:0/626175178 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f77d80092e0 con 0x7f77e81038f0 2023-12-16T21:33:20.364 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2023-12-16T21:33:19.909+0000 7f77eca1f700 1 -- 172.21.15.45:0/626175178 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(26 keys) v1 ==== 1009+0+0 (secure 0 0 0) 0x7f77d800be90 con 0x7f77e81038f0 2023-12-16T21:33:20.364 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2023-12-16T21:33:19.909+0000 7f77eca1f700 1 -- 172.21.15.45:0/626175178 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f77d8007400 con 0x7f77e81038f0 2023-12-16T21:33:20.364 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2023-12-16T21:33:19.910+0000 7f77efc85700 1 -- 172.21.15.45:0/626175178 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f77e81038f0 msgr2=0x7f77e8103d00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:33:20.364 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2023-12-16T21:33:19.910+0000 7f77efc85700 1 --2- 172.21.15.45:0/626175178 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f77e81038f0 0x7f77e8103d00 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7f77d8009e50 tx=0x7f77d8009840 comp rx=0 tx=0).stop 2023-12-16T21:33:20.364 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2023-12-16T21:33:19.910+0000 7f77efc85700 1 -- 172.21.15.45:0/626175178 shutdown_connections 2023-12-16T21:33:20.364 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2023-12-16T21:33:19.910+0000 7f77efc85700 1 --2- 172.21.15.45:0/626175178 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f77e81038f0 0x7f77e8103d00 unknown :-1 s=CLOSED pgs=43 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:33:20.364 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2023-12-16T21:33:19.910+0000 7f77efc85700 1 -- 172.21.15.45:0/626175178 >> 172.21.15.45:0/626175178 conn(0x7f77e80fc0b0 msgr2=0x7f77e80fe4e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-16T21:33:20.364 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2023-12-16T21:33:19.910+0000 7f77efc85700 1 -- 172.21.15.45:0/626175178 shutdown_connections 2023-12-16T21:33:20.364 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2023-12-16T21:33:19.910+0000 7f77efc85700 1 -- 172.21.15.45:0/626175178 wait complete. 2023-12-16T21:33:20.364 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2023-12-16T21:33:19.910+0000 7f77efc85700 1 Processor -- start 2023-12-16T21:33:20.364 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2023-12-16T21:33:19.910+0000 7f77efc85700 1 -- start start 2023-12-16T21:33:20.364 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2023-12-16T21:33:19.911+0000 7f77efc85700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f77e81038f0 0x7f77e8111140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:33:20.365 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2023-12-16T21:33:19.911+0000 7f77efc85700 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f77e8111680 con 0x7f77e81038f0 2023-12-16T21:33:20.365 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2023-12-16T21:33:19.911+0000 7f77eda21700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f77e81038f0 0x7f77e8111140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:33:20.365 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2023-12-16T21:33:19.911+0000 7f77eda21700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f77e81038f0 0x7f77e8111140 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:59148/0 (socket says 172.21.15.45:59148) 2023-12-16T21:33:20.365 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2023-12-16T21:33:19.911+0000 7f77eda21700 1 -- 172.21.15.45:0/741302064 learned_addr learned my addr 172.21.15.45:0/741302064 (peer_addr_for_me v2:172.21.15.45:0/0) 2023-12-16T21:33:20.365 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2023-12-16T21:33:19.911+0000 7f77eda21700 1 -- 172.21.15.45:0/741302064 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f77d8009ae0 con 0x7f77e81038f0 2023-12-16T21:33:20.365 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2023-12-16T21:33:19.911+0000 7f77eda21700 1 --2- 172.21.15.45:0/741302064 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f77e81038f0 0x7f77e8111140 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7f77d8009fd0 tx=0x7f77d8009840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:33:20.365 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2023-12-16T21:33:19.911+0000 7f77e67fc700 1 -- 172.21.15.45:0/741302064 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f77d801c620 con 0x7f77e81038f0 2023-12-16T21:33:20.365 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2023-12-16T21:33:19.911+0000 7f77e67fc700 1 -- 172.21.15.45:0/741302064 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(26 keys) v1 ==== 1009+0+0 (secure 0 0 0) 0x7f77d801cbf0 con 0x7f77e81038f0 2023-12-16T21:33:20.365 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2023-12-16T21:33:19.911+0000 7f77efc85700 1 -- 172.21.15.45:0/741302064 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f77e8111880 con 0x7f77e81038f0 2023-12-16T21:33:20.365 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2023-12-16T21:33:19.911+0000 7f77e67fc700 1 -- 172.21.15.45:0/741302064 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f77d8007bd0 con 0x7f77e81038f0 2023-12-16T21:33:20.365 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2023-12-16T21:33:19.911+0000 7f77efc85700 1 -- 172.21.15.45:0/741302064 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f77e8112520 con 0x7f77e81038f0 2023-12-16T21:33:20.365 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2023-12-16T21:33:19.912+0000 7f77e67fc700 1 -- 172.21.15.45:0/741302064 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 8) v1 ==== 43281+0+0 (secure 0 0 0) 0x7f77d8019070 con 0x7f77e81038f0 2023-12-16T21:33:20.365 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2023-12-16T21:33:19.912+0000 7f77e67fc700 1 --2- 172.21.15.45:0/741302064 >> [v2:172.21.15.45:6800/2502560408,v1:172.21.15.45:6801/2502560408] conn(0x7f77d0036640 0x7f77d0038af0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:33:20.365 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2023-12-16T21:33:19.912+0000 7f77efc85700 1 -- 172.21.15.45:0/741302064 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f77e805f270 con 0x7f77e81038f0 2023-12-16T21:33:20.366 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2023-12-16T21:33:19.912+0000 7f77e67fc700 1 -- 172.21.15.45:0/741302064 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 936+0+0 (secure 0 0 0) 0x7f77d8049e70 con 0x7f77e81038f0 2023-12-16T21:33:20.366 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2023-12-16T21:33:19.913+0000 7f77ed220700 1 --2- 172.21.15.45:0/741302064 >> [v2:172.21.15.45:6800/2502560408,v1:172.21.15.45:6801/2502560408] conn(0x7f77d0036640 0x7f77d0038af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:33:20.366 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2023-12-16T21:33:19.913+0000 7f77ed220700 1 --2- 172.21.15.45:0/741302064 >> [v2:172.21.15.45:6800/2502560408,v1:172.21.15.45:6801/2502560408] conn(0x7f77d0036640 0x7f77d0038af0 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7f77dc009990 tx=0x7f77dc006e40 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:33:20.366 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2023-12-16T21:33:19.917+0000 7f77e67fc700 1 -- 172.21.15.45:0/741302064 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+174570 (secure 0 0 0) 0x7f77d801c780 con 0x7f77e81038f0 2023-12-16T21:33:20.366 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2023-12-16T21:33:20.115+0000 7f77efc85700 1 -- 172.21.15.45:0/741302064 --> [v2:172.21.15.45:6800/2502560408,v1:172.21.15.45:6801/2502560408] -- mgr_command(tid 0: {"prefix": "cephadm get-pub-key", "target": ["mon-mgr", ""]}) v1 -- 0x7f77e8113040 con 0x7f77d0036640 2023-12-16T21:33:20.366 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2023-12-16T21:33:20.115+0000 7f77e67fc700 1 -- 172.21.15.45:0/741302064 <== mgr.14120 v2:172.21.15.45:6800/2502560408 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+595 (secure 0 0 0) 0x7f77e8113040 con 0x7f77d0036640 2023-12-16T21:33:20.366 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2023-12-16T21:33:20.117+0000 7f77efc85700 1 -- 172.21.15.45:0/741302064 >> [v2:172.21.15.45:6800/2502560408,v1:172.21.15.45:6801/2502560408] conn(0x7f77d0036640 msgr2=0x7f77d0038af0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:33:20.366 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2023-12-16T21:33:20.117+0000 7f77efc85700 1 --2- 172.21.15.45:0/741302064 >> [v2:172.21.15.45:6800/2502560408,v1:172.21.15.45:6801/2502560408] conn(0x7f77d0036640 0x7f77d0038af0 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7f77dc009990 tx=0x7f77dc006e40 comp rx=0 tx=0).stop 2023-12-16T21:33:20.366 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2023-12-16T21:33:20.117+0000 7f77efc85700 1 -- 172.21.15.45:0/741302064 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f77e81038f0 msgr2=0x7f77e8111140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:33:20.366 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2023-12-16T21:33:20.117+0000 7f77efc85700 1 --2- 172.21.15.45:0/741302064 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f77e81038f0 0x7f77e8111140 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7f77d8009fd0 tx=0x7f77d8009840 comp rx=0 tx=0).stop 2023-12-16T21:33:20.366 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2023-12-16T21:33:20.117+0000 7f77efc85700 1 -- 172.21.15.45:0/741302064 shutdown_connections 2023-12-16T21:33:20.367 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2023-12-16T21:33:20.117+0000 7f77efc85700 1 --2- 172.21.15.45:0/741302064 >> [v2:172.21.15.45:6800/2502560408,v1:172.21.15.45:6801/2502560408] conn(0x7f77d0036640 0x7f77d0038af0 unknown :-1 s=CLOSED pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:33:20.367 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2023-12-16T21:33:20.117+0000 7f77efc85700 1 --2- 172.21.15.45:0/741302064 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f77e81038f0 0x7f77e8111140 unknown :-1 s=CLOSED pgs=44 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:33:20.367 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2023-12-16T21:33:20.117+0000 7f77efc85700 1 -- 172.21.15.45:0/741302064 >> 172.21.15.45:0/741302064 conn(0x7f77e80fc0b0 msgr2=0x7f77e80fd850 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-16T21:33:20.367 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2023-12-16T21:33:20.118+0000 7f77efc85700 1 -- 172.21.15.45:0/741302064 shutdown_connections 2023-12-16T21:33:20.367 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2023-12-16T21:33:20.118+0000 7f77efc85700 1 -- 172.21.15.45:0/741302064 wait complete. 2023-12-16T21:33:20.367 INFO:teuthology.orchestra.run.smithi045.stdout:Wrote public SSH key to /home/ubuntu/cephtest/ceph.pub 2023-12-16T21:33:20.367 INFO:teuthology.orchestra.run.smithi045.stdout:Adding key to root@localhost authorized_keys... 2023-12-16T21:33:20.367 INFO:teuthology.orchestra.run.smithi045.stdout:Adding host smithi045... 2023-12-16T21:33:20.437 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Dec 16 21:33:20 smithi045 ceph-mon[80763]: from='client.14134 -' entity='client.admin' cmd=[{"prefix": "cephadm set-user", "user": "root", "target": ["mon-mgr", ""]}]: dispatch 2023-12-16T21:33:20.437 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Dec 16 21:33:20 smithi045 ceph-mon[80763]: from='client.14136 -' entity='client.admin' cmd=[{"prefix": "cephadm generate-key", "target": ["mon-mgr", ""]}]: dispatch 2023-12-16T21:33:20.437 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Dec 16 21:33:20 smithi045 ceph-mon[80763]: Generating ssh key... 2023-12-16T21:33:20.437 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Dec 16 21:33:20 smithi045 ceph-mon[80763]: from='mgr.14120 172.21.15.45:0/902863156' entity='mgr.smithi045.nhtmwc' 2023-12-16T21:33:20.437 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Dec 16 21:33:20 smithi045 ceph-mon[80763]: from='mgr.14120 172.21.15.45:0/902863156' entity='mgr.smithi045.nhtmwc' 2023-12-16T21:33:21.431 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Dec 16 21:33:21 smithi045 ceph-mon[80763]: from='client.14138 -' entity='client.admin' cmd=[{"prefix": "cephadm get-pub-key", "target": ["mon-mgr", ""]}]: dispatch 2023-12-16T21:33:22.513 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Dec 16 21:33:22 smithi045 ceph-mon[80763]: from='client.14140 -' entity='client.admin' cmd=[{"prefix": "orch host add", "hostname": "smithi045", "addr": "172.21.15.45", "target": ["mon-mgr", ""]}]: dispatch 2023-12-16T21:33:23.129 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stdout Added host 'smithi045' with addr '172.21.15.45' 2023-12-16T21:33:23.129 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2023-12-16T21:33:20.652+0000 7fe7f2cbd700 1 Processor -- start 2023-12-16T21:33:23.129 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2023-12-16T21:33:20.652+0000 7fe7f2cbd700 1 -- start start 2023-12-16T21:33:23.130 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2023-12-16T21:33:20.652+0000 7fe7f2cbd700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fe7ec1039c0 0x7fe7ec103dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:33:23.130 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2023-12-16T21:33:20.652+0000 7fe7f2cbd700 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe7ec104310 con 0x7fe7ec1039c0 2023-12-16T21:33:23.130 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2023-12-16T21:33:20.653+0000 7fe7f0a59700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fe7ec1039c0 0x7fe7ec103dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:33:23.130 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2023-12-16T21:33:20.653+0000 7fe7f0a59700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fe7ec1039c0 0x7fe7ec103dd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:59158/0 (socket says 172.21.15.45:59158) 2023-12-16T21:33:23.130 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2023-12-16T21:33:20.653+0000 7fe7f0a59700 1 -- 172.21.15.45:0/2045051497 learned_addr learned my addr 172.21.15.45:0/2045051497 (peer_addr_for_me v2:172.21.15.45:0/0) 2023-12-16T21:33:23.130 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2023-12-16T21:33:20.653+0000 7fe7f0a59700 1 -- 172.21.15.45:0/2045051497 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe7ec101fb0 con 0x7fe7ec1039c0 2023-12-16T21:33:23.130 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2023-12-16T21:33:20.653+0000 7fe7f0a59700 1 --2- 172.21.15.45:0/2045051497 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fe7ec1039c0 0x7fe7ec103dd0 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7fe7e4009e50 tx=0x7fe7e4009840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5d8b3a70c9281e37 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:33:23.130 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2023-12-16T21:33:20.653+0000 7fe7eb7fe700 1 -- 172.21.15.45:0/2045051497 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fe7e40092e0 con 0x7fe7ec1039c0 2023-12-16T21:33:23.130 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2023-12-16T21:33:20.653+0000 7fe7eb7fe700 1 -- 172.21.15.45:0/2045051497 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(26 keys) v1 ==== 1009+0+0 (secure 0 0 0) 0x7fe7e400be90 con 0x7fe7ec1039c0 2023-12-16T21:33:23.130 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2023-12-16T21:33:20.653+0000 7fe7eb7fe700 1 -- 172.21.15.45:0/2045051497 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fe7e4007400 con 0x7fe7ec1039c0 2023-12-16T21:33:23.130 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2023-12-16T21:33:20.654+0000 7fe7f2cbd700 1 -- 172.21.15.45:0/2045051497 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fe7ec1039c0 msgr2=0x7fe7ec103dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:33:23.130 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2023-12-16T21:33:20.654+0000 7fe7f2cbd700 1 --2- 172.21.15.45:0/2045051497 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fe7ec1039c0 0x7fe7ec103dd0 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7fe7e4009e50 tx=0x7fe7e4009840 comp rx=0 tx=0).stop 2023-12-16T21:33:23.130 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2023-12-16T21:33:20.654+0000 7fe7f2cbd700 1 -- 172.21.15.45:0/2045051497 shutdown_connections 2023-12-16T21:33:23.131 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2023-12-16T21:33:20.654+0000 7fe7f2cbd700 1 --2- 172.21.15.45:0/2045051497 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fe7ec1039c0 0x7fe7ec103dd0 unknown :-1 s=CLOSED pgs=45 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:33:23.131 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2023-12-16T21:33:20.654+0000 7fe7f2cbd700 1 -- 172.21.15.45:0/2045051497 >> 172.21.15.45:0/2045051497 conn(0x7fe7ec0fda40 msgr2=0x7fe7ec0ffe70 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-16T21:33:23.131 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2023-12-16T21:33:20.654+0000 7fe7f2cbd700 1 -- 172.21.15.45:0/2045051497 shutdown_connections 2023-12-16T21:33:23.131 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2023-12-16T21:33:20.654+0000 7fe7f2cbd700 1 -- 172.21.15.45:0/2045051497 wait complete. 2023-12-16T21:33:23.131 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2023-12-16T21:33:20.654+0000 7fe7f2cbd700 1 Processor -- start 2023-12-16T21:33:23.131 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2023-12-16T21:33:20.654+0000 7fe7f2cbd700 1 -- start start 2023-12-16T21:33:23.131 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2023-12-16T21:33:20.655+0000 7fe7f2cbd700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fe7ec1039c0 0x7fe7ec111120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:33:23.131 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2023-12-16T21:33:20.655+0000 7fe7f2cbd700 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe7ec111660 con 0x7fe7ec1039c0 2023-12-16T21:33:23.131 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2023-12-16T21:33:20.655+0000 7fe7f0a59700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fe7ec1039c0 0x7fe7ec111120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:33:23.131 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2023-12-16T21:33:20.655+0000 7fe7f0a59700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fe7ec1039c0 0x7fe7ec111120 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:59174/0 (socket says 172.21.15.45:59174) 2023-12-16T21:33:23.131 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2023-12-16T21:33:20.655+0000 7fe7f0a59700 1 -- 172.21.15.45:0/62930004 learned_addr learned my addr 172.21.15.45:0/62930004 (peer_addr_for_me v2:172.21.15.45:0/0) 2023-12-16T21:33:23.131 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2023-12-16T21:33:20.655+0000 7fe7f0a59700 1 -- 172.21.15.45:0/62930004 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe7e4009ae0 con 0x7fe7ec1039c0 2023-12-16T21:33:23.131 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2023-12-16T21:33:20.655+0000 7fe7f0a59700 1 --2- 172.21.15.45:0/62930004 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fe7ec1039c0 0x7fe7ec111120 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7fe7e4009540 tx=0x7fe7e4009840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:33:23.131 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2023-12-16T21:33:20.656+0000 7fe7e97fa700 1 -- 172.21.15.45:0/62930004 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fe7e401c620 con 0x7fe7ec1039c0 2023-12-16T21:33:23.131 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2023-12-16T21:33:20.656+0000 7fe7e97fa700 1 -- 172.21.15.45:0/62930004 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(26 keys) v1 ==== 1009+0+0 (secure 0 0 0) 0x7fe7e401cbf0 con 0x7fe7ec1039c0 2023-12-16T21:33:23.132 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2023-12-16T21:33:20.656+0000 7fe7f2cbd700 1 -- 172.21.15.45:0/62930004 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe7ec111860 con 0x7fe7ec1039c0 2023-12-16T21:33:23.132 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2023-12-16T21:33:20.656+0000 7fe7e97fa700 1 -- 172.21.15.45:0/62930004 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fe7e4007b50 con 0x7fe7ec1039c0 2023-12-16T21:33:23.132 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2023-12-16T21:33:20.656+0000 7fe7f2cbd700 1 -- 172.21.15.45:0/62930004 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe7ec112500 con 0x7fe7ec1039c0 2023-12-16T21:33:23.132 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2023-12-16T21:33:20.657+0000 7fe7e97fa700 1 -- 172.21.15.45:0/62930004 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 8) v1 ==== 43281+0+0 (secure 0 0 0) 0x7fe7e4019070 con 0x7fe7ec1039c0 2023-12-16T21:33:23.132 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2023-12-16T21:33:20.657+0000 7fe7f2cbd700 1 -- 172.21.15.45:0/62930004 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe7ec111cd0 con 0x7fe7ec1039c0 2023-12-16T21:33:23.132 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2023-12-16T21:33:20.657+0000 7fe7e97fa700 1 --2- 172.21.15.45:0/62930004 >> [v2:172.21.15.45:6800/2502560408,v1:172.21.15.45:6801/2502560408] conn(0x7fe7d40365f0 0x7fe7d4038aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:33:23.132 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2023-12-16T21:33:20.657+0000 7fe7e97fa700 1 -- 172.21.15.45:0/62930004 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 936+0+0 (secure 0 0 0) 0x7fe7e404a000 con 0x7fe7ec1039c0 2023-12-16T21:33:23.132 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2023-12-16T21:33:20.657+0000 7fe7ebfff700 1 --2- 172.21.15.45:0/62930004 >> [v2:172.21.15.45:6800/2502560408,v1:172.21.15.45:6801/2502560408] conn(0x7fe7d40365f0 0x7fe7d4038aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:33:23.132 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2023-12-16T21:33:20.657+0000 7fe7ebfff700 1 --2- 172.21.15.45:0/62930004 >> [v2:172.21.15.45:6800/2502560408,v1:172.21.15.45:6801/2502560408] conn(0x7fe7d40365f0 0x7fe7d4038aa0 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7fe7e0009990 tx=0x7fe7e0006e40 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:33:23.132 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2023-12-16T21:33:20.662+0000 7fe7e97fa700 1 -- 172.21.15.45:0/62930004 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+174570 (secure 0 0 0) 0x7fe7e4019910 con 0x7fe7ec1039c0 2023-12-16T21:33:23.132 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2023-12-16T21:33:20.858+0000 7fe7f2cbd700 1 -- 172.21.15.45:0/62930004 --> [v2:172.21.15.45:6800/2502560408,v1:172.21.15.45:6801/2502560408] -- mgr_command(tid 0: {"prefix": "orch host add", "hostname": "smithi045", "addr": "172.21.15.45", "target": ["mon-mgr", ""]}) v1 -- 0x7fe7ec112fb0 con 0x7fe7d40365f0 2023-12-16T21:33:23.132 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2023-12-16T21:33:22.889+0000 7fe7e97fa700 1 -- 172.21.15.45:0/62930004 <== mgr.14120 v2:172.21.15.45:6800/2502560408 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+48 (secure 0 0 0) 0x7fe7ec112fb0 con 0x7fe7d40365f0 2023-12-16T21:33:23.132 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2023-12-16T21:33:22.891+0000 7fe7f2cbd700 1 -- 172.21.15.45:0/62930004 >> [v2:172.21.15.45:6800/2502560408,v1:172.21.15.45:6801/2502560408] conn(0x7fe7d40365f0 msgr2=0x7fe7d4038aa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:33:23.132 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2023-12-16T21:33:22.891+0000 7fe7f2cbd700 1 --2- 172.21.15.45:0/62930004 >> [v2:172.21.15.45:6800/2502560408,v1:172.21.15.45:6801/2502560408] conn(0x7fe7d40365f0 0x7fe7d4038aa0 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7fe7e0009990 tx=0x7fe7e0006e40 comp rx=0 tx=0).stop 2023-12-16T21:33:23.133 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2023-12-16T21:33:22.891+0000 7fe7f2cbd700 1 -- 172.21.15.45:0/62930004 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fe7ec1039c0 msgr2=0x7fe7ec111120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:33:23.133 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2023-12-16T21:33:22.891+0000 7fe7f2cbd700 1 --2- 172.21.15.45:0/62930004 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fe7ec1039c0 0x7fe7ec111120 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7fe7e4009540 tx=0x7fe7e4009840 comp rx=0 tx=0).stop 2023-12-16T21:33:23.133 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2023-12-16T21:33:22.891+0000 7fe7f2cbd700 1 -- 172.21.15.45:0/62930004 shutdown_connections 2023-12-16T21:33:23.133 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2023-12-16T21:33:22.891+0000 7fe7f2cbd700 1 --2- 172.21.15.45:0/62930004 >> [v2:172.21.15.45:6800/2502560408,v1:172.21.15.45:6801/2502560408] conn(0x7fe7d40365f0 0x7fe7d4038aa0 unknown :-1 s=CLOSED pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:33:23.133 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2023-12-16T21:33:22.891+0000 7fe7f2cbd700 1 --2- 172.21.15.45:0/62930004 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fe7ec1039c0 0x7fe7ec111120 unknown :-1 s=CLOSED pgs=46 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:33:23.133 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2023-12-16T21:33:22.891+0000 7fe7f2cbd700 1 -- 172.21.15.45:0/62930004 >> 172.21.15.45:0/62930004 conn(0x7fe7ec0fda40 msgr2=0x7fe7ec0ffe60 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-16T21:33:23.133 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2023-12-16T21:33:22.891+0000 7fe7f2cbd700 1 -- 172.21.15.45:0/62930004 shutdown_connections 2023-12-16T21:33:23.133 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2023-12-16T21:33:22.891+0000 7fe7f2cbd700 1 -- 172.21.15.45:0/62930004 wait complete. 2023-12-16T21:33:23.133 INFO:teuthology.orchestra.run.smithi045.stdout:Deploying mon service with default placement... 2023-12-16T21:33:23.566 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Dec 16 21:33:23 smithi045 ceph-mon[80763]: Deploying cephadm binary to smithi045 2023-12-16T21:33:23.566 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Dec 16 21:33:23 smithi045 ceph-mon[80763]: from='mgr.14120 172.21.15.45:0/902863156' entity='mgr.smithi045.nhtmwc' 2023-12-16T21:33:23.566 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Dec 16 21:33:23 smithi045 ceph-mon[80763]: from='mgr.14120 172.21.15.45:0/902863156' entity='mgr.smithi045.nhtmwc' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-12-16T21:33:24.086 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stdout Scheduled mon update... 2023-12-16T21:33:24.086 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2023-12-16T21:33:23.400+0000 7effdc0c6700 1 Processor -- start 2023-12-16T21:33:24.086 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2023-12-16T21:33:23.400+0000 7effdc0c6700 1 -- start start 2023-12-16T21:33:24.086 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2023-12-16T21:33:23.400+0000 7effdc0c6700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7effd41011d0 0x7effd41015e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:33:24.087 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2023-12-16T21:33:23.400+0000 7effdc0c6700 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7effd4101b20 con 0x7effd41011d0 2023-12-16T21:33:24.087 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2023-12-16T21:33:23.400+0000 7effd9e62700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7effd41011d0 0x7effd41015e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:33:24.087 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2023-12-16T21:33:23.400+0000 7effd9e62700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7effd41011d0 0x7effd41015e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:32916/0 (socket says 172.21.15.45:32916) 2023-12-16T21:33:24.087 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2023-12-16T21:33:23.400+0000 7effd9e62700 1 -- 172.21.15.45:0/852030335 learned_addr learned my addr 172.21.15.45:0/852030335 (peer_addr_for_me v2:172.21.15.45:0/0) 2023-12-16T21:33:24.087 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2023-12-16T21:33:23.401+0000 7effd9e62700 1 -- 172.21.15.45:0/852030335 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7effd4101c60 con 0x7effd41011d0 2023-12-16T21:33:24.087 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2023-12-16T21:33:23.401+0000 7effd9e62700 1 --2- 172.21.15.45:0/852030335 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7effd41011d0 0x7effd41015e0 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7effc80098e0 tx=0x7effc8009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e8a4367a475bee08 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:33:24.087 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2023-12-16T21:33:23.401+0000 7effd8e60700 1 -- 172.21.15.45:0/852030335 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7effc800b860 con 0x7effd41011d0 2023-12-16T21:33:24.087 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2023-12-16T21:33:23.401+0000 7effd8e60700 1 -- 172.21.15.45:0/852030335 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(26 keys) v1 ==== 1009+0+0 (secure 0 0 0) 0x7effc800be30 con 0x7effd41011d0 2023-12-16T21:33:24.087 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2023-12-16T21:33:23.401+0000 7effd8e60700 1 -- 172.21.15.45:0/852030335 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7effc8010380 con 0x7effd41011d0 2023-12-16T21:33:24.087 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2023-12-16T21:33:23.401+0000 7effdc0c6700 1 -- 172.21.15.45:0/852030335 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7effd41011d0 msgr2=0x7effd41015e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:33:24.087 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2023-12-16T21:33:23.401+0000 7effdc0c6700 1 --2- 172.21.15.45:0/852030335 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7effd41011d0 0x7effd41015e0 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7effc80098e0 tx=0x7effc8009250 comp rx=0 tx=0).stop 2023-12-16T21:33:24.087 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2023-12-16T21:33:23.401+0000 7effdc0c6700 1 -- 172.21.15.45:0/852030335 shutdown_connections 2023-12-16T21:33:24.088 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2023-12-16T21:33:23.401+0000 7effdc0c6700 1 --2- 172.21.15.45:0/852030335 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7effd41011d0 0x7effd41015e0 unknown :-1 s=CLOSED pgs=47 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:33:24.088 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2023-12-16T21:33:23.402+0000 7effdc0c6700 1 -- 172.21.15.45:0/852030335 >> 172.21.15.45:0/852030335 conn(0x7effd40fc970 msgr2=0x7effd40fed80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-16T21:33:24.088 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2023-12-16T21:33:23.402+0000 7effdc0c6700 1 -- 172.21.15.45:0/852030335 shutdown_connections 2023-12-16T21:33:24.088 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2023-12-16T21:33:23.402+0000 7effdc0c6700 1 -- 172.21.15.45:0/852030335 wait complete. 2023-12-16T21:33:24.088 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2023-12-16T21:33:23.402+0000 7effdc0c6700 1 Processor -- start 2023-12-16T21:33:24.088 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2023-12-16T21:33:23.402+0000 7effdc0c6700 1 -- start start 2023-12-16T21:33:24.088 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2023-12-16T21:33:23.402+0000 7effdc0c6700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7effd41011d0 0x7effd4113340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:33:24.088 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2023-12-16T21:33:23.402+0000 7effdc0c6700 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7effd4113880 con 0x7effd41011d0 2023-12-16T21:33:24.088 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2023-12-16T21:33:23.402+0000 7effd9e62700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7effd41011d0 0x7effd4113340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:33:24.088 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2023-12-16T21:33:23.403+0000 7effd9e62700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7effd41011d0 0x7effd4113340 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:32930/0 (socket says 172.21.15.45:32930) 2023-12-16T21:33:24.088 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2023-12-16T21:33:23.403+0000 7effd9e62700 1 -- 172.21.15.45:0/2237703247 learned_addr learned my addr 172.21.15.45:0/2237703247 (peer_addr_for_me v2:172.21.15.45:0/0) 2023-12-16T21:33:24.088 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2023-12-16T21:33:23.403+0000 7effd9e62700 1 -- 172.21.15.45:0/2237703247 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7effc80094f0 con 0x7effd41011d0 2023-12-16T21:33:24.089 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2023-12-16T21:33:23.403+0000 7effd9e62700 1 --2- 172.21.15.45:0/2237703247 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7effd41011d0 0x7effd4113340 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7effc8006b00 tx=0x7effc8009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:33:24.089 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2023-12-16T21:33:23.403+0000 7effc67fc700 1 -- 172.21.15.45:0/2237703247 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7effc80109b0 con 0x7effd41011d0 2023-12-16T21:33:24.089 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2023-12-16T21:33:23.403+0000 7effc67fc700 1 -- 172.21.15.45:0/2237703247 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(26 keys) v1 ==== 1009+0+0 (secure 0 0 0) 0x7effc8024440 con 0x7effd41011d0 2023-12-16T21:33:24.089 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2023-12-16T21:33:23.403+0000 7effdc0c6700 1 -- 172.21.15.45:0/2237703247 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7effd4113a80 con 0x7effd41011d0 2023-12-16T21:33:24.089 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2023-12-16T21:33:23.403+0000 7effc67fc700 1 -- 172.21.15.45:0/2237703247 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7effc801a5f0 con 0x7effd41011d0 2023-12-16T21:33:24.089 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2023-12-16T21:33:23.403+0000 7effdc0c6700 1 -- 172.21.15.45:0/2237703247 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7effd4113e90 con 0x7effd41011d0 2023-12-16T21:33:24.089 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2023-12-16T21:33:23.404+0000 7effc67fc700 1 -- 172.21.15.45:0/2237703247 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 8) v1 ==== 43281+0+0 (secure 0 0 0) 0x7effc8024990 con 0x7effd41011d0 2023-12-16T21:33:24.089 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2023-12-16T21:33:23.404+0000 7effc67fc700 1 --2- 172.21.15.45:0/2237703247 >> [v2:172.21.15.45:6800/2502560408,v1:172.21.15.45:6801/2502560408] conn(0x7effc0036630 0x7effc0038ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:33:24.089 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2023-12-16T21:33:23.404+0000 7effc67fc700 1 -- 172.21.15.45:0/2237703247 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 936+0+0 (secure 0 0 0) 0x7effc804a630 con 0x7effd41011d0 2023-12-16T21:33:24.090 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2023-12-16T21:33:23.404+0000 7effdc0c6700 1 -- 172.21.15.45:0/2237703247 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7effd404bb80 con 0x7effd41011d0 2023-12-16T21:33:24.090 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2023-12-16T21:33:23.404+0000 7effd9661700 1 --2- 172.21.15.45:0/2237703247 >> [v2:172.21.15.45:6800/2502560408,v1:172.21.15.45:6801/2502560408] conn(0x7effc0036630 0x7effc0038ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:33:24.090 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2023-12-16T21:33:23.405+0000 7effd9661700 1 --2- 172.21.15.45:0/2237703247 >> [v2:172.21.15.45:6800/2502560408,v1:172.21.15.45:6801/2502560408] conn(0x7effc0036630 0x7effc0038ae0 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7effd0009990 tx=0x7effd0006e40 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:33:24.090 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2023-12-16T21:33:23.413+0000 7effc67fc700 1 -- 172.21.15.45:0/2237703247 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+174570 (secure 0 0 0) 0x7effc80245b0 con 0x7effd41011d0 2023-12-16T21:33:24.090 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2023-12-16T21:33:23.630+0000 7effdc0c6700 1 -- 172.21.15.45:0/2237703247 --> [v2:172.21.15.45:6800/2502560408,v1:172.21.15.45:6801/2502560408] -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "mon", "target": ["mon-mgr", ""]}) v1 -- 0x7effd4108b20 con 0x7effc0036630 2023-12-16T21:33:24.090 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2023-12-16T21:33:23.634+0000 7effc67fc700 1 -- 172.21.15.45:0/2237703247 <== mgr.14120 v2:172.21.15.45:6800/2502560408 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+24 (secure 0 0 0) 0x7effd4108b20 con 0x7effc0036630 2023-12-16T21:33:24.090 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2023-12-16T21:33:23.636+0000 7effdc0c6700 1 -- 172.21.15.45:0/2237703247 >> [v2:172.21.15.45:6800/2502560408,v1:172.21.15.45:6801/2502560408] conn(0x7effc0036630 msgr2=0x7effc0038ae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:33:24.090 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2023-12-16T21:33:23.636+0000 7effdc0c6700 1 --2- 172.21.15.45:0/2237703247 >> [v2:172.21.15.45:6800/2502560408,v1:172.21.15.45:6801/2502560408] conn(0x7effc0036630 0x7effc0038ae0 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7effd0009990 tx=0x7effd0006e40 comp rx=0 tx=0).stop 2023-12-16T21:33:24.090 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2023-12-16T21:33:23.636+0000 7effdc0c6700 1 -- 172.21.15.45:0/2237703247 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7effd41011d0 msgr2=0x7effd4113340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:33:24.090 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2023-12-16T21:33:23.636+0000 7effdc0c6700 1 --2- 172.21.15.45:0/2237703247 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7effd41011d0 0x7effd4113340 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7effc8006b00 tx=0x7effc8009250 comp rx=0 tx=0).stop 2023-12-16T21:33:24.090 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2023-12-16T21:33:23.636+0000 7effdc0c6700 1 -- 172.21.15.45:0/2237703247 shutdown_connections 2023-12-16T21:33:24.091 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2023-12-16T21:33:23.636+0000 7effdc0c6700 1 --2- 172.21.15.45:0/2237703247 >> [v2:172.21.15.45:6800/2502560408,v1:172.21.15.45:6801/2502560408] conn(0x7effc0036630 0x7effc0038ae0 unknown :-1 s=CLOSED pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:33:24.091 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2023-12-16T21:33:23.636+0000 7effdc0c6700 1 --2- 172.21.15.45:0/2237703247 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7effd41011d0 0x7effd4113340 unknown :-1 s=CLOSED pgs=48 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:33:24.091 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2023-12-16T21:33:23.636+0000 7effdc0c6700 1 -- 172.21.15.45:0/2237703247 >> 172.21.15.45:0/2237703247 conn(0x7effd40fc970 msgr2=0x7effd4105f40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-16T21:33:24.091 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2023-12-16T21:33:23.636+0000 7effdc0c6700 1 -- 172.21.15.45:0/2237703247 shutdown_connections 2023-12-16T21:33:24.091 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2023-12-16T21:33:23.636+0000 7effdc0c6700 1 -- 172.21.15.45:0/2237703247 wait complete. 2023-12-16T21:33:24.091 INFO:teuthology.orchestra.run.smithi045.stdout:Deploying mgr service with default placement... 2023-12-16T21:33:24.511 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Dec 16 21:33:24 smithi045 ceph-mon[80763]: Added host smithi045 2023-12-16T21:33:24.511 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Dec 16 21:33:24 smithi045 ceph-mon[80763]: from='mgr.14120 172.21.15.45:0/902863156' entity='mgr.smithi045.nhtmwc' 2023-12-16T21:33:24.910 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stdout Scheduled mgr update... 2023-12-16T21:33:24.910 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2023-12-16T21:33:24.343+0000 7f950c06b700 1 Processor -- start 2023-12-16T21:33:24.910 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2023-12-16T21:33:24.343+0000 7f950c06b700 1 -- start start 2023-12-16T21:33:24.910 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2023-12-16T21:33:24.343+0000 7f950c06b700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f95041039d0 0x7f9504103de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:33:24.910 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2023-12-16T21:33:24.343+0000 7f950c06b700 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9504104320 con 0x7f95041039d0 2023-12-16T21:33:24.910 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2023-12-16T21:33:24.344+0000 7f9509e07700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f95041039d0 0x7f9504103de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:33:24.910 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2023-12-16T21:33:24.344+0000 7f9509e07700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f95041039d0 0x7f9504103de0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:32942/0 (socket says 172.21.15.45:32942) 2023-12-16T21:33:24.910 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2023-12-16T21:33:24.344+0000 7f9509e07700 1 -- 172.21.15.45:0/287939818 learned_addr learned my addr 172.21.15.45:0/287939818 (peer_addr_for_me v2:172.21.15.45:0/0) 2023-12-16T21:33:24.910 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2023-12-16T21:33:24.344+0000 7f9509e07700 1 -- 172.21.15.45:0/287939818 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9504104460 con 0x7f95041039d0 2023-12-16T21:33:24.911 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2023-12-16T21:33:24.344+0000 7f9509e07700 1 --2- 172.21.15.45:0/287939818 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f95041039d0 0x7f9504103de0 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7f94f80098e0 tx=0x7f94f8009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e24f4ab67e2ab687 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:33:24.911 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2023-12-16T21:33:24.344+0000 7f9508e05700 1 -- 172.21.15.45:0/287939818 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f94f800b860 con 0x7f95041039d0 2023-12-16T21:33:24.911 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2023-12-16T21:33:24.344+0000 7f9508e05700 1 -- 172.21.15.45:0/287939818 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(26 keys) v1 ==== 1009+0+0 (secure 0 0 0) 0x7f94f800be30 con 0x7f95041039d0 2023-12-16T21:33:24.911 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2023-12-16T21:33:24.344+0000 7f9508e05700 1 -- 172.21.15.45:0/287939818 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f94f8010380 con 0x7f95041039d0 2023-12-16T21:33:24.911 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2023-12-16T21:33:24.345+0000 7f950c06b700 1 -- 172.21.15.45:0/287939818 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f95041039d0 msgr2=0x7f9504103de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:33:24.911 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2023-12-16T21:33:24.345+0000 7f950c06b700 1 --2- 172.21.15.45:0/287939818 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f95041039d0 0x7f9504103de0 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7f94f80098e0 tx=0x7f94f8009250 comp rx=0 tx=0).stop 2023-12-16T21:33:24.911 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2023-12-16T21:33:24.345+0000 7f950c06b700 1 -- 172.21.15.45:0/287939818 shutdown_connections 2023-12-16T21:33:24.911 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2023-12-16T21:33:24.345+0000 7f950c06b700 1 --2- 172.21.15.45:0/287939818 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f95041039d0 0x7f9504103de0 unknown :-1 s=CLOSED pgs=49 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:33:24.911 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2023-12-16T21:33:24.345+0000 7f950c06b700 1 -- 172.21.15.45:0/287939818 >> 172.21.15.45:0/287939818 conn(0x7f95040fc0b0 msgr2=0x7f95040fe4e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-16T21:33:24.911 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2023-12-16T21:33:24.345+0000 7f950c06b700 1 -- 172.21.15.45:0/287939818 shutdown_connections 2023-12-16T21:33:24.911 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2023-12-16T21:33:24.345+0000 7f950c06b700 1 -- 172.21.15.45:0/287939818 wait complete. 2023-12-16T21:33:24.911 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2023-12-16T21:33:24.346+0000 7f950c06b700 1 Processor -- start 2023-12-16T21:33:24.911 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2023-12-16T21:33:24.346+0000 7f950c06b700 1 -- start start 2023-12-16T21:33:24.912 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2023-12-16T21:33:24.346+0000 7f950c06b700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f95041039d0 0x7f9504111150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:33:24.912 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2023-12-16T21:33:24.346+0000 7f950c06b700 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9504111690 con 0x7f95041039d0 2023-12-16T21:33:24.912 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2023-12-16T21:33:24.346+0000 7f9509e07700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f95041039d0 0x7f9504111150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:33:24.912 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2023-12-16T21:33:24.346+0000 7f9509e07700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f95041039d0 0x7f9504111150 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:32954/0 (socket says 172.21.15.45:32954) 2023-12-16T21:33:24.912 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2023-12-16T21:33:24.346+0000 7f9509e07700 1 -- 172.21.15.45:0/3095508740 learned_addr learned my addr 172.21.15.45:0/3095508740 (peer_addr_for_me v2:172.21.15.45:0/0) 2023-12-16T21:33:24.912 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2023-12-16T21:33:24.347+0000 7f9509e07700 1 -- 172.21.15.45:0/3095508740 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f94f80094f0 con 0x7f95041039d0 2023-12-16T21:33:24.912 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2023-12-16T21:33:24.347+0000 7f9509e07700 1 --2- 172.21.15.45:0/3095508740 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f95041039d0 0x7f9504111150 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7f94f8006b00 tx=0x7f94f8009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:33:24.912 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2023-12-16T21:33:24.347+0000 7f94f67fc700 1 -- 172.21.15.45:0/3095508740 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f94f80109b0 con 0x7f95041039d0 2023-12-16T21:33:24.912 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2023-12-16T21:33:24.347+0000 7f950c06b700 1 -- 172.21.15.45:0/3095508740 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9504111890 con 0x7f95041039d0 2023-12-16T21:33:24.912 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2023-12-16T21:33:24.347+0000 7f950c06b700 1 -- 172.21.15.45:0/3095508740 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9504111ca0 con 0x7f95041039d0 2023-12-16T21:33:24.912 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2023-12-16T21:33:24.347+0000 7f94f67fc700 1 -- 172.21.15.45:0/3095508740 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(26 keys) v1 ==== 1009+0+0 (secure 0 0 0) 0x7f94f8024440 con 0x7f95041039d0 2023-12-16T21:33:24.912 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2023-12-16T21:33:24.347+0000 7f94f67fc700 1 -- 172.21.15.45:0/3095508740 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f94f801a5f0 con 0x7f95041039d0 2023-12-16T21:33:24.912 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2023-12-16T21:33:24.348+0000 7f94f67fc700 1 -- 172.21.15.45:0/3095508740 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 8) v1 ==== 43281+0+0 (secure 0 0 0) 0x7f94f801a860 con 0x7f95041039d0 2023-12-16T21:33:24.912 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2023-12-16T21:33:24.348+0000 7f950c06b700 1 -- 172.21.15.45:0/3095508740 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f94e80052e0 con 0x7f95041039d0 2023-12-16T21:33:24.913 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2023-12-16T21:33:24.348+0000 7f94f67fc700 1 --2- 172.21.15.45:0/3095508740 >> [v2:172.21.15.45:6800/2502560408,v1:172.21.15.45:6801/2502560408] conn(0x7f94f0036640 0x7f94f0038af0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:33:24.913 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2023-12-16T21:33:24.348+0000 7f94f67fc700 1 -- 172.21.15.45:0/3095508740 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 936+0+0 (secure 0 0 0) 0x7f94f804a520 con 0x7f95041039d0 2023-12-16T21:33:24.913 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2023-12-16T21:33:24.348+0000 7f9509606700 1 --2- 172.21.15.45:0/3095508740 >> [v2:172.21.15.45:6800/2502560408,v1:172.21.15.45:6801/2502560408] conn(0x7f94f0036640 0x7f94f0038af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:33:24.913 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2023-12-16T21:33:24.349+0000 7f9509606700 1 --2- 172.21.15.45:0/3095508740 >> [v2:172.21.15.45:6800/2502560408,v1:172.21.15.45:6801/2502560408] conn(0x7f94f0036640 0x7f94f0038af0 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7f9500009990 tx=0x7f9500006e40 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:33:24.913 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2023-12-16T21:33:24.353+0000 7f94f67fc700 1 -- 172.21.15.45:0/3095508740 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+174570 (secure 0 0 0) 0x7f94f8022340 con 0x7f95041039d0 2023-12-16T21:33:24.913 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2023-12-16T21:33:24.566+0000 7f950c06b700 1 -- 172.21.15.45:0/3095508740 --> [v2:172.21.15.45:6800/2502560408,v1:172.21.15.45:6801/2502560408] -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "mgr", "target": ["mon-mgr", ""]}) v1 -- 0x7f94e8000bf0 con 0x7f94f0036640 2023-12-16T21:33:24.913 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2023-12-16T21:33:24.570+0000 7f94f67fc700 1 -- 172.21.15.45:0/3095508740 <== mgr.14120 v2:172.21.15.45:6800/2502560408 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+24 (secure 0 0 0) 0x7f94e8000bf0 con 0x7f94f0036640 2023-12-16T21:33:24.913 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2023-12-16T21:33:24.572+0000 7f950c06b700 1 -- 172.21.15.45:0/3095508740 >> [v2:172.21.15.45:6800/2502560408,v1:172.21.15.45:6801/2502560408] conn(0x7f94f0036640 msgr2=0x7f94f0038af0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:33:24.913 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2023-12-16T21:33:24.572+0000 7f950c06b700 1 --2- 172.21.15.45:0/3095508740 >> [v2:172.21.15.45:6800/2502560408,v1:172.21.15.45:6801/2502560408] conn(0x7f94f0036640 0x7f94f0038af0 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7f9500009990 tx=0x7f9500006e40 comp rx=0 tx=0).stop 2023-12-16T21:33:24.913 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2023-12-16T21:33:24.572+0000 7f950c06b700 1 -- 172.21.15.45:0/3095508740 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f95041039d0 msgr2=0x7f9504111150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:33:24.913 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2023-12-16T21:33:24.572+0000 7f950c06b700 1 --2- 172.21.15.45:0/3095508740 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f95041039d0 0x7f9504111150 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7f94f8006b00 tx=0x7f94f8009250 comp rx=0 tx=0).stop 2023-12-16T21:33:24.913 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2023-12-16T21:33:24.573+0000 7f950c06b700 1 -- 172.21.15.45:0/3095508740 shutdown_connections 2023-12-16T21:33:24.913 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2023-12-16T21:33:24.573+0000 7f950c06b700 1 --2- 172.21.15.45:0/3095508740 >> [v2:172.21.15.45:6800/2502560408,v1:172.21.15.45:6801/2502560408] conn(0x7f94f0036640 0x7f94f0038af0 unknown :-1 s=CLOSED pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:33:24.913 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2023-12-16T21:33:24.573+0000 7f950c06b700 1 --2- 172.21.15.45:0/3095508740 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f95041039d0 0x7f9504111150 unknown :-1 s=CLOSED pgs=50 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:33:24.914 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2023-12-16T21:33:24.573+0000 7f950c06b700 1 -- 172.21.15.45:0/3095508740 >> 172.21.15.45:0/3095508740 conn(0x7f95040fc0b0 msgr2=0x7f95040fd800 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-16T21:33:24.914 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2023-12-16T21:33:24.573+0000 7f950c06b700 1 -- 172.21.15.45:0/3095508740 shutdown_connections 2023-12-16T21:33:24.914 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2023-12-16T21:33:24.573+0000 7f950c06b700 1 -- 172.21.15.45:0/3095508740 wait complete. 2023-12-16T21:33:24.914 INFO:teuthology.orchestra.run.smithi045.stdout:Deploying crash service with default placement... 2023-12-16T21:33:25.513 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Dec 16 21:33:25 smithi045 ceph-mon[80763]: from='client.14142 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mon", "target": ["mon-mgr", ""]}]: dispatch 2023-12-16T21:33:25.514 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Dec 16 21:33:25 smithi045 ceph-mon[80763]: Saving service mon spec with placement count:5 2023-12-16T21:33:25.514 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Dec 16 21:33:25 smithi045 ceph-mon[80763]: from='mgr.14120 172.21.15.45:0/902863156' entity='mgr.smithi045.nhtmwc' 2023-12-16T21:33:25.514 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Dec 16 21:33:25 smithi045 ceph-mon[80763]: from='mgr.14120 172.21.15.45:0/902863156' entity='mgr.smithi045.nhtmwc' 2023-12-16T21:33:25.770 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stdout Scheduled crash update... 2023-12-16T21:33:25.770 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2023-12-16T21:33:25.294+0000 7f8f763c3700 1 Processor -- start 2023-12-16T21:33:25.770 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2023-12-16T21:33:25.294+0000 7f8f763c3700 1 -- start start 2023-12-16T21:33:25.770 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2023-12-16T21:33:25.294+0000 7f8f763c3700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f8f70076c60 0x7f8f70104e90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:33:25.771 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2023-12-16T21:33:25.294+0000 7f8f763c3700 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8f701053d0 con 0x7f8f70076c60 2023-12-16T21:33:25.771 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2023-12-16T21:33:25.294+0000 7f8f6ffff700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f8f70076c60 0x7f8f70104e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:33:25.771 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2023-12-16T21:33:25.294+0000 7f8f6ffff700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f8f70076c60 0x7f8f70104e90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:32966/0 (socket says 172.21.15.45:32966) 2023-12-16T21:33:25.771 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2023-12-16T21:33:25.294+0000 7f8f6ffff700 1 -- 172.21.15.45:0/547410855 learned_addr learned my addr 172.21.15.45:0/547410855 (peer_addr_for_me v2:172.21.15.45:0/0) 2023-12-16T21:33:25.771 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2023-12-16T21:33:25.295+0000 7f8f6ffff700 1 -- 172.21.15.45:0/547410855 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8f70105510 con 0x7f8f70076c60 2023-12-16T21:33:25.771 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2023-12-16T21:33:25.295+0000 7f8f6ffff700 1 --2- 172.21.15.45:0/547410855 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f8f70076c60 0x7f8f70104e90 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7f8f600098e0 tx=0x7f8f60009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b9b7a4662838c4d2 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:33:25.771 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2023-12-16T21:33:25.295+0000 7f8f6effd700 1 -- 172.21.15.45:0/547410855 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f8f6000b860 con 0x7f8f70076c60 2023-12-16T21:33:25.771 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2023-12-16T21:33:25.295+0000 7f8f6effd700 1 -- 172.21.15.45:0/547410855 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f8f6000be70 con 0x7f8f70076c60 2023-12-16T21:33:25.771 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2023-12-16T21:33:25.295+0000 7f8f6effd700 1 -- 172.21.15.45:0/547410855 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f8f60010380 con 0x7f8f70076c60 2023-12-16T21:33:25.771 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2023-12-16T21:33:25.295+0000 7f8f763c3700 1 -- 172.21.15.45:0/547410855 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f8f70076c60 msgr2=0x7f8f70104e90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:33:25.771 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2023-12-16T21:33:25.295+0000 7f8f763c3700 1 --2- 172.21.15.45:0/547410855 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f8f70076c60 0x7f8f70104e90 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7f8f600098e0 tx=0x7f8f60009250 comp rx=0 tx=0).stop 2023-12-16T21:33:25.771 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2023-12-16T21:33:25.296+0000 7f8f763c3700 1 -- 172.21.15.45:0/547410855 shutdown_connections 2023-12-16T21:33:25.772 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2023-12-16T21:33:25.296+0000 7f8f763c3700 1 --2- 172.21.15.45:0/547410855 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f8f70076c60 0x7f8f70104e90 unknown :-1 s=CLOSED pgs=51 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:33:25.772 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2023-12-16T21:33:25.296+0000 7f8f763c3700 1 -- 172.21.15.45:0/547410855 >> 172.21.15.45:0/547410855 conn(0x7f8f700fc980 msgr2=0x7f8f700fed90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-16T21:33:25.772 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2023-12-16T21:33:25.296+0000 7f8f763c3700 1 -- 172.21.15.45:0/547410855 shutdown_connections 2023-12-16T21:33:25.772 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2023-12-16T21:33:25.296+0000 7f8f763c3700 1 -- 172.21.15.45:0/547410855 wait complete. 2023-12-16T21:33:25.772 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2023-12-16T21:33:25.296+0000 7f8f763c3700 1 Processor -- start 2023-12-16T21:33:25.772 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2023-12-16T21:33:25.296+0000 7f8f763c3700 1 -- start start 2023-12-16T21:33:25.772 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2023-12-16T21:33:25.297+0000 7f8f763c3700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f8f70076c60 0x7f8f70111160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:33:25.772 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2023-12-16T21:33:25.297+0000 7f8f763c3700 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8f701116a0 con 0x7f8f70076c60 2023-12-16T21:33:25.772 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2023-12-16T21:33:25.297+0000 7f8f6ffff700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f8f70076c60 0x7f8f70111160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:33:25.772 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2023-12-16T21:33:25.297+0000 7f8f6ffff700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f8f70076c60 0x7f8f70111160 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:32980/0 (socket says 172.21.15.45:32980) 2023-12-16T21:33:25.772 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2023-12-16T21:33:25.297+0000 7f8f6ffff700 1 -- 172.21.15.45:0/4100148426 learned_addr learned my addr 172.21.15.45:0/4100148426 (peer_addr_for_me v2:172.21.15.45:0/0) 2023-12-16T21:33:25.772 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2023-12-16T21:33:25.297+0000 7f8f6ffff700 1 -- 172.21.15.45:0/4100148426 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8f600094f0 con 0x7f8f70076c60 2023-12-16T21:33:25.772 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2023-12-16T21:33:25.297+0000 7f8f6ffff700 1 --2- 172.21.15.45:0/4100148426 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f8f70076c60 0x7f8f70111160 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f8f60006d40 tx=0x7f8f60009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:33:25.772 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2023-12-16T21:33:25.297+0000 7f8f6cff9700 1 -- 172.21.15.45:0/4100148426 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f8f60010960 con 0x7f8f70076c60 2023-12-16T21:33:25.773 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2023-12-16T21:33:25.297+0000 7f8f763c3700 1 -- 172.21.15.45:0/4100148426 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8f701118a0 con 0x7f8f70076c60 2023-12-16T21:33:25.773 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2023-12-16T21:33:25.297+0000 7f8f6cff9700 1 -- 172.21.15.45:0/4100148426 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f8f6001a430 con 0x7f8f70076c60 2023-12-16T21:33:25.773 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2023-12-16T21:33:25.297+0000 7f8f6cff9700 1 -- 172.21.15.45:0/4100148426 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f8f600245b0 con 0x7f8f70076c60 2023-12-16T21:33:25.773 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2023-12-16T21:33:25.297+0000 7f8f763c3700 1 -- 172.21.15.45:0/4100148426 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8f70111cb0 con 0x7f8f70076c60 2023-12-16T21:33:25.773 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2023-12-16T21:33:25.298+0000 7f8f6cff9700 1 -- 172.21.15.45:0/4100148426 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 8) v1 ==== 43281+0+0 (secure 0 0 0) 0x7f8f6002d3f0 con 0x7f8f70076c60 2023-12-16T21:33:25.773 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2023-12-16T21:33:25.298+0000 7f8f6cff9700 1 --2- 172.21.15.45:0/4100148426 >> [v2:172.21.15.45:6800/2502560408,v1:172.21.15.45:6801/2502560408] conn(0x7f8f5c0365f0 0x7f8f5c038aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:33:25.773 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2023-12-16T21:33:25.298+0000 7f8f6cff9700 1 -- 172.21.15.45:0/4100148426 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 936+0+0 (secure 0 0 0) 0x7f8f6004b0e0 con 0x7f8f70076c60 2023-12-16T21:33:25.773 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2023-12-16T21:33:25.299+0000 7f8f6f7fe700 1 --2- 172.21.15.45:0/4100148426 >> [v2:172.21.15.45:6800/2502560408,v1:172.21.15.45:6801/2502560408] conn(0x7f8f5c0365f0 0x7f8f5c038aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:33:25.773 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2023-12-16T21:33:25.299+0000 7f8f763c3700 1 -- 172.21.15.45:0/4100148426 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8f7004bb80 con 0x7f8f70076c60 2023-12-16T21:33:25.773 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2023-12-16T21:33:25.299+0000 7f8f6f7fe700 1 --2- 172.21.15.45:0/4100148426 >> [v2:172.21.15.45:6800/2502560408,v1:172.21.15.45:6801/2502560408] conn(0x7f8f5c0365f0 0x7f8f5c038aa0 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7f8f58009990 tx=0x7f8f58006e40 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:33:25.773 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2023-12-16T21:33:25.303+0000 7f8f6cff9700 1 -- 172.21.15.45:0/4100148426 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+174570 (secure 0 0 0) 0x7f8f6002b8e0 con 0x7f8f70076c60 2023-12-16T21:33:25.773 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2023-12-16T21:33:25.511+0000 7f8f763c3700 1 -- 172.21.15.45:0/4100148426 --> [v2:172.21.15.45:6800/2502560408,v1:172.21.15.45:6801/2502560408] -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "crash", "target": ["mon-mgr", ""]}) v1 -- 0x7f8f7010ad00 con 0x7f8f5c0365f0 2023-12-16T21:33:25.773 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2023-12-16T21:33:25.516+0000 7f8f6cff9700 1 -- 172.21.15.45:0/4100148426 <== mgr.14120 v2:172.21.15.45:6800/2502560408 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+26 (secure 0 0 0) 0x7f8f7010ad00 con 0x7f8f5c0365f0 2023-12-16T21:33:25.774 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2023-12-16T21:33:25.518+0000 7f8f763c3700 1 -- 172.21.15.45:0/4100148426 >> [v2:172.21.15.45:6800/2502560408,v1:172.21.15.45:6801/2502560408] conn(0x7f8f5c0365f0 msgr2=0x7f8f5c038aa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:33:25.774 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2023-12-16T21:33:25.518+0000 7f8f763c3700 1 --2- 172.21.15.45:0/4100148426 >> [v2:172.21.15.45:6800/2502560408,v1:172.21.15.45:6801/2502560408] conn(0x7f8f5c0365f0 0x7f8f5c038aa0 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7f8f58009990 tx=0x7f8f58006e40 comp rx=0 tx=0).stop 2023-12-16T21:33:25.774 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2023-12-16T21:33:25.518+0000 7f8f763c3700 1 -- 172.21.15.45:0/4100148426 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f8f70076c60 msgr2=0x7f8f70111160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:33:25.774 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2023-12-16T21:33:25.518+0000 7f8f763c3700 1 --2- 172.21.15.45:0/4100148426 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f8f70076c60 0x7f8f70111160 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f8f60006d40 tx=0x7f8f60009250 comp rx=0 tx=0).stop 2023-12-16T21:33:25.774 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2023-12-16T21:33:25.518+0000 7f8f763c3700 1 -- 172.21.15.45:0/4100148426 shutdown_connections 2023-12-16T21:33:25.774 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2023-12-16T21:33:25.518+0000 7f8f763c3700 1 --2- 172.21.15.45:0/4100148426 >> [v2:172.21.15.45:6800/2502560408,v1:172.21.15.45:6801/2502560408] conn(0x7f8f5c0365f0 0x7f8f5c038aa0 unknown :-1 s=CLOSED pgs=14 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:33:25.774 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2023-12-16T21:33:25.518+0000 7f8f763c3700 1 --2- 172.21.15.45:0/4100148426 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f8f70076c60 0x7f8f70111160 unknown :-1 s=CLOSED pgs=52 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:33:25.774 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2023-12-16T21:33:25.518+0000 7f8f763c3700 1 -- 172.21.15.45:0/4100148426 >> 172.21.15.45:0/4100148426 conn(0x7f8f700fc980 msgr2=0x7f8f700fed70 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-16T21:33:25.774 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2023-12-16T21:33:25.519+0000 7f8f763c3700 1 -- 172.21.15.45:0/4100148426 shutdown_connections 2023-12-16T21:33:25.774 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2023-12-16T21:33:25.519+0000 7f8f763c3700 1 -- 172.21.15.45:0/4100148426 wait complete. 2023-12-16T21:33:25.774 INFO:teuthology.orchestra.run.smithi045.stdout:Deploying prometheus service with default placement... 2023-12-16T21:33:26.497 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Dec 16 21:33:26 smithi045 ceph-mon[80763]: from='client.14144 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mgr", "target": ["mon-mgr", ""]}]: dispatch 2023-12-16T21:33:26.497 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Dec 16 21:33:26 smithi045 ceph-mon[80763]: Saving service mgr spec with placement count:2 2023-12-16T21:33:26.497 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Dec 16 21:33:26 smithi045 ceph-mon[80763]: from='mgr.14120 172.21.15.45:0/902863156' entity='mgr.smithi045.nhtmwc' 2023-12-16T21:33:26.497 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Dec 16 21:33:26 smithi045 ceph-mon[80763]: from='mgr.14120 172.21.15.45:0/902863156' entity='mgr.smithi045.nhtmwc' 2023-12-16T21:33:26.619 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stdout Scheduled prometheus update... 2023-12-16T21:33:26.619 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2023-12-16T21:33:26.054+0000 7f6b6719c700 1 Processor -- start 2023-12-16T21:33:26.619 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2023-12-16T21:33:26.054+0000 7f6b6719c700 1 -- start start 2023-12-16T21:33:26.619 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2023-12-16T21:33:26.054+0000 7f6b6719c700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f6b601011d0 0x7f6b601015e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:33:26.619 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2023-12-16T21:33:26.054+0000 7f6b6719c700 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6b60101b20 con 0x7f6b601011d0 2023-12-16T21:33:26.619 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2023-12-16T21:33:26.054+0000 7f6b64f38700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f6b601011d0 0x7f6b601015e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:33:26.619 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2023-12-16T21:33:26.054+0000 7f6b64f38700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f6b601011d0 0x7f6b601015e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:32982/0 (socket says 172.21.15.45:32982) 2023-12-16T21:33:26.619 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2023-12-16T21:33:26.054+0000 7f6b64f38700 1 -- 172.21.15.45:0/2612753205 learned_addr learned my addr 172.21.15.45:0/2612753205 (peer_addr_for_me v2:172.21.15.45:0/0) 2023-12-16T21:33:26.620 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2023-12-16T21:33:26.054+0000 7f6b64f38700 1 -- 172.21.15.45:0/2612753205 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6b60101c60 con 0x7f6b601011d0 2023-12-16T21:33:26.620 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2023-12-16T21:33:26.054+0000 7f6b64f38700 1 --2- 172.21.15.45:0/2612753205 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f6b601011d0 0x7f6b601015e0 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f6b5000bb80 tx=0x7f6b5000b4f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=42a6e22d29fbb2f9 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:33:26.620 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2023-12-16T21:33:26.055+0000 7f6b5f7fe700 1 -- 172.21.15.45:0/2612753205 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f6b5000dd00 con 0x7f6b601011d0 2023-12-16T21:33:26.620 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2023-12-16T21:33:26.055+0000 7f6b5f7fe700 1 -- 172.21.15.45:0/2612753205 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f6b5000de60 con 0x7f6b601011d0 2023-12-16T21:33:26.620 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2023-12-16T21:33:26.055+0000 7f6b5f7fe700 1 -- 172.21.15.45:0/2612753205 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f6b500188d0 con 0x7f6b601011d0 2023-12-16T21:33:26.620 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2023-12-16T21:33:26.055+0000 7f6b6719c700 1 -- 172.21.15.45:0/2612753205 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f6b601011d0 msgr2=0x7f6b601015e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:33:26.620 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2023-12-16T21:33:26.055+0000 7f6b6719c700 1 --2- 172.21.15.45:0/2612753205 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f6b601011d0 0x7f6b601015e0 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f6b5000bb80 tx=0x7f6b5000b4f0 comp rx=0 tx=0).stop 2023-12-16T21:33:26.620 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2023-12-16T21:33:26.055+0000 7f6b6719c700 1 -- 172.21.15.45:0/2612753205 shutdown_connections 2023-12-16T21:33:26.620 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2023-12-16T21:33:26.055+0000 7f6b6719c700 1 --2- 172.21.15.45:0/2612753205 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f6b601011d0 0x7f6b601015e0 unknown :-1 s=CLOSED pgs=53 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:33:26.620 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2023-12-16T21:33:26.055+0000 7f6b6719c700 1 -- 172.21.15.45:0/2612753205 >> 172.21.15.45:0/2612753205 conn(0x7f6b600fc950 msgr2=0x7f6b600fed60 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-16T21:33:26.620 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2023-12-16T21:33:26.055+0000 7f6b6719c700 1 -- 172.21.15.45:0/2612753205 shutdown_connections 2023-12-16T21:33:26.620 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2023-12-16T21:33:26.055+0000 7f6b6719c700 1 -- 172.21.15.45:0/2612753205 wait complete. 2023-12-16T21:33:26.620 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2023-12-16T21:33:26.056+0000 7f6b6719c700 1 Processor -- start 2023-12-16T21:33:26.620 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2023-12-16T21:33:26.056+0000 7f6b6719c700 1 -- start start 2023-12-16T21:33:26.621 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2023-12-16T21:33:26.056+0000 7f6b6719c700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f6b601011d0 0x7f6b6010ef00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:33:26.621 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2023-12-16T21:33:26.056+0000 7f6b6719c700 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6b6010f440 con 0x7f6b601011d0 2023-12-16T21:33:26.621 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2023-12-16T21:33:26.056+0000 7f6b64f38700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f6b601011d0 0x7f6b6010ef00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:33:26.621 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2023-12-16T21:33:26.056+0000 7f6b64f38700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f6b601011d0 0x7f6b6010ef00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:32996/0 (socket says 172.21.15.45:32996) 2023-12-16T21:33:26.621 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2023-12-16T21:33:26.056+0000 7f6b64f38700 1 -- 172.21.15.45:0/4227261771 learned_addr learned my addr 172.21.15.45:0/4227261771 (peer_addr_for_me v2:172.21.15.45:0/0) 2023-12-16T21:33:26.621 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2023-12-16T21:33:26.056+0000 7f6b64f38700 1 -- 172.21.15.45:0/4227261771 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6b5000b790 con 0x7f6b601011d0 2023-12-16T21:33:26.621 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2023-12-16T21:33:26.057+0000 7f6b64f38700 1 --2- 172.21.15.45:0/4227261771 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f6b601011d0 0x7f6b6010ef00 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f6b500184f0 tx=0x7f6b5000b4f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:33:26.621 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2023-12-16T21:33:26.057+0000 7f6b5d7fa700 1 -- 172.21.15.45:0/4227261771 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f6b50010bd0 con 0x7f6b601011d0 2023-12-16T21:33:26.621 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2023-12-16T21:33:26.057+0000 7f6b5d7fa700 1 -- 172.21.15.45:0/4227261771 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f6b5001d050 con 0x7f6b601011d0 2023-12-16T21:33:26.621 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2023-12-16T21:33:26.057+0000 7f6b6719c700 1 -- 172.21.15.45:0/4227261771 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6b6010f640 con 0x7f6b601011d0 2023-12-16T21:33:26.621 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2023-12-16T21:33:26.057+0000 7f6b5d7fa700 1 -- 172.21.15.45:0/4227261771 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f6b50022590 con 0x7f6b601011d0 2023-12-16T21:33:26.621 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2023-12-16T21:33:26.057+0000 7f6b6719c700 1 -- 172.21.15.45:0/4227261771 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6b6010fa50 con 0x7f6b601011d0 2023-12-16T21:33:26.621 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2023-12-16T21:33:26.058+0000 7f6b5d7fa700 1 -- 172.21.15.45:0/4227261771 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 8) v1 ==== 43281+0+0 (secure 0 0 0) 0x7f6b50022850 con 0x7f6b601011d0 2023-12-16T21:33:26.621 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2023-12-16T21:33:26.058+0000 7f6b6719c700 1 -- 172.21.15.45:0/4227261771 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6b6004bb80 con 0x7f6b601011d0 2023-12-16T21:33:26.621 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2023-12-16T21:33:26.058+0000 7f6b5d7fa700 1 --2- 172.21.15.45:0/4227261771 >> [v2:172.21.15.45:6800/2502560408,v1:172.21.15.45:6801/2502560408] conn(0x7f6b4c0365f0 0x7f6b4c038aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:33:26.622 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2023-12-16T21:33:26.058+0000 7f6b5d7fa700 1 -- 172.21.15.45:0/4227261771 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 936+0+0 (secure 0 0 0) 0x7f6b50014070 con 0x7f6b601011d0 2023-12-16T21:33:26.622 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2023-12-16T21:33:26.058+0000 7f6b5ffff700 1 --2- 172.21.15.45:0/4227261771 >> [v2:172.21.15.45:6800/2502560408,v1:172.21.15.45:6801/2502560408] conn(0x7f6b4c0365f0 0x7f6b4c038aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:33:26.622 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2023-12-16T21:33:26.058+0000 7f6b5ffff700 1 --2- 172.21.15.45:0/4227261771 >> [v2:172.21.15.45:6800/2502560408,v1:172.21.15.45:6801/2502560408] conn(0x7f6b4c0365f0 0x7f6b4c038aa0 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0x7f6b48009990 tx=0x7f6b48006e40 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:33:26.622 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2023-12-16T21:33:26.063+0000 7f6b5d7fa700 1 -- 172.21.15.45:0/4227261771 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+174570 (secure 0 0 0) 0x7f6b5002d980 con 0x7f6b601011d0 2023-12-16T21:33:26.622 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2023-12-16T21:33:26.276+0000 7f6b6719c700 1 -- 172.21.15.45:0/4227261771 --> [v2:172.21.15.45:6800/2502560408,v1:172.21.15.45:6801/2502560408] -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "prometheus", "target": ["mon-mgr", ""]}) v1 -- 0x7f6b600008d0 con 0x7f6b4c0365f0 2023-12-16T21:33:26.622 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2023-12-16T21:33:26.279+0000 7f6b5d7fa700 1 -- 172.21.15.45:0/4227261771 <== mgr.14120 v2:172.21.15.45:6800/2502560408 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+31 (secure 0 0 0) 0x7f6b600008d0 con 0x7f6b4c0365f0 2023-12-16T21:33:26.622 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2023-12-16T21:33:26.282+0000 7f6b6719c700 1 -- 172.21.15.45:0/4227261771 >> [v2:172.21.15.45:6800/2502560408,v1:172.21.15.45:6801/2502560408] conn(0x7f6b4c0365f0 msgr2=0x7f6b4c038aa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:33:26.622 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2023-12-16T21:33:26.282+0000 7f6b6719c700 1 --2- 172.21.15.45:0/4227261771 >> [v2:172.21.15.45:6800/2502560408,v1:172.21.15.45:6801/2502560408] conn(0x7f6b4c0365f0 0x7f6b4c038aa0 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0x7f6b48009990 tx=0x7f6b48006e40 comp rx=0 tx=0).stop 2023-12-16T21:33:26.622 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2023-12-16T21:33:26.282+0000 7f6b6719c700 1 -- 172.21.15.45:0/4227261771 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f6b601011d0 msgr2=0x7f6b6010ef00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:33:26.622 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2023-12-16T21:33:26.282+0000 7f6b6719c700 1 --2- 172.21.15.45:0/4227261771 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f6b601011d0 0x7f6b6010ef00 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f6b500184f0 tx=0x7f6b5000b4f0 comp rx=0 tx=0).stop 2023-12-16T21:33:26.622 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2023-12-16T21:33:26.282+0000 7f6b6719c700 1 -- 172.21.15.45:0/4227261771 shutdown_connections 2023-12-16T21:33:26.622 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2023-12-16T21:33:26.282+0000 7f6b6719c700 1 --2- 172.21.15.45:0/4227261771 >> [v2:172.21.15.45:6800/2502560408,v1:172.21.15.45:6801/2502560408] conn(0x7f6b4c0365f0 0x7f6b4c038aa0 unknown :-1 s=CLOSED pgs=15 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:33:26.622 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2023-12-16T21:33:26.282+0000 7f6b6719c700 1 --2- 172.21.15.45:0/4227261771 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f6b601011d0 0x7f6b6010ef00 unknown :-1 s=CLOSED pgs=54 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:33:26.623 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2023-12-16T21:33:26.282+0000 7f6b6719c700 1 -- 172.21.15.45:0/4227261771 >> 172.21.15.45:0/4227261771 conn(0x7f6b600fc950 msgr2=0x7f6b60105f40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-16T21:33:26.623 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2023-12-16T21:33:26.282+0000 7f6b6719c700 1 -- 172.21.15.45:0/4227261771 shutdown_connections 2023-12-16T21:33:26.623 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2023-12-16T21:33:26.282+0000 7f6b6719c700 1 -- 172.21.15.45:0/4227261771 wait complete. 2023-12-16T21:33:26.623 INFO:teuthology.orchestra.run.smithi045.stdout:Deploying grafana service with default placement... 2023-12-16T21:33:27.339 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stdout Scheduled grafana update... 2023-12-16T21:33:27.339 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2023-12-16T21:33:26.851+0000 7f42f3af6700 1 Processor -- start 2023-12-16T21:33:27.339 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2023-12-16T21:33:26.851+0000 7f42f3af6700 1 -- start start 2023-12-16T21:33:27.339 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2023-12-16T21:33:26.851+0000 7f42f3af6700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f42ec070fa0 0x7f42ec107300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:33:27.339 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2023-12-16T21:33:26.851+0000 7f42f3af6700 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f42ec102cf0 con 0x7f42ec070fa0 2023-12-16T21:33:27.340 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2023-12-16T21:33:26.852+0000 7f42f1892700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f42ec070fa0 0x7f42ec107300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:33:27.340 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2023-12-16T21:33:26.852+0000 7f42f1892700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f42ec070fa0 0x7f42ec107300 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:33002/0 (socket says 172.21.15.45:33002) 2023-12-16T21:33:27.340 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2023-12-16T21:33:26.852+0000 7f42f1892700 1 -- 172.21.15.45:0/3582253773 learned_addr learned my addr 172.21.15.45:0/3582253773 (peer_addr_for_me v2:172.21.15.45:0/0) 2023-12-16T21:33:27.340 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2023-12-16T21:33:26.852+0000 7f42f1892700 1 -- 172.21.15.45:0/3582253773 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f42ec107840 con 0x7f42ec070fa0 2023-12-16T21:33:27.340 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2023-12-16T21:33:26.852+0000 7f42f1892700 1 --2- 172.21.15.45:0/3582253773 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f42ec070fa0 0x7f42ec107300 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f42dc0098e0 tx=0x7f42dc009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bced3a80018537fb server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:33:27.340 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2023-12-16T21:33:26.852+0000 7f42f0890700 1 -- 172.21.15.45:0/3582253773 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f42dc00b860 con 0x7f42ec070fa0 2023-12-16T21:33:27.341 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2023-12-16T21:33:26.852+0000 7f42f0890700 1 -- 172.21.15.45:0/3582253773 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f42dc00be70 con 0x7f42ec070fa0 2023-12-16T21:33:27.341 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2023-12-16T21:33:26.852+0000 7f42f0890700 1 -- 172.21.15.45:0/3582253773 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f42dc010380 con 0x7f42ec070fa0 2023-12-16T21:33:27.341 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2023-12-16T21:33:26.853+0000 7f42f3af6700 1 -- 172.21.15.45:0/3582253773 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f42ec070fa0 msgr2=0x7f42ec107300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:33:27.341 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2023-12-16T21:33:26.853+0000 7f42f3af6700 1 --2- 172.21.15.45:0/3582253773 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f42ec070fa0 0x7f42ec107300 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f42dc0098e0 tx=0x7f42dc009250 comp rx=0 tx=0).stop 2023-12-16T21:33:27.341 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2023-12-16T21:33:26.853+0000 7f42f3af6700 1 -- 172.21.15.45:0/3582253773 shutdown_connections 2023-12-16T21:33:27.341 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2023-12-16T21:33:26.853+0000 7f42f3af6700 1 --2- 172.21.15.45:0/3582253773 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f42ec070fa0 0x7f42ec107300 unknown :-1 s=CLOSED pgs=55 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:33:27.342 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2023-12-16T21:33:26.853+0000 7f42f3af6700 1 -- 172.21.15.45:0/3582253773 >> 172.21.15.45:0/3582253773 conn(0x7f42ec0fc970 msgr2=0x7f42ec0fed80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-16T21:33:27.342 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2023-12-16T21:33:26.853+0000 7f42f3af6700 1 -- 172.21.15.45:0/3582253773 shutdown_connections 2023-12-16T21:33:27.342 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2023-12-16T21:33:26.853+0000 7f42f3af6700 1 -- 172.21.15.45:0/3582253773 wait complete. 2023-12-16T21:33:27.342 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2023-12-16T21:33:26.854+0000 7f42f3af6700 1 Processor -- start 2023-12-16T21:33:27.342 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2023-12-16T21:33:26.854+0000 7f42f3af6700 1 -- start start 2023-12-16T21:33:27.342 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2023-12-16T21:33:26.854+0000 7f42f3af6700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f42ec070fa0 0x7f42ec1133c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:33:27.343 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2023-12-16T21:33:26.854+0000 7f42f3af6700 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f42ec113900 con 0x7f42ec070fa0 2023-12-16T21:33:27.343 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2023-12-16T21:33:26.854+0000 7f42f1892700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f42ec070fa0 0x7f42ec1133c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:33:27.344 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2023-12-16T21:33:26.854+0000 7f42f1892700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f42ec070fa0 0x7f42ec1133c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:33016/0 (socket says 172.21.15.45:33016) 2023-12-16T21:33:27.344 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2023-12-16T21:33:26.854+0000 7f42f1892700 1 -- 172.21.15.45:0/2239398462 learned_addr learned my addr 172.21.15.45:0/2239398462 (peer_addr_for_me v2:172.21.15.45:0/0) 2023-12-16T21:33:27.344 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2023-12-16T21:33:26.854+0000 7f42f1892700 1 -- 172.21.15.45:0/2239398462 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f42dc0094f0 con 0x7f42ec070fa0 2023-12-16T21:33:27.344 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2023-12-16T21:33:26.854+0000 7f42f1892700 1 --2- 172.21.15.45:0/2239398462 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f42ec070fa0 0x7f42ec1133c0 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f42dc0094c0 tx=0x7f42dc009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:33:27.344 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2023-12-16T21:33:26.855+0000 7f42e27fc700 1 -- 172.21.15.45:0/2239398462 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f42dc010850 con 0x7f42ec070fa0 2023-12-16T21:33:27.344 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2023-12-16T21:33:26.855+0000 7f42f3af6700 1 -- 172.21.15.45:0/2239398462 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f42ec113b00 con 0x7f42ec070fa0 2023-12-16T21:33:27.344 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2023-12-16T21:33:26.855+0000 7f42e27fc700 1 -- 172.21.15.45:0/2239398462 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f42dc010e60 con 0x7f42ec070fa0 2023-12-16T21:33:27.344 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2023-12-16T21:33:26.855+0000 7f42e27fc700 1 -- 172.21.15.45:0/2239398462 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f42dc01a430 con 0x7f42ec070fa0 2023-12-16T21:33:27.345 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2023-12-16T21:33:26.855+0000 7f42f3af6700 1 -- 172.21.15.45:0/2239398462 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f42ec113f10 con 0x7f42ec070fa0 2023-12-16T21:33:27.345 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2023-12-16T21:33:26.856+0000 7f42e27fc700 1 -- 172.21.15.45:0/2239398462 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 8) v1 ==== 43281+0+0 (secure 0 0 0) 0x7f42dc0247b0 con 0x7f42ec070fa0 2023-12-16T21:33:27.345 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2023-12-16T21:33:26.856+0000 7f42f3af6700 1 -- 172.21.15.45:0/2239398462 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f42ec04bb80 con 0x7f42ec070fa0 2023-12-16T21:33:27.345 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2023-12-16T21:33:26.856+0000 7f42e27fc700 1 --2- 172.21.15.45:0/2239398462 >> [v2:172.21.15.45:6800/2502560408,v1:172.21.15.45:6801/2502560408] conn(0x7f42d8036640 0x7f42d8038af0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:33:27.346 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2023-12-16T21:33:26.856+0000 7f42e27fc700 1 -- 172.21.15.45:0/2239398462 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 936+0+0 (secure 0 0 0) 0x7f42dc04a3b0 con 0x7f42ec070fa0 2023-12-16T21:33:27.346 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2023-12-16T21:33:26.856+0000 7f42f1091700 1 --2- 172.21.15.45:0/2239398462 >> [v2:172.21.15.45:6800/2502560408,v1:172.21.15.45:6801/2502560408] conn(0x7f42d8036640 0x7f42d8038af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:33:27.346 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2023-12-16T21:33:26.856+0000 7f42f1091700 1 --2- 172.21.15.45:0/2239398462 >> [v2:172.21.15.45:6800/2502560408,v1:172.21.15.45:6801/2502560408] conn(0x7f42d8036640 0x7f42d8038af0 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7f42e8009990 tx=0x7f42e8006e40 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:33:27.347 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2023-12-16T21:33:26.861+0000 7f42e27fc700 1 -- 172.21.15.45:0/2239398462 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+174570 (secure 0 0 0) 0x7f42dc022910 con 0x7f42ec070fa0 2023-12-16T21:33:27.347 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2023-12-16T21:33:27.071+0000 7f42f3af6700 1 -- 172.21.15.45:0/2239398462 --> [v2:172.21.15.45:6800/2502560408,v1:172.21.15.45:6801/2502560408] -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "grafana", "target": ["mon-mgr", ""]}) v1 -- 0x7f42ec0008d0 con 0x7f42d8036640 2023-12-16T21:33:27.349 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2023-12-16T21:33:27.075+0000 7f42e27fc700 1 -- 172.21.15.45:0/2239398462 <== mgr.14120 v2:172.21.15.45:6800/2502560408 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+28 (secure 0 0 0) 0x7f42ec0008d0 con 0x7f42d8036640 2023-12-16T21:33:27.349 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2023-12-16T21:33:27.077+0000 7f42f3af6700 1 -- 172.21.15.45:0/2239398462 >> [v2:172.21.15.45:6800/2502560408,v1:172.21.15.45:6801/2502560408] conn(0x7f42d8036640 msgr2=0x7f42d8038af0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:33:27.349 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2023-12-16T21:33:27.077+0000 7f42f3af6700 1 --2- 172.21.15.45:0/2239398462 >> [v2:172.21.15.45:6800/2502560408,v1:172.21.15.45:6801/2502560408] conn(0x7f42d8036640 0x7f42d8038af0 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7f42e8009990 tx=0x7f42e8006e40 comp rx=0 tx=0).stop 2023-12-16T21:33:27.349 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2023-12-16T21:33:27.077+0000 7f42f3af6700 1 -- 172.21.15.45:0/2239398462 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f42ec070fa0 msgr2=0x7f42ec1133c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:33:27.349 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2023-12-16T21:33:27.077+0000 7f42f3af6700 1 --2- 172.21.15.45:0/2239398462 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f42ec070fa0 0x7f42ec1133c0 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f42dc0094c0 tx=0x7f42dc009250 comp rx=0 tx=0).stop 2023-12-16T21:33:27.349 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2023-12-16T21:33:27.077+0000 7f42f3af6700 1 -- 172.21.15.45:0/2239398462 shutdown_connections 2023-12-16T21:33:27.349 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2023-12-16T21:33:27.077+0000 7f42f3af6700 1 --2- 172.21.15.45:0/2239398462 >> [v2:172.21.15.45:6800/2502560408,v1:172.21.15.45:6801/2502560408] conn(0x7f42d8036640 0x7f42d8038af0 unknown :-1 s=CLOSED pgs=16 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:33:27.349 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2023-12-16T21:33:27.077+0000 7f42f3af6700 1 --2- 172.21.15.45:0/2239398462 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f42ec070fa0 0x7f42ec1133c0 unknown :-1 s=CLOSED pgs=56 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:33:27.349 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2023-12-16T21:33:27.077+0000 7f42f3af6700 1 -- 172.21.15.45:0/2239398462 >> 172.21.15.45:0/2239398462 conn(0x7f42ec0fc970 msgr2=0x7f42ec0fe1d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-16T21:33:27.350 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2023-12-16T21:33:27.078+0000 7f42f3af6700 1 -- 172.21.15.45:0/2239398462 shutdown_connections 2023-12-16T21:33:27.350 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2023-12-16T21:33:27.078+0000 7f42f3af6700 1 -- 172.21.15.45:0/2239398462 wait complete. 2023-12-16T21:33:27.350 INFO:teuthology.orchestra.run.smithi045.stdout:Deploying node-exporter service with default placement... 2023-12-16T21:33:27.411 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Dec 16 21:33:27 smithi045 ceph-mon[80763]: from='client.14146 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "crash", "target": ["mon-mgr", ""]}]: dispatch 2023-12-16T21:33:27.411 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Dec 16 21:33:27 smithi045 ceph-mon[80763]: Saving service crash spec with placement * 2023-12-16T21:33:27.411 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Dec 16 21:33:27 smithi045 ceph-mon[80763]: from='client.14148 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "prometheus", "target": ["mon-mgr", ""]}]: dispatch 2023-12-16T21:33:27.411 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Dec 16 21:33:27 smithi045 ceph-mon[80763]: Saving service prometheus spec with placement count:1 2023-12-16T21:33:27.411 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Dec 16 21:33:27 smithi045 ceph-mon[80763]: from='mgr.14120 172.21.15.45:0/902863156' entity='mgr.smithi045.nhtmwc' 2023-12-16T21:33:27.411 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Dec 16 21:33:27 smithi045 ceph-mon[80763]: from='mgr.14120 172.21.15.45:0/902863156' entity='mgr.smithi045.nhtmwc' 2023-12-16T21:33:27.411 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Dec 16 21:33:27 smithi045 ceph-mon[80763]: from='mgr.14120 172.21.15.45:0/902863156' entity='mgr.smithi045.nhtmwc' 2023-12-16T21:33:28.119 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stdout Scheduled node-exporter update... 2023-12-16T21:33:28.119 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2023-12-16T21:33:27.599+0000 7fe3020b5700 1 Processor -- start 2023-12-16T21:33:28.119 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2023-12-16T21:33:27.599+0000 7fe3020b5700 1 -- start start 2023-12-16T21:33:28.119 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2023-12-16T21:33:27.599+0000 7fe3020b5700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fe2fc1011d0 0x7fe2fc1015e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:33:28.119 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2023-12-16T21:33:27.599+0000 7fe3020b5700 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe2fc101b20 con 0x7fe2fc1011d0 2023-12-16T21:33:28.119 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2023-12-16T21:33:27.600+0000 7fe2fb7fe700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fe2fc1011d0 0x7fe2fc1015e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:33:28.119 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2023-12-16T21:33:27.600+0000 7fe2fb7fe700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fe2fc1011d0 0x7fe2fc1015e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:33018/0 (socket says 172.21.15.45:33018) 2023-12-16T21:33:28.119 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2023-12-16T21:33:27.600+0000 7fe2fb7fe700 1 -- 172.21.15.45:0/243422305 learned_addr learned my addr 172.21.15.45:0/243422305 (peer_addr_for_me v2:172.21.15.45:0/0) 2023-12-16T21:33:28.120 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2023-12-16T21:33:27.600+0000 7fe2fb7fe700 1 -- 172.21.15.45:0/243422305 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe2fc101c60 con 0x7fe2fc1011d0 2023-12-16T21:33:28.120 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2023-12-16T21:33:27.600+0000 7fe2fb7fe700 1 --2- 172.21.15.45:0/243422305 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fe2fc1011d0 0x7fe2fc1015e0 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7fe2ec00bb80 tx=0x7fe2ec00b4f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b1c12406b74680e6 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:33:28.120 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2023-12-16T21:33:27.600+0000 7fe2fa7fc700 1 -- 172.21.15.45:0/243422305 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fe2ec00dd00 con 0x7fe2fc1011d0 2023-12-16T21:33:28.120 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2023-12-16T21:33:27.600+0000 7fe2fa7fc700 1 -- 172.21.15.45:0/243422305 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fe2ec00de60 con 0x7fe2fc1011d0 2023-12-16T21:33:28.120 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2023-12-16T21:33:27.600+0000 7fe2fa7fc700 1 -- 172.21.15.45:0/243422305 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fe2ec0188d0 con 0x7fe2fc1011d0 2023-12-16T21:33:28.120 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2023-12-16T21:33:27.601+0000 7fe3020b5700 1 -- 172.21.15.45:0/243422305 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fe2fc1011d0 msgr2=0x7fe2fc1015e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:33:28.120 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2023-12-16T21:33:27.601+0000 7fe3020b5700 1 --2- 172.21.15.45:0/243422305 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fe2fc1011d0 0x7fe2fc1015e0 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7fe2ec00bb80 tx=0x7fe2ec00b4f0 comp rx=0 tx=0).stop 2023-12-16T21:33:28.120 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2023-12-16T21:33:27.601+0000 7fe3020b5700 1 -- 172.21.15.45:0/243422305 shutdown_connections 2023-12-16T21:33:28.120 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2023-12-16T21:33:27.601+0000 7fe3020b5700 1 --2- 172.21.15.45:0/243422305 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fe2fc1011d0 0x7fe2fc1015e0 unknown :-1 s=CLOSED pgs=57 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:33:28.120 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2023-12-16T21:33:27.601+0000 7fe3020b5700 1 -- 172.21.15.45:0/243422305 >> 172.21.15.45:0/243422305 conn(0x7fe2fc0fc970 msgr2=0x7fe2fc0fed80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-16T21:33:28.120 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2023-12-16T21:33:27.601+0000 7fe3020b5700 1 -- 172.21.15.45:0/243422305 shutdown_connections 2023-12-16T21:33:28.120 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2023-12-16T21:33:27.601+0000 7fe3020b5700 1 -- 172.21.15.45:0/243422305 wait complete. 2023-12-16T21:33:28.120 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2023-12-16T21:33:27.602+0000 7fe3020b5700 1 Processor -- start 2023-12-16T21:33:28.120 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2023-12-16T21:33:27.602+0000 7fe3020b5700 1 -- start start 2023-12-16T21:33:28.121 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2023-12-16T21:33:27.602+0000 7fe3020b5700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fe2fc1011d0 0x7fe2fc10ef60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:33:28.121 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2023-12-16T21:33:27.602+0000 7fe3020b5700 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe2fc10f4a0 con 0x7fe2fc1011d0 2023-12-16T21:33:28.121 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2023-12-16T21:33:27.602+0000 7fe2fb7fe700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fe2fc1011d0 0x7fe2fc10ef60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:33:28.121 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2023-12-16T21:33:27.602+0000 7fe2fb7fe700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fe2fc1011d0 0x7fe2fc10ef60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:33032/0 (socket says 172.21.15.45:33032) 2023-12-16T21:33:28.121 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2023-12-16T21:33:27.602+0000 7fe2fb7fe700 1 -- 172.21.15.45:0/952145566 learned_addr learned my addr 172.21.15.45:0/952145566 (peer_addr_for_me v2:172.21.15.45:0/0) 2023-12-16T21:33:28.121 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2023-12-16T21:33:27.602+0000 7fe2fb7fe700 1 -- 172.21.15.45:0/952145566 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe2ec00b790 con 0x7fe2fc1011d0 2023-12-16T21:33:28.121 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2023-12-16T21:33:27.602+0000 7fe2fb7fe700 1 --2- 172.21.15.45:0/952145566 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fe2fc1011d0 0x7fe2fc10ef60 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7fe2ec0184f0 tx=0x7fe2ec00b4f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:33:28.121 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2023-12-16T21:33:27.603+0000 7fe2e3fff700 1 -- 172.21.15.45:0/952145566 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fe2ec010bd0 con 0x7fe2fc1011d0 2023-12-16T21:33:28.121 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2023-12-16T21:33:27.603+0000 7fe2e3fff700 1 -- 172.21.15.45:0/952145566 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fe2ec01d050 con 0x7fe2fc1011d0 2023-12-16T21:33:28.121 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2023-12-16T21:33:27.603+0000 7fe3020b5700 1 -- 172.21.15.45:0/952145566 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe2fc10f6a0 con 0x7fe2fc1011d0 2023-12-16T21:33:28.121 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2023-12-16T21:33:27.603+0000 7fe2e3fff700 1 -- 172.21.15.45:0/952145566 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fe2ec022590 con 0x7fe2fc1011d0 2023-12-16T21:33:28.121 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2023-12-16T21:33:27.603+0000 7fe3020b5700 1 -- 172.21.15.45:0/952145566 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe2fc10fab0 con 0x7fe2fc1011d0 2023-12-16T21:33:28.121 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2023-12-16T21:33:27.604+0000 7fe2e3fff700 1 -- 172.21.15.45:0/952145566 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 8) v1 ==== 43281+0+0 (secure 0 0 0) 0x7fe2ec022850 con 0x7fe2fc1011d0 2023-12-16T21:33:28.121 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2023-12-16T21:33:27.604+0000 7fe3020b5700 1 -- 172.21.15.45:0/952145566 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe2fc04bb80 con 0x7fe2fc1011d0 2023-12-16T21:33:28.122 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2023-12-16T21:33:27.604+0000 7fe2e3fff700 1 --2- 172.21.15.45:0/952145566 >> [v2:172.21.15.45:6800/2502560408,v1:172.21.15.45:6801/2502560408] conn(0x7fe2e40365f0 0x7fe2e4038aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:33:28.122 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2023-12-16T21:33:27.604+0000 7fe2e3fff700 1 -- 172.21.15.45:0/952145566 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 936+0+0 (secure 0 0 0) 0x7fe2ec014070 con 0x7fe2fc1011d0 2023-12-16T21:33:28.122 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2023-12-16T21:33:27.604+0000 7fe2faffd700 1 --2- 172.21.15.45:0/952145566 >> [v2:172.21.15.45:6800/2502560408,v1:172.21.15.45:6801/2502560408] conn(0x7fe2e40365f0 0x7fe2e4038aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:33:28.122 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2023-12-16T21:33:27.604+0000 7fe2faffd700 1 --2- 172.21.15.45:0/952145566 >> [v2:172.21.15.45:6800/2502560408,v1:172.21.15.45:6801/2502560408] conn(0x7fe2e40365f0 0x7fe2e4038aa0 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0x7fe2f0009990 tx=0x7fe2f0006e40 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:33:28.122 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2023-12-16T21:33:27.608+0000 7fe2e3fff700 1 -- 172.21.15.45:0/952145566 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+174570 (secure 0 0 0) 0x7fe2ec021b90 con 0x7fe2fc1011d0 2023-12-16T21:33:28.122 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2023-12-16T21:33:27.825+0000 7fe3020b5700 1 -- 172.21.15.45:0/952145566 --> [v2:172.21.15.45:6800/2502560408,v1:172.21.15.45:6801/2502560408] -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "node-exporter", "target": ["mon-mgr", ""]}) v1 -- 0x7fe2fc108c30 con 0x7fe2e40365f0 2023-12-16T21:33:28.122 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2023-12-16T21:33:27.830+0000 7fe2e3fff700 1 -- 172.21.15.45:0/952145566 <== mgr.14120 v2:172.21.15.45:6800/2502560408 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+34 (secure 0 0 0) 0x7fe2fc108c30 con 0x7fe2e40365f0 2023-12-16T21:33:28.122 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2023-12-16T21:33:27.832+0000 7fe3020b5700 1 -- 172.21.15.45:0/952145566 >> [v2:172.21.15.45:6800/2502560408,v1:172.21.15.45:6801/2502560408] conn(0x7fe2e40365f0 msgr2=0x7fe2e4038aa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:33:28.122 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2023-12-16T21:33:27.832+0000 7fe3020b5700 1 --2- 172.21.15.45:0/952145566 >> [v2:172.21.15.45:6800/2502560408,v1:172.21.15.45:6801/2502560408] conn(0x7fe2e40365f0 0x7fe2e4038aa0 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0x7fe2f0009990 tx=0x7fe2f0006e40 comp rx=0 tx=0).stop 2023-12-16T21:33:28.122 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2023-12-16T21:33:27.832+0000 7fe3020b5700 1 -- 172.21.15.45:0/952145566 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fe2fc1011d0 msgr2=0x7fe2fc10ef60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:33:28.122 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2023-12-16T21:33:27.832+0000 7fe3020b5700 1 --2- 172.21.15.45:0/952145566 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fe2fc1011d0 0x7fe2fc10ef60 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7fe2ec0184f0 tx=0x7fe2ec00b4f0 comp rx=0 tx=0).stop 2023-12-16T21:33:28.123 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2023-12-16T21:33:27.832+0000 7fe3020b5700 1 -- 172.21.15.45:0/952145566 shutdown_connections 2023-12-16T21:33:28.123 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2023-12-16T21:33:27.832+0000 7fe3020b5700 1 --2- 172.21.15.45:0/952145566 >> [v2:172.21.15.45:6800/2502560408,v1:172.21.15.45:6801/2502560408] conn(0x7fe2e40365f0 0x7fe2e4038aa0 unknown :-1 s=CLOSED pgs=17 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:33:28.123 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2023-12-16T21:33:27.832+0000 7fe3020b5700 1 --2- 172.21.15.45:0/952145566 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fe2fc1011d0 0x7fe2fc10ef60 unknown :-1 s=CLOSED pgs=58 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:33:28.123 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2023-12-16T21:33:27.832+0000 7fe3020b5700 1 -- 172.21.15.45:0/952145566 >> 172.21.15.45:0/952145566 conn(0x7fe2fc0fc970 msgr2=0x7fe2fc105f40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-16T21:33:28.123 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2023-12-16T21:33:27.833+0000 7fe3020b5700 1 -- 172.21.15.45:0/952145566 shutdown_connections 2023-12-16T21:33:28.123 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2023-12-16T21:33:27.833+0000 7fe3020b5700 1 -- 172.21.15.45:0/952145566 wait complete. 2023-12-16T21:33:28.123 INFO:teuthology.orchestra.run.smithi045.stdout:Deploying alertmanager service with default placement... 2023-12-16T21:33:28.394 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Dec 16 21:33:28 smithi045 ceph-mon[80763]: from='client.14150 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "grafana", "target": ["mon-mgr", ""]}]: dispatch 2023-12-16T21:33:28.394 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Dec 16 21:33:28 smithi045 ceph-mon[80763]: Saving service grafana spec with placement count:1 2023-12-16T21:33:28.394 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Dec 16 21:33:28 smithi045 ceph-mon[80763]: from='mgr.14120 172.21.15.45:0/902863156' entity='mgr.smithi045.nhtmwc' 2023-12-16T21:33:28.394 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Dec 16 21:33:28 smithi045 ceph-mon[80763]: from='mgr.14120 172.21.15.45:0/902863156' entity='mgr.smithi045.nhtmwc' 2023-12-16T21:33:28.928 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stdout Scheduled alertmanager update... 2023-12-16T21:33:28.928 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2023-12-16T21:33:28.408+0000 7f5592a84700 1 Processor -- start 2023-12-16T21:33:28.929 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2023-12-16T21:33:28.408+0000 7f5592a84700 1 -- start start 2023-12-16T21:33:28.929 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2023-12-16T21:33:28.408+0000 7f5592a84700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f558c1027b0 0x7f558c102bc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:33:28.929 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2023-12-16T21:33:28.408+0000 7f5592a84700 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f558c103100 con 0x7f558c1027b0 2023-12-16T21:33:28.929 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2023-12-16T21:33:28.408+0000 7f5590820700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f558c1027b0 0x7f558c102bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:33:28.929 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2023-12-16T21:33:28.408+0000 7f5590820700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f558c1027b0 0x7f558c102bc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:33036/0 (socket says 172.21.15.45:33036) 2023-12-16T21:33:28.929 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2023-12-16T21:33:28.408+0000 7f5590820700 1 -- 172.21.15.45:0/1513638893 learned_addr learned my addr 172.21.15.45:0/1513638893 (peer_addr_for_me v2:172.21.15.45:0/0) 2023-12-16T21:33:28.929 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2023-12-16T21:33:28.409+0000 7f5590820700 1 -- 172.21.15.45:0/1513638893 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f558c103240 con 0x7f558c1027b0 2023-12-16T21:33:28.929 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2023-12-16T21:33:28.409+0000 7f5590820700 1 --2- 172.21.15.45:0/1513638893 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f558c1027b0 0x7f558c102bc0 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f558000bb80 tx=0x7f558000b4f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=44bcd6f580971092 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:33:28.929 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2023-12-16T21:33:28.409+0000 7f558b7fe700 1 -- 172.21.15.45:0/1513638893 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f558000dd00 con 0x7f558c1027b0 2023-12-16T21:33:28.929 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2023-12-16T21:33:28.409+0000 7f558b7fe700 1 -- 172.21.15.45:0/1513638893 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f558000de60 con 0x7f558c1027b0 2023-12-16T21:33:28.929 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2023-12-16T21:33:28.409+0000 7f558b7fe700 1 -- 172.21.15.45:0/1513638893 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f55800188d0 con 0x7f558c1027b0 2023-12-16T21:33:28.929 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2023-12-16T21:33:28.409+0000 7f5592a84700 1 -- 172.21.15.45:0/1513638893 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f558c1027b0 msgr2=0x7f558c102bc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:33:28.930 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2023-12-16T21:33:28.409+0000 7f5592a84700 1 --2- 172.21.15.45:0/1513638893 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f558c1027b0 0x7f558c102bc0 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f558000bb80 tx=0x7f558000b4f0 comp rx=0 tx=0).stop 2023-12-16T21:33:28.930 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2023-12-16T21:33:28.410+0000 7f5592a84700 1 -- 172.21.15.45:0/1513638893 shutdown_connections 2023-12-16T21:33:28.930 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2023-12-16T21:33:28.410+0000 7f5592a84700 1 --2- 172.21.15.45:0/1513638893 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f558c1027b0 0x7f558c102bc0 unknown :-1 s=CLOSED pgs=59 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:33:28.930 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2023-12-16T21:33:28.410+0000 7f5592a84700 1 -- 172.21.15.45:0/1513638893 >> 172.21.15.45:0/1513638893 conn(0x7f558c0fc0b0 msgr2=0x7f558c0fe4e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-16T21:33:28.930 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2023-12-16T21:33:28.410+0000 7f5592a84700 1 -- 172.21.15.45:0/1513638893 shutdown_connections 2023-12-16T21:33:28.930 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2023-12-16T21:33:28.410+0000 7f5592a84700 1 -- 172.21.15.45:0/1513638893 wait complete. 2023-12-16T21:33:28.930 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2023-12-16T21:33:28.410+0000 7f5592a84700 1 Processor -- start 2023-12-16T21:33:28.930 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2023-12-16T21:33:28.410+0000 7f5592a84700 1 -- start start 2023-12-16T21:33:28.930 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2023-12-16T21:33:28.411+0000 7f5592a84700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f558c1027b0 0x7f558c10ef40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:33:28.930 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2023-12-16T21:33:28.411+0000 7f5592a84700 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f558c10f480 con 0x7f558c1027b0 2023-12-16T21:33:28.930 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2023-12-16T21:33:28.411+0000 7f5590820700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f558c1027b0 0x7f558c10ef40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:33:28.930 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2023-12-16T21:33:28.411+0000 7f5590820700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f558c1027b0 0x7f558c10ef40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:33038/0 (socket says 172.21.15.45:33038) 2023-12-16T21:33:28.930 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2023-12-16T21:33:28.411+0000 7f5590820700 1 -- 172.21.15.45:0/2157110132 learned_addr learned my addr 172.21.15.45:0/2157110132 (peer_addr_for_me v2:172.21.15.45:0/0) 2023-12-16T21:33:28.930 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2023-12-16T21:33:28.411+0000 7f5590820700 1 -- 172.21.15.45:0/2157110132 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f558000b790 con 0x7f558c1027b0 2023-12-16T21:33:28.930 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2023-12-16T21:33:28.411+0000 7f5590820700 1 --2- 172.21.15.45:0/2157110132 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f558c1027b0 0x7f558c10ef40 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f55800184f0 tx=0x7f558000b4f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:33:28.931 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2023-12-16T21:33:28.411+0000 7f55897fa700 1 -- 172.21.15.45:0/2157110132 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f5580010b90 con 0x7f558c1027b0 2023-12-16T21:33:28.931 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2023-12-16T21:33:28.411+0000 7f55897fa700 1 -- 172.21.15.45:0/2157110132 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f558001d050 con 0x7f558c1027b0 2023-12-16T21:33:28.931 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2023-12-16T21:33:28.411+0000 7f5592a84700 1 -- 172.21.15.45:0/2157110132 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f558c10f680 con 0x7f558c1027b0 2023-12-16T21:33:28.931 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2023-12-16T21:33:28.411+0000 7f55897fa700 1 -- 172.21.15.45:0/2157110132 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f5580022590 con 0x7f558c1027b0 2023-12-16T21:33:28.931 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2023-12-16T21:33:28.411+0000 7f5592a84700 1 -- 172.21.15.45:0/2157110132 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f558c10fa90 con 0x7f558c1027b0 2023-12-16T21:33:28.931 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2023-12-16T21:33:28.412+0000 7f55897fa700 1 -- 172.21.15.45:0/2157110132 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 8) v1 ==== 43281+0+0 (secure 0 0 0) 0x7f5580022800 con 0x7f558c1027b0 2023-12-16T21:33:28.931 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2023-12-16T21:33:28.412+0000 7f5592a84700 1 -- 172.21.15.45:0/2157110132 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f558c04bb80 con 0x7f558c1027b0 2023-12-16T21:33:28.931 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2023-12-16T21:33:28.412+0000 7f55897fa700 1 --2- 172.21.15.45:0/2157110132 >> [v2:172.21.15.45:6800/2502560408,v1:172.21.15.45:6801/2502560408] conn(0x7f557c036640 0x7f557c038af0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:33:28.931 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2023-12-16T21:33:28.412+0000 7f55897fa700 1 -- 172.21.15.45:0/2157110132 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 936+0+0 (secure 0 0 0) 0x7f5580014070 con 0x7f558c1027b0 2023-12-16T21:33:28.931 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2023-12-16T21:33:28.413+0000 7f558bfff700 1 --2- 172.21.15.45:0/2157110132 >> [v2:172.21.15.45:6800/2502560408,v1:172.21.15.45:6801/2502560408] conn(0x7f557c036640 0x7f557c038af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:33:28.931 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2023-12-16T21:33:28.413+0000 7f558bfff700 1 --2- 172.21.15.45:0/2157110132 >> [v2:172.21.15.45:6800/2502560408,v1:172.21.15.45:6801/2502560408] conn(0x7f557c036640 0x7f557c038af0 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7f5578009990 tx=0x7f5578006e40 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:33:28.931 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2023-12-16T21:33:28.417+0000 7f55897fa700 1 -- 172.21.15.45:0/2157110132 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+174570 (secure 0 0 0) 0x7f558007ea00 con 0x7f558c1027b0 2023-12-16T21:33:28.931 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2023-12-16T21:33:28.640+0000 7f5592a84700 1 -- 172.21.15.45:0/2157110132 --> [v2:172.21.15.45:6800/2502560408,v1:172.21.15.45:6801/2502560408] -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "alertmanager", "target": ["mon-mgr", ""]}) v1 -- 0x7f558c0008d0 con 0x7f557c036640 2023-12-16T21:33:28.931 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2023-12-16T21:33:28.644+0000 7f55897fa700 1 -- 172.21.15.45:0/2157110132 <== mgr.14120 v2:172.21.15.45:6800/2502560408 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+33 (secure 0 0 0) 0x7f558c0008d0 con 0x7f557c036640 2023-12-16T21:33:28.932 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2023-12-16T21:33:28.646+0000 7f5592a84700 1 -- 172.21.15.45:0/2157110132 >> [v2:172.21.15.45:6800/2502560408,v1:172.21.15.45:6801/2502560408] conn(0x7f557c036640 msgr2=0x7f557c038af0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:33:28.932 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2023-12-16T21:33:28.646+0000 7f5592a84700 1 --2- 172.21.15.45:0/2157110132 >> [v2:172.21.15.45:6800/2502560408,v1:172.21.15.45:6801/2502560408] conn(0x7f557c036640 0x7f557c038af0 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7f5578009990 tx=0x7f5578006e40 comp rx=0 tx=0).stop 2023-12-16T21:33:28.932 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2023-12-16T21:33:28.646+0000 7f5592a84700 1 -- 172.21.15.45:0/2157110132 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f558c1027b0 msgr2=0x7f558c10ef40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:33:28.932 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2023-12-16T21:33:28.646+0000 7f5592a84700 1 --2- 172.21.15.45:0/2157110132 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f558c1027b0 0x7f558c10ef40 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f55800184f0 tx=0x7f558000b4f0 comp rx=0 tx=0).stop 2023-12-16T21:33:28.932 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2023-12-16T21:33:28.646+0000 7f5592a84700 1 -- 172.21.15.45:0/2157110132 shutdown_connections 2023-12-16T21:33:28.932 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2023-12-16T21:33:28.646+0000 7f5592a84700 1 --2- 172.21.15.45:0/2157110132 >> [v2:172.21.15.45:6800/2502560408,v1:172.21.15.45:6801/2502560408] conn(0x7f557c036640 0x7f557c038af0 unknown :-1 s=CLOSED pgs=18 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:33:28.932 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2023-12-16T21:33:28.646+0000 7f5592a84700 1 --2- 172.21.15.45:0/2157110132 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f558c1027b0 0x7f558c10ef40 unknown :-1 s=CLOSED pgs=60 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:33:28.932 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2023-12-16T21:33:28.646+0000 7f5592a84700 1 -- 172.21.15.45:0/2157110132 >> 172.21.15.45:0/2157110132 conn(0x7f558c0fc0b0 msgr2=0x7f558c105fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-16T21:33:28.932 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2023-12-16T21:33:28.647+0000 7f5592a84700 1 -- 172.21.15.45:0/2157110132 shutdown_connections 2023-12-16T21:33:28.932 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2023-12-16T21:33:28.647+0000 7f5592a84700 1 -- 172.21.15.45:0/2157110132 wait complete. 2023-12-16T21:33:29.566 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Dec 16 21:33:29 smithi045 ceph-mon[80763]: from='client.14152 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "node-exporter", "target": ["mon-mgr", ""]}]: dispatch 2023-12-16T21:33:29.566 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Dec 16 21:33:29 smithi045 ceph-mon[80763]: Saving service node-exporter spec with placement * 2023-12-16T21:33:29.566 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Dec 16 21:33:29 smithi045 ceph-mon[80763]: from='mgr.14120 172.21.15.45:0/902863156' entity='mgr.smithi045.nhtmwc' 2023-12-16T21:33:29.620 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2023-12-16T21:33:29.161+0000 7ffa5e429700 1 Processor -- start 2023-12-16T21:33:29.620 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2023-12-16T21:33:29.161+0000 7ffa5e429700 1 -- start start 2023-12-16T21:33:29.620 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2023-12-16T21:33:29.161+0000 7ffa5e429700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ffa581039c0 0x7ffa58103dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:33:29.620 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2023-12-16T21:33:29.161+0000 7ffa5e429700 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffa58104310 con 0x7ffa581039c0 2023-12-16T21:33:29.620 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2023-12-16T21:33:29.161+0000 7ffa57fff700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ffa581039c0 0x7ffa58103dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:33:29.621 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2023-12-16T21:33:29.161+0000 7ffa57fff700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ffa581039c0 0x7ffa58103dd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:33040/0 (socket says 172.21.15.45:33040) 2023-12-16T21:33:29.621 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2023-12-16T21:33:29.161+0000 7ffa57fff700 1 -- 172.21.15.45:0/1546014346 learned_addr learned my addr 172.21.15.45:0/1546014346 (peer_addr_for_me v2:172.21.15.45:0/0) 2023-12-16T21:33:29.621 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2023-12-16T21:33:29.161+0000 7ffa57fff700 1 -- 172.21.15.45:0/1546014346 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffa58101fb0 con 0x7ffa581039c0 2023-12-16T21:33:29.621 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2023-12-16T21:33:29.161+0000 7ffa57fff700 1 --2- 172.21.15.45:0/1546014346 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ffa581039c0 0x7ffa58103dd0 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7ffa480098e0 tx=0x7ffa48009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=90f383db82dfd02d server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:33:29.621 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2023-12-16T21:33:29.162+0000 7ffa56ffd700 1 -- 172.21.15.45:0/1546014346 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7ffa4800b860 con 0x7ffa581039c0 2023-12-16T21:33:29.621 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2023-12-16T21:33:29.162+0000 7ffa56ffd700 1 -- 172.21.15.45:0/1546014346 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7ffa4800be70 con 0x7ffa581039c0 2023-12-16T21:33:29.621 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2023-12-16T21:33:29.162+0000 7ffa56ffd700 1 -- 172.21.15.45:0/1546014346 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7ffa48010380 con 0x7ffa581039c0 2023-12-16T21:33:29.621 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2023-12-16T21:33:29.162+0000 7ffa5e429700 1 -- 172.21.15.45:0/1546014346 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ffa581039c0 msgr2=0x7ffa58103dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:33:29.621 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2023-12-16T21:33:29.162+0000 7ffa5e429700 1 --2- 172.21.15.45:0/1546014346 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ffa581039c0 0x7ffa58103dd0 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7ffa480098e0 tx=0x7ffa48009250 comp rx=0 tx=0).stop 2023-12-16T21:33:29.621 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2023-12-16T21:33:29.162+0000 7ffa5e429700 1 -- 172.21.15.45:0/1546014346 shutdown_connections 2023-12-16T21:33:29.621 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2023-12-16T21:33:29.162+0000 7ffa5e429700 1 --2- 172.21.15.45:0/1546014346 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ffa581039c0 0x7ffa58103dd0 unknown :-1 s=CLOSED pgs=61 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:33:29.622 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2023-12-16T21:33:29.162+0000 7ffa5e429700 1 -- 172.21.15.45:0/1546014346 >> 172.21.15.45:0/1546014346 conn(0x7ffa580fda40 msgr2=0x7ffa580ffe70 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-16T21:33:29.622 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2023-12-16T21:33:29.162+0000 7ffa5e429700 1 -- 172.21.15.45:0/1546014346 shutdown_connections 2023-12-16T21:33:29.622 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2023-12-16T21:33:29.162+0000 7ffa5e429700 1 -- 172.21.15.45:0/1546014346 wait complete. 2023-12-16T21:33:29.622 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2023-12-16T21:33:29.163+0000 7ffa5e429700 1 Processor -- start 2023-12-16T21:33:29.622 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2023-12-16T21:33:29.163+0000 7ffa5e429700 1 -- start start 2023-12-16T21:33:29.622 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2023-12-16T21:33:29.163+0000 7ffa5e429700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ffa581039c0 0x7ffa58111130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:33:29.622 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2023-12-16T21:33:29.163+0000 7ffa5e429700 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffa58111670 con 0x7ffa581039c0 2023-12-16T21:33:29.622 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2023-12-16T21:33:29.163+0000 7ffa57fff700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ffa581039c0 0x7ffa58111130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:33:29.622 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2023-12-16T21:33:29.163+0000 7ffa57fff700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ffa581039c0 0x7ffa58111130 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:33046/0 (socket says 172.21.15.45:33046) 2023-12-16T21:33:29.622 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2023-12-16T21:33:29.163+0000 7ffa57fff700 1 -- 172.21.15.45:0/1108666224 learned_addr learned my addr 172.21.15.45:0/1108666224 (peer_addr_for_me v2:172.21.15.45:0/0) 2023-12-16T21:33:29.622 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2023-12-16T21:33:29.163+0000 7ffa57fff700 1 -- 172.21.15.45:0/1108666224 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffa480094f0 con 0x7ffa581039c0 2023-12-16T21:33:29.622 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2023-12-16T21:33:29.164+0000 7ffa57fff700 1 --2- 172.21.15.45:0/1108666224 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ffa581039c0 0x7ffa58111130 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7ffa4800bac0 tx=0x7ffa48009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:33:29.622 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2023-12-16T21:33:29.164+0000 7ffa54ff9700 1 -- 172.21.15.45:0/1108666224 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7ffa480109b0 con 0x7ffa581039c0 2023-12-16T21:33:29.622 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2023-12-16T21:33:29.164+0000 7ffa54ff9700 1 -- 172.21.15.45:0/1108666224 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7ffa4801a430 con 0x7ffa581039c0 2023-12-16T21:33:29.623 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2023-12-16T21:33:29.164+0000 7ffa5e429700 1 -- 172.21.15.45:0/1108666224 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ffa58111870 con 0x7ffa581039c0 2023-12-16T21:33:29.623 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2023-12-16T21:33:29.164+0000 7ffa54ff9700 1 -- 172.21.15.45:0/1108666224 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7ffa480245b0 con 0x7ffa581039c0 2023-12-16T21:33:29.623 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2023-12-16T21:33:29.164+0000 7ffa5e429700 1 -- 172.21.15.45:0/1108666224 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ffa58111c80 con 0x7ffa581039c0 2023-12-16T21:33:29.623 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2023-12-16T21:33:29.165+0000 7ffa54ff9700 1 -- 172.21.15.45:0/1108666224 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 8) v1 ==== 43281+0+0 (secure 0 0 0) 0x7ffa48024820 con 0x7ffa581039c0 2023-12-16T21:33:29.623 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2023-12-16T21:33:29.165+0000 7ffa54ff9700 1 --2- 172.21.15.45:0/1108666224 >> [v2:172.21.15.45:6800/2502560408,v1:172.21.15.45:6801/2502560408] conn(0x7ffa400365f0 0x7ffa40038aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:33:29.623 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2023-12-16T21:33:29.165+0000 7ffa54ff9700 1 -- 172.21.15.45:0/1108666224 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 936+0+0 (secure 0 0 0) 0x7ffa4801a9f0 con 0x7ffa581039c0 2023-12-16T21:33:29.623 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2023-12-16T21:33:29.165+0000 7ffa577fe700 1 --2- 172.21.15.45:0/1108666224 >> [v2:172.21.15.45:6800/2502560408,v1:172.21.15.45:6801/2502560408] conn(0x7ffa400365f0 0x7ffa40038aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:33:29.623 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2023-12-16T21:33:29.165+0000 7ffa5e429700 1 -- 172.21.15.45:0/1108666224 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ffa58102d30 con 0x7ffa581039c0 2023-12-16T21:33:29.623 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2023-12-16T21:33:29.166+0000 7ffa577fe700 1 --2- 172.21.15.45:0/1108666224 >> [v2:172.21.15.45:6800/2502560408,v1:172.21.15.45:6801/2502560408] conn(0x7ffa400365f0 0x7ffa40038aa0 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7ffa4c009990 tx=0x7ffa4c006e40 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:33:29.623 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2023-12-16T21:33:29.170+0000 7ffa54ff9700 1 -- 172.21.15.45:0/1108666224 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+174570 (secure 0 0 0) 0x7ffa48006d70 con 0x7ffa581039c0 2023-12-16T21:33:29.623 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2023-12-16T21:33:29.368+0000 7ffa5e429700 1 -- 172.21.15.45:0/1108666224 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command([{prefix=config set, name=mgr/cephadm/container_init}] v 0) v1 -- 0x7ffa5804bb80 con 0x7ffa581039c0 2023-12-16T21:33:29.623 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2023-12-16T21:33:29.371+0000 7ffa54ff9700 1 -- 172.21.15.45:0/1108666224 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/cephadm/container_init}]=0 v7)=0 v7) v1 ==== 142+0+0 (secure 0 0 0) 0x7ffa48049020 con 0x7ffa581039c0 2023-12-16T21:33:29.623 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2023-12-16T21:33:29.373+0000 7ffa5e429700 1 -- 172.21.15.45:0/1108666224 >> [v2:172.21.15.45:6800/2502560408,v1:172.21.15.45:6801/2502560408] conn(0x7ffa400365f0 msgr2=0x7ffa40038aa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:33:29.623 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2023-12-16T21:33:29.373+0000 7ffa5e429700 1 --2- 172.21.15.45:0/1108666224 >> [v2:172.21.15.45:6800/2502560408,v1:172.21.15.45:6801/2502560408] conn(0x7ffa400365f0 0x7ffa40038aa0 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7ffa4c009990 tx=0x7ffa4c006e40 comp rx=0 tx=0).stop 2023-12-16T21:33:29.623 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2023-12-16T21:33:29.373+0000 7ffa5e429700 1 -- 172.21.15.45:0/1108666224 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ffa581039c0 msgr2=0x7ffa58111130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:33:29.624 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2023-12-16T21:33:29.373+0000 7ffa5e429700 1 --2- 172.21.15.45:0/1108666224 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ffa581039c0 0x7ffa58111130 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7ffa4800bac0 tx=0x7ffa48009250 comp rx=0 tx=0).stop 2023-12-16T21:33:29.624 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2023-12-16T21:33:29.373+0000 7ffa5e429700 1 -- 172.21.15.45:0/1108666224 shutdown_connections 2023-12-16T21:33:29.624 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2023-12-16T21:33:29.373+0000 7ffa5e429700 1 --2- 172.21.15.45:0/1108666224 >> [v2:172.21.15.45:6800/2502560408,v1:172.21.15.45:6801/2502560408] conn(0x7ffa400365f0 0x7ffa40038aa0 unknown :-1 s=CLOSED pgs=19 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:33:29.624 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2023-12-16T21:33:29.373+0000 7ffa5e429700 1 --2- 172.21.15.45:0/1108666224 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ffa581039c0 0x7ffa58111130 unknown :-1 s=CLOSED pgs=62 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:33:29.624 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2023-12-16T21:33:29.373+0000 7ffa5e429700 1 -- 172.21.15.45:0/1108666224 >> 172.21.15.45:0/1108666224 conn(0x7ffa580fda40 msgr2=0x7ffa580ff190 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-16T21:33:29.624 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2023-12-16T21:33:29.374+0000 7ffa5e429700 1 -- 172.21.15.45:0/1108666224 shutdown_connections 2023-12-16T21:33:29.624 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2023-12-16T21:33:29.374+0000 7ffa5e429700 1 -- 172.21.15.45:0/1108666224 wait complete. 2023-12-16T21:33:30.302 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2023-12-16T21:33:29.852+0000 7f1221ba0700 1 Processor -- start 2023-12-16T21:33:30.302 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2023-12-16T21:33:29.852+0000 7f1221ba0700 1 -- start start 2023-12-16T21:33:30.302 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2023-12-16T21:33:29.852+0000 7f1221ba0700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f121c102100 0x7f121c102510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:33:30.303 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2023-12-16T21:33:29.852+0000 7f1221ba0700 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f121c102a50 con 0x7f121c102100 2023-12-16T21:33:30.303 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2023-12-16T21:33:29.853+0000 7f1220b9e700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f121c102100 0x7f121c102510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:33:30.303 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2023-12-16T21:33:29.853+0000 7f1220b9e700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f121c102100 0x7f121c102510 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:33060/0 (socket says 172.21.15.45:33060) 2023-12-16T21:33:30.303 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2023-12-16T21:33:29.853+0000 7f1220b9e700 1 -- 172.21.15.45:0/2578000939 learned_addr learned my addr 172.21.15.45:0/2578000939 (peer_addr_for_me v2:172.21.15.45:0/0) 2023-12-16T21:33:30.303 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2023-12-16T21:33:29.853+0000 7f1220b9e700 1 -- 172.21.15.45:0/2578000939 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f121c102b90 con 0x7f121c102100 2023-12-16T21:33:30.303 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2023-12-16T21:33:29.853+0000 7f1220b9e700 1 --2- 172.21.15.45:0/2578000939 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f121c102100 0x7f121c102510 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7f1214009e50 tx=0x7f1214009840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=32e4f0c3dd629c29 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:33:30.303 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2023-12-16T21:33:29.853+0000 7f121b7fe700 1 -- 172.21.15.45:0/2578000939 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f12140092e0 con 0x7f121c102100 2023-12-16T21:33:30.303 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2023-12-16T21:33:29.853+0000 7f121b7fe700 1 -- 172.21.15.45:0/2578000939 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f1214012430 con 0x7f121c102100 2023-12-16T21:33:30.303 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2023-12-16T21:33:29.853+0000 7f121b7fe700 1 -- 172.21.15.45:0/2578000939 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f12140075c0 con 0x7f121c102100 2023-12-16T21:33:30.303 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2023-12-16T21:33:29.854+0000 7f1221ba0700 1 -- 172.21.15.45:0/2578000939 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f121c102100 msgr2=0x7f121c102510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:33:30.303 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2023-12-16T21:33:29.854+0000 7f1221ba0700 1 --2- 172.21.15.45:0/2578000939 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f121c102100 0x7f121c102510 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7f1214009e50 tx=0x7f1214009840 comp rx=0 tx=0).stop 2023-12-16T21:33:30.303 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2023-12-16T21:33:29.854+0000 7f1221ba0700 1 -- 172.21.15.45:0/2578000939 shutdown_connections 2023-12-16T21:33:30.303 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2023-12-16T21:33:29.854+0000 7f1221ba0700 1 --2- 172.21.15.45:0/2578000939 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f121c102100 0x7f121c102510 unknown :-1 s=CLOSED pgs=63 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:33:30.303 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2023-12-16T21:33:29.854+0000 7f1221ba0700 1 -- 172.21.15.45:0/2578000939 >> 172.21.15.45:0/2578000939 conn(0x7f121c0fd860 msgr2=0x7f121c0ffc90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-16T21:33:30.304 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2023-12-16T21:33:29.854+0000 7f1221ba0700 1 -- 172.21.15.45:0/2578000939 shutdown_connections 2023-12-16T21:33:30.304 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2023-12-16T21:33:29.854+0000 7f1221ba0700 1 -- 172.21.15.45:0/2578000939 wait complete. 2023-12-16T21:33:30.304 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2023-12-16T21:33:29.854+0000 7f1221ba0700 1 Processor -- start 2023-12-16T21:33:30.304 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2023-12-16T21:33:29.854+0000 7f1221ba0700 1 -- start start 2023-12-16T21:33:30.304 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2023-12-16T21:33:29.855+0000 7f1221ba0700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f121c102100 0x7f121c078d10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:33:30.304 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2023-12-16T21:33:29.855+0000 7f1221ba0700 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f121c077190 con 0x7f121c102100 2023-12-16T21:33:30.304 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2023-12-16T21:33:29.855+0000 7f1220b9e700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f121c102100 0x7f121c078d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:33:30.304 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2023-12-16T21:33:29.855+0000 7f1220b9e700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f121c102100 0x7f121c078d10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:33070/0 (socket says 172.21.15.45:33070) 2023-12-16T21:33:30.304 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2023-12-16T21:33:29.855+0000 7f1220b9e700 1 -- 172.21.15.45:0/1237251184 learned_addr learned my addr 172.21.15.45:0/1237251184 (peer_addr_for_me v2:172.21.15.45:0/0) 2023-12-16T21:33:30.304 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2023-12-16T21:33:29.855+0000 7f1220b9e700 1 -- 172.21.15.45:0/1237251184 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1214009ae0 con 0x7f121c102100 2023-12-16T21:33:30.304 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2023-12-16T21:33:29.855+0000 7f1220b9e700 1 --2- 172.21.15.45:0/1237251184 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f121c102100 0x7f121c078d10 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7f12140127d0 tx=0x7f1214009840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:33:30.304 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2023-12-16T21:33:29.855+0000 7f12197fa700 1 -- 172.21.15.45:0/1237251184 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f12140092e0 con 0x7f121c102100 2023-12-16T21:33:30.304 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2023-12-16T21:33:29.856+0000 7f12197fa700 1 -- 172.21.15.45:0/1237251184 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f1214007ce0 con 0x7f121c102100 2023-12-16T21:33:30.304 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2023-12-16T21:33:29.856+0000 7f12197fa700 1 -- 172.21.15.45:0/1237251184 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f121401c910 con 0x7f121c102100 2023-12-16T21:33:30.305 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2023-12-16T21:33:29.856+0000 7f1221ba0700 1 -- 172.21.15.45:0/1237251184 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f121c077390 con 0x7f121c102100 2023-12-16T21:33:30.305 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2023-12-16T21:33:29.856+0000 7f1221ba0700 1 -- 172.21.15.45:0/1237251184 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f121c077800 con 0x7f121c102100 2023-12-16T21:33:30.305 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2023-12-16T21:33:29.856+0000 7f12197fa700 1 -- 172.21.15.45:0/1237251184 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 8) v1 ==== 43281+0+0 (secure 0 0 0) 0x7f1214019070 con 0x7f121c102100 2023-12-16T21:33:30.305 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2023-12-16T21:33:29.857+0000 7f1221ba0700 1 -- 172.21.15.45:0/1237251184 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f121c10afc0 con 0x7f121c102100 2023-12-16T21:33:30.305 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2023-12-16T21:33:29.857+0000 7f12197fa700 1 --2- 172.21.15.45:0/1237251184 >> [v2:172.21.15.45:6800/2502560408,v1:172.21.15.45:6801/2502560408] conn(0x7f12040365f0 0x7f1204038aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:33:30.305 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2023-12-16T21:33:29.857+0000 7f12197fa700 1 -- 172.21.15.45:0/1237251184 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 936+0+0 (secure 0 0 0) 0x7f1214049c70 con 0x7f121c102100 2023-12-16T21:33:30.305 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2023-12-16T21:33:29.857+0000 7f121bfff700 1 --2- 172.21.15.45:0/1237251184 >> [v2:172.21.15.45:6800/2502560408,v1:172.21.15.45:6801/2502560408] conn(0x7f12040365f0 0x7f1204038aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:33:30.305 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2023-12-16T21:33:29.857+0000 7f121bfff700 1 --2- 172.21.15.45:0/1237251184 >> [v2:172.21.15.45:6800/2502560408,v1:172.21.15.45:6801/2502560408] conn(0x7f12040365f0 0x7f1204038aa0 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7f1210009990 tx=0x7f1210006e40 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:33:30.305 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2023-12-16T21:33:29.861+0000 7f12197fa700 1 -- 172.21.15.45:0/1237251184 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+174570 (secure 0 0 0) 0x7f121400bda0 con 0x7f121c102100 2023-12-16T21:33:30.305 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2023-12-16T21:33:30.058+0000 7f1221ba0700 1 -- 172.21.15.45:0/1237251184 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command([{prefix=config set, name=mgr/dashboard/ssl_server_port}] v 0) v1 -- 0x7f121c04bb80 con 0x7f121c102100 2023-12-16T21:33:30.305 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2023-12-16T21:33:30.061+0000 7f12197fa700 1 -- 172.21.15.45:0/1237251184 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/dashboard/ssl_server_port}]=0 v8)=0 v8) v1 ==== 130+0+0 (secure 0 0 0) 0x7f1214020020 con 0x7f121c102100 2023-12-16T21:33:30.305 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2023-12-16T21:33:30.063+0000 7f1221ba0700 1 -- 172.21.15.45:0/1237251184 >> [v2:172.21.15.45:6800/2502560408,v1:172.21.15.45:6801/2502560408] conn(0x7f12040365f0 msgr2=0x7f1204038aa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:33:30.305 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2023-12-16T21:33:30.063+0000 7f1221ba0700 1 --2- 172.21.15.45:0/1237251184 >> [v2:172.21.15.45:6800/2502560408,v1:172.21.15.45:6801/2502560408] conn(0x7f12040365f0 0x7f1204038aa0 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7f1210009990 tx=0x7f1210006e40 comp rx=0 tx=0).stop 2023-12-16T21:33:30.306 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2023-12-16T21:33:30.063+0000 7f1221ba0700 1 -- 172.21.15.45:0/1237251184 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f121c102100 msgr2=0x7f121c078d10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:33:30.306 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2023-12-16T21:33:30.063+0000 7f1221ba0700 1 --2- 172.21.15.45:0/1237251184 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f121c102100 0x7f121c078d10 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7f12140127d0 tx=0x7f1214009840 comp rx=0 tx=0).stop 2023-12-16T21:33:30.306 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2023-12-16T21:33:30.063+0000 7f1221ba0700 1 -- 172.21.15.45:0/1237251184 shutdown_connections 2023-12-16T21:33:30.306 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2023-12-16T21:33:30.063+0000 7f1221ba0700 1 --2- 172.21.15.45:0/1237251184 >> [v2:172.21.15.45:6800/2502560408,v1:172.21.15.45:6801/2502560408] conn(0x7f12040365f0 0x7f1204038aa0 unknown :-1 s=CLOSED pgs=20 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:33:30.306 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2023-12-16T21:33:30.063+0000 7f1221ba0700 1 --2- 172.21.15.45:0/1237251184 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f121c102100 0x7f121c078d10 unknown :-1 s=CLOSED pgs=64 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:33:30.306 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2023-12-16T21:33:30.063+0000 7f1221ba0700 1 -- 172.21.15.45:0/1237251184 >> 172.21.15.45:0/1237251184 conn(0x7f121c0fd860 msgr2=0x7f121c0fefe0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-16T21:33:30.306 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2023-12-16T21:33:30.063+0000 7f1221ba0700 1 -- 172.21.15.45:0/1237251184 shutdown_connections 2023-12-16T21:33:30.306 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2023-12-16T21:33:30.064+0000 7f1221ba0700 1 -- 172.21.15.45:0/1237251184 wait complete. 2023-12-16T21:33:30.306 INFO:teuthology.orchestra.run.smithi045.stdout:Enabling the dashboard module... 2023-12-16T21:33:30.436 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Dec 16 21:33:30 smithi045 ceph-mon[80763]: from='client.14154 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "alertmanager", "target": ["mon-mgr", ""]}]: dispatch 2023-12-16T21:33:30.436 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Dec 16 21:33:30 smithi045 ceph-mon[80763]: Saving service alertmanager spec with placement count:1 2023-12-16T21:33:30.436 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Dec 16 21:33:30 smithi045 ceph-mon[80763]: from='client.? 172.21.15.45:0/1108666224' entity='client.admin' 2023-12-16T21:33:30.436 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Dec 16 21:33:30 smithi045 ceph-mon[80763]: from='client.? 172.21.15.45:0/1237251184' entity='client.admin' 2023-12-16T21:33:31.536 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Dec 16 21:33:31 smithi045 ceph-mon[80763]: from='client.? 172.21.15.45:0/3944328154' entity='client.admin' cmd=[{"prefix": "mgr module enable", "module": "dashboard"}]: dispatch 2023-12-16T21:33:31.546 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2023-12-16T21:33:30.533+0000 7f982e9cc700 1 Processor -- start 2023-12-16T21:33:31.546 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2023-12-16T21:33:30.533+0000 7f982e9cc700 1 -- start start 2023-12-16T21:33:31.546 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2023-12-16T21:33:30.533+0000 7f982e9cc700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f9828076c00 0x7f9828107300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:33:31.546 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2023-12-16T21:33:30.533+0000 7f982e9cc700 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9828102cf0 con 0x7f9828076c00 2023-12-16T21:33:31.546 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2023-12-16T21:33:30.533+0000 7f9827fff700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f9828076c00 0x7f9828107300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:33:31.546 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2023-12-16T21:33:30.533+0000 7f9827fff700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f9828076c00 0x7f9828107300 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:33086/0 (socket says 172.21.15.45:33086) 2023-12-16T21:33:31.546 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2023-12-16T21:33:30.533+0000 7f9827fff700 1 -- 172.21.15.45:0/4289789644 learned_addr learned my addr 172.21.15.45:0/4289789644 (peer_addr_for_me v2:172.21.15.45:0/0) 2023-12-16T21:33:31.546 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2023-12-16T21:33:30.533+0000 7f9827fff700 1 -- 172.21.15.45:0/4289789644 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9828107840 con 0x7f9828076c00 2023-12-16T21:33:31.546 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2023-12-16T21:33:30.533+0000 7f9827fff700 1 --2- 172.21.15.45:0/4289789644 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f9828076c00 0x7f9828107300 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7f98180098e0 tx=0x7f9818009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=49f7b2fca23244f5 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:33:31.546 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2023-12-16T21:33:30.534+0000 7f9826ffd700 1 -- 172.21.15.45:0/4289789644 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f981800b860 con 0x7f9828076c00 2023-12-16T21:33:31.546 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2023-12-16T21:33:30.534+0000 7f9826ffd700 1 -- 172.21.15.45:0/4289789644 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f981800be70 con 0x7f9828076c00 2023-12-16T21:33:31.547 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2023-12-16T21:33:30.534+0000 7f9826ffd700 1 -- 172.21.15.45:0/4289789644 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f9818010380 con 0x7f9828076c00 2023-12-16T21:33:31.547 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2023-12-16T21:33:30.534+0000 7f982e9cc700 1 -- 172.21.15.45:0/4289789644 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f9828076c00 msgr2=0x7f9828107300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:33:31.547 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2023-12-16T21:33:30.534+0000 7f982e9cc700 1 --2- 172.21.15.45:0/4289789644 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f9828076c00 0x7f9828107300 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7f98180098e0 tx=0x7f9818009250 comp rx=0 tx=0).stop 2023-12-16T21:33:31.547 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2023-12-16T21:33:30.534+0000 7f982e9cc700 1 -- 172.21.15.45:0/4289789644 shutdown_connections 2023-12-16T21:33:31.547 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2023-12-16T21:33:30.534+0000 7f982e9cc700 1 --2- 172.21.15.45:0/4289789644 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f9828076c00 0x7f9828107300 unknown :-1 s=CLOSED pgs=65 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:33:31.547 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2023-12-16T21:33:30.534+0000 7f982e9cc700 1 -- 172.21.15.45:0/4289789644 >> 172.21.15.45:0/4289789644 conn(0x7f98280fc950 msgr2=0x7f98280fed60 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-16T21:33:31.547 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2023-12-16T21:33:30.534+0000 7f982e9cc700 1 -- 172.21.15.45:0/4289789644 shutdown_connections 2023-12-16T21:33:31.547 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2023-12-16T21:33:30.534+0000 7f982e9cc700 1 -- 172.21.15.45:0/4289789644 wait complete. 2023-12-16T21:33:31.547 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2023-12-16T21:33:30.535+0000 7f982e9cc700 1 Processor -- start 2023-12-16T21:33:31.547 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2023-12-16T21:33:30.535+0000 7f982e9cc700 1 -- start start 2023-12-16T21:33:31.547 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2023-12-16T21:33:30.535+0000 7f982e9cc700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f9828076c00 0x7f98281133c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:33:31.547 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2023-12-16T21:33:30.535+0000 7f982e9cc700 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9828113900 con 0x7f9828076c00 2023-12-16T21:33:31.547 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2023-12-16T21:33:30.535+0000 7f9827fff700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f9828076c00 0x7f98281133c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:33:31.547 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2023-12-16T21:33:30.535+0000 7f9827fff700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f9828076c00 0x7f98281133c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:33100/0 (socket says 172.21.15.45:33100) 2023-12-16T21:33:31.548 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2023-12-16T21:33:30.535+0000 7f9827fff700 1 -- 172.21.15.45:0/3944328154 learned_addr learned my addr 172.21.15.45:0/3944328154 (peer_addr_for_me v2:172.21.15.45:0/0) 2023-12-16T21:33:31.548 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2023-12-16T21:33:30.536+0000 7f9827fff700 1 -- 172.21.15.45:0/3944328154 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f98180094f0 con 0x7f9828076c00 2023-12-16T21:33:31.548 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2023-12-16T21:33:30.536+0000 7f9827fff700 1 --2- 172.21.15.45:0/3944328154 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f9828076c00 0x7f98281133c0 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7f9818006d40 tx=0x7f9818009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:33:31.548 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2023-12-16T21:33:30.536+0000 7f9824ff9700 1 -- 172.21.15.45:0/3944328154 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f9818010960 con 0x7f9828076c00 2023-12-16T21:33:31.548 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2023-12-16T21:33:30.536+0000 7f9824ff9700 1 -- 172.21.15.45:0/3944328154 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f981801a430 con 0x7f9828076c00 2023-12-16T21:33:31.548 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2023-12-16T21:33:30.536+0000 7f982e9cc700 1 -- 172.21.15.45:0/3944328154 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9828113b00 con 0x7f9828076c00 2023-12-16T21:33:31.548 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2023-12-16T21:33:30.536+0000 7f9824ff9700 1 -- 172.21.15.45:0/3944328154 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f98180245b0 con 0x7f9828076c00 2023-12-16T21:33:31.548 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2023-12-16T21:33:30.536+0000 7f982e9cc700 1 -- 172.21.15.45:0/3944328154 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9828113f10 con 0x7f9828076c00 2023-12-16T21:33:31.548 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2023-12-16T21:33:30.537+0000 7f9824ff9700 1 -- 172.21.15.45:0/3944328154 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 8) v1 ==== 43281+0+0 (secure 0 0 0) 0x7f981802d3f0 con 0x7f9828076c00 2023-12-16T21:33:31.548 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2023-12-16T21:33:30.537+0000 7f982e9cc700 1 -- 172.21.15.45:0/3944328154 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f982804bb80 con 0x7f9828076c00 2023-12-16T21:33:31.548 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2023-12-16T21:33:30.537+0000 7f9824ff9700 1 --2- 172.21.15.45:0/3944328154 >> [v2:172.21.15.45:6800/2502560408,v1:172.21.15.45:6801/2502560408] conn(0x7f98100365f0 0x7f9810038aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:33:31.548 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2023-12-16T21:33:30.537+0000 7f9824ff9700 1 -- 172.21.15.45:0/3944328154 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 936+0+0 (secure 0 0 0) 0x7f981804a0f0 con 0x7f9828076c00 2023-12-16T21:33:31.548 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2023-12-16T21:33:30.537+0000 7f98277fe700 1 --2- 172.21.15.45:0/3944328154 >> [v2:172.21.15.45:6800/2502560408,v1:172.21.15.45:6801/2502560408] conn(0x7f98100365f0 0x7f9810038aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:33:31.549 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2023-12-16T21:33:30.537+0000 7f98277fe700 1 --2- 172.21.15.45:0/3944328154 >> [v2:172.21.15.45:6800/2502560408,v1:172.21.15.45:6801/2502560408] conn(0x7f98100365f0 0x7f9810038aa0 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7f981c009990 tx=0x7f981c006e40 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:33:31.549 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2023-12-16T21:33:30.542+0000 7f9824ff9700 1 -- 172.21.15.45:0/3944328154 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+174570 (secure 0 0 0) 0x7f981802b8e0 con 0x7f9828076c00 2023-12-16T21:33:31.549 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2023-12-16T21:33:30.792+0000 7f982e9cc700 1 -- 172.21.15.45:0/3944328154 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "mgr module enable", "module": "dashboard"} v 0) v1 -- 0x7f98280008d0 con 0x7f9828076c00 2023-12-16T21:33:31.549 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2023-12-16T21:33:31.285+0000 7f9824ff9700 1 -- 172.21.15.45:0/3944328154 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mgrmap(e 9) v1 ==== 43294+0+0 (secure 0 0 0) 0x7f981801a9f0 con 0x7f9828076c00 2023-12-16T21:33:31.549 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2023-12-16T21:33:31.286+0000 7f9824ff9700 1 -- 172.21.15.45:0/3944328154 <== mon.0 v2:172.21.15.45:3300/0 8 ==== mon_command_ack([{"prefix": "mgr module enable", "module": "dashboard"}]=0 v9) v1 ==== 88+0+0 (secure 0 0 0) 0x7f9818010e80 con 0x7f9828076c00 2023-12-16T21:33:31.549 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2023-12-16T21:33:31.289+0000 7f982e9cc700 1 -- 172.21.15.45:0/3944328154 >> [v2:172.21.15.45:6800/2502560408,v1:172.21.15.45:6801/2502560408] conn(0x7f98100365f0 msgr2=0x7f9810038aa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:33:31.549 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2023-12-16T21:33:31.289+0000 7f982e9cc700 1 --2- 172.21.15.45:0/3944328154 >> [v2:172.21.15.45:6800/2502560408,v1:172.21.15.45:6801/2502560408] conn(0x7f98100365f0 0x7f9810038aa0 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7f981c009990 tx=0x7f981c006e40 comp rx=0 tx=0).stop 2023-12-16T21:33:31.549 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2023-12-16T21:33:31.289+0000 7f982e9cc700 1 -- 172.21.15.45:0/3944328154 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f9828076c00 msgr2=0x7f98281133c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:33:31.549 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2023-12-16T21:33:31.289+0000 7f982e9cc700 1 --2- 172.21.15.45:0/3944328154 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f9828076c00 0x7f98281133c0 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7f9818006d40 tx=0x7f9818009250 comp rx=0 tx=0).stop 2023-12-16T21:33:31.549 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2023-12-16T21:33:31.289+0000 7f982e9cc700 1 -- 172.21.15.45:0/3944328154 shutdown_connections 2023-12-16T21:33:31.549 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2023-12-16T21:33:31.289+0000 7f982e9cc700 1 --2- 172.21.15.45:0/3944328154 >> [v2:172.21.15.45:6800/2502560408,v1:172.21.15.45:6801/2502560408] conn(0x7f98100365f0 0x7f9810038aa0 unknown :-1 s=CLOSED pgs=21 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:33:31.549 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2023-12-16T21:33:31.289+0000 7f982e9cc700 1 --2- 172.21.15.45:0/3944328154 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f9828076c00 0x7f98281133c0 unknown :-1 s=CLOSED pgs=66 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:33:31.549 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2023-12-16T21:33:31.289+0000 7f982e9cc700 1 -- 172.21.15.45:0/3944328154 >> 172.21.15.45:0/3944328154 conn(0x7f98280fc950 msgr2=0x7f98280fed50 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-16T21:33:31.549 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2023-12-16T21:33:31.289+0000 7f982e9cc700 1 -- 172.21.15.45:0/3944328154 shutdown_connections 2023-12-16T21:33:31.550 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2023-12-16T21:33:31.289+0000 7f982e9cc700 1 -- 172.21.15.45:0/3944328154 wait complete. 2023-12-16T21:33:32.287 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stdout { 2023-12-16T21:33:32.287 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stdout "epoch": 9, 2023-12-16T21:33:32.287 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stdout "available": true, 2023-12-16T21:33:32.287 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stdout "active_name": "smithi045.nhtmwc", 2023-12-16T21:33:32.287 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stdout "num_standby": 0 2023-12-16T21:33:32.287 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stdout } 2023-12-16T21:33:32.287 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2023-12-16T21:33:31.783+0000 7fa87ea49700 1 Processor -- start 2023-12-16T21:33:32.287 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2023-12-16T21:33:31.783+0000 7fa87ea49700 1 -- start start 2023-12-16T21:33:32.288 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2023-12-16T21:33:31.783+0000 7fa87ea49700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fa8781006d0 0x7fa878100ae0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:33:32.288 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2023-12-16T21:33:31.783+0000 7fa87ea49700 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa878101020 con 0x7fa8781006d0 2023-12-16T21:33:32.288 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2023-12-16T21:33:31.783+0000 7fa877fff700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fa8781006d0 0x7fa878100ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:33:32.288 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2023-12-16T21:33:31.783+0000 7fa877fff700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fa8781006d0 0x7fa878100ae0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:33128/0 (socket says 172.21.15.45:33128) 2023-12-16T21:33:32.288 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2023-12-16T21:33:31.783+0000 7fa877fff700 1 -- 172.21.15.45:0/1786958546 learned_addr learned my addr 172.21.15.45:0/1786958546 (peer_addr_for_me v2:172.21.15.45:0/0) 2023-12-16T21:33:32.288 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2023-12-16T21:33:31.784+0000 7fa877fff700 1 -- 172.21.15.45:0/1786958546 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa878101160 con 0x7fa8781006d0 2023-12-16T21:33:32.288 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2023-12-16T21:33:31.784+0000 7fa877fff700 1 --2- 172.21.15.45:0/1786958546 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fa8781006d0 0x7fa878100ae0 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7fa8680098e0 tx=0x7fa868009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e295154bc90184e0 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:33:32.288 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2023-12-16T21:33:31.784+0000 7fa876ffd700 1 -- 172.21.15.45:0/1786958546 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fa86800b860 con 0x7fa8781006d0 2023-12-16T21:33:32.288 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2023-12-16T21:33:31.784+0000 7fa876ffd700 1 -- 172.21.15.45:0/1786958546 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fa86800be70 con 0x7fa8781006d0 2023-12-16T21:33:32.288 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2023-12-16T21:33:31.784+0000 7fa876ffd700 1 -- 172.21.15.45:0/1786958546 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fa868010380 con 0x7fa8781006d0 2023-12-16T21:33:32.288 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2023-12-16T21:33:31.785+0000 7fa87ea49700 1 -- 172.21.15.45:0/1786958546 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fa8781006d0 msgr2=0x7fa878100ae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:33:32.288 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2023-12-16T21:33:31.785+0000 7fa87ea49700 1 --2- 172.21.15.45:0/1786958546 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fa8781006d0 0x7fa878100ae0 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7fa8680098e0 tx=0x7fa868009250 comp rx=0 tx=0).stop 2023-12-16T21:33:32.288 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2023-12-16T21:33:31.785+0000 7fa87ea49700 1 -- 172.21.15.45:0/1786958546 shutdown_connections 2023-12-16T21:33:32.288 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2023-12-16T21:33:31.785+0000 7fa87ea49700 1 --2- 172.21.15.45:0/1786958546 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fa8781006d0 0x7fa878100ae0 unknown :-1 s=CLOSED pgs=69 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:33:32.288 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2023-12-16T21:33:31.785+0000 7fa87ea49700 1 -- 172.21.15.45:0/1786958546 >> 172.21.15.45:0/1786958546 conn(0x7fa8780fc0b0 msgr2=0x7fa8780fe4e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-16T21:33:32.289 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2023-12-16T21:33:31.785+0000 7fa87ea49700 1 -- 172.21.15.45:0/1786958546 shutdown_connections 2023-12-16T21:33:32.289 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2023-12-16T21:33:31.785+0000 7fa87ea49700 1 -- 172.21.15.45:0/1786958546 wait complete. 2023-12-16T21:33:32.289 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2023-12-16T21:33:31.786+0000 7fa87ea49700 1 Processor -- start 2023-12-16T21:33:32.289 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2023-12-16T21:33:31.786+0000 7fa87ea49700 1 -- start start 2023-12-16T21:33:32.289 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2023-12-16T21:33:31.786+0000 7fa87ea49700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fa8781006d0 0x7fa878111390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:33:32.289 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2023-12-16T21:33:31.786+0000 7fa87ea49700 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa8781118d0 con 0x7fa8781006d0 2023-12-16T21:33:32.289 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2023-12-16T21:33:31.786+0000 7fa877fff700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fa8781006d0 0x7fa878111390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:33:32.289 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2023-12-16T21:33:31.786+0000 7fa877fff700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fa8781006d0 0x7fa878111390 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:33136/0 (socket says 172.21.15.45:33136) 2023-12-16T21:33:32.289 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2023-12-16T21:33:31.786+0000 7fa877fff700 1 -- 172.21.15.45:0/2981324625 learned_addr learned my addr 172.21.15.45:0/2981324625 (peer_addr_for_me v2:172.21.15.45:0/0) 2023-12-16T21:33:32.289 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2023-12-16T21:33:31.787+0000 7fa877fff700 1 -- 172.21.15.45:0/2981324625 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa8680094f0 con 0x7fa8781006d0 2023-12-16T21:33:32.289 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2023-12-16T21:33:31.787+0000 7fa877fff700 1 --2- 172.21.15.45:0/2981324625 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fa8781006d0 0x7fa878111390 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7fa86800c010 tx=0x7fa868009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:33:32.289 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2023-12-16T21:33:31.787+0000 7fa874ff9700 1 -- 172.21.15.45:0/2981324625 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fa868010960 con 0x7fa8781006d0 2023-12-16T21:33:32.289 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2023-12-16T21:33:31.787+0000 7fa874ff9700 1 -- 172.21.15.45:0/2981324625 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fa86801a430 con 0x7fa8781006d0 2023-12-16T21:33:32.290 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2023-12-16T21:33:31.787+0000 7fa87ea49700 1 -- 172.21.15.45:0/2981324625 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa878111ad0 con 0x7fa8781006d0 2023-12-16T21:33:32.290 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2023-12-16T21:33:31.787+0000 7fa874ff9700 1 -- 172.21.15.45:0/2981324625 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fa8680245b0 con 0x7fa8781006d0 2023-12-16T21:33:32.290 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2023-12-16T21:33:31.787+0000 7fa87ea49700 1 -- 172.21.15.45:0/2981324625 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa878111ee0 con 0x7fa8781006d0 2023-12-16T21:33:32.290 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2023-12-16T21:33:31.788+0000 7fa874ff9700 1 -- 172.21.15.45:0/2981324625 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 9) v1 ==== 43294+0+0 (secure 0 0 0) 0x7fa8680247a0 con 0x7fa8781006d0 2023-12-16T21:33:32.290 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2023-12-16T21:33:31.788+0000 7fa87ea49700 1 -- 172.21.15.45:0/2981324625 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa87804bb80 con 0x7fa8781006d0 2023-12-16T21:33:32.290 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2023-12-16T21:33:31.788+0000 7fa874ff9700 1 --2- 172.21.15.45:0/2981324625 >> [v2:172.21.15.45:6800/2502560408,v1:172.21.15.45:6801/2502560408] conn(0x7fa860036690 0x7fa860038b40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:33:32.290 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2023-12-16T21:33:31.788+0000 7fa874ff9700 1 -- 172.21.15.45:0/2981324625 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 936+0+0 (secure 0 0 0) 0x7fa86804a650 con 0x7fa8781006d0 2023-12-16T21:33:32.290 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2023-12-16T21:33:31.788+0000 7fa8777fe700 1 -- 172.21.15.45:0/2981324625 >> [v2:172.21.15.45:6800/2502560408,v1:172.21.15.45:6801/2502560408] conn(0x7fa860036690 msgr2=0x7fa860038b40 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.45:6800/2502560408 2023-12-16T21:33:32.290 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2023-12-16T21:33:31.788+0000 7fa8777fe700 1 --2- 172.21.15.45:0/2981324625 >> [v2:172.21.15.45:6800/2502560408,v1:172.21.15.45:6801/2502560408] conn(0x7fa860036690 0x7fa860038b40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-12-16T21:33:32.290 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2023-12-16T21:33:31.793+0000 7fa874ff9700 1 -- 172.21.15.45:0/2981324625 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+174570 (secure 0 0 0) 0x7fa8680189a0 con 0x7fa8781006d0 2023-12-16T21:33:32.290 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2023-12-16T21:33:31.989+0000 7fa8777fe700 1 -- 172.21.15.45:0/2981324625 >> [v2:172.21.15.45:6800/2502560408,v1:172.21.15.45:6801/2502560408] conn(0x7fa860036690 msgr2=0x7fa860038b40 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.45:6800/2502560408 2023-12-16T21:33:32.290 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2023-12-16T21:33:31.989+0000 7fa8777fe700 1 --2- 172.21.15.45:0/2981324625 >> [v2:172.21.15.45:6800/2502560408,v1:172.21.15.45:6801/2502560408] conn(0x7fa860036690 0x7fa860038b40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-12-16T21:33:32.290 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2023-12-16T21:33:32.044+0000 7fa87ea49700 1 -- 172.21.15.45:0/2981324625 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "mgr stat"} v 0) v1 -- 0x7fa87810ae90 con 0x7fa8781006d0 2023-12-16T21:33:32.291 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2023-12-16T21:33:32.044+0000 7fa874ff9700 1 -- 172.21.15.45:0/2981324625 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mon_command_ack([{"prefix": "mgr stat"}]=0 v9) v1 ==== 56+0+103 (secure 0 0 0) 0x7fa868018350 con 0x7fa8781006d0 2023-12-16T21:33:32.291 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2023-12-16T21:33:32.047+0000 7fa87ea49700 1 -- 172.21.15.45:0/2981324625 >> [v2:172.21.15.45:6800/2502560408,v1:172.21.15.45:6801/2502560408] conn(0x7fa860036690 msgr2=0x7fa860038b40 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-12-16T21:33:32.291 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2023-12-16T21:33:32.047+0000 7fa87ea49700 1 --2- 172.21.15.45:0/2981324625 >> [v2:172.21.15.45:6800/2502560408,v1:172.21.15.45:6801/2502560408] conn(0x7fa860036690 0x7fa860038b40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:33:32.291 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2023-12-16T21:33:32.047+0000 7fa87ea49700 1 -- 172.21.15.45:0/2981324625 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fa8781006d0 msgr2=0x7fa878111390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:33:32.291 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2023-12-16T21:33:32.047+0000 7fa87ea49700 1 --2- 172.21.15.45:0/2981324625 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fa8781006d0 0x7fa878111390 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7fa86800c010 tx=0x7fa868009250 comp rx=0 tx=0).stop 2023-12-16T21:33:32.291 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2023-12-16T21:33:32.047+0000 7fa87ea49700 1 -- 172.21.15.45:0/2981324625 shutdown_connections 2023-12-16T21:33:32.291 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2023-12-16T21:33:32.047+0000 7fa87ea49700 1 --2- 172.21.15.45:0/2981324625 >> [v2:172.21.15.45:6800/2502560408,v1:172.21.15.45:6801/2502560408] conn(0x7fa860036690 0x7fa860038b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:33:32.291 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2023-12-16T21:33:32.047+0000 7fa87ea49700 1 --2- 172.21.15.45:0/2981324625 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fa8781006d0 0x7fa878111390 unknown :-1 s=CLOSED pgs=70 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:33:32.291 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2023-12-16T21:33:32.047+0000 7fa87ea49700 1 -- 172.21.15.45:0/2981324625 >> 172.21.15.45:0/2981324625 conn(0x7fa8780fc0b0 msgr2=0x7fa8780fd830 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-16T21:33:32.291 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2023-12-16T21:33:32.048+0000 7fa87ea49700 1 -- 172.21.15.45:0/2981324625 shutdown_connections 2023-12-16T21:33:32.291 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2023-12-16T21:33:32.048+0000 7fa87ea49700 1 -- 172.21.15.45:0/2981324625 wait complete. 2023-12-16T21:33:32.291 INFO:teuthology.orchestra.run.smithi045.stdout:Waiting for the mgr to restart... 2023-12-16T21:33:32.291 INFO:teuthology.orchestra.run.smithi045.stdout:Waiting for mgr epoch 9... 2023-12-16T21:33:32.315 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Dec 16 21:33:32 smithi045 ceph-mon[80763]: from='client.? 172.21.15.45:0/3944328154' entity='client.admin' cmd='[{"prefix": "mgr module enable", "module": "dashboard"}]': finished 2023-12-16T21:33:32.316 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Dec 16 21:33:32 smithi045 ceph-mon[80763]: mgrmap e9: smithi045.nhtmwc(active, since 15s) 2023-12-16T21:33:32.316 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Dec 16 21:33:32 smithi045 ceph-mon[80763]: from='client.? 172.21.15.45:0/2981324625' entity='client.admin' cmd=[{"prefix": "mgr stat"}]: dispatch 2023-12-16T21:33:38.566 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Dec 16 21:33:38 smithi045 ceph-mon[80763]: Active manager daemon smithi045.nhtmwc restarted 2023-12-16T21:33:38.566 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Dec 16 21:33:38 smithi045 ceph-mon[80763]: Activating manager daemon smithi045.nhtmwc 2023-12-16T21:33:38.566 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Dec 16 21:33:38 smithi045 ceph-mon[80763]: osdmap e3: 0 total, 0 up, 0 in 2023-12-16T21:33:38.566 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Dec 16 21:33:38 smithi045 ceph-mon[80763]: mgrmap e10: smithi045.nhtmwc(active, starting, since 0.00348446s) 2023-12-16T21:33:38.566 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Dec 16 21:33:38 smithi045 ceph-mon[80763]: from='mgr.14162 172.21.15.45:0/142909557' entity='mgr.smithi045.nhtmwc' cmd=[{"prefix": "mon metadata", "id": "smithi045"}]: dispatch 2023-12-16T21:33:38.566 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Dec 16 21:33:38 smithi045 ceph-mon[80763]: from='mgr.14162 172.21.15.45:0/142909557' entity='mgr.smithi045.nhtmwc' cmd=[{"prefix": "mgr metadata", "who": "smithi045.nhtmwc", "id": "smithi045.nhtmwc"}]: dispatch 2023-12-16T21:33:38.566 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Dec 16 21:33:38 smithi045 ceph-mon[80763]: from='mgr.14162 172.21.15.45:0/142909557' entity='mgr.smithi045.nhtmwc' cmd=[{"prefix": "mds metadata"}]: dispatch 2023-12-16T21:33:38.566 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Dec 16 21:33:38 smithi045 ceph-mon[80763]: from='mgr.14162 172.21.15.45:0/142909557' entity='mgr.smithi045.nhtmwc' cmd=[{"prefix": "osd metadata"}]: dispatch 2023-12-16T21:33:38.566 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Dec 16 21:33:38 smithi045 ceph-mon[80763]: from='mgr.14162 172.21.15.45:0/142909557' entity='mgr.smithi045.nhtmwc' cmd=[{"prefix": "mon metadata"}]: dispatch 2023-12-16T21:33:38.566 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Dec 16 21:33:38 smithi045 ceph-mon[80763]: Manager daemon smithi045.nhtmwc is now available 2023-12-16T21:33:38.567 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Dec 16 21:33:38 smithi045 ceph-mon[80763]: from='mgr.14162 172.21.15.45:0/142909557' entity='mgr.smithi045.nhtmwc' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-12-16T21:33:38.567 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Dec 16 21:33:38 smithi045 ceph-mon[80763]: from='mgr.14162 172.21.15.45:0/142909557' entity='mgr.smithi045.nhtmwc' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-12-16T21:33:39.345 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stdout { 2023-12-16T21:33:39.345 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stdout "mgrmap_epoch": 11, 2023-12-16T21:33:39.345 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stdout "initialized": true 2023-12-16T21:33:39.345 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stdout } 2023-12-16T21:33:39.345 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2023-12-16T21:33:32.542+0000 7f0b331fe700 1 Processor -- start 2023-12-16T21:33:39.345 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2023-12-16T21:33:32.542+0000 7f0b331fe700 1 -- start start 2023-12-16T21:33:39.345 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2023-12-16T21:33:32.542+0000 7f0b331fe700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f0b2c1011c0 0x7f0b2c1015d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:33:39.345 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2023-12-16T21:33:32.542+0000 7f0b331fe700 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0b2c101b10 con 0x7f0b2c1011c0 2023-12-16T21:33:39.345 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2023-12-16T21:33:32.542+0000 7f0b30f9a700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f0b2c1011c0 0x7f0b2c1015d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:33:39.345 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2023-12-16T21:33:32.542+0000 7f0b30f9a700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f0b2c1011c0 0x7f0b2c1015d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:57626/0 (socket says 172.21.15.45:57626) 2023-12-16T21:33:39.346 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2023-12-16T21:33:32.542+0000 7f0b30f9a700 1 -- 172.21.15.45:0/2076024636 learned_addr learned my addr 172.21.15.45:0/2076024636 (peer_addr_for_me v2:172.21.15.45:0/0) 2023-12-16T21:33:39.346 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2023-12-16T21:33:32.542+0000 7f0b30f9a700 1 -- 172.21.15.45:0/2076024636 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0b2c101c50 con 0x7f0b2c1011c0 2023-12-16T21:33:39.346 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2023-12-16T21:33:32.543+0000 7f0b30f9a700 1 --2- 172.21.15.45:0/2076024636 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f0b2c1011c0 0x7f0b2c1015d0 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7f0b200098e0 tx=0x7f0b20009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ad23c6835aef6b4d server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:33:39.346 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2023-12-16T21:33:32.543+0000 7f0b2b7fe700 1 -- 172.21.15.45:0/2076024636 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f0b2000b860 con 0x7f0b2c1011c0 2023-12-16T21:33:39.346 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2023-12-16T21:33:32.543+0000 7f0b2b7fe700 1 -- 172.21.15.45:0/2076024636 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f0b2000be70 con 0x7f0b2c1011c0 2023-12-16T21:33:39.346 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2023-12-16T21:33:32.543+0000 7f0b2b7fe700 1 -- 172.21.15.45:0/2076024636 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f0b20010380 con 0x7f0b2c1011c0 2023-12-16T21:33:39.346 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2023-12-16T21:33:32.543+0000 7f0b331fe700 1 -- 172.21.15.45:0/2076024636 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f0b2c1011c0 msgr2=0x7f0b2c1015d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:33:39.346 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2023-12-16T21:33:32.543+0000 7f0b331fe700 1 --2- 172.21.15.45:0/2076024636 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f0b2c1011c0 0x7f0b2c1015d0 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7f0b200098e0 tx=0x7f0b20009250 comp rx=0 tx=0).stop 2023-12-16T21:33:39.346 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2023-12-16T21:33:32.543+0000 7f0b331fe700 1 -- 172.21.15.45:0/2076024636 shutdown_connections 2023-12-16T21:33:39.346 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2023-12-16T21:33:32.543+0000 7f0b331fe700 1 --2- 172.21.15.45:0/2076024636 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f0b2c1011c0 0x7f0b2c1015d0 unknown :-1 s=CLOSED pgs=71 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:33:39.346 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2023-12-16T21:33:32.543+0000 7f0b331fe700 1 -- 172.21.15.45:0/2076024636 >> 172.21.15.45:0/2076024636 conn(0x7f0b2c0fc980 msgr2=0x7f0b2c0fed90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-16T21:33:39.346 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2023-12-16T21:33:32.544+0000 7f0b331fe700 1 -- 172.21.15.45:0/2076024636 shutdown_connections 2023-12-16T21:33:39.346 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2023-12-16T21:33:32.544+0000 7f0b331fe700 1 -- 172.21.15.45:0/2076024636 wait complete. 2023-12-16T21:33:39.346 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2023-12-16T21:33:32.544+0000 7f0b331fe700 1 Processor -- start 2023-12-16T21:33:39.346 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2023-12-16T21:33:32.544+0000 7f0b331fe700 1 -- start start 2023-12-16T21:33:39.347 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2023-12-16T21:33:32.544+0000 7f0b331fe700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f0b2c1011c0 0x7f0b2c1133d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:33:39.347 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2023-12-16T21:33:32.544+0000 7f0b331fe700 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0b2c113910 con 0x7f0b2c1011c0 2023-12-16T21:33:39.347 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2023-12-16T21:33:32.545+0000 7f0b30f9a700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f0b2c1011c0 0x7f0b2c1133d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:33:39.347 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2023-12-16T21:33:32.545+0000 7f0b30f9a700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f0b2c1011c0 0x7f0b2c1133d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:57636/0 (socket says 172.21.15.45:57636) 2023-12-16T21:33:39.347 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2023-12-16T21:33:32.545+0000 7f0b30f9a700 1 -- 172.21.15.45:0/119807499 learned_addr learned my addr 172.21.15.45:0/119807499 (peer_addr_for_me v2:172.21.15.45:0/0) 2023-12-16T21:33:39.347 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2023-12-16T21:33:32.545+0000 7f0b30f9a700 1 -- 172.21.15.45:0/119807499 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0b200094f0 con 0x7f0b2c1011c0 2023-12-16T21:33:39.347 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2023-12-16T21:33:32.545+0000 7f0b30f9a700 1 --2- 172.21.15.45:0/119807499 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f0b2c1011c0 0x7f0b2c1133d0 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f0b2000c010 tx=0x7f0b20009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:33:39.347 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2023-12-16T21:33:32.545+0000 7f0b297fa700 1 -- 172.21.15.45:0/119807499 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f0b20010960 con 0x7f0b2c1011c0 2023-12-16T21:33:39.347 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2023-12-16T21:33:32.545+0000 7f0b297fa700 1 -- 172.21.15.45:0/119807499 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f0b2001a430 con 0x7f0b2c1011c0 2023-12-16T21:33:39.347 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2023-12-16T21:33:32.545+0000 7f0b331fe700 1 -- 172.21.15.45:0/119807499 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0b2c113b10 con 0x7f0b2c1011c0 2023-12-16T21:33:39.347 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2023-12-16T21:33:32.545+0000 7f0b297fa700 1 -- 172.21.15.45:0/119807499 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f0b20019580 con 0x7f0b2c1011c0 2023-12-16T21:33:39.347 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2023-12-16T21:33:32.545+0000 7f0b331fe700 1 -- 172.21.15.45:0/119807499 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0b2c113f20 con 0x7f0b2c1011c0 2023-12-16T21:33:39.347 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2023-12-16T21:33:32.546+0000 7f0b297fa700 1 -- 172.21.15.45:0/119807499 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 9) v1 ==== 43294+0+0 (secure 0 0 0) 0x7f0b20019800 con 0x7f0b2c1011c0 2023-12-16T21:33:39.347 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2023-12-16T21:33:32.546+0000 7f0b297fa700 1 --2- 172.21.15.45:0/119807499 >> [v2:172.21.15.45:6800/2502560408,v1:172.21.15.45:6801/2502560408] conn(0x7f0b1c036630 0x7f0b1c038ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:33:39.348 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2023-12-16T21:33:32.546+0000 7f0b297fa700 1 -- 172.21.15.45:0/119807499 --> [v2:172.21.15.45:6800/2502560408,v1:172.21.15.45:6801/2502560408] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x7f0b1c0391f0 con 0x7f0b1c036630 2023-12-16T21:33:39.348 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2023-12-16T21:33:32.546+0000 7f0b297fa700 1 -- 172.21.15.45:0/119807499 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 936+0+0 (secure 0 0 0) 0x7f0b2004a3a0 con 0x7f0b2c1011c0 2023-12-16T21:33:39.348 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2023-12-16T21:33:32.546+0000 7f0b2bfff700 1 -- 172.21.15.45:0/119807499 >> [v2:172.21.15.45:6800/2502560408,v1:172.21.15.45:6801/2502560408] conn(0x7f0b1c036630 msgr2=0x7f0b1c038ae0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.45:6800/2502560408 2023-12-16T21:33:39.348 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2023-12-16T21:33:32.546+0000 7f0b2bfff700 1 --2- 172.21.15.45:0/119807499 >> [v2:172.21.15.45:6800/2502560408,v1:172.21.15.45:6801/2502560408] conn(0x7f0b1c036630 0x7f0b1c038ae0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-12-16T21:33:39.348 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2023-12-16T21:33:32.747+0000 7f0b2bfff700 1 -- 172.21.15.45:0/119807499 >> [v2:172.21.15.45:6800/2502560408,v1:172.21.15.45:6801/2502560408] conn(0x7f0b1c036630 msgr2=0x7f0b1c038ae0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.45:6800/2502560408 2023-12-16T21:33:39.348 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2023-12-16T21:33:32.747+0000 7f0b2bfff700 1 --2- 172.21.15.45:0/119807499 >> [v2:172.21.15.45:6800/2502560408,v1:172.21.15.45:6801/2502560408] conn(0x7f0b1c036630 0x7f0b1c038ae0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-12-16T21:33:39.348 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2023-12-16T21:33:33.148+0000 7f0b2bfff700 1 -- 172.21.15.45:0/119807499 >> [v2:172.21.15.45:6800/2502560408,v1:172.21.15.45:6801/2502560408] conn(0x7f0b1c036630 msgr2=0x7f0b1c038ae0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.45:6800/2502560408 2023-12-16T21:33:39.348 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2023-12-16T21:33:33.148+0000 7f0b2bfff700 1 --2- 172.21.15.45:0/119807499 >> [v2:172.21.15.45:6800/2502560408,v1:172.21.15.45:6801/2502560408] conn(0x7f0b1c036630 0x7f0b1c038ae0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2023-12-16T21:33:39.348 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2023-12-16T21:33:33.949+0000 7f0b2bfff700 1 -- 172.21.15.45:0/119807499 >> [v2:172.21.15.45:6800/2502560408,v1:172.21.15.45:6801/2502560408] conn(0x7f0b1c036630 msgr2=0x7f0b1c038ae0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.45:6800/2502560408 2023-12-16T21:33:39.348 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2023-12-16T21:33:33.949+0000 7f0b2bfff700 1 --2- 172.21.15.45:0/119807499 >> [v2:172.21.15.45:6800/2502560408,v1:172.21.15.45:6801/2502560408] conn(0x7f0b1c036630 0x7f0b1c038ae0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2023-12-16T21:33:39.348 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2023-12-16T21:33:35.550+0000 7f0b2bfff700 1 -- 172.21.15.45:0/119807499 >> [v2:172.21.15.45:6800/2502560408,v1:172.21.15.45:6801/2502560408] conn(0x7f0b1c036630 msgr2=0x7f0b1c038ae0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.45:6800/2502560408 2023-12-16T21:33:39.348 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2023-12-16T21:33:35.550+0000 7f0b2bfff700 1 --2- 172.21.15.45:0/119807499 >> [v2:172.21.15.45:6800/2502560408,v1:172.21.15.45:6801/2502560408] conn(0x7f0b1c036630 0x7f0b1c038ae0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 3.200000 2023-12-16T21:33:39.348 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2023-12-16T21:33:38.103+0000 7f0b297fa700 1 -- 172.21.15.45:0/119807499 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mgrmap(e 10) v1 ==== 43061+0+0 (secure 0 0 0) 0x7f0b2000fd80 con 0x7f0b2c1011c0 2023-12-16T21:33:39.349 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2023-12-16T21:33:38.103+0000 7f0b297fa700 1 -- 172.21.15.45:0/119807499 >> [v2:172.21.15.45:6800/2502560408,v1:172.21.15.45:6801/2502560408] conn(0x7f0b1c036630 msgr2=0x7f0b1c038ae0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-12-16T21:33:39.349 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2023-12-16T21:33:38.103+0000 7f0b297fa700 1 --2- 172.21.15.45:0/119807499 >> [v2:172.21.15.45:6800/2502560408,v1:172.21.15.45:6801/2502560408] conn(0x7f0b1c036630 0x7f0b1c038ae0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:33:39.349 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2023-12-16T21:33:39.105+0000 7f0b297fa700 1 -- 172.21.15.45:0/119807499 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mgrmap(e 11) v1 ==== 43188+0+0 (secure 0 0 0) 0x7f0b2004b9f0 con 0x7f0b2c1011c0 2023-12-16T21:33:39.349 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2023-12-16T21:33:39.105+0000 7f0b297fa700 1 --2- 172.21.15.45:0/119807499 >> [v2:172.21.15.45:6800/2215417738,v1:172.21.15.45:6801/2215417738] conn(0x7f0b1c039ff0 0x7f0b1c03c3d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:33:39.349 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2023-12-16T21:33:39.105+0000 7f0b297fa700 1 -- 172.21.15.45:0/119807499 --> [v2:172.21.15.45:6800/2215417738,v1:172.21.15.45:6801/2215417738] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x7f0b1c0391f0 con 0x7f0b1c039ff0 2023-12-16T21:33:39.349 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2023-12-16T21:33:39.106+0000 7f0b2bfff700 1 --2- 172.21.15.45:0/119807499 >> [v2:172.21.15.45:6800/2215417738,v1:172.21.15.45:6801/2215417738] conn(0x7f0b1c039ff0 0x7f0b1c03c3d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:33:39.349 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2023-12-16T21:33:39.106+0000 7f0b2bfff700 1 --2- 172.21.15.45:0/119807499 >> [v2:172.21.15.45:6800/2215417738,v1:172.21.15.45:6801/2215417738] conn(0x7f0b1c039ff0 0x7f0b1c03c3d0 secure :-1 s=READY pgs=1 cs=0 l=1 rev1=1 crypto rx=0x7f0b180039d0 tx=0x7f0b180092f0 comp rx=0 tx=0).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:33:39.349 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2023-12-16T21:33:39.107+0000 7f0b297fa700 1 -- 172.21.15.45:0/119807499 <== mgr.14162 v2:172.21.15.45:6800/2215417738 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+6549 (secure 0 0 0) 0x7f0b1c0391f0 con 0x7f0b1c039ff0 2023-12-16T21:33:39.349 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2023-12-16T21:33:39.113+0000 7f0b331fe700 1 -- 172.21.15.45:0/119807499 --> [v2:172.21.15.45:6800/2215417738,v1:172.21.15.45:6801/2215417738] -- command(tid 1: {"prefix": "mgr_status"}) v1 -- 0x7f0b100025f0 con 0x7f0b1c039ff0 2023-12-16T21:33:39.349 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2023-12-16T21:33:39.113+0000 7f0b297fa700 1 -- 172.21.15.45:0/119807499 <== mgr.14162 v2:172.21.15.45:6800/2215417738 2 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+52 (secure 0 0 0) 0x7f0b100025f0 con 0x7f0b1c039ff0 2023-12-16T21:33:39.349 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2023-12-16T21:33:39.113+0000 7f0b331fe700 1 -- 172.21.15.45:0/119807499 >> [v2:172.21.15.45:6800/2215417738,v1:172.21.15.45:6801/2215417738] conn(0x7f0b1c039ff0 msgr2=0x7f0b1c03c3d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:33:39.349 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2023-12-16T21:33:39.113+0000 7f0b331fe700 1 --2- 172.21.15.45:0/119807499 >> [v2:172.21.15.45:6800/2215417738,v1:172.21.15.45:6801/2215417738] conn(0x7f0b1c039ff0 0x7f0b1c03c3d0 secure :-1 s=READY pgs=1 cs=0 l=1 rev1=1 crypto rx=0x7f0b180039d0 tx=0x7f0b180092f0 comp rx=0 tx=0).stop 2023-12-16T21:33:39.349 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2023-12-16T21:33:39.113+0000 7f0b331fe700 1 -- 172.21.15.45:0/119807499 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f0b2c1011c0 msgr2=0x7f0b2c1133d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:33:39.350 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2023-12-16T21:33:39.113+0000 7f0b331fe700 1 --2- 172.21.15.45:0/119807499 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f0b2c1011c0 0x7f0b2c1133d0 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f0b2000c010 tx=0x7f0b20009250 comp rx=0 tx=0).stop 2023-12-16T21:33:39.350 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2023-12-16T21:33:39.113+0000 7f0b331fe700 1 -- 172.21.15.45:0/119807499 shutdown_connections 2023-12-16T21:33:39.350 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2023-12-16T21:33:39.113+0000 7f0b331fe700 1 --2- 172.21.15.45:0/119807499 >> [v2:172.21.15.45:6800/2215417738,v1:172.21.15.45:6801/2215417738] conn(0x7f0b1c039ff0 0x7f0b1c03c3d0 unknown :-1 s=CLOSED pgs=1 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:33:39.350 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2023-12-16T21:33:39.113+0000 7f0b331fe700 1 --2- 172.21.15.45:0/119807499 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f0b2c1011c0 0x7f0b2c1133d0 unknown :-1 s=CLOSED pgs=72 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:33:39.350 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2023-12-16T21:33:39.113+0000 7f0b331fe700 1 --2- 172.21.15.45:0/119807499 >> [v2:172.21.15.45:6800/2502560408,v1:172.21.15.45:6801/2502560408] conn(0x7f0b1c036630 0x7f0b1c038ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:33:39.350 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2023-12-16T21:33:39.113+0000 7f0b331fe700 1 -- 172.21.15.45:0/119807499 >> 172.21.15.45:0/119807499 conn(0x7f0b2c0fc980 msgr2=0x7f0b2c10a340 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-16T21:33:39.350 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2023-12-16T21:33:39.113+0000 7f0b331fe700 1 -- 172.21.15.45:0/119807499 shutdown_connections 2023-12-16T21:33:39.350 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2023-12-16T21:33:39.113+0000 7f0b331fe700 1 -- 172.21.15.45:0/119807499 wait complete. 2023-12-16T21:33:39.350 INFO:teuthology.orchestra.run.smithi045.stdout:mgr epoch 9 is available 2023-12-16T21:33:39.350 INFO:teuthology.orchestra.run.smithi045.stdout:Generating a dashboard self-signed certificate... 2023-12-16T21:33:39.409 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Dec 16 21:33:39 smithi045 ceph-mon[80763]: from='mgr.14162 172.21.15.45:0/142909557' entity='mgr.smithi045.nhtmwc' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi045.nhtmwc/mirror_snapshot_schedule"}]: dispatch 2023-12-16T21:33:39.409 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Dec 16 21:33:39 smithi045 ceph-mon[80763]: from='mgr.14162 172.21.15.45:0/142909557' entity='mgr.smithi045.nhtmwc' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi045.nhtmwc/trash_purge_schedule"}]: dispatch 2023-12-16T21:33:39.409 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Dec 16 21:33:39 smithi045 ceph-mon[80763]: mgrmap e11: smithi045.nhtmwc(active, since 1.00625s) 2023-12-16T21:33:39.409 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Dec 16 21:33:39 smithi045 ceph-mon[80763]: from='mgr.14162 172.21.15.45:0/142909557' entity='mgr.smithi045.nhtmwc' 2023-12-16T21:33:40.192 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stdout Self-signed certificate created 2023-12-16T21:33:40.193 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2023-12-16T21:33:39.591+0000 7f794d978700 1 Processor -- start 2023-12-16T21:33:40.193 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2023-12-16T21:33:39.591+0000 7f794d978700 1 -- start start 2023-12-16T21:33:40.193 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2023-12-16T21:33:39.591+0000 7f794d978700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f7948103990 0x7f7948103da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:33:40.193 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2023-12-16T21:33:39.591+0000 7f794d978700 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f79481042e0 con 0x7f7948103990 2023-12-16T21:33:40.193 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2023-12-16T21:33:39.592+0000 7f7946ffd700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f7948103990 0x7f7948103da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:33:40.193 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2023-12-16T21:33:39.592+0000 7f7946ffd700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f7948103990 0x7f7948103da0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:57698/0 (socket says 172.21.15.45:57698) 2023-12-16T21:33:40.193 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2023-12-16T21:33:39.592+0000 7f7946ffd700 1 -- 172.21.15.45:0/303232632 learned_addr learned my addr 172.21.15.45:0/303232632 (peer_addr_for_me v2:172.21.15.45:0/0) 2023-12-16T21:33:40.193 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2023-12-16T21:33:39.592+0000 7f7946ffd700 1 -- 172.21.15.45:0/303232632 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7948104420 con 0x7f7948103990 2023-12-16T21:33:40.193 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2023-12-16T21:33:39.592+0000 7f7946ffd700 1 --2- 172.21.15.45:0/303232632 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f7948103990 0x7f7948103da0 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7f79300098e0 tx=0x7f7930009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=508022c056d3e651 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:33:40.193 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2023-12-16T21:33:39.592+0000 7f7945ffb700 1 -- 172.21.15.45:0/303232632 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f793000b860 con 0x7f7948103990 2023-12-16T21:33:40.193 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2023-12-16T21:33:39.592+0000 7f7945ffb700 1 -- 172.21.15.45:0/303232632 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f793000be70 con 0x7f7948103990 2023-12-16T21:33:40.194 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2023-12-16T21:33:39.592+0000 7f7945ffb700 1 -- 172.21.15.45:0/303232632 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f7930010380 con 0x7f7948103990 2023-12-16T21:33:40.194 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2023-12-16T21:33:39.593+0000 7f794d978700 1 -- 172.21.15.45:0/303232632 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f7948103990 msgr2=0x7f7948103da0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:33:40.194 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2023-12-16T21:33:39.593+0000 7f794d978700 1 --2- 172.21.15.45:0/303232632 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f7948103990 0x7f7948103da0 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7f79300098e0 tx=0x7f7930009250 comp rx=0 tx=0).stop 2023-12-16T21:33:40.194 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2023-12-16T21:33:39.593+0000 7f794d978700 1 -- 172.21.15.45:0/303232632 shutdown_connections 2023-12-16T21:33:40.194 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2023-12-16T21:33:39.593+0000 7f794d978700 1 --2- 172.21.15.45:0/303232632 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f7948103990 0x7f7948103da0 unknown :-1 s=CLOSED pgs=80 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:33:40.194 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2023-12-16T21:33:39.593+0000 7f794d978700 1 -- 172.21.15.45:0/303232632 >> 172.21.15.45:0/303232632 conn(0x7f79480fc0b0 msgr2=0x7f79480fe4e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-16T21:33:40.194 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2023-12-16T21:33:39.593+0000 7f794d978700 1 -- 172.21.15.45:0/303232632 shutdown_connections 2023-12-16T21:33:40.194 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2023-12-16T21:33:39.593+0000 7f794d978700 1 -- 172.21.15.45:0/303232632 wait complete. 2023-12-16T21:33:40.194 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2023-12-16T21:33:39.594+0000 7f794d978700 1 Processor -- start 2023-12-16T21:33:40.194 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2023-12-16T21:33:39.594+0000 7f794d978700 1 -- start start 2023-12-16T21:33:40.194 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2023-12-16T21:33:39.594+0000 7f794d978700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f7948103990 0x7f79481031d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:33:40.194 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2023-12-16T21:33:39.594+0000 7f794d978700 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7948101650 con 0x7f7948103990 2023-12-16T21:33:40.194 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2023-12-16T21:33:39.594+0000 7f7946ffd700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f7948103990 0x7f79481031d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:33:40.194 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2023-12-16T21:33:39.594+0000 7f7946ffd700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f7948103990 0x7f79481031d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:57710/0 (socket says 172.21.15.45:57710) 2023-12-16T21:33:40.194 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2023-12-16T21:33:39.594+0000 7f7946ffd700 1 -- 172.21.15.45:0/2295179010 learned_addr learned my addr 172.21.15.45:0/2295179010 (peer_addr_for_me v2:172.21.15.45:0/0) 2023-12-16T21:33:40.195 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2023-12-16T21:33:39.594+0000 7f7946ffd700 1 -- 172.21.15.45:0/2295179010 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f79300094f0 con 0x7f7948103990 2023-12-16T21:33:40.195 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2023-12-16T21:33:39.595+0000 7f7946ffd700 1 --2- 172.21.15.45:0/2295179010 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f7948103990 0x7f79481031d0 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7f7930011b60 tx=0x7f7930009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:33:40.195 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2023-12-16T21:33:39.595+0000 7f793f7fe700 1 -- 172.21.15.45:0/2295179010 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f7930010850 con 0x7f7948103990 2023-12-16T21:33:40.195 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2023-12-16T21:33:39.595+0000 7f793f7fe700 1 -- 172.21.15.45:0/2295179010 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f7930010e60 con 0x7f7948103990 2023-12-16T21:33:40.195 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2023-12-16T21:33:39.595+0000 7f794d978700 1 -- 172.21.15.45:0/2295179010 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7948101850 con 0x7f7948103990 2023-12-16T21:33:40.195 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2023-12-16T21:33:39.595+0000 7f793f7fe700 1 -- 172.21.15.45:0/2295179010 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f793001ae20 con 0x7f7948103990 2023-12-16T21:33:40.195 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2023-12-16T21:33:39.595+0000 7f794d978700 1 -- 172.21.15.45:0/2295179010 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7948101c60 con 0x7f7948103990 2023-12-16T21:33:40.195 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2023-12-16T21:33:39.596+0000 7f793f7fe700 1 -- 172.21.15.45:0/2295179010 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 11) v1 ==== 43188+0+0 (secure 0 0 0) 0x7f793002d3f0 con 0x7f7948103990 2023-12-16T21:33:40.195 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2023-12-16T21:33:39.596+0000 7f794d978700 1 -- 172.21.15.45:0/2295179010 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f794804bb80 con 0x7f7948103990 2023-12-16T21:33:40.195 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2023-12-16T21:33:39.596+0000 7f793f7fe700 1 --2- 172.21.15.45:0/2295179010 >> [v2:172.21.15.45:6800/2215417738,v1:172.21.15.45:6801/2215417738] conn(0x7f7934036510 0x7f79340389c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:33:40.195 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2023-12-16T21:33:39.596+0000 7f793f7fe700 1 -- 172.21.15.45:0/2295179010 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1151+0+0 (secure 0 0 0) 0x7f793004a1c0 con 0x7f7948103990 2023-12-16T21:33:40.195 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2023-12-16T21:33:39.596+0000 7f79467fc700 1 --2- 172.21.15.45:0/2295179010 >> [v2:172.21.15.45:6800/2215417738,v1:172.21.15.45:6801/2215417738] conn(0x7f7934036510 0x7f79340389c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:33:40.195 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2023-12-16T21:33:39.597+0000 7f79467fc700 1 --2- 172.21.15.45:0/2295179010 >> [v2:172.21.15.45:6800/2215417738,v1:172.21.15.45:6801/2215417738] conn(0x7f7934036510 0x7f79340389c0 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7f7938009990 tx=0x7f7938006e40 comp rx=0 tx=0).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:33:40.195 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2023-12-16T21:33:39.601+0000 7f793f7fe700 1 -- 172.21.15.45:0/2295179010 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+174570 (secure 0 0 0) 0x7f793002b6b0 con 0x7f7948103990 2023-12-16T21:33:40.196 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2023-12-16T21:33:39.846+0000 7f794d978700 1 -- 172.21.15.45:0/2295179010 --> [v2:172.21.15.45:6800/2215417738,v1:172.21.15.45:6801/2215417738] -- mgr_command(tid 0: {"prefix": "dashboard create-self-signed-cert", "target": ["mon-mgr", ""]}) v1 -- 0x7f794810ae20 con 0x7f7934036510 2023-12-16T21:33:40.196 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2023-12-16T21:33:39.939+0000 7f793f7fe700 1 -- 172.21.15.45:0/2295179010 <== mgr.14162 v2:172.21.15.45:6800/2215417738 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+32 (secure 0 0 0) 0x7f794810ae20 con 0x7f7934036510 2023-12-16T21:33:40.196 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2023-12-16T21:33:39.941+0000 7f794d978700 1 -- 172.21.15.45:0/2295179010 >> [v2:172.21.15.45:6800/2215417738,v1:172.21.15.45:6801/2215417738] conn(0x7f7934036510 msgr2=0x7f79340389c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:33:40.196 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2023-12-16T21:33:39.941+0000 7f794d978700 1 --2- 172.21.15.45:0/2295179010 >> [v2:172.21.15.45:6800/2215417738,v1:172.21.15.45:6801/2215417738] conn(0x7f7934036510 0x7f79340389c0 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7f7938009990 tx=0x7f7938006e40 comp rx=0 tx=0).stop 2023-12-16T21:33:40.196 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2023-12-16T21:33:39.941+0000 7f794d978700 1 -- 172.21.15.45:0/2295179010 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f7948103990 msgr2=0x7f79481031d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:33:40.196 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2023-12-16T21:33:39.941+0000 7f794d978700 1 --2- 172.21.15.45:0/2295179010 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f7948103990 0x7f79481031d0 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7f7930011b60 tx=0x7f7930009250 comp rx=0 tx=0).stop 2023-12-16T21:33:40.196 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2023-12-16T21:33:39.941+0000 7f794d978700 1 -- 172.21.15.45:0/2295179010 shutdown_connections 2023-12-16T21:33:40.196 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2023-12-16T21:33:39.941+0000 7f794d978700 1 --2- 172.21.15.45:0/2295179010 >> [v2:172.21.15.45:6800/2215417738,v1:172.21.15.45:6801/2215417738] conn(0x7f7934036510 0x7f79340389c0 unknown :-1 s=CLOSED pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:33:40.196 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2023-12-16T21:33:39.941+0000 7f794d978700 1 --2- 172.21.15.45:0/2295179010 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f7948103990 0x7f79481031d0 unknown :-1 s=CLOSED pgs=81 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:33:40.196 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2023-12-16T21:33:39.941+0000 7f794d978700 1 -- 172.21.15.45:0/2295179010 >> 172.21.15.45:0/2295179010 conn(0x7f79480fc0b0 msgr2=0x7f79480fe490 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-16T21:33:40.196 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2023-12-16T21:33:39.941+0000 7f794d978700 1 -- 172.21.15.45:0/2295179010 shutdown_connections 2023-12-16T21:33:40.196 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2023-12-16T21:33:39.941+0000 7f794d978700 1 -- 172.21.15.45:0/2295179010 wait complete. 2023-12-16T21:33:40.196 INFO:teuthology.orchestra.run.smithi045.stdout:Creating initial admin user... 2023-12-16T21:33:40.475 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Dec 16 21:33:40 smithi045 ceph-mon[80763]: [16/Dec/2023:21:33:39] ENGINE Bus STARTING 2023-12-16T21:33:40.476 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Dec 16 21:33:40 smithi045 ceph-mon[80763]: [16/Dec/2023:21:33:39] ENGINE Serving on https://172.21.15.45:7150 2023-12-16T21:33:40.476 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Dec 16 21:33:40 smithi045 ceph-mon[80763]: [16/Dec/2023:21:33:39] ENGINE Bus STARTED 2023-12-16T21:33:40.476 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Dec 16 21:33:40 smithi045 ceph-mon[80763]: from='mgr.14162 172.21.15.45:0/142909557' entity='mgr.smithi045.nhtmwc' 2023-12-16T21:33:40.476 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Dec 16 21:33:40 smithi045 ceph-mon[80763]: from='mgr.14162 172.21.15.45:0/142909557' entity='mgr.smithi045.nhtmwc' 2023-12-16T21:33:40.476 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Dec 16 21:33:40 smithi045 ceph-mon[80763]: from='mgr.14162 172.21.15.45:0/142909557' entity='mgr.smithi045.nhtmwc' 2023-12-16T21:33:40.476 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Dec 16 21:33:40 smithi045 ceph-mon[80763]: from='mgr.14162 172.21.15.45:0/142909557' entity='mgr.smithi045.nhtmwc' 2023-12-16T21:33:41.210 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stdout {"username": "admin", "password": "$2b$12$6eKsIsGrp.VoG3jI5gfjnuC0bT/T9zgFbEvt/DUQxOeUR3My1D/DK", "roles": ["administrator"], "name": null, "email": null, "lastUpdate": 1702762420, "enabled": true, "pwdExpirationDate": null, "pwdUpdateRequired": true} 2023-12-16T21:33:41.210 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2023-12-16T21:33:40.493+0000 7fb8024a0700 1 Processor -- start 2023-12-16T21:33:41.210 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2023-12-16T21:33:40.493+0000 7fb8024a0700 1 -- start start 2023-12-16T21:33:41.210 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2023-12-16T21:33:40.493+0000 7fb8024a0700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fb7fc1011d0 0x7fb7fc1015e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:33:41.210 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2023-12-16T21:33:40.493+0000 7fb8024a0700 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb7fc101b20 con 0x7fb7fc1011d0 2023-12-16T21:33:41.210 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2023-12-16T21:33:40.493+0000 7fb7fbfff700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fb7fc1011d0 0x7fb7fc1015e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:33:41.210 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2023-12-16T21:33:40.493+0000 7fb7fbfff700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fb7fc1011d0 0x7fb7fc1015e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:57718/0 (socket says 172.21.15.45:57718) 2023-12-16T21:33:41.210 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2023-12-16T21:33:40.493+0000 7fb7fbfff700 1 -- 172.21.15.45:0/1231703181 learned_addr learned my addr 172.21.15.45:0/1231703181 (peer_addr_for_me v2:172.21.15.45:0/0) 2023-12-16T21:33:41.210 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2023-12-16T21:33:40.493+0000 7fb7fbfff700 1 -- 172.21.15.45:0/1231703181 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb7fc101c60 con 0x7fb7fc1011d0 2023-12-16T21:33:41.211 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2023-12-16T21:33:40.494+0000 7fb7fbfff700 1 --2- 172.21.15.45:0/1231703181 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fb7fc1011d0 0x7fb7fc1015e0 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7fb7ec0098e0 tx=0x7fb7ec009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=861b4b56bb735f9b server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:33:41.211 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2023-12-16T21:33:40.494+0000 7fb7faffd700 1 -- 172.21.15.45:0/1231703181 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fb7ec00b860 con 0x7fb7fc1011d0 2023-12-16T21:33:41.211 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2023-12-16T21:33:40.494+0000 7fb7faffd700 1 -- 172.21.15.45:0/1231703181 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fb7ec00be70 con 0x7fb7fc1011d0 2023-12-16T21:33:41.211 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2023-12-16T21:33:40.494+0000 7fb7faffd700 1 -- 172.21.15.45:0/1231703181 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fb7ec010380 con 0x7fb7fc1011d0 2023-12-16T21:33:41.211 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2023-12-16T21:33:40.494+0000 7fb8024a0700 1 -- 172.21.15.45:0/1231703181 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fb7fc1011d0 msgr2=0x7fb7fc1015e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:33:41.211 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2023-12-16T21:33:40.494+0000 7fb8024a0700 1 --2- 172.21.15.45:0/1231703181 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fb7fc1011d0 0x7fb7fc1015e0 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7fb7ec0098e0 tx=0x7fb7ec009250 comp rx=0 tx=0).stop 2023-12-16T21:33:41.211 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2023-12-16T21:33:40.494+0000 7fb8024a0700 1 -- 172.21.15.45:0/1231703181 shutdown_connections 2023-12-16T21:33:41.211 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2023-12-16T21:33:40.494+0000 7fb8024a0700 1 --2- 172.21.15.45:0/1231703181 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fb7fc1011d0 0x7fb7fc1015e0 unknown :-1 s=CLOSED pgs=82 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:33:41.211 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2023-12-16T21:33:40.494+0000 7fb8024a0700 1 -- 172.21.15.45:0/1231703181 >> 172.21.15.45:0/1231703181 conn(0x7fb7fc0fc950 msgr2=0x7fb7fc0fed60 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-16T21:33:41.211 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2023-12-16T21:33:40.494+0000 7fb8024a0700 1 -- 172.21.15.45:0/1231703181 shutdown_connections 2023-12-16T21:33:41.211 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2023-12-16T21:33:40.494+0000 7fb8024a0700 1 -- 172.21.15.45:0/1231703181 wait complete. 2023-12-16T21:33:41.211 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2023-12-16T21:33:40.495+0000 7fb8024a0700 1 Processor -- start 2023-12-16T21:33:41.211 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2023-12-16T21:33:40.495+0000 7fb8024a0700 1 -- start start 2023-12-16T21:33:41.211 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2023-12-16T21:33:40.495+0000 7fb8024a0700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fb7fc1011d0 0x7fb7fc1075b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:33:41.212 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2023-12-16T21:33:40.495+0000 7fb8024a0700 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb7fc105a30 con 0x7fb7fc1011d0 2023-12-16T21:33:41.212 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2023-12-16T21:33:40.495+0000 7fb7fbfff700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fb7fc1011d0 0x7fb7fc1075b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:33:41.212 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2023-12-16T21:33:40.495+0000 7fb7fbfff700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fb7fc1011d0 0x7fb7fc1075b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:57724/0 (socket says 172.21.15.45:57724) 2023-12-16T21:33:41.212 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2023-12-16T21:33:40.495+0000 7fb7fbfff700 1 -- 172.21.15.45:0/3519543325 learned_addr learned my addr 172.21.15.45:0/3519543325 (peer_addr_for_me v2:172.21.15.45:0/0) 2023-12-16T21:33:41.212 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2023-12-16T21:33:40.496+0000 7fb7fbfff700 1 -- 172.21.15.45:0/3519543325 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb7ec0094f0 con 0x7fb7fc1011d0 2023-12-16T21:33:41.212 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2023-12-16T21:33:40.496+0000 7fb7fbfff700 1 --2- 172.21.15.45:0/3519543325 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fb7fc1011d0 0x7fb7fc1075b0 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7fb7ec00c010 tx=0x7fb7ec009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:33:41.213 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2023-12-16T21:33:40.496+0000 7fb7f8ff9700 1 -- 172.21.15.45:0/3519543325 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fb7ec010960 con 0x7fb7fc1011d0 2023-12-16T21:33:41.213 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2023-12-16T21:33:40.496+0000 7fb7f8ff9700 1 -- 172.21.15.45:0/3519543325 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fb7ec01a430 con 0x7fb7fc1011d0 2023-12-16T21:33:41.213 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2023-12-16T21:33:40.496+0000 7fb8024a0700 1 -- 172.21.15.45:0/3519543325 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb7fc105c30 con 0x7fb7fc1011d0 2023-12-16T21:33:41.213 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2023-12-16T21:33:40.496+0000 7fb7f8ff9700 1 -- 172.21.15.45:0/3519543325 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fb7ec019580 con 0x7fb7fc1011d0 2023-12-16T21:33:41.213 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2023-12-16T21:33:40.496+0000 7fb8024a0700 1 -- 172.21.15.45:0/3519543325 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb7fc106040 con 0x7fb7fc1011d0 2023-12-16T21:33:41.214 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2023-12-16T21:33:40.497+0000 7fb7f8ff9700 1 -- 172.21.15.45:0/3519543325 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 11) v1 ==== 43188+0+0 (secure 0 0 0) 0x7fb7ec0197b0 con 0x7fb7fc1011d0 2023-12-16T21:33:41.214 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2023-12-16T21:33:40.497+0000 7fb7f8ff9700 1 --2- 172.21.15.45:0/3519543325 >> [v2:172.21.15.45:6800/2215417738,v1:172.21.15.45:6801/2215417738] conn(0x7fb7e4036570 0x7fb7e4038a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:33:41.214 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2023-12-16T21:33:40.497+0000 7fb7f8ff9700 1 -- 172.21.15.45:0/3519543325 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1151+0+0 (secure 0 0 0) 0x7fb7ec04a2c0 con 0x7fb7fc1011d0 2023-12-16T21:33:41.214 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2023-12-16T21:33:40.497+0000 7fb8024a0700 1 -- 172.21.15.45:0/3519543325 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb7fc04bb80 con 0x7fb7fc1011d0 2023-12-16T21:33:41.214 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2023-12-16T21:33:40.497+0000 7fb7fb7fe700 1 --2- 172.21.15.45:0/3519543325 >> [v2:172.21.15.45:6800/2215417738,v1:172.21.15.45:6801/2215417738] conn(0x7fb7e4036570 0x7fb7e4038a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:33:41.214 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2023-12-16T21:33:40.497+0000 7fb7fb7fe700 1 --2- 172.21.15.45:0/3519543325 >> [v2:172.21.15.45:6800/2215417738,v1:172.21.15.45:6801/2215417738] conn(0x7fb7e4036570 0x7fb7e4038a20 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7fb7f0009990 tx=0x7fb7f0006e40 comp rx=0 tx=0).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:33:41.214 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2023-12-16T21:33:40.502+0000 7fb7f8ff9700 1 -- 172.21.15.45:0/3519543325 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+174570 (secure 0 0 0) 0x7fb7ec019b50 con 0x7fb7fc1011d0 2023-12-16T21:33:41.214 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2023-12-16T21:33:40.718+0000 7fb8024a0700 1 -- 172.21.15.45:0/3519543325 --> [v2:172.21.15.45:6800/2215417738,v1:172.21.15.45:6801/2215417738] -- mgr_command(tid 0: {"prefix": "dashboard ac-user-create", "username": "admin", "rolename": "administrator", "force_password": true, "pwd_update_required": true, "target": ["mon-mgr", ""]}) v1 -- 0x7fb7fc05f090 con 0x7fb7e4036570 2023-12-16T21:33:41.214 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2023-12-16T21:33:40.940+0000 7fb7f8ff9700 1 -- 172.21.15.45:0/3519543325 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mgrmap(e 12) v1 ==== 43294+0+0 (secure 0 0 0) 0x7fb7ec019d30 con 0x7fb7fc1011d0 2023-12-16T21:33:41.214 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2023-12-16T21:33:40.964+0000 7fb7f8ff9700 1 -- 172.21.15.45:0/3519543325 <== mgr.14162 v2:172.21.15.45:6800/2215417738 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+252 (secure 0 0 0) 0x7fb7fc05f090 con 0x7fb7e4036570 2023-12-16T21:33:41.215 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2023-12-16T21:33:40.966+0000 7fb8024a0700 1 -- 172.21.15.45:0/3519543325 >> [v2:172.21.15.45:6800/2215417738,v1:172.21.15.45:6801/2215417738] conn(0x7fb7e4036570 msgr2=0x7fb7e4038a20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:33:41.215 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2023-12-16T21:33:40.966+0000 7fb8024a0700 1 --2- 172.21.15.45:0/3519543325 >> [v2:172.21.15.45:6800/2215417738,v1:172.21.15.45:6801/2215417738] conn(0x7fb7e4036570 0x7fb7e4038a20 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7fb7f0009990 tx=0x7fb7f0006e40 comp rx=0 tx=0).stop 2023-12-16T21:33:41.215 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2023-12-16T21:33:40.966+0000 7fb8024a0700 1 -- 172.21.15.45:0/3519543325 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fb7fc1011d0 msgr2=0x7fb7fc1075b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:33:41.215 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2023-12-16T21:33:40.966+0000 7fb8024a0700 1 --2- 172.21.15.45:0/3519543325 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fb7fc1011d0 0x7fb7fc1075b0 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7fb7ec00c010 tx=0x7fb7ec009250 comp rx=0 tx=0).stop 2023-12-16T21:33:41.215 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2023-12-16T21:33:40.966+0000 7fb8024a0700 1 -- 172.21.15.45:0/3519543325 shutdown_connections 2023-12-16T21:33:41.215 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2023-12-16T21:33:40.966+0000 7fb8024a0700 1 --2- 172.21.15.45:0/3519543325 >> [v2:172.21.15.45:6800/2215417738,v1:172.21.15.45:6801/2215417738] conn(0x7fb7e4036570 0x7fb7e4038a20 unknown :-1 s=CLOSED pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:33:41.215 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2023-12-16T21:33:40.966+0000 7fb8024a0700 1 --2- 172.21.15.45:0/3519543325 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fb7fc1011d0 0x7fb7fc1075b0 unknown :-1 s=CLOSED pgs=83 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:33:41.215 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2023-12-16T21:33:40.966+0000 7fb8024a0700 1 -- 172.21.15.45:0/3519543325 >> 172.21.15.45:0/3519543325 conn(0x7fb7fc0fc950 msgr2=0x7fb7fc0767f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-16T21:33:41.215 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2023-12-16T21:33:40.967+0000 7fb8024a0700 1 -- 172.21.15.45:0/3519543325 shutdown_connections 2023-12-16T21:33:41.215 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2023-12-16T21:33:40.967+0000 7fb8024a0700 1 -- 172.21.15.45:0/3519543325 wait complete. 2023-12-16T21:33:41.215 INFO:teuthology.orchestra.run.smithi045.stdout:Fetching dashboard port number... 2023-12-16T21:33:41.451 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Dec 16 21:33:41 smithi045 ceph-mon[80763]: from='client.14174 -' entity='client.admin' cmd=[{"prefix": "dashboard create-self-signed-cert", "target": ["mon-mgr", ""]}]: dispatch 2023-12-16T21:33:41.451 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Dec 16 21:33:41 smithi045 ceph-mon[80763]: mgrmap e12: smithi045.nhtmwc(active, since 2s) 2023-12-16T21:33:41.451 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Dec 16 21:33:41 smithi045 ceph-mon[80763]: from='mgr.14162 172.21.15.45:0/142909557' entity='mgr.smithi045.nhtmwc' 2023-12-16T21:33:41.934 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stdout 8443 2023-12-16T21:33:41.934 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2023-12-16T21:33:41.448+0000 7fb03b30a700 1 Processor -- start 2023-12-16T21:33:41.934 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2023-12-16T21:33:41.449+0000 7fb03b30a700 1 -- start start 2023-12-16T21:33:41.934 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2023-12-16T21:33:41.449+0000 7fb03b30a700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fb034076c60 0x7fb034104ea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:33:41.934 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2023-12-16T21:33:41.449+0000 7fb03b30a700 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb0341053e0 con 0x7fb034076c60 2023-12-16T21:33:41.934 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2023-12-16T21:33:41.449+0000 7fb0390a6700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fb034076c60 0x7fb034104ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:33:41.934 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2023-12-16T21:33:41.449+0000 7fb0390a6700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fb034076c60 0x7fb034104ea0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:57734/0 (socket says 172.21.15.45:57734) 2023-12-16T21:33:41.934 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2023-12-16T21:33:41.449+0000 7fb0390a6700 1 -- 172.21.15.45:0/1638501023 learned_addr learned my addr 172.21.15.45:0/1638501023 (peer_addr_for_me v2:172.21.15.45:0/0) 2023-12-16T21:33:41.934 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2023-12-16T21:33:41.449+0000 7fb0390a6700 1 -- 172.21.15.45:0/1638501023 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb034105520 con 0x7fb034076c60 2023-12-16T21:33:41.934 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2023-12-16T21:33:41.450+0000 7fb0390a6700 1 --2- 172.21.15.45:0/1638501023 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fb034076c60 0x7fb034104ea0 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7fb0300098e0 tx=0x7fb030009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f96d4f28091e2b8c server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:33:41.934 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2023-12-16T21:33:41.450+0000 7fb02bfff700 1 -- 172.21.15.45:0/1638501023 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fb03000b860 con 0x7fb034076c60 2023-12-16T21:33:41.934 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2023-12-16T21:33:41.450+0000 7fb02bfff700 1 -- 172.21.15.45:0/1638501023 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fb03000be70 con 0x7fb034076c60 2023-12-16T21:33:41.935 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2023-12-16T21:33:41.450+0000 7fb02bfff700 1 -- 172.21.15.45:0/1638501023 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fb030010380 con 0x7fb034076c60 2023-12-16T21:33:41.935 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2023-12-16T21:33:41.450+0000 7fb03b30a700 1 -- 172.21.15.45:0/1638501023 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fb034076c60 msgr2=0x7fb034104ea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:33:41.935 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2023-12-16T21:33:41.450+0000 7fb03b30a700 1 --2- 172.21.15.45:0/1638501023 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fb034076c60 0x7fb034104ea0 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7fb0300098e0 tx=0x7fb030009250 comp rx=0 tx=0).stop 2023-12-16T21:33:41.935 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2023-12-16T21:33:41.450+0000 7fb03b30a700 1 -- 172.21.15.45:0/1638501023 shutdown_connections 2023-12-16T21:33:41.935 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2023-12-16T21:33:41.450+0000 7fb03b30a700 1 --2- 172.21.15.45:0/1638501023 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fb034076c60 0x7fb034104ea0 unknown :-1 s=CLOSED pgs=84 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:33:41.935 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2023-12-16T21:33:41.450+0000 7fb03b30a700 1 -- 172.21.15.45:0/1638501023 >> 172.21.15.45:0/1638501023 conn(0x7fb0340fc950 msgr2=0x7fb0340fed60 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-16T21:33:41.935 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2023-12-16T21:33:41.451+0000 7fb03b30a700 1 -- 172.21.15.45:0/1638501023 shutdown_connections 2023-12-16T21:33:41.935 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2023-12-16T21:33:41.451+0000 7fb03b30a700 1 -- 172.21.15.45:0/1638501023 wait complete. 2023-12-16T21:33:41.935 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2023-12-16T21:33:41.451+0000 7fb03b30a700 1 Processor -- start 2023-12-16T21:33:41.935 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2023-12-16T21:33:41.451+0000 7fb03b30a700 1 -- start start 2023-12-16T21:33:41.935 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2023-12-16T21:33:41.451+0000 7fb03b30a700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fb034076c60 0x7fb034102920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:33:41.935 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2023-12-16T21:33:41.451+0000 7fb03b30a700 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb034100da0 con 0x7fb034076c60 2023-12-16T21:33:41.935 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2023-12-16T21:33:41.451+0000 7fb0390a6700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fb034076c60 0x7fb034102920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:33:41.935 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2023-12-16T21:33:41.451+0000 7fb0390a6700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fb034076c60 0x7fb034102920 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:57744/0 (socket says 172.21.15.45:57744) 2023-12-16T21:33:41.935 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2023-12-16T21:33:41.451+0000 7fb0390a6700 1 -- 172.21.15.45:0/4009726959 learned_addr learned my addr 172.21.15.45:0/4009726959 (peer_addr_for_me v2:172.21.15.45:0/0) 2023-12-16T21:33:41.936 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2023-12-16T21:33:41.452+0000 7fb0390a6700 1 -- 172.21.15.45:0/4009726959 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb0300094f0 con 0x7fb034076c60 2023-12-16T21:33:41.936 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2023-12-16T21:33:41.452+0000 7fb0390a6700 1 --2- 172.21.15.45:0/4009726959 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fb034076c60 0x7fb034102920 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7fb030006d40 tx=0x7fb030009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:33:41.936 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2023-12-16T21:33:41.452+0000 7fb029ffb700 1 -- 172.21.15.45:0/4009726959 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fb030010960 con 0x7fb034076c60 2023-12-16T21:33:41.936 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2023-12-16T21:33:41.452+0000 7fb029ffb700 1 -- 172.21.15.45:0/4009726959 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fb03001a430 con 0x7fb034076c60 2023-12-16T21:33:41.936 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2023-12-16T21:33:41.452+0000 7fb03b30a700 1 -- 172.21.15.45:0/4009726959 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb034100fa0 con 0x7fb034076c60 2023-12-16T21:33:41.936 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2023-12-16T21:33:41.452+0000 7fb029ffb700 1 -- 172.21.15.45:0/4009726959 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fb0300245b0 con 0x7fb034076c60 2023-12-16T21:33:41.936 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2023-12-16T21:33:41.452+0000 7fb03b30a700 1 -- 172.21.15.45:0/4009726959 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb0341013b0 con 0x7fb034076c60 2023-12-16T21:33:41.936 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2023-12-16T21:33:41.453+0000 7fb029ffb700 1 -- 172.21.15.45:0/4009726959 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 12) v1 ==== 43294+0+0 (secure 0 0 0) 0x7fb03002d3f0 con 0x7fb034076c60 2023-12-16T21:33:41.936 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2023-12-16T21:33:41.453+0000 7fb03b30a700 1 -- 172.21.15.45:0/4009726959 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb03410b180 con 0x7fb034076c60 2023-12-16T21:33:41.936 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2023-12-16T21:33:41.453+0000 7fb029ffb700 1 --2- 172.21.15.45:0/4009726959 >> [v2:172.21.15.45:6800/2215417738,v1:172.21.15.45:6801/2215417738] conn(0x7fb020036640 0x7fb020038af0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:33:41.936 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2023-12-16T21:33:41.453+0000 7fb029ffb700 1 -- 172.21.15.45:0/4009726959 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1151+0+0 (secure 0 0 0) 0x7fb03004a280 con 0x7fb034076c60 2023-12-16T21:33:41.936 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2023-12-16T21:33:41.453+0000 7fb0388a5700 1 --2- 172.21.15.45:0/4009726959 >> [v2:172.21.15.45:6800/2215417738,v1:172.21.15.45:6801/2215417738] conn(0x7fb020036640 0x7fb020038af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:33:41.936 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2023-12-16T21:33:41.454+0000 7fb0388a5700 1 --2- 172.21.15.45:0/4009726959 >> [v2:172.21.15.45:6800/2215417738,v1:172.21.15.45:6801/2215417738] conn(0x7fb020036640 0x7fb020038af0 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7fb024009990 tx=0x7fb024006e40 comp rx=0 tx=0).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:33:41.936 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2023-12-16T21:33:41.458+0000 7fb029ffb700 1 -- 172.21.15.45:0/4009726959 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+174570 (secure 0 0 0) 0x7fb03002b8e0 con 0x7fb034076c60 2023-12-16T21:33:41.937 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2023-12-16T21:33:41.653+0000 7fb03b30a700 1 -- 172.21.15.45:0/4009726959 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "config get", "who": "mgr", "key": "mgr/dashboard/ssl_server_port"} v 0) v1 -- 0x7fb03404bb80 con 0x7fb034076c60 2023-12-16T21:33:41.937 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2023-12-16T21:33:41.653+0000 7fb029ffb700 1 -- 172.21.15.45:0/4009726959 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mon_command_ack([{"prefix": "config get", "who": "mgr", "key": "mgr/dashboard/ssl_server_port"}]=0 v8) v1 ==== 112+0+5 (secure 0 0 0) 0x7fb03002b2f0 con 0x7fb034076c60 2023-12-16T21:33:41.937 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2023-12-16T21:33:41.655+0000 7fb03b30a700 1 -- 172.21.15.45:0/4009726959 >> [v2:172.21.15.45:6800/2215417738,v1:172.21.15.45:6801/2215417738] conn(0x7fb020036640 msgr2=0x7fb020038af0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:33:41.937 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2023-12-16T21:33:41.655+0000 7fb03b30a700 1 --2- 172.21.15.45:0/4009726959 >> [v2:172.21.15.45:6800/2215417738,v1:172.21.15.45:6801/2215417738] conn(0x7fb020036640 0x7fb020038af0 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7fb024009990 tx=0x7fb024006e40 comp rx=0 tx=0).stop 2023-12-16T21:33:41.937 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2023-12-16T21:33:41.655+0000 7fb03b30a700 1 -- 172.21.15.45:0/4009726959 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fb034076c60 msgr2=0x7fb034102920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:33:41.937 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2023-12-16T21:33:41.655+0000 7fb03b30a700 1 --2- 172.21.15.45:0/4009726959 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fb034076c60 0x7fb034102920 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7fb030006d40 tx=0x7fb030009250 comp rx=0 tx=0).stop 2023-12-16T21:33:41.937 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2023-12-16T21:33:41.655+0000 7fb03b30a700 1 -- 172.21.15.45:0/4009726959 shutdown_connections 2023-12-16T21:33:41.937 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2023-12-16T21:33:41.655+0000 7fb03b30a700 1 --2- 172.21.15.45:0/4009726959 >> [v2:172.21.15.45:6800/2215417738,v1:172.21.15.45:6801/2215417738] conn(0x7fb020036640 0x7fb020038af0 unknown :-1 s=CLOSED pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:33:41.937 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2023-12-16T21:33:41.655+0000 7fb03b30a700 1 --2- 172.21.15.45:0/4009726959 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fb034076c60 0x7fb034102920 unknown :-1 s=CLOSED pgs=85 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:33:41.937 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2023-12-16T21:33:41.655+0000 7fb03b30a700 1 -- 172.21.15.45:0/4009726959 >> 172.21.15.45:0/4009726959 conn(0x7fb0340fc950 msgr2=0x7fb0340fed10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-16T21:33:41.937 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2023-12-16T21:33:41.655+0000 7fb03b30a700 1 -- 172.21.15.45:0/4009726959 shutdown_connections 2023-12-16T21:33:41.937 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2023-12-16T21:33:41.655+0000 7fb03b30a700 1 -- 172.21.15.45:0/4009726959 wait complete. 2023-12-16T21:33:41.941 INFO:teuthology.orchestra.run.smithi045.stdout:Non-zero exit code 1 from systemctl is-enabled firewalld.service 2023-12-16T21:33:41.941 INFO:teuthology.orchestra.run.smithi045.stdout:systemctl: stdout disabled 2023-12-16T21:33:41.948 INFO:teuthology.orchestra.run.smithi045.stdout:Non-zero exit code 3 from systemctl is-active firewalld.service 2023-12-16T21:33:41.948 INFO:teuthology.orchestra.run.smithi045.stdout:systemctl: stdout inactive 2023-12-16T21:33:41.948 INFO:teuthology.orchestra.run.smithi045.stdout:firewalld.service is not enabled 2023-12-16T21:33:41.948 INFO:teuthology.orchestra.run.smithi045.stdout:Not possible to open ports <[8443]>. firewalld.service is not available 2023-12-16T21:33:41.950 INFO:teuthology.orchestra.run.smithi045.stdout:Ceph Dashboard is now available at: 2023-12-16T21:33:41.950 INFO:teuthology.orchestra.run.smithi045.stdout: 2023-12-16T21:33:41.950 INFO:teuthology.orchestra.run.smithi045.stdout: URL: https://smithi045.front.sepia.ceph.com:8443/ 2023-12-16T21:33:41.950 INFO:teuthology.orchestra.run.smithi045.stdout: User: admin 2023-12-16T21:33:41.951 INFO:teuthology.orchestra.run.smithi045.stdout: Password: fxnj1ckyz6 2023-12-16T21:33:41.951 INFO:teuthology.orchestra.run.smithi045.stdout: 2023-12-16T21:33:41.951 INFO:teuthology.orchestra.run.smithi045.stdout:Saving cluster configuration to /var/lib/ceph/88ff363c-9c5a-11ee-95a5-87774f69a715/config directory 2023-12-16T21:33:41.951 INFO:teuthology.orchestra.run.smithi045.stdout:Enabling autotune for osd_memory_target 2023-12-16T21:33:42.608 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Dec 16 21:33:42 smithi045 ceph-mon[80763]: from='client.14176 -' entity='client.admin' cmd=[{"prefix": "dashboard ac-user-create", "username": "admin", "rolename": "administrator", "force_password": true, "pwd_update_required": true, "target": ["mon-mgr", ""]}]: dispatch 2023-12-16T21:33:42.609 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Dec 16 21:33:42 smithi045 ceph-mon[80763]: from='client.? 172.21.15.45:0/4009726959' entity='client.admin' cmd=[{"prefix": "config get", "who": "mgr", "key": "mgr/dashboard/ssl_server_port"}]: dispatch 2023-12-16T21:33:42.616 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2023-12-16T21:33:42.183+0000 7fda7ec19700 1 Processor -- start 2023-12-16T21:33:42.616 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2023-12-16T21:33:42.183+0000 7fda7ec19700 1 -- start start 2023-12-16T21:33:42.617 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2023-12-16T21:33:42.184+0000 7fda7ec19700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fda78076be0 0x7fda78104f20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:33:42.617 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2023-12-16T21:33:42.184+0000 7fda7ec19700 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fda78105460 con 0x7fda78076be0 2023-12-16T21:33:42.617 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2023-12-16T21:33:42.184+0000 7fda7c9b5700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fda78076be0 0x7fda78104f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:33:42.617 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2023-12-16T21:33:42.184+0000 7fda7c9b5700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fda78076be0 0x7fda78104f20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:57756/0 (socket says 172.21.15.45:57756) 2023-12-16T21:33:42.617 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2023-12-16T21:33:42.184+0000 7fda7c9b5700 1 -- 172.21.15.45:0/3159196820 learned_addr learned my addr 172.21.15.45:0/3159196820 (peer_addr_for_me v2:172.21.15.45:0/0) 2023-12-16T21:33:42.617 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2023-12-16T21:33:42.184+0000 7fda7c9b5700 1 -- 172.21.15.45:0/3159196820 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fda781055a0 con 0x7fda78076be0 2023-12-16T21:33:42.617 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2023-12-16T21:33:42.184+0000 7fda7c9b5700 1 --2- 172.21.15.45:0/3159196820 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fda78076be0 0x7fda78104f20 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7fda6c0098e0 tx=0x7fda6c009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=baec9b221bca4c67 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:33:42.617 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2023-12-16T21:33:42.184+0000 7fda777fe700 1 -- 172.21.15.45:0/3159196820 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fda6c00b860 con 0x7fda78076be0 2023-12-16T21:33:42.617 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2023-12-16T21:33:42.184+0000 7fda777fe700 1 -- 172.21.15.45:0/3159196820 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fda6c00be70 con 0x7fda78076be0 2023-12-16T21:33:42.617 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2023-12-16T21:33:42.184+0000 7fda777fe700 1 -- 172.21.15.45:0/3159196820 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fda6c010380 con 0x7fda78076be0 2023-12-16T21:33:42.617 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2023-12-16T21:33:42.185+0000 7fda7ec19700 1 -- 172.21.15.45:0/3159196820 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fda78076be0 msgr2=0x7fda78104f20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:33:42.617 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2023-12-16T21:33:42.185+0000 7fda7ec19700 1 --2- 172.21.15.45:0/3159196820 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fda78076be0 0x7fda78104f20 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7fda6c0098e0 tx=0x7fda6c009250 comp rx=0 tx=0).stop 2023-12-16T21:33:42.618 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2023-12-16T21:33:42.185+0000 7fda7ec19700 1 -- 172.21.15.45:0/3159196820 shutdown_connections 2023-12-16T21:33:42.618 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2023-12-16T21:33:42.185+0000 7fda7ec19700 1 --2- 172.21.15.45:0/3159196820 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fda78076be0 0x7fda78104f20 unknown :-1 s=CLOSED pgs=86 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:33:42.618 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2023-12-16T21:33:42.185+0000 7fda7ec19700 1 -- 172.21.15.45:0/3159196820 >> 172.21.15.45:0/3159196820 conn(0x7fda780fc950 msgr2=0x7fda780fed60 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-16T21:33:42.618 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2023-12-16T21:33:42.185+0000 7fda7ec19700 1 -- 172.21.15.45:0/3159196820 shutdown_connections 2023-12-16T21:33:42.618 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2023-12-16T21:33:42.185+0000 7fda7ec19700 1 -- 172.21.15.45:0/3159196820 wait complete. 2023-12-16T21:33:42.618 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2023-12-16T21:33:42.185+0000 7fda7ec19700 1 Processor -- start 2023-12-16T21:33:42.618 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2023-12-16T21:33:42.186+0000 7fda7ec19700 1 -- start start 2023-12-16T21:33:42.618 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2023-12-16T21:33:42.186+0000 7fda7ec19700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fda78076be0 0x7fda781111c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:33:42.618 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2023-12-16T21:33:42.186+0000 7fda7ec19700 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fda78111700 con 0x7fda78076be0 2023-12-16T21:33:42.618 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2023-12-16T21:33:42.186+0000 7fda7c9b5700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fda78076be0 0x7fda781111c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:33:42.618 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2023-12-16T21:33:42.186+0000 7fda7c9b5700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fda78076be0 0x7fda781111c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:57762/0 (socket says 172.21.15.45:57762) 2023-12-16T21:33:42.618 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2023-12-16T21:33:42.186+0000 7fda7c9b5700 1 -- 172.21.15.45:0/1721833416 learned_addr learned my addr 172.21.15.45:0/1721833416 (peer_addr_for_me v2:172.21.15.45:0/0) 2023-12-16T21:33:42.618 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2023-12-16T21:33:42.186+0000 7fda7c9b5700 1 -- 172.21.15.45:0/1721833416 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fda6c0094f0 con 0x7fda78076be0 2023-12-16T21:33:42.618 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2023-12-16T21:33:42.186+0000 7fda7c9b5700 1 --2- 172.21.15.45:0/1721833416 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fda78076be0 0x7fda781111c0 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7fda6c0094c0 tx=0x7fda6c009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:33:42.619 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2023-12-16T21:33:42.187+0000 7fda757fa700 1 -- 172.21.15.45:0/1721833416 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fda6c010850 con 0x7fda78076be0 2023-12-16T21:33:42.619 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2023-12-16T21:33:42.187+0000 7fda757fa700 1 -- 172.21.15.45:0/1721833416 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fda6c010e60 con 0x7fda78076be0 2023-12-16T21:33:42.619 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2023-12-16T21:33:42.187+0000 7fda7ec19700 1 -- 172.21.15.45:0/1721833416 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fda78111900 con 0x7fda78076be0 2023-12-16T21:33:42.619 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2023-12-16T21:33:42.187+0000 7fda757fa700 1 -- 172.21.15.45:0/1721833416 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fda6c01a430 con 0x7fda78076be0 2023-12-16T21:33:42.619 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2023-12-16T21:33:42.187+0000 7fda7ec19700 1 -- 172.21.15.45:0/1721833416 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fda78111d10 con 0x7fda78076be0 2023-12-16T21:33:42.619 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2023-12-16T21:33:42.187+0000 7fda757fa700 1 -- 172.21.15.45:0/1721833416 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 12) v1 ==== 43294+0+0 (secure 0 0 0) 0x7fda6c01a6b0 con 0x7fda78076be0 2023-12-16T21:33:42.619 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2023-12-16T21:33:42.188+0000 7fda757fa700 1 --2- 172.21.15.45:0/1721833416 >> [v2:172.21.15.45:6800/2215417738,v1:172.21.15.45:6801/2215417738] conn(0x7fda68036640 0x7fda68038af0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:33:42.619 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2023-12-16T21:33:42.188+0000 7fda757fa700 1 -- 172.21.15.45:0/1721833416 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1151+0+0 (secure 0 0 0) 0x7fda6c04a590 con 0x7fda78076be0 2023-12-16T21:33:42.619 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2023-12-16T21:33:42.188+0000 7fda77fff700 1 --2- 172.21.15.45:0/1721833416 >> [v2:172.21.15.45:6800/2215417738,v1:172.21.15.45:6801/2215417738] conn(0x7fda68036640 0x7fda68038af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:33:42.619 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2023-12-16T21:33:42.188+0000 7fda7ec19700 1 -- 172.21.15.45:0/1721833416 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fda7804bb80 con 0x7fda78076be0 2023-12-16T21:33:42.619 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2023-12-16T21:33:42.188+0000 7fda77fff700 1 --2- 172.21.15.45:0/1721833416 >> [v2:172.21.15.45:6800/2215417738,v1:172.21.15.45:6801/2215417738] conn(0x7fda68036640 0x7fda68038af0 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7fda64009990 tx=0x7fda64006e40 comp rx=0 tx=0).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:33:42.619 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2023-12-16T21:33:42.192+0000 7fda757fa700 1 -- 172.21.15.45:0/1721833416 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+174570 (secure 0 0 0) 0x7fda6c0189a0 con 0x7fda78076be0 2023-12-16T21:33:42.619 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2023-12-16T21:33:42.388+0000 7fda7ec19700 1 -- 172.21.15.45:0/1721833416 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command([{prefix=config set, name=osd_memory_target_autotune}] v 0) v1 -- 0x7fda781062c0 con 0x7fda78076be0 2023-12-16T21:33:42.620 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2023-12-16T21:33:42.388+0000 7fda757fa700 1 -- 172.21.15.45:0/1721833416 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mon_command_ack([{prefix=config set, name=osd_memory_target_autotune}]=0 v8)=0 v8) v1 ==== 127+0+0 (secure 0 0 0) 0x7fda6c023ed0 con 0x7fda78076be0 2023-12-16T21:33:42.620 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2023-12-16T21:33:42.390+0000 7fda7ec19700 1 -- 172.21.15.45:0/1721833416 >> [v2:172.21.15.45:6800/2215417738,v1:172.21.15.45:6801/2215417738] conn(0x7fda68036640 msgr2=0x7fda68038af0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:33:42.620 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2023-12-16T21:33:42.390+0000 7fda7ec19700 1 --2- 172.21.15.45:0/1721833416 >> [v2:172.21.15.45:6800/2215417738,v1:172.21.15.45:6801/2215417738] conn(0x7fda68036640 0x7fda68038af0 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7fda64009990 tx=0x7fda64006e40 comp rx=0 tx=0).stop 2023-12-16T21:33:42.620 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2023-12-16T21:33:42.390+0000 7fda7ec19700 1 -- 172.21.15.45:0/1721833416 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fda78076be0 msgr2=0x7fda781111c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:33:42.620 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2023-12-16T21:33:42.390+0000 7fda7ec19700 1 --2- 172.21.15.45:0/1721833416 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fda78076be0 0x7fda781111c0 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7fda6c0094c0 tx=0x7fda6c009250 comp rx=0 tx=0).stop 2023-12-16T21:33:42.620 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2023-12-16T21:33:42.390+0000 7fda7ec19700 1 -- 172.21.15.45:0/1721833416 shutdown_connections 2023-12-16T21:33:42.620 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2023-12-16T21:33:42.390+0000 7fda7ec19700 1 --2- 172.21.15.45:0/1721833416 >> [v2:172.21.15.45:6800/2215417738,v1:172.21.15.45:6801/2215417738] conn(0x7fda68036640 0x7fda68038af0 unknown :-1 s=CLOSED pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:33:42.620 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2023-12-16T21:33:42.390+0000 7fda7ec19700 1 --2- 172.21.15.45:0/1721833416 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fda78076be0 0x7fda781111c0 unknown :-1 s=CLOSED pgs=87 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:33:42.620 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2023-12-16T21:33:42.390+0000 7fda7ec19700 1 -- 172.21.15.45:0/1721833416 >> 172.21.15.45:0/1721833416 conn(0x7fda780fc950 msgr2=0x7fda780fed50 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-16T21:33:42.620 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2023-12-16T21:33:42.390+0000 7fda7ec19700 1 -- 172.21.15.45:0/1721833416 shutdown_connections 2023-12-16T21:33:42.620 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2023-12-16T21:33:42.390+0000 7fda7ec19700 1 -- 172.21.15.45:0/1721833416 wait complete. 2023-12-16T21:33:43.468 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2023-12-16T21:33:42.899+0000 7fdaf8d2e700 1 Processor -- start 2023-12-16T21:33:43.468 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2023-12-16T21:33:42.900+0000 7fdaf8d2e700 1 -- start start 2023-12-16T21:33:43.468 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2023-12-16T21:33:42.900+0000 7fdaf8d2e700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fdaf41011d0 0x7fdaf41015e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:33:43.468 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2023-12-16T21:33:42.900+0000 7fdaf8d2e700 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdaf4101b20 con 0x7fdaf41011d0 2023-12-16T21:33:43.468 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2023-12-16T21:33:42.900+0000 7fdaf259c700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fdaf41011d0 0x7fdaf41015e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:33:43.468 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2023-12-16T21:33:42.900+0000 7fdaf259c700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fdaf41011d0 0x7fdaf41015e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:55188/0 (socket says 172.21.15.45:55188) 2023-12-16T21:33:43.468 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2023-12-16T21:33:42.900+0000 7fdaf259c700 1 -- 172.21.15.45:0/2573030256 learned_addr learned my addr 172.21.15.45:0/2573030256 (peer_addr_for_me v2:172.21.15.45:0/0) 2023-12-16T21:33:43.469 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2023-12-16T21:33:42.900+0000 7fdaf259c700 1 -- 172.21.15.45:0/2573030256 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdaf4101c60 con 0x7fdaf41011d0 2023-12-16T21:33:43.469 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2023-12-16T21:33:42.901+0000 7fdaf259c700 1 --2- 172.21.15.45:0/2573030256 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fdaf41011d0 0x7fdaf41015e0 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7fdae400bb80 tx=0x7fdae400b4f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3b7824aeab80e7e3 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:33:43.469 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2023-12-16T21:33:42.901+0000 7fdaf159a700 1 -- 172.21.15.45:0/2573030256 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fdae400dd00 con 0x7fdaf41011d0 2023-12-16T21:33:43.469 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2023-12-16T21:33:42.901+0000 7fdaf159a700 1 -- 172.21.15.45:0/2573030256 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fdae400de60 con 0x7fdaf41011d0 2023-12-16T21:33:43.469 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2023-12-16T21:33:42.901+0000 7fdaf159a700 1 -- 172.21.15.45:0/2573030256 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fdae40188d0 con 0x7fdaf41011d0 2023-12-16T21:33:43.469 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2023-12-16T21:33:42.901+0000 7fdaf8d2e700 1 -- 172.21.15.45:0/2573030256 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fdaf41011d0 msgr2=0x7fdaf41015e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:33:43.469 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2023-12-16T21:33:42.901+0000 7fdaf8d2e700 1 --2- 172.21.15.45:0/2573030256 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fdaf41011d0 0x7fdaf41015e0 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7fdae400bb80 tx=0x7fdae400b4f0 comp rx=0 tx=0).stop 2023-12-16T21:33:43.469 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2023-12-16T21:33:42.901+0000 7fdaf8d2e700 1 -- 172.21.15.45:0/2573030256 shutdown_connections 2023-12-16T21:33:43.469 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2023-12-16T21:33:42.901+0000 7fdaf8d2e700 1 --2- 172.21.15.45:0/2573030256 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fdaf41011d0 0x7fdaf41015e0 unknown :-1 s=CLOSED pgs=88 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:33:43.469 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2023-12-16T21:33:42.901+0000 7fdaf8d2e700 1 -- 172.21.15.45:0/2573030256 >> 172.21.15.45:0/2573030256 conn(0x7fdaf40fc950 msgr2=0x7fdaf40fed60 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-16T21:33:43.469 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2023-12-16T21:33:42.902+0000 7fdaf8d2e700 1 -- 172.21.15.45:0/2573030256 shutdown_connections 2023-12-16T21:33:43.469 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2023-12-16T21:33:42.902+0000 7fdaf8d2e700 1 -- 172.21.15.45:0/2573030256 wait complete. 2023-12-16T21:33:43.470 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2023-12-16T21:33:42.902+0000 7fdaf8d2e700 1 Processor -- start 2023-12-16T21:33:43.470 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2023-12-16T21:33:42.902+0000 7fdaf8d2e700 1 -- start start 2023-12-16T21:33:43.470 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2023-12-16T21:33:42.902+0000 7fdaf8d2e700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fdaf41011d0 0x7fdaf410ef30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:33:43.470 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2023-12-16T21:33:42.902+0000 7fdaf8d2e700 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdaf410f470 con 0x7fdaf41011d0 2023-12-16T21:33:43.470 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2023-12-16T21:33:42.902+0000 7fdaf259c700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fdaf41011d0 0x7fdaf410ef30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:33:43.470 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2023-12-16T21:33:42.902+0000 7fdaf259c700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fdaf41011d0 0x7fdaf410ef30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:55192/0 (socket says 172.21.15.45:55192) 2023-12-16T21:33:43.470 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2023-12-16T21:33:42.902+0000 7fdaf259c700 1 -- 172.21.15.45:0/195932510 learned_addr learned my addr 172.21.15.45:0/195932510 (peer_addr_for_me v2:172.21.15.45:0/0) 2023-12-16T21:33:43.470 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2023-12-16T21:33:42.903+0000 7fdaf259c700 1 -- 172.21.15.45:0/195932510 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdae400b790 con 0x7fdaf41011d0 2023-12-16T21:33:43.470 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2023-12-16T21:33:42.903+0000 7fdaf259c700 1 --2- 172.21.15.45:0/195932510 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fdaf41011d0 0x7fdaf410ef30 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7fdae40184f0 tx=0x7fdae400b4f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:33:43.470 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2023-12-16T21:33:42.903+0000 7fdae2ffd700 1 -- 172.21.15.45:0/195932510 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fdae4010bd0 con 0x7fdaf41011d0 2023-12-16T21:33:43.470 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2023-12-16T21:33:42.903+0000 7fdae2ffd700 1 -- 172.21.15.45:0/195932510 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fdae401d050 con 0x7fdaf41011d0 2023-12-16T21:33:43.470 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2023-12-16T21:33:42.903+0000 7fdaf8d2e700 1 -- 172.21.15.45:0/195932510 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdaf410f670 con 0x7fdaf41011d0 2023-12-16T21:33:43.470 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2023-12-16T21:33:42.903+0000 7fdae2ffd700 1 -- 172.21.15.45:0/195932510 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fdae4022590 con 0x7fdaf41011d0 2023-12-16T21:33:43.470 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2023-12-16T21:33:42.903+0000 7fdaf8d2e700 1 -- 172.21.15.45:0/195932510 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdaf410fa80 con 0x7fdaf41011d0 2023-12-16T21:33:43.471 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2023-12-16T21:33:42.904+0000 7fdae2ffd700 1 -- 172.21.15.45:0/195932510 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 12) v1 ==== 43294+0+0 (secure 0 0 0) 0x7fdae4022800 con 0x7fdaf41011d0 2023-12-16T21:33:43.471 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2023-12-16T21:33:42.904+0000 7fdaf8d2e700 1 -- 172.21.15.45:0/195932510 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdaf404bb80 con 0x7fdaf41011d0 2023-12-16T21:33:43.471 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2023-12-16T21:33:42.904+0000 7fdae2ffd700 1 --2- 172.21.15.45:0/195932510 >> [v2:172.21.15.45:6800/2215417738,v1:172.21.15.45:6801/2215417738] conn(0x7fdadc036690 0x7fdadc038b40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:33:43.471 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2023-12-16T21:33:42.904+0000 7fdaf1d9b700 1 --2- 172.21.15.45:0/195932510 >> [v2:172.21.15.45:6800/2215417738,v1:172.21.15.45:6801/2215417738] conn(0x7fdadc036690 0x7fdadc038b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:33:43.471 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2023-12-16T21:33:42.904+0000 7fdae2ffd700 1 -- 172.21.15.45:0/195932510 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1151+0+0 (secure 0 0 0) 0x7fdae4014070 con 0x7fdaf41011d0 2023-12-16T21:33:43.471 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2023-12-16T21:33:42.905+0000 7fdaf1d9b700 1 --2- 172.21.15.45:0/195932510 >> [v2:172.21.15.45:6800/2215417738,v1:172.21.15.45:6801/2215417738] conn(0x7fdadc036690 0x7fdadc038b40 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7fdae8009990 tx=0x7fdae8006e40 comp rx=0 tx=0).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:33:43.471 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2023-12-16T21:33:42.909+0000 7fdae2ffd700 1 -- 172.21.15.45:0/195932510 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+174570 (secure 0 0 0) 0x7fdae4021d90 con 0x7fdaf41011d0 2023-12-16T21:33:43.471 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2023-12-16T21:33:43.225+0000 7fdaf8d2e700 1 -- 172.21.15.45:0/195932510 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command([{prefix=config-key set, key=mgr/dashboard/cluster/status}] v 0) v1 -- 0x7fdaf4102fb0 con 0x7fdaf41011d0 2023-12-16T21:33:43.471 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2023-12-16T21:33:43.228+0000 7fdae2ffd700 1 -- 172.21.15.45:0/195932510 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mon_command_ack([{prefix=config-key set, key=mgr/dashboard/cluster/status}]=0 set mgr/dashboard/cluster/status v33)=0 set mgr/dashboard/cluster/status v33) v1 ==== 153+0+0 (secure 0 0 0) 0x7fdae402be90 con 0x7fdaf41011d0 2023-12-16T21:33:43.471 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2023-12-16T21:33:43.230+0000 7fdaf8d2e700 1 -- 172.21.15.45:0/195932510 >> [v2:172.21.15.45:6800/2215417738,v1:172.21.15.45:6801/2215417738] conn(0x7fdadc036690 msgr2=0x7fdadc038b40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:33:43.472 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2023-12-16T21:33:43.230+0000 7fdaf8d2e700 1 --2- 172.21.15.45:0/195932510 >> [v2:172.21.15.45:6800/2215417738,v1:172.21.15.45:6801/2215417738] conn(0x7fdadc036690 0x7fdadc038b40 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7fdae8009990 tx=0x7fdae8006e40 comp rx=0 tx=0).stop 2023-12-16T21:33:43.472 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2023-12-16T21:33:43.230+0000 7fdaf8d2e700 1 -- 172.21.15.45:0/195932510 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fdaf41011d0 msgr2=0x7fdaf410ef30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:33:43.472 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2023-12-16T21:33:43.230+0000 7fdaf8d2e700 1 --2- 172.21.15.45:0/195932510 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fdaf41011d0 0x7fdaf410ef30 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7fdae40184f0 tx=0x7fdae400b4f0 comp rx=0 tx=0).stop 2023-12-16T21:33:43.472 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2023-12-16T21:33:43.230+0000 7fdaf8d2e700 1 -- 172.21.15.45:0/195932510 shutdown_connections 2023-12-16T21:33:43.472 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2023-12-16T21:33:43.230+0000 7fdaf8d2e700 1 --2- 172.21.15.45:0/195932510 >> [v2:172.21.15.45:6800/2215417738,v1:172.21.15.45:6801/2215417738] conn(0x7fdadc036690 0x7fdadc038b40 unknown :-1 s=CLOSED pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:33:43.472 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2023-12-16T21:33:43.230+0000 7fdaf8d2e700 1 --2- 172.21.15.45:0/195932510 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fdaf41011d0 0x7fdaf410ef30 unknown :-1 s=CLOSED pgs=89 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:33:43.472 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2023-12-16T21:33:43.230+0000 7fdaf8d2e700 1 -- 172.21.15.45:0/195932510 >> 172.21.15.45:0/195932510 conn(0x7fdaf40fc950 msgr2=0x7fdaf4105f40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-16T21:33:43.472 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2023-12-16T21:33:43.230+0000 7fdaf8d2e700 1 -- 172.21.15.45:0/195932510 shutdown_connections 2023-12-16T21:33:43.472 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr 2023-12-16T21:33:43.230+0000 7fdaf8d2e700 1 -- 172.21.15.45:0/195932510 wait complete. 2023-12-16T21:33:43.472 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr set mgr/dashboard/cluster/status 2023-12-16T21:33:43.472 INFO:teuthology.orchestra.run.smithi045.stdout:You can access the Ceph CLI as following in case of multi-cluster or non-default config: 2023-12-16T21:33:43.472 INFO:teuthology.orchestra.run.smithi045.stdout: 2023-12-16T21:33:43.472 INFO:teuthology.orchestra.run.smithi045.stdout: sudo /home/ubuntu/cephtest/cephadm shell --fsid 88ff363c-9c5a-11ee-95a5-87774f69a715 -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring 2023-12-16T21:33:43.472 INFO:teuthology.orchestra.run.smithi045.stdout: 2023-12-16T21:33:43.473 INFO:teuthology.orchestra.run.smithi045.stdout:Or, if you are only running a single cluster on this host: 2023-12-16T21:33:43.473 INFO:teuthology.orchestra.run.smithi045.stdout: 2023-12-16T21:33:43.473 INFO:teuthology.orchestra.run.smithi045.stdout: sudo /home/ubuntu/cephtest/cephadm shell 2023-12-16T21:33:43.473 INFO:teuthology.orchestra.run.smithi045.stdout: 2023-12-16T21:33:43.473 INFO:teuthology.orchestra.run.smithi045.stdout:Please consider enabling telemetry to help improve Ceph: 2023-12-16T21:33:43.473 INFO:teuthology.orchestra.run.smithi045.stdout: 2023-12-16T21:33:43.473 INFO:teuthology.orchestra.run.smithi045.stdout: ceph telemetry on 2023-12-16T21:33:43.473 INFO:teuthology.orchestra.run.smithi045.stdout: 2023-12-16T21:33:43.473 INFO:teuthology.orchestra.run.smithi045.stdout:For more information see: 2023-12-16T21:33:43.473 INFO:teuthology.orchestra.run.smithi045.stdout: 2023-12-16T21:33:43.473 INFO:teuthology.orchestra.run.smithi045.stdout: https://docs.ceph.com/docs/master/mgr/telemetry/ 2023-12-16T21:33:43.473 INFO:teuthology.orchestra.run.smithi045.stdout: 2023-12-16T21:33:43.473 INFO:teuthology.orchestra.run.smithi045.stdout:Bootstrap complete. 2023-12-16T21:33:43.508 INFO:tasks.cephadm:Fetching config... 2023-12-16T21:33:43.508 DEBUG:teuthology.orchestra.run.smithi045:> set -ex 2023-12-16T21:33:43.508 DEBUG:teuthology.orchestra.run.smithi045:> dd if=/etc/ceph/ceph.conf of=/dev/stdout 2023-12-16T21:33:43.563 INFO:tasks.cephadm:Fetching client.admin keyring... 2023-12-16T21:33:43.564 DEBUG:teuthology.orchestra.run.smithi045:> set -ex 2023-12-16T21:33:43.564 DEBUG:teuthology.orchestra.run.smithi045:> dd if=/etc/ceph/ceph.client.admin.keyring of=/dev/stdout 2023-12-16T21:33:43.619 INFO:tasks.cephadm:Fetching mon keyring... 2023-12-16T21:33:43.619 DEBUG:teuthology.orchestra.run.smithi045:> set -ex 2023-12-16T21:33:43.619 DEBUG:teuthology.orchestra.run.smithi045:> sudo dd if=/var/lib/ceph/88ff363c-9c5a-11ee-95a5-87774f69a715/mon.smithi045/keyring of=/dev/stdout 2023-12-16T21:33:43.692 INFO:tasks.cephadm:Fetching pub ssh key... 2023-12-16T21:33:43.692 DEBUG:teuthology.orchestra.run.smithi045:> set -ex 2023-12-16T21:33:43.693 DEBUG:teuthology.orchestra.run.smithi045:> dd if=/home/ubuntu/cephtest/ceph.pub of=/dev/stdout 2023-12-16T21:33:43.748 INFO:tasks.cephadm:Installing pub ssh key for root users... 2023-12-16T21:33:43.749 DEBUG:teuthology.orchestra.run.smithi045:> sudo install -d -m 0700 /root/.ssh && echo 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDY/F0Hv6LE/uJ1jo40L0WUNThVihgYwfPy+mLt2OQbJWonIqKHi+B4LiwYorryFM9Fy0fuR5Rwyf230hg1kl1sZmhdvsmFa2huLbc9543VlR8SlXiLiQJXCvjE9Gm2Ba3h2W7LhS1+S8ZOTJOhms6VTb7IO2WF8OofHyymCM8a7zO0N2x8f+Khkh7/IoBI8cDQzHXB73NqS4+ONXOhoGPAwyvgp3EJXcy23eKD1Ss9lml8XU3Ek5oRsfhSeyIXJHicj6EFwHnZNuL8dooEwEdiaJ00Oeunrp5EksuK7TblA1lfDpZT3gMWvY+PP92eKruWbC4Ai7ZAAg/L+WKpj5/za4+cQqSCtwmaT6cWsavGVY0SwVQmyl5z/E3JLDqyPAMH0YnHE+vcn/0VFIUbLMt//8c3LOTwBnTGFSFz5uThRxxOdNAKXZouRU4nt7+L/gl+TWhovSyoBdNPS0lHmIo7jDc1QaJVpt5cmAvN0zvh4qRdsNEUV13xQrq6wPqDgyM= ceph-88ff363c-9c5a-11ee-95a5-87774f69a715' | sudo tee -a /root/.ssh/authorized_keys && sudo chmod 0600 /root/.ssh/authorized_keys 2023-12-16T21:33:43.835 INFO:teuthology.orchestra.run.smithi045.stdout:ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDY/F0Hv6LE/uJ1jo40L0WUNThVihgYwfPy+mLt2OQbJWonIqKHi+B4LiwYorryFM9Fy0fuR5Rwyf230hg1kl1sZmhdvsmFa2huLbc9543VlR8SlXiLiQJXCvjE9Gm2Ba3h2W7LhS1+S8ZOTJOhms6VTb7IO2WF8OofHyymCM8a7zO0N2x8f+Khkh7/IoBI8cDQzHXB73NqS4+ONXOhoGPAwyvgp3EJXcy23eKD1Ss9lml8XU3Ek5oRsfhSeyIXJHicj6EFwHnZNuL8dooEwEdiaJ00Oeunrp5EksuK7TblA1lfDpZT3gMWvY+PP92eKruWbC4Ai7ZAAg/L+WKpj5/za4+cQqSCtwmaT6cWsavGVY0SwVQmyl5z/E3JLDqyPAMH0YnHE+vcn/0VFIUbLMt//8c3LOTwBnTGFSFz5uThRxxOdNAKXZouRU4nt7+L/gl+TWhovSyoBdNPS0lHmIo7jDc1QaJVpt5cmAvN0zvh4qRdsNEUV13xQrq6wPqDgyM= ceph-88ff363c-9c5a-11ee-95a5-87774f69a715 2023-12-16T21:33:43.852 DEBUG:teuthology.orchestra.run.smithi169:> sudo install -d -m 0700 /root/.ssh && echo 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDY/F0Hv6LE/uJ1jo40L0WUNThVihgYwfPy+mLt2OQbJWonIqKHi+B4LiwYorryFM9Fy0fuR5Rwyf230hg1kl1sZmhdvsmFa2huLbc9543VlR8SlXiLiQJXCvjE9Gm2Ba3h2W7LhS1+S8ZOTJOhms6VTb7IO2WF8OofHyymCM8a7zO0N2x8f+Khkh7/IoBI8cDQzHXB73NqS4+ONXOhoGPAwyvgp3EJXcy23eKD1Ss9lml8XU3Ek5oRsfhSeyIXJHicj6EFwHnZNuL8dooEwEdiaJ00Oeunrp5EksuK7TblA1lfDpZT3gMWvY+PP92eKruWbC4Ai7ZAAg/L+WKpj5/za4+cQqSCtwmaT6cWsavGVY0SwVQmyl5z/E3JLDqyPAMH0YnHE+vcn/0VFIUbLMt//8c3LOTwBnTGFSFz5uThRxxOdNAKXZouRU4nt7+L/gl+TWhovSyoBdNPS0lHmIo7jDc1QaJVpt5cmAvN0zvh4qRdsNEUV13xQrq6wPqDgyM= ceph-88ff363c-9c5a-11ee-95a5-87774f69a715' | sudo tee -a /root/.ssh/authorized_keys && sudo chmod 0600 /root/.ssh/authorized_keys 2023-12-16T21:33:43.898 INFO:teuthology.orchestra.run.smithi169.stdout:ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDY/F0Hv6LE/uJ1jo40L0WUNThVihgYwfPy+mLt2OQbJWonIqKHi+B4LiwYorryFM9Fy0fuR5Rwyf230hg1kl1sZmhdvsmFa2huLbc9543VlR8SlXiLiQJXCvjE9Gm2Ba3h2W7LhS1+S8ZOTJOhms6VTb7IO2WF8OofHyymCM8a7zO0N2x8f+Khkh7/IoBI8cDQzHXB73NqS4+ONXOhoGPAwyvgp3EJXcy23eKD1Ss9lml8XU3Ek5oRsfhSeyIXJHicj6EFwHnZNuL8dooEwEdiaJ00Oeunrp5EksuK7TblA1lfDpZT3gMWvY+PP92eKruWbC4Ai7ZAAg/L+WKpj5/za4+cQqSCtwmaT6cWsavGVY0SwVQmyl5z/E3JLDqyPAMH0YnHE+vcn/0VFIUbLMt//8c3LOTwBnTGFSFz5uThRxxOdNAKXZouRU4nt7+L/gl+TWhovSyoBdNPS0lHmIo7jDc1QaJVpt5cmAvN0zvh4qRdsNEUV13xQrq6wPqDgyM= ceph-88ff363c-9c5a-11ee-95a5-87774f69a715 2023-12-16T21:33:43.913 DEBUG:teuthology.orchestra.run.smithi045:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 88ff363c-9c5a-11ee-95a5-87774f69a715 -- ceph config set mgr mgr/cephadm/allow_ptrace true 2023-12-16T21:33:44.177 INFO:teuthology.orchestra.run.smithi045.stderr:Inferring config /var/lib/ceph/88ff363c-9c5a-11ee-95a5-87774f69a715/mon.smithi045/config 2023-12-16T21:33:44.334 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Dec 16 21:33:44 smithi045 ceph-mon[80763]: from='client.? 172.21.15.45:0/195932510' entity='client.admin' 2023-12-16T21:33:45.334 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:33:45.329+0000 7fee4b46d700 1 -- 172.21.15.45:0/3508875275 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fee44066490 msgr2=0x7fee440ff450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:33:45.334 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:33:45.329+0000 7fee4b46d700 1 --2- 172.21.15.45:0/3508875275 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fee44066490 0x7fee440ff450 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7fee400048c0 tx=0x7fee40009e30 comp rx=0 tx=0).stop 2023-12-16T21:33:45.334 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:33:45.329+0000 7fee4b46d700 1 -- 172.21.15.45:0/3508875275 shutdown_connections 2023-12-16T21:33:45.334 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:33:45.329+0000 7fee4b46d700 1 --2- 172.21.15.45:0/3508875275 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fee44066490 0x7fee440ff450 unknown :-1 s=CLOSED pgs=90 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:33:45.334 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:33:45.329+0000 7fee4b46d700 1 -- 172.21.15.45:0/3508875275 >> 172.21.15.45:0/3508875275 conn(0x7fee440f6ec0 msgr2=0x7fee440f92f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-16T21:33:45.334 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:33:45.330+0000 7fee4b46d700 1 -- 172.21.15.45:0/3508875275 shutdown_connections 2023-12-16T21:33:45.334 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:33:45.330+0000 7fee4b46d700 1 -- 172.21.15.45:0/3508875275 wait complete. 2023-12-16T21:33:45.334 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:33:45.330+0000 7fee4b46d700 1 Processor -- start 2023-12-16T21:33:45.334 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:33:45.330+0000 7fee4b46d700 1 -- start start 2023-12-16T21:33:45.334 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:33:45.330+0000 7fee4b46d700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fee44066490 0x7fee4410ca90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:33:45.334 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:33:45.330+0000 7fee4b46d700 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fee4410cfd0 con 0x7fee44066490 2023-12-16T21:33:45.334 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:33:45.330+0000 7fee49209700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fee44066490 0x7fee4410ca90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:33:45.335 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:33:45.330+0000 7fee49209700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fee44066490 0x7fee4410ca90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:55202/0 (socket says 172.21.15.45:55202) 2023-12-16T21:33:45.335 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:33:45.330+0000 7fee49209700 1 -- 172.21.15.45:0/3524658377 learned_addr learned my addr 172.21.15.45:0/3524658377 (peer_addr_for_me v2:172.21.15.45:0/0) 2023-12-16T21:33:45.335 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:33:45.331+0000 7fee49209700 1 -- 172.21.15.45:0/3524658377 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fee40005040 con 0x7fee44066490 2023-12-16T21:33:45.335 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:33:45.331+0000 7fee49209700 1 --2- 172.21.15.45:0/3524658377 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fee44066490 0x7fee4410ca90 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7fee40009330 tx=0x7fee40009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:33:45.335 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:33:45.331+0000 7fee39ffb700 1 -- 172.21.15.45:0/3524658377 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fee400096b0 con 0x7fee44066490 2023-12-16T21:33:45.335 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:33:45.331+0000 7fee39ffb700 1 -- 172.21.15.45:0/3524658377 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fee40009810 con 0x7fee44066490 2023-12-16T21:33:45.335 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:33:45.331+0000 7fee39ffb700 1 -- 172.21.15.45:0/3524658377 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fee4001c5b0 con 0x7fee44066490 2023-12-16T21:33:45.335 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:33:45.331+0000 7fee4b46d700 1 -- 172.21.15.45:0/3524658377 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fee4410d1d0 con 0x7fee44066490 2023-12-16T21:33:45.335 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:33:45.332+0000 7fee4b46d700 1 -- 172.21.15.45:0/3524658377 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fee4410de70 con 0x7fee44066490 2023-12-16T21:33:45.336 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:33:45.332+0000 7fee39ffb700 1 -- 172.21.15.45:0/3524658377 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 13) v1 ==== 43337+0+0 (secure 0 0 0) 0x7fee4001c820 con 0x7fee44066490 2023-12-16T21:33:45.336 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:33:45.333+0000 7fee4b46d700 1 -- 172.21.15.45:0/3524658377 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fee4405f270 con 0x7fee44066490 2023-12-16T21:33:45.336 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:33:45.333+0000 7fee39ffb700 1 --2- 172.21.15.45:0/3524658377 >> [v2:172.21.15.45:6800/2215417738,v1:172.21.15.45:6801/2215417738] conn(0x7fee30036720 0x7fee30038bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:33:45.336 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:33:45.333+0000 7fee39ffb700 1 -- 172.21.15.45:0/3524658377 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1151+0+0 (secure 0 0 0) 0x7fee40049990 con 0x7fee44066490 2023-12-16T21:33:45.336 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:33:45.333+0000 7fee48a08700 1 --2- 172.21.15.45:0/3524658377 >> [v2:172.21.15.45:6800/2215417738,v1:172.21.15.45:6801/2215417738] conn(0x7fee30036720 0x7fee30038bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:33:45.336 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:33:45.333+0000 7fee48a08700 1 --2- 172.21.15.45:0/3524658377 >> [v2:172.21.15.45:6800/2215417738,v1:172.21.15.45:6801/2215417738] conn(0x7fee30036720 0x7fee30038bd0 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7fee34009990 tx=0x7fee34006e40 comp rx=0 tx=0).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:33:45.340 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:33:45.338+0000 7fee39ffb700 1 -- 172.21.15.45:0/3524658377 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+174570 (secure 0 0 0) 0x7fee40023300 con 0x7fee44066490 2023-12-16T21:33:45.470 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Dec 16 21:33:45 smithi045 ceph-mon[80763]: mgrmap e13: smithi045.nhtmwc(active, since 6s) 2023-12-16T21:33:45.537 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:33:45.533+0000 7fee4b46d700 1 -- 172.21.15.45:0/3524658377 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command([{prefix=config set, name=mgr/cephadm/allow_ptrace}] v 0) v1 -- 0x7fee4404bb80 con 0x7fee44066490 2023-12-16T21:33:45.539 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:33:45.536+0000 7fee39ffb700 1 -- 172.21.15.45:0/3524658377 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/cephadm/allow_ptrace}]=0 v9)=0 v9) v1 ==== 125+0+0 (secure 0 0 0) 0x7fee40011b60 con 0x7fee44066490 2023-12-16T21:33:45.542 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:33:45.539+0000 7fee4b46d700 1 -- 172.21.15.45:0/3524658377 >> [v2:172.21.15.45:6800/2215417738,v1:172.21.15.45:6801/2215417738] conn(0x7fee30036720 msgr2=0x7fee30038bd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:33:45.542 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:33:45.539+0000 7fee4b46d700 1 --2- 172.21.15.45:0/3524658377 >> [v2:172.21.15.45:6800/2215417738,v1:172.21.15.45:6801/2215417738] conn(0x7fee30036720 0x7fee30038bd0 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7fee34009990 tx=0x7fee34006e40 comp rx=0 tx=0).stop 2023-12-16T21:33:45.543 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:33:45.539+0000 7fee4b46d700 1 -- 172.21.15.45:0/3524658377 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fee44066490 msgr2=0x7fee4410ca90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:33:45.543 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:33:45.539+0000 7fee4b46d700 1 --2- 172.21.15.45:0/3524658377 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fee44066490 0x7fee4410ca90 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7fee40009330 tx=0x7fee40009e30 comp rx=0 tx=0).stop 2023-12-16T21:33:45.543 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:33:45.539+0000 7fee4b46d700 1 -- 172.21.15.45:0/3524658377 shutdown_connections 2023-12-16T21:33:45.543 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:33:45.539+0000 7fee4b46d700 1 --2- 172.21.15.45:0/3524658377 >> [v2:172.21.15.45:6800/2215417738,v1:172.21.15.45:6801/2215417738] conn(0x7fee30036720 0x7fee30038bd0 unknown :-1 s=CLOSED pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:33:45.543 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:33:45.539+0000 7fee4b46d700 1 --2- 172.21.15.45:0/3524658377 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fee44066490 0x7fee4410ca90 unknown :-1 s=CLOSED pgs=91 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:33:45.543 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:33:45.539+0000 7fee4b46d700 1 -- 172.21.15.45:0/3524658377 >> 172.21.15.45:0/3524658377 conn(0x7fee440f6ec0 msgr2=0x7fee440f7c50 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-16T21:33:45.543 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:33:45.539+0000 7fee4b46d700 1 -- 172.21.15.45:0/3524658377 shutdown_connections 2023-12-16T21:33:45.543 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:33:45.539+0000 7fee4b46d700 1 -- 172.21.15.45:0/3524658377 wait complete. 2023-12-16T21:33:45.852 INFO:tasks.cephadm:Distributing conf and client.admin keyring to all hosts + 0755 2023-12-16T21:33:45.853 DEBUG:teuthology.orchestra.run.smithi045:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 88ff363c-9c5a-11ee-95a5-87774f69a715 -- ceph orch client-keyring set client.admin '*' --mode 0755 2023-12-16T21:33:46.177 INFO:teuthology.orchestra.run.smithi045.stderr:Inferring config /var/lib/ceph/88ff363c-9c5a-11ee-95a5-87774f69a715/mon.smithi045/config 2023-12-16T21:33:46.633 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Dec 16 21:33:46 smithi045 ceph-mon[80763]: from='client.? 172.21.15.45:0/3524658377' entity='client.admin' 2023-12-16T21:33:46.633 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Dec 16 21:33:46 smithi045 ceph-mon[80763]: from='mgr.14162 172.21.15.45:0/142909557' entity='mgr.smithi045.nhtmwc' 2023-12-16T21:33:46.633 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Dec 16 21:33:46 smithi045 ceph-mon[80763]: from='mgr.14162 172.21.15.45:0/142909557' entity='mgr.smithi045.nhtmwc' 2023-12-16T21:33:46.633 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Dec 16 21:33:46 smithi045 ceph-mon[80763]: from='mgr.14162 172.21.15.45:0/142909557' entity='mgr.smithi045.nhtmwc' cmd=[{"prefix": "config rm", "who": "osd/host:smithi045", "name": "osd_memory_target"}]: dispatch 2023-12-16T21:33:46.634 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Dec 16 21:33:46 smithi045 ceph-mon[80763]: from='mgr.14162 172.21.15.45:0/142909557' entity='mgr.smithi045.nhtmwc' 2023-12-16T21:33:46.634 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Dec 16 21:33:46 smithi045 ceph-mon[80763]: from='mgr.14162 172.21.15.45:0/142909557' entity='mgr.smithi045.nhtmwc' cmd=[{"prefix": "auth get-or-create", "entity": "client.crash.smithi045", "caps": ["mon", "profile crash", "mgr", "profile crash"]}]: dispatch 2023-12-16T21:33:46.634 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Dec 16 21:33:46 smithi045 ceph-mon[80763]: from='mgr.14162 172.21.15.45:0/142909557' entity='mgr.smithi045.nhtmwc' cmd='[{"prefix": "auth get-or-create", "entity": "client.crash.smithi045", "caps": ["mon", "profile crash", "mgr", "profile crash"]}]': finished 2023-12-16T21:33:46.634 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Dec 16 21:33:46 smithi045 ceph-mon[80763]: from='mgr.14162 172.21.15.45:0/142909557' entity='mgr.smithi045.nhtmwc' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-12-16T21:33:46.634 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Dec 16 21:33:46 smithi045 ceph-mon[80763]: Deploying daemon crash.smithi045 on smithi045 2023-12-16T21:33:47.153 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:33:47.149+0000 7f0bed458700 1 -- 172.21.15.45:0/836604325 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f0be80fc880 msgr2=0x7f0be80fcc90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:33:47.153 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:33:47.149+0000 7f0bed458700 1 --2- 172.21.15.45:0/836604325 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f0be80fc880 0x7f0be80fcc90 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7f0bd80048c0 tx=0x7f0bd8009e30 comp rx=0 tx=0).stop 2023-12-16T21:33:47.153 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:33:47.149+0000 7f0bed458700 1 -- 172.21.15.45:0/836604325 shutdown_connections 2023-12-16T21:33:47.153 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:33:47.149+0000 7f0bed458700 1 --2- 172.21.15.45:0/836604325 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f0be80fc880 0x7f0be80fcc90 unknown :-1 s=CLOSED pgs=92 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:33:47.153 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:33:47.149+0000 7f0bed458700 1 -- 172.21.15.45:0/836604325 >> 172.21.15.45:0/836604325 conn(0x7f0be80f7f40 msgr2=0x7f0be80fa370 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-16T21:33:47.153 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:33:47.150+0000 7f0bed458700 1 -- 172.21.15.45:0/836604325 shutdown_connections 2023-12-16T21:33:47.153 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:33:47.150+0000 7f0bed458700 1 -- 172.21.15.45:0/836604325 wait complete. 2023-12-16T21:33:47.155 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:33:47.150+0000 7f0bed458700 1 Processor -- start 2023-12-16T21:33:47.155 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:33:47.150+0000 7f0bed458700 1 -- start start 2023-12-16T21:33:47.155 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:33:47.150+0000 7f0bed458700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f0be80fc880 0x7f0be810cac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:33:47.155 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:33:47.150+0000 7f0bed458700 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0be810d000 con 0x7f0be80fc880 2023-12-16T21:33:47.155 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:33:47.151+0000 7f0be6ffd700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f0be80fc880 0x7f0be810cac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:33:47.155 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:33:47.151+0000 7f0be6ffd700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f0be80fc880 0x7f0be810cac0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:55228/0 (socket says 172.21.15.45:55228) 2023-12-16T21:33:47.155 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:33:47.151+0000 7f0be6ffd700 1 -- 172.21.15.45:0/3834928997 learned_addr learned my addr 172.21.15.45:0/3834928997 (peer_addr_for_me v2:172.21.15.45:0/0) 2023-12-16T21:33:47.155 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:33:47.151+0000 7f0be6ffd700 1 -- 172.21.15.45:0/3834928997 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0bd8005040 con 0x7f0be80fc880 2023-12-16T21:33:47.155 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:33:47.151+0000 7f0be6ffd700 1 --2- 172.21.15.45:0/3834928997 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f0be80fc880 0x7f0be810cac0 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7f0bd800b690 tx=0x7f0bd8009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:33:47.155 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:33:47.151+0000 7f0bcf7fe700 1 -- 172.21.15.45:0/3834928997 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f0bd8009650 con 0x7f0be80fc880 2023-12-16T21:33:47.155 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:33:47.151+0000 7f0bcf7fe700 1 -- 172.21.15.45:0/3834928997 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f0bd80097b0 con 0x7f0be80fc880 2023-12-16T21:33:47.155 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:33:47.151+0000 7f0bed458700 1 -- 172.21.15.45:0/3834928997 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0be810d200 con 0x7f0be80fc880 2023-12-16T21:33:47.155 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:33:47.151+0000 7f0bcf7fe700 1 -- 172.21.15.45:0/3834928997 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f0bd801c5d0 con 0x7f0be80fc880 2023-12-16T21:33:47.155 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:33:47.151+0000 7f0bed458700 1 -- 172.21.15.45:0/3834928997 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0be810dea0 con 0x7f0be80fc880 2023-12-16T21:33:47.156 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:33:47.152+0000 7f0bcf7fe700 1 -- 172.21.15.45:0/3834928997 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 13) v1 ==== 43337+0+0 (secure 0 0 0) 0x7f0bd801c840 con 0x7f0be80fc880 2023-12-16T21:33:47.156 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:33:47.152+0000 7f0bcf7fe700 1 --2- 172.21.15.45:0/3834928997 >> [v2:172.21.15.45:6800/2215417738,v1:172.21.15.45:6801/2215417738] conn(0x7f0bd00366d0 0x7f0bd0038b80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:33:47.156 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:33:47.152+0000 7f0bcf7fe700 1 -- 172.21.15.45:0/3834928997 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1151+0+0 (secure 0 0 0) 0x7f0bd8049940 con 0x7f0be80fc880 2023-12-16T21:33:47.156 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:33:47.152+0000 7f0bed458700 1 -- 172.21.15.45:0/3834928997 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0be805f270 con 0x7f0be80fc880 2023-12-16T21:33:47.156 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:33:47.152+0000 7f0be67fc700 1 --2- 172.21.15.45:0/3834928997 >> [v2:172.21.15.45:6800/2215417738,v1:172.21.15.45:6801/2215417738] conn(0x7f0bd00366d0 0x7f0bd0038b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:33:47.156 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:33:47.153+0000 7f0be67fc700 1 --2- 172.21.15.45:0/3834928997 >> [v2:172.21.15.45:6800/2215417738,v1:172.21.15.45:6801/2215417738] conn(0x7f0bd00366d0 0x7f0bd0038b80 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7f0bdc009990 tx=0x7f0bdc006e40 comp rx=0 tx=0).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:33:47.160 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:33:47.157+0000 7f0bcf7fe700 1 -- 172.21.15.45:0/3834928997 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+174570 (secure 0 0 0) 0x7f0bd8023300 con 0x7f0be80fc880 2023-12-16T21:33:47.371 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:33:47.368+0000 7f0bed458700 1 -- 172.21.15.45:0/3834928997 --> [v2:172.21.15.45:6800/2215417738,v1:172.21.15.45:6801/2215417738] -- mgr_command(tid 0: {"prefix": "orch client-keyring set", "entity": "client.admin", "placement": "*", "mode": "0755", "target": ["mon-mgr", ""]}) v1 -- 0x7f0be810e9e0 con 0x7f0bd00366d0 2023-12-16T21:33:47.374 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:33:47.371+0000 7f0bcf7fe700 1 -- 172.21.15.45:0/3834928997 <== mgr.14162 v2:172.21.15.45:6800/2215417738 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+0 (secure 0 0 0) 0x7f0be810e9e0 con 0x7f0bd00366d0 2023-12-16T21:33:47.376 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:33:47.373+0000 7f0bed458700 1 -- 172.21.15.45:0/3834928997 >> [v2:172.21.15.45:6800/2215417738,v1:172.21.15.45:6801/2215417738] conn(0x7f0bd00366d0 msgr2=0x7f0bd0038b80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:33:47.377 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:33:47.373+0000 7f0bed458700 1 --2- 172.21.15.45:0/3834928997 >> [v2:172.21.15.45:6800/2215417738,v1:172.21.15.45:6801/2215417738] conn(0x7f0bd00366d0 0x7f0bd0038b80 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7f0bdc009990 tx=0x7f0bdc006e40 comp rx=0 tx=0).stop 2023-12-16T21:33:47.377 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:33:47.373+0000 7f0bed458700 1 -- 172.21.15.45:0/3834928997 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f0be80fc880 msgr2=0x7f0be810cac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:33:47.377 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:33:47.373+0000 7f0bed458700 1 --2- 172.21.15.45:0/3834928997 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f0be80fc880 0x7f0be810cac0 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7f0bd800b690 tx=0x7f0bd8009e30 comp rx=0 tx=0).stop 2023-12-16T21:33:47.377 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:33:47.373+0000 7f0bed458700 1 -- 172.21.15.45:0/3834928997 shutdown_connections 2023-12-16T21:33:47.377 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:33:47.373+0000 7f0bed458700 1 --2- 172.21.15.45:0/3834928997 >> [v2:172.21.15.45:6800/2215417738,v1:172.21.15.45:6801/2215417738] conn(0x7f0bd00366d0 0x7f0bd0038b80 unknown :-1 s=CLOSED pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:33:47.377 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:33:47.373+0000 7f0bed458700 1 --2- 172.21.15.45:0/3834928997 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f0be80fc880 0x7f0be810cac0 unknown :-1 s=CLOSED pgs=93 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:33:47.377 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:33:47.373+0000 7f0bed458700 1 -- 172.21.15.45:0/3834928997 >> 172.21.15.45:0/3834928997 conn(0x7f0be80f7f40 msgr2=0x7f0be80f8d70 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-16T21:33:47.377 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:33:47.373+0000 7f0bed458700 1 -- 172.21.15.45:0/3834928997 shutdown_connections 2023-12-16T21:33:47.377 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:33:47.373+0000 7f0bed458700 1 -- 172.21.15.45:0/3834928997 wait complete. 2023-12-16T21:33:47.789 INFO:tasks.cephadm:Writing (initial) conf and keyring to smithi169 2023-12-16T21:33:47.789 DEBUG:teuthology.orchestra.run.smithi169:> set -ex 2023-12-16T21:33:47.789 DEBUG:teuthology.orchestra.run.smithi169:> dd of=/etc/ceph/ceph.conf 2023-12-16T21:33:47.807 DEBUG:teuthology.orchestra.run.smithi169:> set -ex 2023-12-16T21:33:47.807 DEBUG:teuthology.orchestra.run.smithi169:> dd of=/etc/ceph/ceph.client.admin.keyring 2023-12-16T21:33:47.864 INFO:tasks.cephadm:Adding host smithi169 to orchestrator... 2023-12-16T21:33:47.864 DEBUG:teuthology.orchestra.run.smithi045:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 88ff363c-9c5a-11ee-95a5-87774f69a715 -- ceph orch host add smithi169 2023-12-16T21:33:48.141 INFO:teuthology.orchestra.run.smithi045.stderr:Inferring config /var/lib/ceph/88ff363c-9c5a-11ee-95a5-87774f69a715/mon.smithi045/config 2023-12-16T21:33:48.566 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Dec 16 21:33:48 smithi045 ceph-mon[80763]: from='client.14186 -' entity='client.admin' cmd=[{"prefix": "orch client-keyring set", "entity": "client.admin", "placement": "*", "mode": "0755", "target": ["mon-mgr", ""]}]: dispatch 2023-12-16T21:33:48.566 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Dec 16 21:33:48 smithi045 ceph-mon[80763]: from='mgr.14162 172.21.15.45:0/142909557' entity='mgr.smithi045.nhtmwc' 2023-12-16T21:33:48.566 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Dec 16 21:33:48 smithi045 ceph-mon[80763]: from='mgr.14162 172.21.15.45:0/142909557' entity='mgr.smithi045.nhtmwc' 2023-12-16T21:33:48.566 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Dec 16 21:33:48 smithi045 ceph-mon[80763]: from='mgr.14162 172.21.15.45:0/142909557' entity='mgr.smithi045.nhtmwc' 2023-12-16T21:33:48.566 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Dec 16 21:33:48 smithi045 ceph-mon[80763]: from='mgr.14162 172.21.15.45:0/142909557' entity='mgr.smithi045.nhtmwc' 2023-12-16T21:33:48.566 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Dec 16 21:33:48 smithi045 ceph-mon[80763]: from='mgr.14162 172.21.15.45:0/142909557' entity='mgr.smithi045.nhtmwc' 2023-12-16T21:33:48.566 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Dec 16 21:33:48 smithi045 ceph-mon[80763]: Deploying daemon node-exporter.smithi045 on smithi045 2023-12-16T21:33:48.566 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Dec 16 21:33:48 smithi045 ceph-mon[80763]: from='mgr.14162 172.21.15.45:0/142909557' entity='mgr.smithi045.nhtmwc' 2023-12-16T21:33:49.078 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:33:49.073+0000 7fa3b74a9700 1 -- 172.21.15.45:0/2777645024 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fa3b00fb780 msgr2=0x7fa3b00fbb90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:33:49.078 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:33:49.073+0000 7fa3b74a9700 1 --2- 172.21.15.45:0/2777645024 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fa3b00fb780 0x7fa3b00fbb90 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7fa3ac0048c0 tx=0x7fa3ac009e30 comp rx=0 tx=0).stop 2023-12-16T21:33:49.078 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:33:49.073+0000 7fa3b74a9700 1 -- 172.21.15.45:0/2777645024 shutdown_connections 2023-12-16T21:33:49.078 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:33:49.073+0000 7fa3b74a9700 1 --2- 172.21.15.45:0/2777645024 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fa3b00fb780 0x7fa3b00fbb90 unknown :-1 s=CLOSED pgs=94 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:33:49.078 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:33:49.073+0000 7fa3b74a9700 1 -- 172.21.15.45:0/2777645024 >> 172.21.15.45:0/2777645024 conn(0x7fa3b00f6e40 msgr2=0x7fa3b00f9270 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-16T21:33:49.079 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:33:49.074+0000 7fa3b74a9700 1 -- 172.21.15.45:0/2777645024 shutdown_connections 2023-12-16T21:33:49.079 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:33:49.074+0000 7fa3b74a9700 1 -- 172.21.15.45:0/2777645024 wait complete. 2023-12-16T21:33:49.079 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:33:49.074+0000 7fa3b74a9700 1 Processor -- start 2023-12-16T21:33:49.079 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:33:49.074+0000 7fa3b74a9700 1 -- start start 2023-12-16T21:33:49.079 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:33:49.074+0000 7fa3b74a9700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fa3b00fb780 0x7fa3b010ed40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:33:49.079 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:33:49.074+0000 7fa3b74a9700 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa3b010f280 con 0x7fa3b00fb780 2023-12-16T21:33:49.079 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:33:49.074+0000 7fa3b5245700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fa3b00fb780 0x7fa3b010ed40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:33:49.079 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:33:49.074+0000 7fa3b5245700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fa3b00fb780 0x7fa3b010ed40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:55248/0 (socket says 172.21.15.45:55248) 2023-12-16T21:33:49.079 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:33:49.074+0000 7fa3b5245700 1 -- 172.21.15.45:0/4228118219 learned_addr learned my addr 172.21.15.45:0/4228118219 (peer_addr_for_me v2:172.21.15.45:0/0) 2023-12-16T21:33:49.079 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:33:49.075+0000 7fa3b5245700 1 -- 172.21.15.45:0/4228118219 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa3ac005040 con 0x7fa3b00fb780 2023-12-16T21:33:49.079 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:33:49.075+0000 7fa3b5245700 1 --2- 172.21.15.45:0/4228118219 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fa3b00fb780 0x7fa3b010ed40 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7fa3ac00b690 tx=0x7fa3ac009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:33:49.079 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:33:49.075+0000 7fa3a1ffb700 1 -- 172.21.15.45:0/4228118219 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fa3ac009650 con 0x7fa3b00fb780 2023-12-16T21:33:49.079 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:33:49.075+0000 7fa3a1ffb700 1 -- 172.21.15.45:0/4228118219 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fa3ac0097b0 con 0x7fa3b00fb780 2023-12-16T21:33:49.079 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:33:49.075+0000 7fa3b74a9700 1 -- 172.21.15.45:0/4228118219 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa3b010f480 con 0x7fa3b00fb780 2023-12-16T21:33:49.080 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:33:49.075+0000 7fa3a1ffb700 1 -- 172.21.15.45:0/4228118219 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fa3ac01c5d0 con 0x7fa3b00fb780 2023-12-16T21:33:49.080 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:33:49.075+0000 7fa3b74a9700 1 -- 172.21.15.45:0/4228118219 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa3b010ffe0 con 0x7fa3b00fb780 2023-12-16T21:33:49.080 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:33:49.076+0000 7fa3a1ffb700 1 -- 172.21.15.45:0/4228118219 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 13) v1 ==== 43337+0+0 (secure 0 0 0) 0x7fa3ac01c840 con 0x7fa3b00fb780 2023-12-16T21:33:49.080 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:33:49.076+0000 7fa3a1ffb700 1 --2- 172.21.15.45:0/4228118219 >> [v2:172.21.15.45:6800/2215417738,v1:172.21.15.45:6801/2215417738] conn(0x7fa39c0366d0 0x7fa39c038b80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:33:49.081 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:33:49.076+0000 7fa3a1ffb700 1 -- 172.21.15.45:0/4228118219 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1151+0+0 (secure 0 0 0) 0x7fa3ac0139d0 con 0x7fa3b00fb780 2023-12-16T21:33:49.081 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:33:49.076+0000 7fa3b4a44700 1 --2- 172.21.15.45:0/4228118219 >> [v2:172.21.15.45:6800/2215417738,v1:172.21.15.45:6801/2215417738] conn(0x7fa39c0366d0 0x7fa39c038b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:33:49.081 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:33:49.076+0000 7fa3b74a9700 1 -- 172.21.15.45:0/4228118219 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa3b005f270 con 0x7fa3b00fb780 2023-12-16T21:33:49.081 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:33:49.077+0000 7fa3b4a44700 1 --2- 172.21.15.45:0/4228118219 >> [v2:172.21.15.45:6800/2215417738,v1:172.21.15.45:6801/2215417738] conn(0x7fa39c0366d0 0x7fa39c038b80 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7fa3a4009990 tx=0x7fa3a4006e40 comp rx=0 tx=0).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:33:49.084 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:33:49.081+0000 7fa3a1ffb700 1 -- 172.21.15.45:0/4228118219 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+174570 (secure 0 0 0) 0x7fa3ac023300 con 0x7fa3b00fb780 2023-12-16T21:33:49.305 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:33:49.301+0000 7fa3b74a9700 1 -- 172.21.15.45:0/4228118219 --> [v2:172.21.15.45:6800/2215417738,v1:172.21.15.45:6801/2215417738] -- mgr_command(tid 0: {"prefix": "orch host add", "hostname": "smithi169", "target": ["mon-mgr", ""]}) v1 -- 0x7fa3b0110ab0 con 0x7fa39c0366d0 2023-12-16T21:33:49.566 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Dec 16 21:33:49 smithi045 ceph-mon[80763]: from='client.14188 -' entity='client.admin' cmd=[{"prefix": "orch host add", "hostname": "smithi169", "target": ["mon-mgr", ""]}]: dispatch 2023-12-16T21:33:50.816 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Dec 16 21:33:50 smithi045 ceph-mon[80763]: Deploying cephadm binary to smithi169 2023-12-16T21:33:51.290 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:33:51.288+0000 7fa3a1ffb700 1 -- 172.21.15.45:0/4228118219 <== mgr.14162 v2:172.21.15.45:6800/2215417738 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+49 (secure 0 0 0) 0x7fa3b0110ab0 con 0x7fa39c0366d0 2023-12-16T21:33:51.291 INFO:teuthology.orchestra.run.smithi045.stdout:Added host 'smithi169' with addr '172.21.15.169' 2023-12-16T21:33:51.292 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:33:51.290+0000 7fa3b74a9700 1 -- 172.21.15.45:0/4228118219 >> [v2:172.21.15.45:6800/2215417738,v1:172.21.15.45:6801/2215417738] conn(0x7fa39c0366d0 msgr2=0x7fa39c038b80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:33:51.292 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:33:51.290+0000 7fa3b74a9700 1 --2- 172.21.15.45:0/4228118219 >> [v2:172.21.15.45:6800/2215417738,v1:172.21.15.45:6801/2215417738] conn(0x7fa39c0366d0 0x7fa39c038b80 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7fa3a4009990 tx=0x7fa3a4006e40 comp rx=0 tx=0).stop 2023-12-16T21:33:51.292 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:33:51.290+0000 7fa3b74a9700 1 -- 172.21.15.45:0/4228118219 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fa3b00fb780 msgr2=0x7fa3b010ed40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:33:51.293 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:33:51.290+0000 7fa3b74a9700 1 --2- 172.21.15.45:0/4228118219 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fa3b00fb780 0x7fa3b010ed40 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7fa3ac00b690 tx=0x7fa3ac009e30 comp rx=0 tx=0).stop 2023-12-16T21:33:51.293 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:33:51.290+0000 7fa3b74a9700 1 -- 172.21.15.45:0/4228118219 shutdown_connections 2023-12-16T21:33:51.293 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:33:51.290+0000 7fa3b74a9700 1 --2- 172.21.15.45:0/4228118219 >> [v2:172.21.15.45:6800/2215417738,v1:172.21.15.45:6801/2215417738] conn(0x7fa39c0366d0 0x7fa39c038b80 unknown :-1 s=CLOSED pgs=14 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:33:51.293 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:33:51.290+0000 7fa3b74a9700 1 --2- 172.21.15.45:0/4228118219 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fa3b00fb780 0x7fa3b010ed40 unknown :-1 s=CLOSED pgs=95 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:33:51.293 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:33:51.290+0000 7fa3b74a9700 1 -- 172.21.15.45:0/4228118219 >> 172.21.15.45:0/4228118219 conn(0x7fa3b00f6e40 msgr2=0x7fa3b0065690 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-16T21:33:51.293 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:33:51.290+0000 7fa3b74a9700 1 -- 172.21.15.45:0/4228118219 shutdown_connections 2023-12-16T21:33:51.293 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:33:51.290+0000 7fa3b74a9700 1 -- 172.21.15.45:0/4228118219 wait complete. 2023-12-16T21:33:51.930 DEBUG:teuthology.orchestra.run.smithi045:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 88ff363c-9c5a-11ee-95a5-87774f69a715 -- ceph orch host ls --format=json 2023-12-16T21:33:52.209 INFO:teuthology.orchestra.run.smithi045.stderr:Inferring config /var/lib/ceph/88ff363c-9c5a-11ee-95a5-87774f69a715/mon.smithi045/config 2023-12-16T21:33:52.566 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Dec 16 21:33:52 smithi045 ceph-mon[80763]: from='mgr.14162 172.21.15.45:0/142909557' entity='mgr.smithi045.nhtmwc' 2023-12-16T21:33:52.566 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Dec 16 21:33:52 smithi045 ceph-mon[80763]: Added host smithi169 2023-12-16T21:33:52.566 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Dec 16 21:33:52 smithi045 ceph-mon[80763]: from='mgr.14162 172.21.15.45:0/142909557' entity='mgr.smithi045.nhtmwc' 2023-12-16T21:33:52.567 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Dec 16 21:33:52 smithi045 ceph-mon[80763]: from='mgr.14162 172.21.15.45:0/142909557' entity='mgr.smithi045.nhtmwc' 2023-12-16T21:33:52.567 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Dec 16 21:33:52 smithi045 ceph-mon[80763]: from='mgr.14162 172.21.15.45:0/142909557' entity='mgr.smithi045.nhtmwc' 2023-12-16T21:33:52.567 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Dec 16 21:33:52 smithi045 ceph-mon[80763]: from='mgr.14162 172.21.15.45:0/142909557' entity='mgr.smithi045.nhtmwc' 2023-12-16T21:33:53.011 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:33:53.007+0000 7f558867c700 1 -- 172.21.15.45:0/313292580 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f55800fc880 msgr2=0x7f55800fcc90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:33:53.011 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:33:53.007+0000 7f558867c700 1 --2- 172.21.15.45:0/313292580 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f55800fc880 0x7f55800fcc90 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7f557c0048c0 tx=0x7f557c009e30 comp rx=0 tx=0).stop 2023-12-16T21:33:53.011 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:33:53.007+0000 7f558867c700 1 -- 172.21.15.45:0/313292580 shutdown_connections 2023-12-16T21:33:53.011 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:33:53.007+0000 7f558867c700 1 --2- 172.21.15.45:0/313292580 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f55800fc880 0x7f55800fcc90 unknown :-1 s=CLOSED pgs=96 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:33:53.011 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:33:53.007+0000 7f558867c700 1 -- 172.21.15.45:0/313292580 >> 172.21.15.45:0/313292580 conn(0x7f55800f7f40 msgr2=0x7f55800fa370 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-16T21:33:53.011 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:33:53.007+0000 7f558867c700 1 -- 172.21.15.45:0/313292580 shutdown_connections 2023-12-16T21:33:53.011 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:33:53.007+0000 7f558867c700 1 -- 172.21.15.45:0/313292580 wait complete. 2023-12-16T21:33:53.012 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:33:53.007+0000 7f558867c700 1 Processor -- start 2023-12-16T21:33:53.012 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:33:53.007+0000 7f558867c700 1 -- start start 2023-12-16T21:33:53.012 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:33:53.008+0000 7f558867c700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f55800fc880 0x7f5580071b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:33:53.012 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:33:53.008+0000 7f558867c700 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f558006ff90 con 0x7f55800fc880 2023-12-16T21:33:53.012 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:33:53.008+0000 7f5586418700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f55800fc880 0x7f5580071b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:33:53.012 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:33:53.008+0000 7f5586418700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f55800fc880 0x7f5580071b10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:38280/0 (socket says 172.21.15.45:38280) 2023-12-16T21:33:53.012 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:33:53.008+0000 7f5586418700 1 -- 172.21.15.45:0/635998772 learned_addr learned my addr 172.21.15.45:0/635998772 (peer_addr_for_me v2:172.21.15.45:0/0) 2023-12-16T21:33:53.012 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:33:53.008+0000 7f5586418700 1 -- 172.21.15.45:0/635998772 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f557c005040 con 0x7f55800fc880 2023-12-16T21:33:53.012 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:33:53.008+0000 7f5586418700 1 --2- 172.21.15.45:0/635998772 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f55800fc880 0x7f5580071b10 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7f557c009330 tx=0x7f557c009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:33:53.012 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:33:53.008+0000 7f5576ffd700 1 -- 172.21.15.45:0/635998772 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f557c0096b0 con 0x7f55800fc880 2023-12-16T21:33:53.012 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:33:53.008+0000 7f5576ffd700 1 -- 172.21.15.45:0/635998772 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f557c009810 con 0x7f55800fc880 2023-12-16T21:33:53.012 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:33:53.008+0000 7f558867c700 1 -- 172.21.15.45:0/635998772 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5580070190 con 0x7f55800fc880 2023-12-16T21:33:53.012 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:33:53.008+0000 7f5576ffd700 1 -- 172.21.15.45:0/635998772 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f557c01c5b0 con 0x7f55800fc880 2023-12-16T21:33:53.012 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:33:53.008+0000 7f558867c700 1 -- 172.21.15.45:0/635998772 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5580070680 con 0x7f55800fc880 2023-12-16T21:33:53.013 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:33:53.009+0000 7f5576ffd700 1 -- 172.21.15.45:0/635998772 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 13) v1 ==== 43337+0+0 (secure 0 0 0) 0x7f557c013420 con 0x7f55800fc880 2023-12-16T21:33:53.013 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:33:53.009+0000 7f558867c700 1 -- 172.21.15.45:0/635998772 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5580106510 con 0x7f55800fc880 2023-12-16T21:33:53.013 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:33:53.009+0000 7f5576ffd700 1 --2- 172.21.15.45:0/635998772 >> [v2:172.21.15.45:6800/2215417738,v1:172.21.15.45:6801/2215417738] conn(0x7f556c0366d0 0x7f556c038b80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:33:53.014 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:33:53.009+0000 7f5576ffd700 1 -- 172.21.15.45:0/635998772 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1151+0+0 (secure 0 0 0) 0x7f557c049a00 con 0x7f55800fc880 2023-12-16T21:33:53.014 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:33:53.010+0000 7f5585c17700 1 --2- 172.21.15.45:0/635998772 >> [v2:172.21.15.45:6800/2215417738,v1:172.21.15.45:6801/2215417738] conn(0x7f556c0366d0 0x7f556c038b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:33:53.014 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:33:53.010+0000 7f5585c17700 1 --2- 172.21.15.45:0/635998772 >> [v2:172.21.15.45:6800/2215417738,v1:172.21.15.45:6801/2215417738] conn(0x7f556c0366d0 0x7f556c038b80 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0x7f5570009990 tx=0x7f5570006e40 comp rx=0 tx=0).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:33:53.017 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:33:53.014+0000 7f5576ffd700 1 -- 172.21.15.45:0/635998772 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+174570 (secure 0 0 0) 0x7f557c0119c0 con 0x7f55800fc880 2023-12-16T21:33:53.221 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:33:53.219+0000 7f558867c700 1 -- 172.21.15.45:0/635998772 --> [v2:172.21.15.45:6800/2215417738,v1:172.21.15.45:6801/2215417738] -- mgr_command(tid 0: {"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f558002b250 con 0x7f556c0366d0 2023-12-16T21:33:53.223 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:33:53.221+0000 7f5576ffd700 1 -- 172.21.15.45:0/635998772 <== mgr.14162 v2:172.21.15.45:6800/2215417738 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+160 (secure 0 0 0) 0x7f558002b250 con 0x7f556c0366d0 2023-12-16T21:33:53.223 INFO:teuthology.orchestra.run.smithi045.stdout: 2023-12-16T21:33:53.223 INFO:teuthology.orchestra.run.smithi045.stdout:[{"addr": "172.21.15.45", "hostname": "smithi045", "labels": [], "status": ""}, {"addr": "172.21.15.169", "hostname": "smithi169", "labels": [], "status": ""}] 2023-12-16T21:33:53.226 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:33:53.223+0000 7f558867c700 1 -- 172.21.15.45:0/635998772 >> [v2:172.21.15.45:6800/2215417738,v1:172.21.15.45:6801/2215417738] conn(0x7f556c0366d0 msgr2=0x7f556c038b80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:33:53.226 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:33:53.223+0000 7f558867c700 1 --2- 172.21.15.45:0/635998772 >> [v2:172.21.15.45:6800/2215417738,v1:172.21.15.45:6801/2215417738] conn(0x7f556c0366d0 0x7f556c038b80 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0x7f5570009990 tx=0x7f5570006e40 comp rx=0 tx=0).stop 2023-12-16T21:33:53.226 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:33:53.223+0000 7f558867c700 1 -- 172.21.15.45:0/635998772 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f55800fc880 msgr2=0x7f5580071b10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:33:53.226 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:33:53.223+0000 7f558867c700 1 --2- 172.21.15.45:0/635998772 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f55800fc880 0x7f5580071b10 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7f557c009330 tx=0x7f557c009e30 comp rx=0 tx=0).stop 2023-12-16T21:33:53.227 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:33:53.223+0000 7f558867c700 1 -- 172.21.15.45:0/635998772 shutdown_connections 2023-12-16T21:33:53.227 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:33:53.223+0000 7f558867c700 1 --2- 172.21.15.45:0/635998772 >> [v2:172.21.15.45:6800/2215417738,v1:172.21.15.45:6801/2215417738] conn(0x7f556c0366d0 0x7f556c038b80 unknown :-1 s=CLOSED pgs=15 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:33:53.227 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:33:53.223+0000 7f558867c700 1 --2- 172.21.15.45:0/635998772 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f55800fc880 0x7f5580071b10 unknown :-1 s=CLOSED pgs=97 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:33:53.227 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:33:53.223+0000 7f558867c700 1 -- 172.21.15.45:0/635998772 >> 172.21.15.45:0/635998772 conn(0x7f55800f7f40 msgr2=0x7f55800f8d70 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-16T21:33:53.227 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:33:53.223+0000 7f558867c700 1 -- 172.21.15.45:0/635998772 shutdown_connections 2023-12-16T21:33:53.227 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:33:53.223+0000 7f558867c700 1 -- 172.21.15.45:0/635998772 wait complete. 2023-12-16T21:33:53.566 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Dec 16 21:33:53 smithi045 ceph-mon[80763]: Deploying daemon alertmanager.smithi045 on smithi045 2023-12-16T21:33:53.566 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Dec 16 21:33:53 smithi045 ceph-mon[80763]: from='mgr.14162 172.21.15.45:0/142909557' entity='mgr.smithi045.nhtmwc' 2023-12-16T21:33:53.635 INFO:tasks.cephadm:Setting crush tunables to default 2023-12-16T21:33:53.636 DEBUG:teuthology.orchestra.run.smithi045:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 88ff363c-9c5a-11ee-95a5-87774f69a715 -- ceph osd crush tunables default 2023-12-16T21:33:53.900 INFO:teuthology.orchestra.run.smithi045.stderr:Inferring config /var/lib/ceph/88ff363c-9c5a-11ee-95a5-87774f69a715/mon.smithi045/config 2023-12-16T21:33:54.566 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Dec 16 21:33:54 smithi045 ceph-mon[80763]: from='client.14190 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-12-16T21:33:55.436 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:33:55.430+0000 7f936452d700 1 -- 172.21.15.45:0/858293344 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f935c0fd3c0 msgr2=0x7f935c0ff7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:33:55.437 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:33:55.430+0000 7f936452d700 1 --2- 172.21.15.45:0/858293344 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f935c0fd3c0 0x7f935c0ff7a0 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7f93580048c0 tx=0x7f9358009e30 comp rx=0 tx=0).stop 2023-12-16T21:33:55.437 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:33:55.431+0000 7f936452d700 1 -- 172.21.15.45:0/858293344 shutdown_connections 2023-12-16T21:33:55.437 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:33:55.431+0000 7f936452d700 1 --2- 172.21.15.45:0/858293344 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f935c0fd3c0 0x7f935c0ff7a0 unknown :-1 s=CLOSED pgs=98 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:33:55.437 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:33:55.431+0000 7f936452d700 1 -- 172.21.15.45:0/858293344 >> 172.21.15.45:0/858293344 conn(0x7f935c0f6e40 msgr2=0x7f935c0f9270 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-16T21:33:55.437 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:33:55.431+0000 7f936452d700 1 -- 172.21.15.45:0/858293344 shutdown_connections 2023-12-16T21:33:55.438 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:33:55.431+0000 7f936452d700 1 -- 172.21.15.45:0/858293344 wait complete. 2023-12-16T21:33:55.438 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:33:55.431+0000 7f936452d700 1 Processor -- start 2023-12-16T21:33:55.438 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:33:55.431+0000 7f936452d700 1 -- start start 2023-12-16T21:33:55.438 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:33:55.431+0000 7f936452d700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f935c0fd3c0 0x7f935c10ca60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:33:55.438 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:33:55.431+0000 7f936452d700 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f935c10cfa0 con 0x7f935c0fd3c0 2023-12-16T21:33:55.439 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:33:55.432+0000 7f93622c9700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f935c0fd3c0 0x7f935c10ca60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:33:55.439 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:33:55.432+0000 7f93622c9700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f935c0fd3c0 0x7f935c10ca60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:38304/0 (socket says 172.21.15.45:38304) 2023-12-16T21:33:55.439 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:33:55.432+0000 7f93622c9700 1 -- 172.21.15.45:0/327486920 learned_addr learned my addr 172.21.15.45:0/327486920 (peer_addr_for_me v2:172.21.15.45:0/0) 2023-12-16T21:33:55.439 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:33:55.432+0000 7f93622c9700 1 -- 172.21.15.45:0/327486920 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9358005040 con 0x7f935c0fd3c0 2023-12-16T21:33:55.439 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:33:55.432+0000 7f93622c9700 1 --2- 172.21.15.45:0/327486920 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f935c0fd3c0 0x7f935c10ca60 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7f935800b690 tx=0x7f9358009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:33:55.439 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:33:55.432+0000 7f9352ffd700 1 -- 172.21.15.45:0/327486920 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f9358009650 con 0x7f935c0fd3c0 2023-12-16T21:33:55.439 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:33:55.432+0000 7f9352ffd700 1 -- 172.21.15.45:0/327486920 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f93580097b0 con 0x7f935c0fd3c0 2023-12-16T21:33:55.440 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:33:55.432+0000 7f936452d700 1 -- 172.21.15.45:0/327486920 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f935c10d1a0 con 0x7f935c0fd3c0 2023-12-16T21:33:55.440 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:33:55.432+0000 7f9352ffd700 1 -- 172.21.15.45:0/327486920 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f935801c5b0 con 0x7f935c0fd3c0 2023-12-16T21:33:55.440 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:33:55.432+0000 7f936452d700 1 -- 172.21.15.45:0/327486920 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f935c10de40 con 0x7f935c0fd3c0 2023-12-16T21:33:55.440 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:33:55.433+0000 7f9352ffd700 1 -- 172.21.15.45:0/327486920 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 13) v1 ==== 43337+0+0 (secure 0 0 0) 0x7f935801c870 con 0x7f935c0fd3c0 2023-12-16T21:33:55.440 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:33:55.433+0000 7f9352ffd700 1 --2- 172.21.15.45:0/327486920 >> [v2:172.21.15.45:6800/2215417738,v1:172.21.15.45:6801/2215417738] conn(0x7f93480366d0 0x7f9348038b80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:33:55.440 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:33:55.433+0000 7f9352ffd700 1 -- 172.21.15.45:0/327486920 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1151+0+0 (secure 0 0 0) 0x7f9358049880 con 0x7f935c0fd3c0 2023-12-16T21:33:55.440 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:33:55.433+0000 7f9361ac8700 1 --2- 172.21.15.45:0/327486920 >> [v2:172.21.15.45:6800/2215417738,v1:172.21.15.45:6801/2215417738] conn(0x7f93480366d0 0x7f9348038b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:33:55.440 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:33:55.433+0000 7f936452d700 1 -- 172.21.15.45:0/327486920 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f935c05f270 con 0x7f935c0fd3c0 2023-12-16T21:33:55.441 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:33:55.434+0000 7f9361ac8700 1 --2- 172.21.15.45:0/327486920 >> [v2:172.21.15.45:6800/2215417738,v1:172.21.15.45:6801/2215417738] conn(0x7f93480366d0 0x7f9348038b80 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7f934c009990 tx=0x7f934c006e40 comp rx=0 tx=0).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:33:55.441 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:33:55.438+0000 7f9352ffd700 1 -- 172.21.15.45:0/327486920 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+174570 (secure 0 0 0) 0x7f9358011db0 con 0x7f935c0fd3c0 2023-12-16T21:33:55.694 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:33:55.692+0000 7f936452d700 1 -- 172.21.15.45:0/327486920 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "osd crush tunables", "profile": "default"} v 0) v1 -- 0x7f935c10e850 con 0x7f935c0fd3c0 2023-12-16T21:33:55.747 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:33:55.744+0000 7f9352ffd700 1 -- 172.21.15.45:0/327486920 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mon_command_ack([{"prefix": "osd crush tunables", "profile": "default"}]=0 adjusted tunables profile to default v4) v1 ==== 124+0+0 (secure 0 0 0) 0x7f9358023990 con 0x7f935c0fd3c0 2023-12-16T21:33:55.749 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:33:55.747+0000 7f936452d700 1 -- 172.21.15.45:0/327486920 >> [v2:172.21.15.45:6800/2215417738,v1:172.21.15.45:6801/2215417738] conn(0x7f93480366d0 msgr2=0x7f9348038b80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:33:55.749 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:33:55.747+0000 7f936452d700 1 --2- 172.21.15.45:0/327486920 >> [v2:172.21.15.45:6800/2215417738,v1:172.21.15.45:6801/2215417738] conn(0x7f93480366d0 0x7f9348038b80 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7f934c009990 tx=0x7f934c006e40 comp rx=0 tx=0).stop 2023-12-16T21:33:55.749 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:33:55.747+0000 7f936452d700 1 -- 172.21.15.45:0/327486920 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f935c0fd3c0 msgr2=0x7f935c10ca60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:33:55.749 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:33:55.747+0000 7f936452d700 1 --2- 172.21.15.45:0/327486920 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f935c0fd3c0 0x7f935c10ca60 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7f935800b690 tx=0x7f9358009e30 comp rx=0 tx=0).stop 2023-12-16T21:33:55.750 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:33:55.747+0000 7f936452d700 1 -- 172.21.15.45:0/327486920 shutdown_connections 2023-12-16T21:33:55.750 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:33:55.747+0000 7f936452d700 1 --2- 172.21.15.45:0/327486920 >> [v2:172.21.15.45:6800/2215417738,v1:172.21.15.45:6801/2215417738] conn(0x7f93480366d0 0x7f9348038b80 unknown :-1 s=CLOSED pgs=16 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:33:55.750 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:33:55.747+0000 7f936452d700 1 --2- 172.21.15.45:0/327486920 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f935c0fd3c0 0x7f935c10ca60 unknown :-1 s=CLOSED pgs=99 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:33:55.750 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:33:55.747+0000 7f936452d700 1 -- 172.21.15.45:0/327486920 >> 172.21.15.45:0/327486920 conn(0x7f935c0f6e40 msgr2=0x7f935c0f7b50 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-16T21:33:55.750 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:33:55.747+0000 7f936452d700 1 -- 172.21.15.45:0/327486920 shutdown_connections 2023-12-16T21:33:55.750 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:33:55.747+0000 7f936452d700 1 -- 172.21.15.45:0/327486920 wait complete. 2023-12-16T21:33:55.750 INFO:teuthology.orchestra.run.smithi045.stderr:adjusted tunables profile to default 2023-12-16T21:33:55.954 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Dec 16 21:33:55 smithi045 ceph-mon[80763]: from='client.? 172.21.15.45:0/327486920' entity='client.admin' cmd=[{"prefix": "osd crush tunables", "profile": "default"}]: dispatch 2023-12-16T21:33:56.807 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Dec 16 21:33:56 smithi045 ceph-mon[80763]: from='client.? 172.21.15.45:0/327486920' entity='client.admin' cmd='[{"prefix": "osd crush tunables", "profile": "default"}]': finished 2023-12-16T21:33:56.807 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Dec 16 21:33:56 smithi045 ceph-mon[80763]: osdmap e4: 0 total, 0 up, 0 in 2023-12-16T21:33:56.808 INFO:tasks.cephadm:Adding mon.smithi045 on smithi045 2023-12-16T21:33:56.808 INFO:tasks.cephadm:Adding mon.smithi169 on smithi169 2023-12-16T21:33:56.809 DEBUG:teuthology.orchestra.run.smithi169:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 88ff363c-9c5a-11ee-95a5-87774f69a715 -- ceph orch apply mon '2;smithi045:172.21.15.45=smithi045;smithi169:172.21.15.169=smithi169' 2023-12-16T21:33:57.059 INFO:teuthology.orchestra.run.smithi169.stderr:Inferring config /etc/ceph/ceph.conf 2023-12-16T21:33:57.106 INFO:teuthology.orchestra.run.smithi169.stderr:Inferring config /etc/ceph/ceph.conf 2023-12-16T21:33:58.370 INFO:teuthology.orchestra.run.smithi169.stderr:2023-12-16T21:33:58.365+0000 7f4054098700 1 -- 172.21.15.169:0/3854789541 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f404c0fc860 msgr2=0x7f404c0fcc70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:33:58.370 INFO:teuthology.orchestra.run.smithi169.stderr:2023-12-16T21:33:58.365+0000 7f4054098700 1 --2- 172.21.15.169:0/3854789541 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f404c0fc860 0x7f404c0fcc70 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7f4048009c60 tx=0x7f4048004d10 comp rx=0 tx=0).stop 2023-12-16T21:33:58.371 INFO:teuthology.orchestra.run.smithi169.stderr:2023-12-16T21:33:58.365+0000 7f4054098700 1 -- 172.21.15.169:0/3854789541 shutdown_connections 2023-12-16T21:33:58.371 INFO:teuthology.orchestra.run.smithi169.stderr:2023-12-16T21:33:58.365+0000 7f4054098700 1 --2- 172.21.15.169:0/3854789541 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f404c0fc860 0x7f404c0fcc70 unknown :-1 s=CLOSED pgs=100 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:33:58.371 INFO:teuthology.orchestra.run.smithi169.stderr:2023-12-16T21:33:58.365+0000 7f4054098700 1 -- 172.21.15.169:0/3854789541 >> 172.21.15.169:0/3854789541 conn(0x7f404c0f7fe0 msgr2=0x7f404c0fa3f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-16T21:33:58.371 INFO:teuthology.orchestra.run.smithi169.stderr:2023-12-16T21:33:58.365+0000 7f4054098700 1 -- 172.21.15.169:0/3854789541 shutdown_connections 2023-12-16T21:33:58.371 INFO:teuthology.orchestra.run.smithi169.stderr:2023-12-16T21:33:58.365+0000 7f4054098700 1 -- 172.21.15.169:0/3854789541 wait complete. 2023-12-16T21:33:58.371 INFO:teuthology.orchestra.run.smithi169.stderr:2023-12-16T21:33:58.365+0000 7f4054098700 1 Processor -- start 2023-12-16T21:33:58.371 INFO:teuthology.orchestra.run.smithi169.stderr:2023-12-16T21:33:58.365+0000 7f4054098700 1 -- start start 2023-12-16T21:33:58.371 INFO:teuthology.orchestra.run.smithi169.stderr:2023-12-16T21:33:58.365+0000 7f4054098700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f404c0fc860 0x7f404c10cb00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:33:58.372 INFO:teuthology.orchestra.run.smithi169.stderr:2023-12-16T21:33:58.365+0000 7f4054098700 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f404c10d040 con 0x7f404c0fc860 2023-12-16T21:33:58.372 INFO:teuthology.orchestra.run.smithi169.stderr:2023-12-16T21:33:58.366+0000 7f4051e34700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f404c0fc860 0x7f404c10cb00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:33:58.372 INFO:teuthology.orchestra.run.smithi169.stderr:2023-12-16T21:33:58.366+0000 7f4051e34700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f404c0fc860 0x7f404c10cb00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.169:42408/0 (socket says 172.21.15.169:42408) 2023-12-16T21:33:58.372 INFO:teuthology.orchestra.run.smithi169.stderr:2023-12-16T21:33:58.366+0000 7f4051e34700 1 -- 172.21.15.169:0/116147483 learned_addr learned my addr 172.21.15.169:0/116147483 (peer_addr_for_me v2:172.21.15.169:0/0) 2023-12-16T21:33:58.372 INFO:teuthology.orchestra.run.smithi169.stderr:2023-12-16T21:33:58.366+0000 7f4051e34700 1 -- 172.21.15.169:0/116147483 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4048009940 con 0x7f404c0fc860 2023-12-16T21:33:58.372 INFO:teuthology.orchestra.run.smithi169.stderr:2023-12-16T21:33:58.366+0000 7f4051e34700 1 --2- 172.21.15.169:0/116147483 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f404c0fc860 0x7f404c10cb00 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7f4048015040 tx=0x7f4048004d10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:33:58.372 INFO:teuthology.orchestra.run.smithi169.stderr:2023-12-16T21:33:58.366+0000 7f40427fc700 1 -- 172.21.15.169:0/116147483 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f404801d440 con 0x7f404c0fc860 2023-12-16T21:33:58.373 INFO:teuthology.orchestra.run.smithi169.stderr:2023-12-16T21:33:58.366+0000 7f40427fc700 1 -- 172.21.15.169:0/116147483 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f404801da50 con 0x7f404c0fc860 2023-12-16T21:33:58.373 INFO:teuthology.orchestra.run.smithi169.stderr:2023-12-16T21:33:58.366+0000 7f4054098700 1 -- 172.21.15.169:0/116147483 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f404c10d240 con 0x7f404c0fc860 2023-12-16T21:33:58.373 INFO:teuthology.orchestra.run.smithi169.stderr:2023-12-16T21:33:58.366+0000 7f40427fc700 1 -- 172.21.15.169:0/116147483 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f40480128e0 con 0x7f404c0fc860 2023-12-16T21:33:58.373 INFO:teuthology.orchestra.run.smithi169.stderr:2023-12-16T21:33:58.366+0000 7f4054098700 1 -- 172.21.15.169:0/116147483 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f404c10dee0 con 0x7f404c0fc860 2023-12-16T21:33:58.373 INFO:teuthology.orchestra.run.smithi169.stderr:2023-12-16T21:33:58.367+0000 7f40427fc700 1 -- 172.21.15.169:0/116147483 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 13) v1 ==== 43337+0+0 (secure 0 0 0) 0x7f4048019040 con 0x7f404c0fc860 2023-12-16T21:33:58.373 INFO:teuthology.orchestra.run.smithi169.stderr:2023-12-16T21:33:58.367+0000 7f4054098700 1 -- 172.21.15.169:0/116147483 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f404c05f270 con 0x7f404c0fc860 2023-12-16T21:33:58.374 INFO:teuthology.orchestra.run.smithi169.stderr:2023-12-16T21:33:58.367+0000 7f40427fc700 1 --2- 172.21.15.169:0/116147483 >> [v2:172.21.15.45:6800/2215417738,v1:172.21.15.45:6801/2215417738] conn(0x7f40380366d0 0x7f4038038b80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:33:58.374 INFO:teuthology.orchestra.run.smithi169.stderr:2023-12-16T21:33:58.367+0000 7f40427fc700 1 -- 172.21.15.169:0/116147483 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1151+0+0 (secure 0 0 0) 0x7f404804a1e0 con 0x7f404c0fc860 2023-12-16T21:33:58.374 INFO:teuthology.orchestra.run.smithi169.stderr:2023-12-16T21:33:58.368+0000 7f4051633700 1 --2- 172.21.15.169:0/116147483 >> [v2:172.21.15.45:6800/2215417738,v1:172.21.15.45:6801/2215417738] conn(0x7f40380366d0 0x7f4038038b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:33:58.374 INFO:teuthology.orchestra.run.smithi169.stderr:2023-12-16T21:33:58.368+0000 7f4051633700 1 --2- 172.21.15.169:0/116147483 >> [v2:172.21.15.45:6800/2215417738,v1:172.21.15.45:6801/2215417738] conn(0x7f40380366d0 0x7f4038038b80 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0x7f403c009990 tx=0x7f403c006e40 comp rx=0 tx=0).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:33:58.375 INFO:teuthology.orchestra.run.smithi169.stderr:2023-12-16T21:33:58.372+0000 7f40427fc700 1 -- 172.21.15.169:0/116147483 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+174570 (secure 0 0 0) 0x7f404802eb40 con 0x7f404c0fc860 2023-12-16T21:33:58.565 INFO:teuthology.orchestra.run.smithi169.stderr:2023-12-16T21:33:58.563+0000 7f4054098700 1 -- 172.21.15.169:0/116147483 --> [v2:172.21.15.45:6800/2215417738,v1:172.21.15.45:6801/2215417738] -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "mon", "placement": "2;smithi045:172.21.15.45=smithi045;smithi169:172.21.15.169=smithi169", "target": ["mon-mgr", ""]}) v1 -- 0x7f404c10eaf0 con 0x7f40380366d0 2023-12-16T21:33:58.572 INFO:teuthology.orchestra.run.smithi169.stderr:2023-12-16T21:33:58.570+0000 7f40427fc700 1 -- 172.21.15.169:0/116147483 <== mgr.14162 v2:172.21.15.45:6800/2215417738 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+24 (secure 0 0 0) 0x7f404c10eaf0 con 0x7f40380366d0 2023-12-16T21:33:58.572 INFO:teuthology.orchestra.run.smithi169.stdout:Scheduled mon update... 2023-12-16T21:33:58.574 INFO:teuthology.orchestra.run.smithi169.stderr:2023-12-16T21:33:58.572+0000 7f4054098700 1 -- 172.21.15.169:0/116147483 >> [v2:172.21.15.45:6800/2215417738,v1:172.21.15.45:6801/2215417738] conn(0x7f40380366d0 msgr2=0x7f4038038b80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:33:58.574 INFO:teuthology.orchestra.run.smithi169.stderr:2023-12-16T21:33:58.572+0000 7f4054098700 1 --2- 172.21.15.169:0/116147483 >> [v2:172.21.15.45:6800/2215417738,v1:172.21.15.45:6801/2215417738] conn(0x7f40380366d0 0x7f4038038b80 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0x7f403c009990 tx=0x7f403c006e40 comp rx=0 tx=0).stop 2023-12-16T21:33:58.574 INFO:teuthology.orchestra.run.smithi169.stderr:2023-12-16T21:33:58.572+0000 7f4054098700 1 -- 172.21.15.169:0/116147483 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f404c0fc860 msgr2=0x7f404c10cb00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:33:58.574 INFO:teuthology.orchestra.run.smithi169.stderr:2023-12-16T21:33:58.572+0000 7f4054098700 1 --2- 172.21.15.169:0/116147483 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f404c0fc860 0x7f404c10cb00 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7f4048015040 tx=0x7f4048004d10 comp rx=0 tx=0).stop 2023-12-16T21:33:58.574 INFO:teuthology.orchestra.run.smithi169.stderr:2023-12-16T21:33:58.572+0000 7f4054098700 1 -- 172.21.15.169:0/116147483 shutdown_connections 2023-12-16T21:33:58.574 INFO:teuthology.orchestra.run.smithi169.stderr:2023-12-16T21:33:58.572+0000 7f4054098700 1 --2- 172.21.15.169:0/116147483 >> [v2:172.21.15.45:6800/2215417738,v1:172.21.15.45:6801/2215417738] conn(0x7f40380366d0 0x7f4038038b80 unknown :-1 s=CLOSED pgs=17 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:33:58.574 INFO:teuthology.orchestra.run.smithi169.stderr:2023-12-16T21:33:58.572+0000 7f4054098700 1 --2- 172.21.15.169:0/116147483 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f404c0fc860 0x7f404c10cb00 unknown :-1 s=CLOSED pgs=101 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:33:58.575 INFO:teuthology.orchestra.run.smithi169.stderr:2023-12-16T21:33:58.572+0000 7f4054098700 1 -- 172.21.15.169:0/116147483 >> 172.21.15.169:0/116147483 conn(0x7f404c0f7fe0 msgr2=0x7f404c0f8df0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-16T21:33:58.575 INFO:teuthology.orchestra.run.smithi169.stderr:2023-12-16T21:33:58.572+0000 7f4054098700 1 -- 172.21.15.169:0/116147483 shutdown_connections 2023-12-16T21:33:58.575 INFO:teuthology.orchestra.run.smithi169.stderr:2023-12-16T21:33:58.572+0000 7f4054098700 1 -- 172.21.15.169:0/116147483 wait complete. 2023-12-16T21:33:58.849 DEBUG:teuthology.orchestra.run.smithi169:mon.smithi169> sudo journalctl -f -n 0 -u ceph-88ff363c-9c5a-11ee-95a5-87774f69a715@mon.smithi169.service 2023-12-16T21:33:58.851 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-12-16T21:33:58.851 DEBUG:teuthology.orchestra.run.smithi169:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 88ff363c-9c5a-11ee-95a5-87774f69a715 -- ceph mon dump -f json 2023-12-16T21:33:59.129 INFO:teuthology.orchestra.run.smithi169.stderr:Inferring config /etc/ceph/ceph.conf 2023-12-16T21:33:59.175 INFO:teuthology.orchestra.run.smithi169.stderr:Inferring config /etc/ceph/ceph.conf 2023-12-16T21:33:59.316 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Dec 16 21:33:59 smithi045 ceph-mon[80763]: pgmap v4: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-16T21:33:59.316 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Dec 16 21:33:59 smithi045 ceph-mon[80763]: from='mgr.14162 172.21.15.45:0/142909557' entity='mgr.smithi045.nhtmwc' 2023-12-16T21:33:59.316 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Dec 16 21:33:59 smithi045 ceph-mon[80763]: from='mgr.14162 172.21.15.45:0/142909557' entity='mgr.smithi045.nhtmwc' 2023-12-16T21:33:59.316 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Dec 16 21:33:59 smithi045 ceph-mon[80763]: from='mgr.14162 172.21.15.45:0/142909557' entity='mgr.smithi045.nhtmwc' 2023-12-16T21:33:59.887 INFO:teuthology.orchestra.run.smithi169.stderr:2023-12-16T21:33:59.884+0000 7ff84f087700 1 -- 172.21.15.169:0/2643988803 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff8480664f0 msgr2=0x7ff8480ff480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:33:59.887 INFO:teuthology.orchestra.run.smithi169.stderr:2023-12-16T21:33:59.884+0000 7ff84f087700 1 --2- 172.21.15.169:0/2643988803 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff8480664f0 0x7ff8480ff480 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7ff83c0048c0 tx=0x7ff83c009e30 comp rx=0 tx=0).stop 2023-12-16T21:33:59.887 INFO:teuthology.orchestra.run.smithi169.stderr:2023-12-16T21:33:59.884+0000 7ff84f087700 1 -- 172.21.15.169:0/2643988803 shutdown_connections 2023-12-16T21:33:59.887 INFO:teuthology.orchestra.run.smithi169.stderr:2023-12-16T21:33:59.884+0000 7ff84f087700 1 --2- 172.21.15.169:0/2643988803 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff8480664f0 0x7ff8480ff480 unknown :-1 s=CLOSED pgs=102 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:33:59.887 INFO:teuthology.orchestra.run.smithi169.stderr:2023-12-16T21:33:59.884+0000 7ff84f087700 1 -- 172.21.15.169:0/2643988803 >> 172.21.15.169:0/2643988803 conn(0x7ff8480f6e50 msgr2=0x7ff8480f9260 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-16T21:33:59.887 INFO:teuthology.orchestra.run.smithi169.stderr:2023-12-16T21:33:59.884+0000 7ff84f087700 1 -- 172.21.15.169:0/2643988803 shutdown_connections 2023-12-16T21:33:59.887 INFO:teuthology.orchestra.run.smithi169.stderr:2023-12-16T21:33:59.884+0000 7ff84f087700 1 -- 172.21.15.169:0/2643988803 wait complete. 2023-12-16T21:33:59.888 INFO:teuthology.orchestra.run.smithi169.stderr:2023-12-16T21:33:59.885+0000 7ff84f087700 1 Processor -- start 2023-12-16T21:33:59.888 INFO:teuthology.orchestra.run.smithi169.stderr:2023-12-16T21:33:59.885+0000 7ff84f087700 1 -- start start 2023-12-16T21:33:59.888 INFO:teuthology.orchestra.run.smithi169.stderr:2023-12-16T21:33:59.885+0000 7ff84f087700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff8480664f0 0x7ff8480fcf00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:33:59.888 INFO:teuthology.orchestra.run.smithi169.stderr:2023-12-16T21:33:59.885+0000 7ff84f087700 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff8480fb380 con 0x7ff8480664f0 2023-12-16T21:33:59.890 INFO:teuthology.orchestra.run.smithi169.stderr:2023-12-16T21:33:59.885+0000 7ff84ce23700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff8480664f0 0x7ff8480fcf00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:33:59.890 INFO:teuthology.orchestra.run.smithi169.stderr:2023-12-16T21:33:59.885+0000 7ff84ce23700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff8480664f0 0x7ff8480fcf00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.169:42418/0 (socket says 172.21.15.169:42418) 2023-12-16T21:33:59.890 INFO:teuthology.orchestra.run.smithi169.stderr:2023-12-16T21:33:59.885+0000 7ff84ce23700 1 -- 172.21.15.169:0/2137488713 learned_addr learned my addr 172.21.15.169:0/2137488713 (peer_addr_for_me v2:172.21.15.169:0/0) 2023-12-16T21:33:59.890 INFO:teuthology.orchestra.run.smithi169.stderr:2023-12-16T21:33:59.886+0000 7ff84ce23700 1 -- 172.21.15.169:0/2137488713 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff83c005040 con 0x7ff8480664f0 2023-12-16T21:33:59.890 INFO:teuthology.orchestra.run.smithi169.stderr:2023-12-16T21:33:59.886+0000 7ff84ce23700 1 --2- 172.21.15.169:0/2137488713 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff8480664f0 0x7ff8480fcf00 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7ff83c009040 tx=0x7ff83c009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:33:59.890 INFO:teuthology.orchestra.run.smithi169.stderr:2023-12-16T21:33:59.886+0000 7ff8457fa700 1 -- 172.21.15.169:0/2137488713 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7ff83c009620 con 0x7ff8480664f0 2023-12-16T21:33:59.890 INFO:teuthology.orchestra.run.smithi169.stderr:2023-12-16T21:33:59.886+0000 7ff84f087700 1 -- 172.21.15.169:0/2137488713 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff8480fb580 con 0x7ff8480664f0 2023-12-16T21:33:59.890 INFO:teuthology.orchestra.run.smithi169.stderr:2023-12-16T21:33:59.886+0000 7ff8457fa700 1 -- 172.21.15.169:0/2137488713 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7ff83c009780 con 0x7ff8480664f0 2023-12-16T21:33:59.890 INFO:teuthology.orchestra.run.smithi169.stderr:2023-12-16T21:33:59.886+0000 7ff8457fa700 1 -- 172.21.15.169:0/2137488713 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7ff83c01c5b0 con 0x7ff8480664f0 2023-12-16T21:33:59.890 INFO:teuthology.orchestra.run.smithi169.stderr:2023-12-16T21:33:59.886+0000 7ff84f087700 1 -- 172.21.15.169:0/2137488713 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff8480fba70 con 0x7ff8480664f0 2023-12-16T21:33:59.890 INFO:teuthology.orchestra.run.smithi169.stderr:2023-12-16T21:33:59.887+0000 7ff8457fa700 1 -- 172.21.15.169:0/2137488713 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 13) v1 ==== 43337+0+0 (secure 0 0 0) 0x7ff83c013420 con 0x7ff8480664f0 2023-12-16T21:33:59.890 INFO:teuthology.orchestra.run.smithi169.stderr:2023-12-16T21:33:59.887+0000 7ff84f087700 1 -- 172.21.15.169:0/2137488713 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff848106500 con 0x7ff8480664f0 2023-12-16T21:33:59.890 INFO:teuthology.orchestra.run.smithi169.stderr:2023-12-16T21:33:59.887+0000 7ff8457fa700 1 --2- 172.21.15.169:0/2137488713 >> [v2:172.21.15.45:6800/2215417738,v1:172.21.15.45:6801/2215417738] conn(0x7ff8380366d0 0x7ff838038b80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:33:59.891 INFO:teuthology.orchestra.run.smithi169.stderr:2023-12-16T21:33:59.887+0000 7ff8457fa700 1 -- 172.21.15.169:0/2137488713 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1151+0+0 (secure 0 0 0) 0x7ff83c0498e0 con 0x7ff8480664f0 2023-12-16T21:33:59.891 INFO:teuthology.orchestra.run.smithi169.stderr:2023-12-16T21:33:59.887+0000 7ff847fff700 1 --2- 172.21.15.169:0/2137488713 >> [v2:172.21.15.45:6800/2215417738,v1:172.21.15.45:6801/2215417738] conn(0x7ff8380366d0 0x7ff838038b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:33:59.891 INFO:teuthology.orchestra.run.smithi169.stderr:2023-12-16T21:33:59.888+0000 7ff847fff700 1 --2- 172.21.15.169:0/2137488713 >> [v2:172.21.15.45:6800/2215417738,v1:172.21.15.45:6801/2215417738] conn(0x7ff8380366d0 0x7ff838038b80 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7ff834009990 tx=0x7ff834006e40 comp rx=0 tx=0).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:33:59.894 INFO:teuthology.orchestra.run.smithi169.stderr:2023-12-16T21:33:59.892+0000 7ff8457fa700 1 -- 172.21.15.169:0/2137488713 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+174570 (secure 0 0 0) 0x7ff83c011370 con 0x7ff8480664f0 2023-12-16T21:34:00.158 INFO:teuthology.orchestra.run.smithi169.stderr:2023-12-16T21:34:00.156+0000 7ff84f087700 1 -- 172.21.15.169:0/2137488713 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7ff84802b180 con 0x7ff8480664f0 2023-12-16T21:34:00.158 INFO:teuthology.orchestra.run.smithi169.stderr:2023-12-16T21:34:00.156+0000 7ff8457fa700 1 -- 172.21.15.169:0/2137488713 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+739 (secure 0 0 0) 0x7ff83c011370 con 0x7ff8480664f0 2023-12-16T21:34:00.159 INFO:teuthology.orchestra.run.smithi169.stdout: 2023-12-16T21:34:00.159 INFO:teuthology.orchestra.run.smithi169.stdout:{"epoch":1,"fsid":"88ff363c-9c5a-11ee-95a5-87774f69a715","modified":"2023-12-16T21:32:46.744221Z","created":"2023-12-16T21:32:46.744221Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"smithi045","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:3300","nonce":0},{"type":"v1","addr":"172.21.15.45:6789","nonce":0}]},"addr":"172.21.15.45:6789/0","public_addr":"172.21.15.45:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-12-16T21:34:00.161 INFO:teuthology.orchestra.run.smithi169.stderr:2023-12-16T21:34:00.158+0000 7ff84f087700 1 -- 172.21.15.169:0/2137488713 >> [v2:172.21.15.45:6800/2215417738,v1:172.21.15.45:6801/2215417738] conn(0x7ff8380366d0 msgr2=0x7ff838038b80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:34:00.161 INFO:teuthology.orchestra.run.smithi169.stderr:2023-12-16T21:34:00.158+0000 7ff84f087700 1 --2- 172.21.15.169:0/2137488713 >> [v2:172.21.15.45:6800/2215417738,v1:172.21.15.45:6801/2215417738] conn(0x7ff8380366d0 0x7ff838038b80 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7ff834009990 tx=0x7ff834006e40 comp rx=0 tx=0).stop 2023-12-16T21:34:00.161 INFO:teuthology.orchestra.run.smithi169.stderr:2023-12-16T21:34:00.158+0000 7ff84f087700 1 -- 172.21.15.169:0/2137488713 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff8480664f0 msgr2=0x7ff8480fcf00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:34:00.161 INFO:teuthology.orchestra.run.smithi169.stderr:2023-12-16T21:34:00.158+0000 7ff84f087700 1 --2- 172.21.15.169:0/2137488713 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff8480664f0 0x7ff8480fcf00 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7ff83c009040 tx=0x7ff83c009e30 comp rx=0 tx=0).stop 2023-12-16T21:34:00.161 INFO:teuthology.orchestra.run.smithi169.stderr:2023-12-16T21:34:00.158+0000 7ff84f087700 1 -- 172.21.15.169:0/2137488713 shutdown_connections 2023-12-16T21:34:00.161 INFO:teuthology.orchestra.run.smithi169.stderr:2023-12-16T21:34:00.158+0000 7ff84f087700 1 --2- 172.21.15.169:0/2137488713 >> [v2:172.21.15.45:6800/2215417738,v1:172.21.15.45:6801/2215417738] conn(0x7ff8380366d0 0x7ff838038b80 unknown :-1 s=CLOSED pgs=18 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:34:00.161 INFO:teuthology.orchestra.run.smithi169.stderr:2023-12-16T21:34:00.158+0000 7ff84f087700 1 --2- 172.21.15.169:0/2137488713 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff8480664f0 0x7ff8480fcf00 unknown :-1 s=CLOSED pgs=103 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:34:00.161 INFO:teuthology.orchestra.run.smithi169.stderr:2023-12-16T21:34:00.158+0000 7ff84f087700 1 -- 172.21.15.169:0/2137488713 >> 172.21.15.169:0/2137488713 conn(0x7ff8480f6e50 msgr2=0x7ff8480f7bf0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-16T21:34:00.161 INFO:teuthology.orchestra.run.smithi169.stderr:2023-12-16T21:34:00.158+0000 7ff84f087700 1 -- 172.21.15.169:0/2137488713 shutdown_connections 2023-12-16T21:34:00.161 INFO:teuthology.orchestra.run.smithi169.stderr:2023-12-16T21:34:00.158+0000 7ff84f087700 1 -- 172.21.15.169:0/2137488713 wait complete. 2023-12-16T21:34:00.162 INFO:teuthology.orchestra.run.smithi169.stderr:dumped monmap epoch 1 2023-12-16T21:34:00.566 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Dec 16 21:34:00 smithi045 ceph-mon[80763]: from='client.14194 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mon", "placement": "2;smithi045:172.21.15.45=smithi045;smithi169:172.21.15.169=smithi169", "target": ["mon-mgr", ""]}]: dispatch 2023-12-16T21:34:00.566 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Dec 16 21:34:00 smithi045 ceph-mon[80763]: Saving service mon spec with placement smithi045:172.21.15.45=smithi045;smithi169:172.21.15.169=smithi169;count:2 2023-12-16T21:34:00.566 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Dec 16 21:34:00 smithi045 ceph-mon[80763]: from='mgr.14162 172.21.15.45:0/142909557' entity='mgr.smithi045.nhtmwc' 2023-12-16T21:34:00.566 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Dec 16 21:34:00 smithi045 ceph-mon[80763]: from='mgr.14162 172.21.15.45:0/142909557' entity='mgr.smithi045.nhtmwc' 2023-12-16T21:34:00.566 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Dec 16 21:34:00 smithi045 ceph-mon[80763]: from='mgr.14162 172.21.15.45:0/142909557' entity='mgr.smithi045.nhtmwc' 2023-12-16T21:34:00.566 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Dec 16 21:34:00 smithi045 ceph-mon[80763]: from='mgr.14162 172.21.15.45:0/142909557' entity='mgr.smithi045.nhtmwc' 2023-12-16T21:34:00.566 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Dec 16 21:34:00 smithi045 ceph-mon[80763]: from='mgr.14162 172.21.15.45:0/142909557' entity='mgr.smithi045.nhtmwc' cmd=[{"prefix": "dashboard set-grafana-api-ssl-verify", "value": "false"}]: dispatch 2023-12-16T21:34:00.566 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Dec 16 21:34:00 smithi045 ceph-mon[80763]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard set-grafana-api-ssl-verify", "value": "false"}]: dispatch 2023-12-16T21:34:00.566 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Dec 16 21:34:00 smithi045 ceph-mon[80763]: from='mgr.14162 172.21.15.45:0/142909557' entity='mgr.smithi045.nhtmwc' 2023-12-16T21:34:00.566 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Dec 16 21:34:00 smithi045 ceph-mon[80763]: Deploying daemon grafana.smithi045 on smithi045 2023-12-16T21:34:01.316 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Dec 16 21:34:01 smithi045 ceph-mon[80763]: pgmap v5: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-16T21:34:01.316 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Dec 16 21:34:01 smithi045 ceph-mon[80763]: from='client.? 172.21.15.169:0/2137488713' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-12-16T21:34:01.432 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-12-16T21:34:01.432 DEBUG:teuthology.orchestra.run.smithi169:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 88ff363c-9c5a-11ee-95a5-87774f69a715 -- ceph mon dump -f json 2023-12-16T21:34:01.679 INFO:teuthology.orchestra.run.smithi169.stderr:Inferring config /etc/ceph/ceph.conf 2023-12-16T21:34:01.725 INFO:teuthology.orchestra.run.smithi169.stderr:Inferring config /etc/ceph/ceph.conf 2023-12-16T21:34:02.430 INFO:teuthology.orchestra.run.smithi169.stderr:2023-12-16T21:34:02.427+0000 7ff287fd2700 1 -- 172.21.15.169:0/2206072543 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff2800fc880 msgr2=0x7ff2800fcc90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:34:02.430 INFO:teuthology.orchestra.run.smithi169.stderr:2023-12-16T21:34:02.427+0000 7ff287fd2700 1 --2- 172.21.15.169:0/2206072543 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff2800fc880 0x7ff2800fcc90 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7ff27c0048c0 tx=0x7ff27c009e30 comp rx=0 tx=0).stop 2023-12-16T21:34:02.430 INFO:teuthology.orchestra.run.smithi169.stderr:2023-12-16T21:34:02.427+0000 7ff287fd2700 1 -- 172.21.15.169:0/2206072543 shutdown_connections 2023-12-16T21:34:02.430 INFO:teuthology.orchestra.run.smithi169.stderr:2023-12-16T21:34:02.427+0000 7ff287fd2700 1 --2- 172.21.15.169:0/2206072543 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff2800fc880 0x7ff2800fcc90 unknown :-1 s=CLOSED pgs=104 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:34:02.430 INFO:teuthology.orchestra.run.smithi169.stderr:2023-12-16T21:34:02.427+0000 7ff287fd2700 1 -- 172.21.15.169:0/2206072543 >> 172.21.15.169:0/2206072543 conn(0x7ff2800f7f40 msgr2=0x7ff2800fa370 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-16T21:34:02.431 INFO:teuthology.orchestra.run.smithi169.stderr:2023-12-16T21:34:02.428+0000 7ff287fd2700 1 -- 172.21.15.169:0/2206072543 shutdown_connections 2023-12-16T21:34:02.431 INFO:teuthology.orchestra.run.smithi169.stderr:2023-12-16T21:34:02.428+0000 7ff287fd2700 1 -- 172.21.15.169:0/2206072543 wait complete. 2023-12-16T21:34:02.433 INFO:teuthology.orchestra.run.smithi169.stderr:2023-12-16T21:34:02.428+0000 7ff287fd2700 1 Processor -- start 2023-12-16T21:34:02.433 INFO:teuthology.orchestra.run.smithi169.stderr:2023-12-16T21:34:02.428+0000 7ff287fd2700 1 -- start start 2023-12-16T21:34:02.433 INFO:teuthology.orchestra.run.smithi169.stderr:2023-12-16T21:34:02.428+0000 7ff287fd2700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff28010ccc0 0x7ff28010d0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:34:02.433 INFO:teuthology.orchestra.run.smithi169.stderr:2023-12-16T21:34:02.428+0000 7ff287fd2700 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff28010d610 con 0x7ff28010ccc0 2023-12-16T21:34:02.433 INFO:teuthology.orchestra.run.smithi169.stderr:2023-12-16T21:34:02.429+0000 7ff285d6e700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff28010ccc0 0x7ff28010d0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:34:02.433 INFO:teuthology.orchestra.run.smithi169.stderr:2023-12-16T21:34:02.429+0000 7ff285d6e700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff28010ccc0 0x7ff28010d0d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.169:39482/0 (socket says 172.21.15.169:39482) 2023-12-16T21:34:02.433 INFO:teuthology.orchestra.run.smithi169.stderr:2023-12-16T21:34:02.429+0000 7ff285d6e700 1 -- 172.21.15.169:0/799541085 learned_addr learned my addr 172.21.15.169:0/799541085 (peer_addr_for_me v2:172.21.15.169:0/0) 2023-12-16T21:34:02.433 INFO:teuthology.orchestra.run.smithi169.stderr:2023-12-16T21:34:02.429+0000 7ff285d6e700 1 -- 172.21.15.169:0/799541085 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff27c005040 con 0x7ff28010ccc0 2023-12-16T21:34:02.433 INFO:teuthology.orchestra.run.smithi169.stderr:2023-12-16T21:34:02.429+0000 7ff285d6e700 1 --2- 172.21.15.169:0/799541085 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff28010ccc0 0x7ff28010d0d0 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7ff27c009040 tx=0x7ff27c009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:34:02.434 INFO:teuthology.orchestra.run.smithi169.stderr:2023-12-16T21:34:02.429+0000 7ff2767fc700 1 -- 172.21.15.169:0/799541085 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7ff27c004e60 con 0x7ff28010ccc0 2023-12-16T21:34:02.434 INFO:teuthology.orchestra.run.smithi169.stderr:2023-12-16T21:34:02.429+0000 7ff2767fc700 1 -- 172.21.15.169:0/799541085 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7ff27c00be50 con 0x7ff28010ccc0 2023-12-16T21:34:02.434 INFO:teuthology.orchestra.run.smithi169.stderr:2023-12-16T21:34:02.429+0000 7ff287fd2700 1 -- 172.21.15.169:0/799541085 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff28010dfa0 con 0x7ff28010ccc0 2023-12-16T21:34:02.434 INFO:teuthology.orchestra.run.smithi169.stderr:2023-12-16T21:34:02.429+0000 7ff2767fc700 1 -- 172.21.15.169:0/799541085 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7ff27c01c3f0 con 0x7ff28010ccc0 2023-12-16T21:34:02.434 INFO:teuthology.orchestra.run.smithi169.stderr:2023-12-16T21:34:02.429+0000 7ff287fd2700 1 -- 172.21.15.169:0/799541085 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff28010e490 con 0x7ff28010ccc0 2023-12-16T21:34:02.434 INFO:teuthology.orchestra.run.smithi169.stderr:2023-12-16T21:34:02.430+0000 7ff2767fc700 1 -- 172.21.15.169:0/799541085 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 13) v1 ==== 43337+0+0 (secure 0 0 0) 0x7ff27c013420 con 0x7ff28010ccc0 2023-12-16T21:34:02.434 INFO:teuthology.orchestra.run.smithi169.stderr:2023-12-16T21:34:02.430+0000 7ff287fd2700 1 -- 172.21.15.169:0/799541085 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff28005f270 con 0x7ff28010ccc0 2023-12-16T21:34:02.434 INFO:teuthology.orchestra.run.smithi169.stderr:2023-12-16T21:34:02.430+0000 7ff2767fc700 1 --2- 172.21.15.169:0/799541085 >> [v2:172.21.15.45:6800/2215417738,v1:172.21.15.45:6801/2215417738] conn(0x7ff26c036720 0x7ff26c038bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:34:02.434 INFO:teuthology.orchestra.run.smithi169.stderr:2023-12-16T21:34:02.430+0000 7ff2767fc700 1 -- 172.21.15.169:0/799541085 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1151+0+0 (secure 0 0 0) 0x7ff27c0498c0 con 0x7ff28010ccc0 2023-12-16T21:34:02.434 INFO:teuthology.orchestra.run.smithi169.stderr:2023-12-16T21:34:02.430+0000 7ff28556d700 1 --2- 172.21.15.169:0/799541085 >> [v2:172.21.15.45:6800/2215417738,v1:172.21.15.45:6801/2215417738] conn(0x7ff26c036720 0x7ff26c038bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:34:02.434 INFO:teuthology.orchestra.run.smithi169.stderr:2023-12-16T21:34:02.431+0000 7ff28556d700 1 --2- 172.21.15.169:0/799541085 >> [v2:172.21.15.45:6800/2215417738,v1:172.21.15.45:6801/2215417738] conn(0x7ff26c036720 0x7ff26c038bd0 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7ff270009990 tx=0x7ff270006e40 comp rx=0 tx=0).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:34:02.437 INFO:teuthology.orchestra.run.smithi169.stderr:2023-12-16T21:34:02.435+0000 7ff2767fc700 1 -- 172.21.15.169:0/799541085 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+174570 (secure 0 0 0) 0x7ff27c0119c0 con 0x7ff28010ccc0 2023-12-16T21:34:02.698 INFO:teuthology.orchestra.run.smithi169.stderr:2023-12-16T21:34:02.696+0000 7ff287fd2700 1 -- 172.21.15.169:0/799541085 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7ff28004bb80 con 0x7ff28010ccc0 2023-12-16T21:34:02.699 INFO:teuthology.orchestra.run.smithi169.stderr:2023-12-16T21:34:02.697+0000 7ff2767fc700 1 -- 172.21.15.169:0/799541085 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+739 (secure 0 0 0) 0x7ff27c011370 con 0x7ff28010ccc0 2023-12-16T21:34:02.699 INFO:teuthology.orchestra.run.smithi169.stdout: 2023-12-16T21:34:02.699 INFO:teuthology.orchestra.run.smithi169.stdout:{"epoch":1,"fsid":"88ff363c-9c5a-11ee-95a5-87774f69a715","modified":"2023-12-16T21:32:46.744221Z","created":"2023-12-16T21:32:46.744221Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"smithi045","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:3300","nonce":0},{"type":"v1","addr":"172.21.15.45:6789","nonce":0}]},"addr":"172.21.15.45:6789/0","public_addr":"172.21.15.45:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-12-16T21:34:02.701 INFO:teuthology.orchestra.run.smithi169.stderr:2023-12-16T21:34:02.699+0000 7ff287fd2700 1 -- 172.21.15.169:0/799541085 >> [v2:172.21.15.45:6800/2215417738,v1:172.21.15.45:6801/2215417738] conn(0x7ff26c036720 msgr2=0x7ff26c038bd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:34:02.701 INFO:teuthology.orchestra.run.smithi169.stderr:2023-12-16T21:34:02.699+0000 7ff287fd2700 1 --2- 172.21.15.169:0/799541085 >> [v2:172.21.15.45:6800/2215417738,v1:172.21.15.45:6801/2215417738] conn(0x7ff26c036720 0x7ff26c038bd0 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7ff270009990 tx=0x7ff270006e40 comp rx=0 tx=0).stop 2023-12-16T21:34:02.701 INFO:teuthology.orchestra.run.smithi169.stderr:2023-12-16T21:34:02.699+0000 7ff287fd2700 1 -- 172.21.15.169:0/799541085 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff28010ccc0 msgr2=0x7ff28010d0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:34:02.701 INFO:teuthology.orchestra.run.smithi169.stderr:2023-12-16T21:34:02.699+0000 7ff287fd2700 1 --2- 172.21.15.169:0/799541085 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff28010ccc0 0x7ff28010d0d0 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7ff27c009040 tx=0x7ff27c009e30 comp rx=0 tx=0).stop 2023-12-16T21:34:02.701 INFO:teuthology.orchestra.run.smithi169.stderr:2023-12-16T21:34:02.699+0000 7ff287fd2700 1 -- 172.21.15.169:0/799541085 shutdown_connections 2023-12-16T21:34:02.701 INFO:teuthology.orchestra.run.smithi169.stderr:2023-12-16T21:34:02.699+0000 7ff287fd2700 1 --2- 172.21.15.169:0/799541085 >> [v2:172.21.15.45:6800/2215417738,v1:172.21.15.45:6801/2215417738] conn(0x7ff26c036720 0x7ff26c038bd0 unknown :-1 s=CLOSED pgs=19 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:34:02.702 INFO:teuthology.orchestra.run.smithi169.stderr:2023-12-16T21:34:02.699+0000 7ff287fd2700 1 --2- 172.21.15.169:0/799541085 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff28010ccc0 0x7ff28010d0d0 unknown :-1 s=CLOSED pgs=105 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:34:02.702 INFO:teuthology.orchestra.run.smithi169.stderr:2023-12-16T21:34:02.699+0000 7ff287fd2700 1 -- 172.21.15.169:0/799541085 >> 172.21.15.169:0/799541085 conn(0x7ff2800f7f40 msgr2=0x7ff2800f8e20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-16T21:34:02.702 INFO:teuthology.orchestra.run.smithi169.stderr:2023-12-16T21:34:02.699+0000 7ff287fd2700 1 -- 172.21.15.169:0/799541085 shutdown_connections 2023-12-16T21:34:02.702 INFO:teuthology.orchestra.run.smithi169.stderr:2023-12-16T21:34:02.699+0000 7ff287fd2700 1 -- 172.21.15.169:0/799541085 wait complete. 2023-12-16T21:34:02.702 INFO:teuthology.orchestra.run.smithi169.stderr:dumped monmap epoch 1 2023-12-16T21:34:03.566 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Dec 16 21:34:03 smithi045 ceph-mon[80763]: pgmap v6: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-16T21:34:03.566 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Dec 16 21:34:03 smithi045 ceph-mon[80763]: from='client.? 172.21.15.169:0/799541085' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-12-16T21:34:03.566 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Dec 16 21:34:03 smithi045 ceph-mon[80763]: from='mgr.14162 172.21.15.45:0/142909557' entity='mgr.smithi045.nhtmwc' 2023-12-16T21:34:03.965 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-12-16T21:34:03.965 DEBUG:teuthology.orchestra.run.smithi169:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 88ff363c-9c5a-11ee-95a5-87774f69a715 -- ceph mon dump -f json 2023-12-16T21:34:04.214 INFO:teuthology.orchestra.run.smithi169.stderr:Inferring config /etc/ceph/ceph.conf 2023-12-16T21:34:04.260 INFO:teuthology.orchestra.run.smithi169.stderr:Inferring config /etc/ceph/ceph.conf 2023-12-16T21:34:05.029 INFO:teuthology.orchestra.run.smithi169.stderr:2023-12-16T21:34:05.026+0000 7f4caf3ae700 1 -- 172.21.15.169:0/626906727 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f4ca80fd3b0 msgr2=0x7f4ca80ff790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:34:05.029 INFO:teuthology.orchestra.run.smithi169.stderr:2023-12-16T21:34:05.026+0000 7f4caf3ae700 1 --2- 172.21.15.169:0/626906727 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f4ca80fd3b0 0x7f4ca80ff790 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7f4c98009c60 tx=0x7f4c98004d10 comp rx=0 tx=0).stop 2023-12-16T21:34:05.029 INFO:teuthology.orchestra.run.smithi169.stderr:2023-12-16T21:34:05.026+0000 7f4caf3ae700 1 -- 172.21.15.169:0/626906727 shutdown_connections 2023-12-16T21:34:05.029 INFO:teuthology.orchestra.run.smithi169.stderr:2023-12-16T21:34:05.026+0000 7f4caf3ae700 1 --2- 172.21.15.169:0/626906727 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f4ca80fd3b0 0x7f4ca80ff790 unknown :-1 s=CLOSED pgs=106 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:34:05.029 INFO:teuthology.orchestra.run.smithi169.stderr:2023-12-16T21:34:05.026+0000 7f4caf3ae700 1 -- 172.21.15.169:0/626906727 >> 172.21.15.169:0/626906727 conn(0x7f4ca80f6e50 msgr2=0x7f4ca80f9260 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-16T21:34:05.029 INFO:teuthology.orchestra.run.smithi169.stderr:2023-12-16T21:34:05.026+0000 7f4caf3ae700 1 -- 172.21.15.169:0/626906727 shutdown_connections 2023-12-16T21:34:05.030 INFO:teuthology.orchestra.run.smithi169.stderr:2023-12-16T21:34:05.026+0000 7f4caf3ae700 1 -- 172.21.15.169:0/626906727 wait complete. 2023-12-16T21:34:05.030 INFO:teuthology.orchestra.run.smithi169.stderr:2023-12-16T21:34:05.026+0000 7f4caf3ae700 1 Processor -- start 2023-12-16T21:34:05.030 INFO:teuthology.orchestra.run.smithi169.stderr:2023-12-16T21:34:05.026+0000 7f4caf3ae700 1 -- start start 2023-12-16T21:34:05.030 INFO:teuthology.orchestra.run.smithi169.stderr:2023-12-16T21:34:05.027+0000 7f4caf3ae700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f4ca80fd3b0 0x7f4ca80fce40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:34:05.030 INFO:teuthology.orchestra.run.smithi169.stderr:2023-12-16T21:34:05.027+0000 7f4caf3ae700 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4ca80fb2c0 con 0x7f4ca80fd3b0 2023-12-16T21:34:05.030 INFO:teuthology.orchestra.run.smithi169.stderr:2023-12-16T21:34:05.027+0000 7f4cad14a700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f4ca80fd3b0 0x7f4ca80fce40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:34:05.030 INFO:teuthology.orchestra.run.smithi169.stderr:2023-12-16T21:34:05.027+0000 7f4cad14a700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f4ca80fd3b0 0x7f4ca80fce40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.169:39510/0 (socket says 172.21.15.169:39510) 2023-12-16T21:34:05.030 INFO:teuthology.orchestra.run.smithi169.stderr:2023-12-16T21:34:05.027+0000 7f4cad14a700 1 -- 172.21.15.169:0/1852407003 learned_addr learned my addr 172.21.15.169:0/1852407003 (peer_addr_for_me v2:172.21.15.169:0/0) 2023-12-16T21:34:05.030 INFO:teuthology.orchestra.run.smithi169.stderr:2023-12-16T21:34:05.027+0000 7f4cad14a700 1 -- 172.21.15.169:0/1852407003 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4c98009940 con 0x7f4ca80fd3b0 2023-12-16T21:34:05.031 INFO:teuthology.orchestra.run.smithi169.stderr:2023-12-16T21:34:05.027+0000 7f4cad14a700 1 --2- 172.21.15.169:0/1852407003 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f4ca80fd3b0 0x7f4ca80fce40 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7f4c98007b30 tx=0x7f4c98004d10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:34:05.032 INFO:teuthology.orchestra.run.smithi169.stderr:2023-12-16T21:34:05.027+0000 7f4ca5ffb700 1 -- 172.21.15.169:0/1852407003 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f4c9800bec0 con 0x7f4ca80fd3b0 2023-12-16T21:34:05.032 INFO:teuthology.orchestra.run.smithi169.stderr:2023-12-16T21:34:05.027+0000 7f4ca5ffb700 1 -- 172.21.15.169:0/1852407003 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f4c9801d8f0 con 0x7f4ca80fd3b0 2023-12-16T21:34:05.032 INFO:teuthology.orchestra.run.smithi169.stderr:2023-12-16T21:34:05.027+0000 7f4caf3ae700 1 -- 172.21.15.169:0/1852407003 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4ca80fb4c0 con 0x7f4ca80fd3b0 2023-12-16T21:34:05.032 INFO:teuthology.orchestra.run.smithi169.stderr:2023-12-16T21:34:05.028+0000 7f4ca5ffb700 1 -- 172.21.15.169:0/1852407003 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f4c980128e0 con 0x7f4ca80fd3b0 2023-12-16T21:34:05.032 INFO:teuthology.orchestra.run.smithi169.stderr:2023-12-16T21:34:05.028+0000 7f4caf3ae700 1 -- 172.21.15.169:0/1852407003 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4ca80fb9b0 con 0x7f4ca80fd3b0 2023-12-16T21:34:05.032 INFO:teuthology.orchestra.run.smithi169.stderr:2023-12-16T21:34:05.028+0000 7f4ca5ffb700 1 -- 172.21.15.169:0/1852407003 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 13) v1 ==== 43337+0+0 (secure 0 0 0) 0x7f4c98019040 con 0x7f4ca80fd3b0 2023-12-16T21:34:05.032 INFO:teuthology.orchestra.run.smithi169.stderr:2023-12-16T21:34:05.028+0000 7f4ca5ffb700 1 --2- 172.21.15.169:0/1852407003 >> [v2:172.21.15.45:6800/2215417738,v1:172.21.15.45:6801/2215417738] conn(0x7f4c90036720 0x7f4c90038bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:34:05.032 INFO:teuthology.orchestra.run.smithi169.stderr:2023-12-16T21:34:05.029+0000 7f4ca5ffb700 1 -- 172.21.15.169:0/1852407003 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1151+0+0 (secure 0 0 0) 0x7f4c9804a340 con 0x7f4ca80fd3b0 2023-12-16T21:34:05.032 INFO:teuthology.orchestra.run.smithi169.stderr:2023-12-16T21:34:05.029+0000 7f4cac949700 1 --2- 172.21.15.169:0/1852407003 >> [v2:172.21.15.45:6800/2215417738,v1:172.21.15.45:6801/2215417738] conn(0x7f4c90036720 0x7f4c90038bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:34:05.032 INFO:teuthology.orchestra.run.smithi169.stderr:2023-12-16T21:34:05.029+0000 7f4caf3ae700 1 -- 172.21.15.169:0/1852407003 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4ca805f270 con 0x7f4ca80fd3b0 2023-12-16T21:34:05.032 INFO:teuthology.orchestra.run.smithi169.stderr:2023-12-16T21:34:05.029+0000 7f4cac949700 1 --2- 172.21.15.169:0/1852407003 >> [v2:172.21.15.45:6800/2215417738,v1:172.21.15.45:6801/2215417738] conn(0x7f4c90036720 0x7f4c90038bd0 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7f4c9c009990 tx=0x7f4c9c006e40 comp rx=0 tx=0).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:34:05.036 INFO:teuthology.orchestra.run.smithi169.stderr:2023-12-16T21:34:05.034+0000 7f4ca5ffb700 1 -- 172.21.15.169:0/1852407003 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+174570 (secure 0 0 0) 0x7f4c98021020 con 0x7f4ca80fd3b0 2023-12-16T21:34:05.296 INFO:teuthology.orchestra.run.smithi169.stderr:2023-12-16T21:34:05.294+0000 7f4caf3ae700 1 -- 172.21.15.169:0/1852407003 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f4ca804bb80 con 0x7f4ca80fd3b0 2023-12-16T21:34:05.296 INFO:teuthology.orchestra.run.smithi169.stderr:2023-12-16T21:34:05.294+0000 7f4ca5ffb700 1 -- 172.21.15.169:0/1852407003 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+739 (secure 0 0 0) 0x7f4c98035090 con 0x7f4ca80fd3b0 2023-12-16T21:34:05.297 INFO:teuthology.orchestra.run.smithi169.stdout: 2023-12-16T21:34:05.297 INFO:teuthology.orchestra.run.smithi169.stdout:{"epoch":1,"fsid":"88ff363c-9c5a-11ee-95a5-87774f69a715","modified":"2023-12-16T21:32:46.744221Z","created":"2023-12-16T21:32:46.744221Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"smithi045","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:3300","nonce":0},{"type":"v1","addr":"172.21.15.45:6789","nonce":0}]},"addr":"172.21.15.45:6789/0","public_addr":"172.21.15.45:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-12-16T21:34:05.298 INFO:teuthology.orchestra.run.smithi169.stderr:2023-12-16T21:34:05.296+0000 7f4caf3ae700 1 -- 172.21.15.169:0/1852407003 >> [v2:172.21.15.45:6800/2215417738,v1:172.21.15.45:6801/2215417738] conn(0x7f4c90036720 msgr2=0x7f4c90038bd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:34:05.298 INFO:teuthology.orchestra.run.smithi169.stderr:2023-12-16T21:34:05.296+0000 7f4caf3ae700 1 --2- 172.21.15.169:0/1852407003 >> [v2:172.21.15.45:6800/2215417738,v1:172.21.15.45:6801/2215417738] conn(0x7f4c90036720 0x7f4c90038bd0 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7f4c9c009990 tx=0x7f4c9c006e40 comp rx=0 tx=0).stop 2023-12-16T21:34:05.298 INFO:teuthology.orchestra.run.smithi169.stderr:2023-12-16T21:34:05.296+0000 7f4caf3ae700 1 -- 172.21.15.169:0/1852407003 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f4ca80fd3b0 msgr2=0x7f4ca80fce40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:34:05.298 INFO:teuthology.orchestra.run.smithi169.stderr:2023-12-16T21:34:05.296+0000 7f4caf3ae700 1 --2- 172.21.15.169:0/1852407003 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f4ca80fd3b0 0x7f4ca80fce40 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7f4c98007b30 tx=0x7f4c98004d10 comp rx=0 tx=0).stop 2023-12-16T21:34:05.299 INFO:teuthology.orchestra.run.smithi169.stderr:2023-12-16T21:34:05.296+0000 7f4caf3ae700 1 -- 172.21.15.169:0/1852407003 shutdown_connections 2023-12-16T21:34:05.299 INFO:teuthology.orchestra.run.smithi169.stderr:2023-12-16T21:34:05.296+0000 7f4caf3ae700 1 --2- 172.21.15.169:0/1852407003 >> [v2:172.21.15.45:6800/2215417738,v1:172.21.15.45:6801/2215417738] conn(0x7f4c90036720 0x7f4c90038bd0 unknown :-1 s=CLOSED pgs=20 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:34:05.299 INFO:teuthology.orchestra.run.smithi169.stderr:2023-12-16T21:34:05.296+0000 7f4caf3ae700 1 --2- 172.21.15.169:0/1852407003 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f4ca80fd3b0 0x7f4ca80fce40 unknown :-1 s=CLOSED pgs=107 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:34:05.299 INFO:teuthology.orchestra.run.smithi169.stderr:2023-12-16T21:34:05.296+0000 7f4caf3ae700 1 -- 172.21.15.169:0/1852407003 >> 172.21.15.169:0/1852407003 conn(0x7f4ca80f6e50 msgr2=0x7f4ca80f7cf0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-16T21:34:05.299 INFO:teuthology.orchestra.run.smithi169.stderr:2023-12-16T21:34:05.296+0000 7f4caf3ae700 1 -- 172.21.15.169:0/1852407003 shutdown_connections 2023-12-16T21:34:05.299 INFO:teuthology.orchestra.run.smithi169.stderr:2023-12-16T21:34:05.296+0000 7f4caf3ae700 1 -- 172.21.15.169:0/1852407003 wait complete. 2023-12-16T21:34:05.299 INFO:teuthology.orchestra.run.smithi169.stderr:dumped monmap epoch 1 2023-12-16T21:34:05.565 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Dec 16 21:34:05 smithi045 ceph-mon[80763]: pgmap v7: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-16T21:34:06.565 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-12-16T21:34:06.565 DEBUG:teuthology.orchestra.run.smithi169:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 88ff363c-9c5a-11ee-95a5-87774f69a715 -- ceph mon dump -f json 2023-12-16T21:34:06.566 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Dec 16 21:34:06 smithi045 ceph-mon[80763]: from='client.? 172.21.15.169:0/1852407003' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-12-16T21:34:06.809 INFO:teuthology.orchestra.run.smithi169.stderr:Inferring config /etc/ceph/ceph.conf 2023-12-16T21:34:06.856 INFO:teuthology.orchestra.run.smithi169.stderr:Inferring config /etc/ceph/ceph.conf 2023-12-16T21:34:07.565 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Dec 16 21:34:07 smithi045 ceph-mon[80763]: pgmap v8: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-16T21:34:07.682 INFO:teuthology.orchestra.run.smithi169.stderr:2023-12-16T21:34:07.676+0000 7f421d3e5700 1 -- 172.21.15.169:0/1343309250 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f42180fb780 msgr2=0x7f42180fbb90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:34:07.682 INFO:teuthology.orchestra.run.smithi169.stderr:2023-12-16T21:34:07.676+0000 7f421d3e5700 1 --2- 172.21.15.169:0/1343309250 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f42180fb780 0x7f42180fbb90 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7f42080048c0 tx=0x7f4208009e30 comp rx=0 tx=0).stop 2023-12-16T21:34:07.682 INFO:teuthology.orchestra.run.smithi169.stderr:2023-12-16T21:34:07.676+0000 7f421d3e5700 1 -- 172.21.15.169:0/1343309250 shutdown_connections 2023-12-16T21:34:07.682 INFO:teuthology.orchestra.run.smithi169.stderr:2023-12-16T21:34:07.676+0000 7f421d3e5700 1 --2- 172.21.15.169:0/1343309250 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f42180fb780 0x7f42180fbb90 unknown :-1 s=CLOSED pgs=108 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:34:07.683 INFO:teuthology.orchestra.run.smithi169.stderr:2023-12-16T21:34:07.676+0000 7f421d3e5700 1 -- 172.21.15.169:0/1343309250 >> 172.21.15.169:0/1343309250 conn(0x7f42180f6e40 msgr2=0x7f42180f9270 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-16T21:34:07.683 INFO:teuthology.orchestra.run.smithi169.stderr:2023-12-16T21:34:07.676+0000 7f421d3e5700 1 -- 172.21.15.169:0/1343309250 shutdown_connections 2023-12-16T21:34:07.683 INFO:teuthology.orchestra.run.smithi169.stderr:2023-12-16T21:34:07.676+0000 7f421d3e5700 1 -- 172.21.15.169:0/1343309250 wait complete. 2023-12-16T21:34:07.683 INFO:teuthology.orchestra.run.smithi169.stderr:2023-12-16T21:34:07.677+0000 7f421d3e5700 1 Processor -- start 2023-12-16T21:34:07.683 INFO:teuthology.orchestra.run.smithi169.stderr:2023-12-16T21:34:07.677+0000 7f421d3e5700 1 -- start start 2023-12-16T21:34:07.683 INFO:teuthology.orchestra.run.smithi169.stderr:2023-12-16T21:34:07.677+0000 7f421d3e5700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f42180fb780 0x7f421810f760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:34:07.683 INFO:teuthology.orchestra.run.smithi169.stderr:2023-12-16T21:34:07.677+0000 7f421d3e5700 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4218065a70 con 0x7f42180fb780 2023-12-16T21:34:07.684 INFO:teuthology.orchestra.run.smithi169.stderr:2023-12-16T21:34:07.677+0000 7f4216ffd700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f42180fb780 0x7f421810f760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:34:07.684 INFO:teuthology.orchestra.run.smithi169.stderr:2023-12-16T21:34:07.677+0000 7f4216ffd700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f42180fb780 0x7f421810f760 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.169:39526/0 (socket says 172.21.15.169:39526) 2023-12-16T21:34:07.684 INFO:teuthology.orchestra.run.smithi169.stderr:2023-12-16T21:34:07.677+0000 7f4216ffd700 1 -- 172.21.15.169:0/161799855 learned_addr learned my addr 172.21.15.169:0/161799855 (peer_addr_for_me v2:172.21.15.169:0/0) 2023-12-16T21:34:07.684 INFO:teuthology.orchestra.run.smithi169.stderr:2023-12-16T21:34:07.678+0000 7f4216ffd700 1 -- 172.21.15.169:0/161799855 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4208005040 con 0x7f42180fb780 2023-12-16T21:34:07.684 INFO:teuthology.orchestra.run.smithi169.stderr:2023-12-16T21:34:07.678+0000 7f4216ffd700 1 --2- 172.21.15.169:0/161799855 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f42180fb780 0x7f421810f760 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7f420800b690 tx=0x7f4208009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:34:07.684 INFO:teuthology.orchestra.run.smithi169.stderr:2023-12-16T21:34:07.678+0000 7f41ff7fe700 1 -- 172.21.15.169:0/161799855 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f4208009650 con 0x7f42180fb780 2023-12-16T21:34:07.685 INFO:teuthology.orchestra.run.smithi169.stderr:2023-12-16T21:34:07.678+0000 7f421d3e5700 1 -- 172.21.15.169:0/161799855 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4218065c70 con 0x7f42180fb780 2023-12-16T21:34:07.685 INFO:teuthology.orchestra.run.smithi169.stderr:2023-12-16T21:34:07.678+0000 7f41ff7fe700 1 -- 172.21.15.169:0/161799855 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f42080097b0 con 0x7f42180fb780 2023-12-16T21:34:07.685 INFO:teuthology.orchestra.run.smithi169.stderr:2023-12-16T21:34:07.678+0000 7f41ff7fe700 1 -- 172.21.15.169:0/161799855 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f420801c5d0 con 0x7f42180fb780 2023-12-16T21:34:07.685 INFO:teuthology.orchestra.run.smithi169.stderr:2023-12-16T21:34:07.678+0000 7f421d3e5700 1 -- 172.21.15.169:0/161799855 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f421810fee0 con 0x7f42180fb780 2023-12-16T21:34:07.685 INFO:teuthology.orchestra.run.smithi169.stderr:2023-12-16T21:34:07.679+0000 7f41ff7fe700 1 -- 172.21.15.169:0/161799855 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 13) v1 ==== 43337+0+0 (secure 0 0 0) 0x7f420801c840 con 0x7f42180fb780 2023-12-16T21:34:07.685 INFO:teuthology.orchestra.run.smithi169.stderr:2023-12-16T21:34:07.679+0000 7f421d3e5700 1 -- 172.21.15.169:0/161799855 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4218105410 con 0x7f42180fb780 2023-12-16T21:34:07.686 INFO:teuthology.orchestra.run.smithi169.stderr:2023-12-16T21:34:07.679+0000 7f41ff7fe700 1 --2- 172.21.15.169:0/161799855 >> [v2:172.21.15.45:6800/2215417738,v1:172.21.15.45:6801/2215417738] conn(0x7f42000366d0 0x7f4200038b80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:34:07.686 INFO:teuthology.orchestra.run.smithi169.stderr:2023-12-16T21:34:07.679+0000 7f41ff7fe700 1 -- 172.21.15.169:0/161799855 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1151+0+0 (secure 0 0 0) 0x7f4208049bb0 con 0x7f42180fb780 2023-12-16T21:34:07.686 INFO:teuthology.orchestra.run.smithi169.stderr:2023-12-16T21:34:07.679+0000 7f42167fc700 1 --2- 172.21.15.169:0/161799855 >> [v2:172.21.15.45:6800/2215417738,v1:172.21.15.45:6801/2215417738] conn(0x7f42000366d0 0x7f4200038b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:34:07.686 INFO:teuthology.orchestra.run.smithi169.stderr:2023-12-16T21:34:07.680+0000 7f42167fc700 1 --2- 172.21.15.169:0/161799855 >> [v2:172.21.15.45:6800/2215417738,v1:172.21.15.45:6801/2215417738] conn(0x7f42000366d0 0x7f4200038b80 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7f420c009990 tx=0x7f420c006e40 comp rx=0 tx=0).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:34:07.687 INFO:teuthology.orchestra.run.smithi169.stderr:2023-12-16T21:34:07.685+0000 7f41ff7fe700 1 -- 172.21.15.169:0/161799855 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+174570 (secure 0 0 0) 0x7f4208023950 con 0x7f42180fb780 2023-12-16T21:34:07.946 INFO:teuthology.orchestra.run.smithi169.stderr:2023-12-16T21:34:07.944+0000 7f421d3e5700 1 -- 172.21.15.169:0/161799855 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f421804bb80 con 0x7f42180fb780 2023-12-16T21:34:07.947 INFO:teuthology.orchestra.run.smithi169.stderr:2023-12-16T21:34:07.945+0000 7f41ff7fe700 1 -- 172.21.15.169:0/161799855 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+739 (secure 0 0 0) 0x7f4208023300 con 0x7f42180fb780 2023-12-16T21:34:07.947 INFO:teuthology.orchestra.run.smithi169.stdout: 2023-12-16T21:34:07.948 INFO:teuthology.orchestra.run.smithi169.stdout:{"epoch":1,"fsid":"88ff363c-9c5a-11ee-95a5-87774f69a715","modified":"2023-12-16T21:32:46.744221Z","created":"2023-12-16T21:32:46.744221Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"smithi045","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:3300","nonce":0},{"type":"v1","addr":"172.21.15.45:6789","nonce":0}]},"addr":"172.21.15.45:6789/0","public_addr":"172.21.15.45:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-12-16T21:34:07.950 INFO:teuthology.orchestra.run.smithi169.stderr:2023-12-16T21:34:07.947+0000 7f421d3e5700 1 -- 172.21.15.169:0/161799855 >> [v2:172.21.15.45:6800/2215417738,v1:172.21.15.45:6801/2215417738] conn(0x7f42000366d0 msgr2=0x7f4200038b80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:34:07.950 INFO:teuthology.orchestra.run.smithi169.stderr:2023-12-16T21:34:07.947+0000 7f421d3e5700 1 --2- 172.21.15.169:0/161799855 >> [v2:172.21.15.45:6800/2215417738,v1:172.21.15.45:6801/2215417738] conn(0x7f42000366d0 0x7f4200038b80 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7f420c009990 tx=0x7f420c006e40 comp rx=0 tx=0).stop 2023-12-16T21:34:07.950 INFO:teuthology.orchestra.run.smithi169.stderr:2023-12-16T21:34:07.947+0000 7f421d3e5700 1 -- 172.21.15.169:0/161799855 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f42180fb780 msgr2=0x7f421810f760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:34:07.950 INFO:teuthology.orchestra.run.smithi169.stderr:2023-12-16T21:34:07.947+0000 7f421d3e5700 1 --2- 172.21.15.169:0/161799855 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f42180fb780 0x7f421810f760 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7f420800b690 tx=0x7f4208009e30 comp rx=0 tx=0).stop 2023-12-16T21:34:07.950 INFO:teuthology.orchestra.run.smithi169.stderr:2023-12-16T21:34:07.947+0000 7f421d3e5700 1 -- 172.21.15.169:0/161799855 shutdown_connections 2023-12-16T21:34:07.950 INFO:teuthology.orchestra.run.smithi169.stderr:2023-12-16T21:34:07.947+0000 7f421d3e5700 1 --2- 172.21.15.169:0/161799855 >> [v2:172.21.15.45:6800/2215417738,v1:172.21.15.45:6801/2215417738] conn(0x7f42000366d0 0x7f4200038b80 unknown :-1 s=CLOSED pgs=21 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:34:07.951 INFO:teuthology.orchestra.run.smithi169.stderr:2023-12-16T21:34:07.947+0000 7f421d3e5700 1 --2- 172.21.15.169:0/161799855 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f42180fb780 0x7f421810f760 unknown :-1 s=CLOSED pgs=109 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:34:07.951 INFO:teuthology.orchestra.run.smithi169.stderr:2023-12-16T21:34:07.947+0000 7f421d3e5700 1 -- 172.21.15.169:0/161799855 >> 172.21.15.169:0/161799855 conn(0x7f42180f6e40 msgr2=0x7f42180f7c30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-16T21:34:07.951 INFO:teuthology.orchestra.run.smithi169.stderr:2023-12-16T21:34:07.947+0000 7f421d3e5700 1 -- 172.21.15.169:0/161799855 shutdown_connections 2023-12-16T21:34:07.951 INFO:teuthology.orchestra.run.smithi169.stderr:2023-12-16T21:34:07.947+0000 7f421d3e5700 1 -- 172.21.15.169:0/161799855 wait complete. 2023-12-16T21:34:07.951 INFO:teuthology.orchestra.run.smithi169.stderr:dumped monmap epoch 1 2023-12-16T21:34:08.566 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Dec 16 21:34:08 smithi045 ceph-mon[80763]: from='client.? 172.21.15.169:0/161799855' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-12-16T21:34:09.298 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-12-16T21:34:09.298 DEBUG:teuthology.orchestra.run.smithi169:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 88ff363c-9c5a-11ee-95a5-87774f69a715 -- ceph mon dump -f json 2023-12-16T21:34:09.546 INFO:teuthology.orchestra.run.smithi169.stderr:Inferring config /etc/ceph/ceph.conf 2023-12-16T21:34:09.566 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Dec 16 21:34:09 smithi045 ceph-mon[80763]: pgmap v9: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-16T21:34:09.592 INFO:teuthology.orchestra.run.smithi169.stderr:Inferring config /etc/ceph/ceph.conf 2023-12-16T21:34:10.319 INFO:teuthology.orchestra.run.smithi169.stderr:2023-12-16T21:34:10.316+0000 7fec1d3a4700 1 -- 172.21.15.169:0/1339438455 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fec180fc880 msgr2=0x7fec180fcc90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:34:10.319 INFO:teuthology.orchestra.run.smithi169.stderr:2023-12-16T21:34:10.316+0000 7fec1d3a4700 1 --2- 172.21.15.169:0/1339438455 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fec180fc880 0x7fec180fcc90 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7fec080048c0 tx=0x7fec08009e30 comp rx=0 tx=0).stop 2023-12-16T21:34:10.319 INFO:teuthology.orchestra.run.smithi169.stderr:2023-12-16T21:34:10.317+0000 7fec1d3a4700 1 -- 172.21.15.169:0/1339438455 shutdown_connections 2023-12-16T21:34:10.319 INFO:teuthology.orchestra.run.smithi169.stderr:2023-12-16T21:34:10.317+0000 7fec1d3a4700 1 --2- 172.21.15.169:0/1339438455 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fec180fc880 0x7fec180fcc90 unknown :-1 s=CLOSED pgs=110 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:34:10.319 INFO:teuthology.orchestra.run.smithi169.stderr:2023-12-16T21:34:10.317+0000 7fec1d3a4700 1 -- 172.21.15.169:0/1339438455 >> 172.21.15.169:0/1339438455 conn(0x7fec180f7f40 msgr2=0x7fec180fa370 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-16T21:34:10.319 INFO:teuthology.orchestra.run.smithi169.stderr:2023-12-16T21:34:10.317+0000 7fec1d3a4700 1 -- 172.21.15.169:0/1339438455 shutdown_connections 2023-12-16T21:34:10.320 INFO:teuthology.orchestra.run.smithi169.stderr:2023-12-16T21:34:10.317+0000 7fec1d3a4700 1 -- 172.21.15.169:0/1339438455 wait complete. 2023-12-16T21:34:10.322 INFO:teuthology.orchestra.run.smithi169.stderr:2023-12-16T21:34:10.317+0000 7fec1d3a4700 1 Processor -- start 2023-12-16T21:34:10.322 INFO:teuthology.orchestra.run.smithi169.stderr:2023-12-16T21:34:10.317+0000 7fec1d3a4700 1 -- start start 2023-12-16T21:34:10.322 INFO:teuthology.orchestra.run.smithi169.stderr:2023-12-16T21:34:10.318+0000 7fec1d3a4700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fec180fc880 0x7fec1810cac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:34:10.322 INFO:teuthology.orchestra.run.smithi169.stderr:2023-12-16T21:34:10.318+0000 7fec1d3a4700 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fec1810d000 con 0x7fec180fc880 2023-12-16T21:34:10.322 INFO:teuthology.orchestra.run.smithi169.stderr:2023-12-16T21:34:10.318+0000 7fec16ffd700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fec180fc880 0x7fec1810cac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:34:10.322 INFO:teuthology.orchestra.run.smithi169.stderr:2023-12-16T21:34:10.318+0000 7fec16ffd700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fec180fc880 0x7fec1810cac0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.169:39556/0 (socket says 172.21.15.169:39556) 2023-12-16T21:34:10.322 INFO:teuthology.orchestra.run.smithi169.stderr:2023-12-16T21:34:10.318+0000 7fec16ffd700 1 -- 172.21.15.169:0/1754624243 learned_addr learned my addr 172.21.15.169:0/1754624243 (peer_addr_for_me v2:172.21.15.169:0/0) 2023-12-16T21:34:10.322 INFO:teuthology.orchestra.run.smithi169.stderr:2023-12-16T21:34:10.318+0000 7fec16ffd700 1 -- 172.21.15.169:0/1754624243 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fec08005040 con 0x7fec180fc880 2023-12-16T21:34:10.322 INFO:teuthology.orchestra.run.smithi169.stderr:2023-12-16T21:34:10.318+0000 7fec16ffd700 1 --2- 172.21.15.169:0/1754624243 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fec180fc880 0x7fec1810cac0 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7fec08009330 tx=0x7fec08009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:34:10.322 INFO:teuthology.orchestra.run.smithi169.stderr:2023-12-16T21:34:10.319+0000 7febff7fe700 1 -- 172.21.15.169:0/1754624243 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fec080096b0 con 0x7fec180fc880 2023-12-16T21:34:10.323 INFO:teuthology.orchestra.run.smithi169.stderr:2023-12-16T21:34:10.319+0000 7fec1d3a4700 1 -- 172.21.15.169:0/1754624243 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fec1810d200 con 0x7fec180fc880 2023-12-16T21:34:10.323 INFO:teuthology.orchestra.run.smithi169.stderr:2023-12-16T21:34:10.319+0000 7febff7fe700 1 -- 172.21.15.169:0/1754624243 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fec08009810 con 0x7fec180fc880 2023-12-16T21:34:10.323 INFO:teuthology.orchestra.run.smithi169.stderr:2023-12-16T21:34:10.319+0000 7febff7fe700 1 -- 172.21.15.169:0/1754624243 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fec0801c5b0 con 0x7fec180fc880 2023-12-16T21:34:10.323 INFO:teuthology.orchestra.run.smithi169.stderr:2023-12-16T21:34:10.319+0000 7fec1d3a4700 1 -- 172.21.15.169:0/1754624243 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fec1810dea0 con 0x7fec180fc880 2023-12-16T21:34:10.323 INFO:teuthology.orchestra.run.smithi169.stderr:2023-12-16T21:34:10.319+0000 7febff7fe700 1 -- 172.21.15.169:0/1754624243 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 13) v1 ==== 43337+0+0 (secure 0 0 0) 0x7fec08013420 con 0x7fec180fc880 2023-12-16T21:34:10.323 INFO:teuthology.orchestra.run.smithi169.stderr:2023-12-16T21:34:10.319+0000 7fec1d3a4700 1 -- 172.21.15.169:0/1754624243 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fec1805f270 con 0x7fec180fc880 2023-12-16T21:34:10.323 INFO:teuthology.orchestra.run.smithi169.stderr:2023-12-16T21:34:10.320+0000 7febff7fe700 1 --2- 172.21.15.169:0/1754624243 >> [v2:172.21.15.45:6800/2215417738,v1:172.21.15.45:6801/2215417738] conn(0x7fec000366d0 0x7fec00038b80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:34:10.323 INFO:teuthology.orchestra.run.smithi169.stderr:2023-12-16T21:34:10.320+0000 7febff7fe700 1 -- 172.21.15.169:0/1754624243 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1151+0+0 (secure 0 0 0) 0x7fec08049ad0 con 0x7fec180fc880 2023-12-16T21:34:10.323 INFO:teuthology.orchestra.run.smithi169.stderr:2023-12-16T21:34:10.320+0000 7fec167fc700 1 --2- 172.21.15.169:0/1754624243 >> [v2:172.21.15.45:6800/2215417738,v1:172.21.15.45:6801/2215417738] conn(0x7fec000366d0 0x7fec00038b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:34:10.323 INFO:teuthology.orchestra.run.smithi169.stderr:2023-12-16T21:34:10.320+0000 7fec167fc700 1 --2- 172.21.15.169:0/1754624243 >> [v2:172.21.15.45:6800/2215417738,v1:172.21.15.45:6801/2215417738] conn(0x7fec000366d0 0x7fec00038b80 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7fec0c009990 tx=0x7fec0c006e40 comp rx=0 tx=0).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:34:10.327 INFO:teuthology.orchestra.run.smithi169.stderr:2023-12-16T21:34:10.325+0000 7febff7fe700 1 -- 172.21.15.169:0/1754624243 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+174570 (secure 0 0 0) 0x7fec080119c0 con 0x7fec180fc880 2023-12-16T21:34:10.589 INFO:teuthology.orchestra.run.smithi169.stderr:2023-12-16T21:34:10.587+0000 7fec1d3a4700 1 -- 172.21.15.169:0/1754624243 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fec1810e9f0 con 0x7fec180fc880 2023-12-16T21:34:10.590 INFO:teuthology.orchestra.run.smithi169.stderr:2023-12-16T21:34:10.588+0000 7febff7fe700 1 -- 172.21.15.169:0/1754624243 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+739 (secure 0 0 0) 0x7fec08011370 con 0x7fec180fc880 2023-12-16T21:34:10.590 INFO:teuthology.orchestra.run.smithi169.stdout: 2023-12-16T21:34:10.590 INFO:teuthology.orchestra.run.smithi169.stdout:{"epoch":1,"fsid":"88ff363c-9c5a-11ee-95a5-87774f69a715","modified":"2023-12-16T21:32:46.744221Z","created":"2023-12-16T21:32:46.744221Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"smithi045","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:3300","nonce":0},{"type":"v1","addr":"172.21.15.45:6789","nonce":0}]},"addr":"172.21.15.45:6789/0","public_addr":"172.21.15.45:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-12-16T21:34:10.592 INFO:teuthology.orchestra.run.smithi169.stderr:2023-12-16T21:34:10.590+0000 7fec1d3a4700 1 -- 172.21.15.169:0/1754624243 >> [v2:172.21.15.45:6800/2215417738,v1:172.21.15.45:6801/2215417738] conn(0x7fec000366d0 msgr2=0x7fec00038b80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:34:10.592 INFO:teuthology.orchestra.run.smithi169.stderr:2023-12-16T21:34:10.590+0000 7fec1d3a4700 1 --2- 172.21.15.169:0/1754624243 >> [v2:172.21.15.45:6800/2215417738,v1:172.21.15.45:6801/2215417738] conn(0x7fec000366d0 0x7fec00038b80 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7fec0c009990 tx=0x7fec0c006e40 comp rx=0 tx=0).stop 2023-12-16T21:34:10.592 INFO:teuthology.orchestra.run.smithi169.stderr:2023-12-16T21:34:10.590+0000 7fec1d3a4700 1 -- 172.21.15.169:0/1754624243 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fec180fc880 msgr2=0x7fec1810cac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:34:10.592 INFO:teuthology.orchestra.run.smithi169.stderr:2023-12-16T21:34:10.590+0000 7fec1d3a4700 1 --2- 172.21.15.169:0/1754624243 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fec180fc880 0x7fec1810cac0 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7fec08009330 tx=0x7fec08009e30 comp rx=0 tx=0).stop 2023-12-16T21:34:10.592 INFO:teuthology.orchestra.run.smithi169.stderr:2023-12-16T21:34:10.590+0000 7fec1d3a4700 1 -- 172.21.15.169:0/1754624243 shutdown_connections 2023-12-16T21:34:10.592 INFO:teuthology.orchestra.run.smithi169.stderr:2023-12-16T21:34:10.590+0000 7fec1d3a4700 1 --2- 172.21.15.169:0/1754624243 >> [v2:172.21.15.45:6800/2215417738,v1:172.21.15.45:6801/2215417738] conn(0x7fec000366d0 0x7fec00038b80 unknown :-1 s=CLOSED pgs=22 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:34:10.592 INFO:teuthology.orchestra.run.smithi169.stderr:2023-12-16T21:34:10.590+0000 7fec1d3a4700 1 --2- 172.21.15.169:0/1754624243 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fec180fc880 0x7fec1810cac0 unknown :-1 s=CLOSED pgs=111 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:34:10.592 INFO:teuthology.orchestra.run.smithi169.stderr:2023-12-16T21:34:10.590+0000 7fec1d3a4700 1 -- 172.21.15.169:0/1754624243 >> 172.21.15.169:0/1754624243 conn(0x7fec180f7f40 msgr2=0x7fec180f8d70 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-16T21:34:10.592 INFO:teuthology.orchestra.run.smithi169.stderr:2023-12-16T21:34:10.590+0000 7fec1d3a4700 1 -- 172.21.15.169:0/1754624243 shutdown_connections 2023-12-16T21:34:10.592 INFO:teuthology.orchestra.run.smithi169.stderr:2023-12-16T21:34:10.590+0000 7fec1d3a4700 1 -- 172.21.15.169:0/1754624243 wait complete. 2023-12-16T21:34:10.593 INFO:teuthology.orchestra.run.smithi169.stderr:dumped monmap epoch 1 2023-12-16T21:34:11.407 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Dec 16 21:34:11 smithi045 ceph-mon[80763]: pgmap v10: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-16T21:34:11.407 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Dec 16 21:34:11 smithi045 ceph-mon[80763]: from='mgr.14162 172.21.15.45:0/142909557' entity='mgr.smithi045.nhtmwc' 2023-12-16T21:34:11.408 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Dec 16 21:34:11 smithi045 ceph-mon[80763]: from='mgr.14162 172.21.15.45:0/142909557' entity='mgr.smithi045.nhtmwc' 2023-12-16T21:34:11.408 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Dec 16 21:34:11 smithi045 ceph-mon[80763]: from='mgr.14162 172.21.15.45:0/142909557' entity='mgr.smithi045.nhtmwc' 2023-12-16T21:34:11.408 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Dec 16 21:34:11 smithi045 ceph-mon[80763]: from='mgr.14162 172.21.15.45:0/142909557' entity='mgr.smithi045.nhtmwc' 2023-12-16T21:34:11.408 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Dec 16 21:34:11 smithi045 ceph-mon[80763]: from='mgr.14162 172.21.15.45:0/142909557' entity='mgr.smithi045.nhtmwc' 2023-12-16T21:34:11.408 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Dec 16 21:34:11 smithi045 ceph-mon[80763]: from='mgr.14162 172.21.15.45:0/142909557' entity='mgr.smithi045.nhtmwc' 2023-12-16T21:34:11.408 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Dec 16 21:34:11 smithi045 ceph-mon[80763]: Deploying daemon prometheus.smithi045 on smithi045 2023-12-16T21:34:11.408 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Dec 16 21:34:11 smithi045 ceph-mon[80763]: from='client.? 172.21.15.169:0/1754624243' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-12-16T21:34:11.839 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-12-16T21:34:11.839 DEBUG:teuthology.orchestra.run.smithi169:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 88ff363c-9c5a-11ee-95a5-87774f69a715 -- ceph mon dump -f json 2023-12-16T21:34:12.086 INFO:teuthology.orchestra.run.smithi169.stderr:Inferring config /etc/ceph/ceph.conf 2023-12-16T21:34:12.132 INFO:teuthology.orchestra.run.smithi169.stderr:Inferring config /etc/ceph/ceph.conf 2023-12-16T21:34:12.910 INFO:teuthology.orchestra.run.smithi169.stderr:2023-12-16T21:34:12.908+0000 7f29225c9700 1 -- 172.21.15.169:0/1884460654 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f291c0fd3d0 msgr2=0x7f291c0ff7b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:34:12.910 INFO:teuthology.orchestra.run.smithi169.stderr:2023-12-16T21:34:12.908+0000 7f29225c9700 1 --2- 172.21.15.169:0/1884460654 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f291c0fd3d0 0x7f291c0ff7b0 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7f290c0048c0 tx=0x7f290c009e30 comp rx=0 tx=0).stop 2023-12-16T21:34:12.910 INFO:teuthology.orchestra.run.smithi169.stderr:2023-12-16T21:34:12.908+0000 7f29225c9700 1 -- 172.21.15.169:0/1884460654 shutdown_connections 2023-12-16T21:34:12.910 INFO:teuthology.orchestra.run.smithi169.stderr:2023-12-16T21:34:12.908+0000 7f29225c9700 1 --2- 172.21.15.169:0/1884460654 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f291c0fd3d0 0x7f291c0ff7b0 unknown :-1 s=CLOSED pgs=112 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:34:12.910 INFO:teuthology.orchestra.run.smithi169.stderr:2023-12-16T21:34:12.908+0000 7f29225c9700 1 -- 172.21.15.169:0/1884460654 >> 172.21.15.169:0/1884460654 conn(0x7f291c0f6e50 msgr2=0x7f291c0f9280 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-16T21:34:12.910 INFO:teuthology.orchestra.run.smithi169.stderr:2023-12-16T21:34:12.908+0000 7f29225c9700 1 -- 172.21.15.169:0/1884460654 shutdown_connections 2023-12-16T21:34:12.911 INFO:teuthology.orchestra.run.smithi169.stderr:2023-12-16T21:34:12.908+0000 7f29225c9700 1 -- 172.21.15.169:0/1884460654 wait complete. 2023-12-16T21:34:12.911 INFO:teuthology.orchestra.run.smithi169.stderr:2023-12-16T21:34:12.908+0000 7f29225c9700 1 Processor -- start 2023-12-16T21:34:12.912 INFO:teuthology.orchestra.run.smithi169.stderr:2023-12-16T21:34:12.908+0000 7f29225c9700 1 -- start start 2023-12-16T21:34:12.912 INFO:teuthology.orchestra.run.smithi169.stderr:2023-12-16T21:34:12.908+0000 7f29225c9700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f291c0fd3d0 0x7f291c10ca70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:34:12.912 INFO:teuthology.orchestra.run.smithi169.stderr:2023-12-16T21:34:12.908+0000 7f29225c9700 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f291c10cfb0 con 0x7f291c0fd3d0 2023-12-16T21:34:12.913 INFO:teuthology.orchestra.run.smithi169.stderr:2023-12-16T21:34:12.909+0000 7f291bfff700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f291c0fd3d0 0x7f291c10ca70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:34:12.913 INFO:teuthology.orchestra.run.smithi169.stderr:2023-12-16T21:34:12.909+0000 7f291bfff700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f291c0fd3d0 0x7f291c10ca70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.169:52162/0 (socket says 172.21.15.169:52162) 2023-12-16T21:34:12.913 INFO:teuthology.orchestra.run.smithi169.stderr:2023-12-16T21:34:12.909+0000 7f291bfff700 1 -- 172.21.15.169:0/3927930416 learned_addr learned my addr 172.21.15.169:0/3927930416 (peer_addr_for_me v2:172.21.15.169:0/0) 2023-12-16T21:34:12.913 INFO:teuthology.orchestra.run.smithi169.stderr:2023-12-16T21:34:12.909+0000 7f291bfff700 1 -- 172.21.15.169:0/3927930416 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f290c005040 con 0x7f291c0fd3d0 2023-12-16T21:34:12.913 INFO:teuthology.orchestra.run.smithi169.stderr:2023-12-16T21:34:12.909+0000 7f291bfff700 1 --2- 172.21.15.169:0/3927930416 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f291c0fd3d0 0x7f291c10ca70 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7f290c00b690 tx=0x7f290c009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:34:12.913 INFO:teuthology.orchestra.run.smithi169.stderr:2023-12-16T21:34:12.909+0000 7f2918ff9700 1 -- 172.21.15.169:0/3927930416 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f290c009650 con 0x7f291c0fd3d0 2023-12-16T21:34:12.913 INFO:teuthology.orchestra.run.smithi169.stderr:2023-12-16T21:34:12.909+0000 7f29225c9700 1 -- 172.21.15.169:0/3927930416 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f291c10d1b0 con 0x7f291c0fd3d0 2023-12-16T21:34:12.913 INFO:teuthology.orchestra.run.smithi169.stderr:2023-12-16T21:34:12.909+0000 7f2918ff9700 1 -- 172.21.15.169:0/3927930416 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f290c0097b0 con 0x7f291c0fd3d0 2023-12-16T21:34:12.913 INFO:teuthology.orchestra.run.smithi169.stderr:2023-12-16T21:34:12.909+0000 7f2918ff9700 1 -- 172.21.15.169:0/3927930416 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f290c01c5b0 con 0x7f291c0fd3d0 2023-12-16T21:34:12.913 INFO:teuthology.orchestra.run.smithi169.stderr:2023-12-16T21:34:12.909+0000 7f29225c9700 1 -- 172.21.15.169:0/3927930416 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f291c10de50 con 0x7f291c0fd3d0 2023-12-16T21:34:12.913 INFO:teuthology.orchestra.run.smithi169.stderr:2023-12-16T21:34:12.910+0000 7f2918ff9700 1 -- 172.21.15.169:0/3927930416 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 13) v1 ==== 43337+0+0 (secure 0 0 0) 0x7f290c013420 con 0x7f291c0fd3d0 2023-12-16T21:34:12.913 INFO:teuthology.orchestra.run.smithi169.stderr:2023-12-16T21:34:12.910+0000 7f29225c9700 1 -- 172.21.15.169:0/3927930416 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f291c05f270 con 0x7f291c0fd3d0 2023-12-16T21:34:12.913 INFO:teuthology.orchestra.run.smithi169.stderr:2023-12-16T21:34:12.910+0000 7f2918ff9700 1 --2- 172.21.15.169:0/3927930416 >> [v2:172.21.15.45:6800/2215417738,v1:172.21.15.45:6801/2215417738] conn(0x7f29040366d0 0x7f2904038b80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:34:12.913 INFO:teuthology.orchestra.run.smithi169.stderr:2023-12-16T21:34:12.910+0000 7f2918ff9700 1 -- 172.21.15.169:0/3927930416 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1151+0+0 (secure 0 0 0) 0x7f290c049910 con 0x7f291c0fd3d0 2023-12-16T21:34:12.914 INFO:teuthology.orchestra.run.smithi169.stderr:2023-12-16T21:34:12.910+0000 7f291b7fe700 1 --2- 172.21.15.169:0/3927930416 >> [v2:172.21.15.45:6800/2215417738,v1:172.21.15.45:6801/2215417738] conn(0x7f29040366d0 0x7f2904038b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:34:12.914 INFO:teuthology.orchestra.run.smithi169.stderr:2023-12-16T21:34:12.911+0000 7f291b7fe700 1 --2- 172.21.15.169:0/3927930416 >> [v2:172.21.15.45:6800/2215417738,v1:172.21.15.45:6801/2215417738] conn(0x7f29040366d0 0x7f2904038b80 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7f2910009990 tx=0x7f2910006e40 comp rx=0 tx=0).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:34:12.917 INFO:teuthology.orchestra.run.smithi169.stderr:2023-12-16T21:34:12.915+0000 7f2918ff9700 1 -- 172.21.15.169:0/3927930416 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+174570 (secure 0 0 0) 0x7f290c013a60 con 0x7f291c0fd3d0 2023-12-16T21:34:13.179 INFO:teuthology.orchestra.run.smithi169.stderr:2023-12-16T21:34:13.176+0000 7f29225c9700 1 -- 172.21.15.169:0/3927930416 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f291c10e9a0 con 0x7f291c0fd3d0 2023-12-16T21:34:13.179 INFO:teuthology.orchestra.run.smithi169.stderr:2023-12-16T21:34:13.177+0000 7f2918ff9700 1 -- 172.21.15.169:0/3927930416 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+739 (secure 0 0 0) 0x7f290c011370 con 0x7f291c0fd3d0 2023-12-16T21:34:13.179 INFO:teuthology.orchestra.run.smithi169.stdout: 2023-12-16T21:34:13.180 INFO:teuthology.orchestra.run.smithi169.stdout:{"epoch":1,"fsid":"88ff363c-9c5a-11ee-95a5-87774f69a715","modified":"2023-12-16T21:32:46.744221Z","created":"2023-12-16T21:32:46.744221Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"smithi045","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:3300","nonce":0},{"type":"v1","addr":"172.21.15.45:6789","nonce":0}]},"addr":"172.21.15.45:6789/0","public_addr":"172.21.15.45:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-12-16T21:34:13.181 INFO:teuthology.orchestra.run.smithi169.stderr:2023-12-16T21:34:13.179+0000 7f29225c9700 1 -- 172.21.15.169:0/3927930416 >> [v2:172.21.15.45:6800/2215417738,v1:172.21.15.45:6801/2215417738] conn(0x7f29040366d0 msgr2=0x7f2904038b80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:34:13.181 INFO:teuthology.orchestra.run.smithi169.stderr:2023-12-16T21:34:13.179+0000 7f29225c9700 1 --2- 172.21.15.169:0/3927930416 >> [v2:172.21.15.45:6800/2215417738,v1:172.21.15.45:6801/2215417738] conn(0x7f29040366d0 0x7f2904038b80 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7f2910009990 tx=0x7f2910006e40 comp rx=0 tx=0).stop 2023-12-16T21:34:13.181 INFO:teuthology.orchestra.run.smithi169.stderr:2023-12-16T21:34:13.179+0000 7f29225c9700 1 -- 172.21.15.169:0/3927930416 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f291c0fd3d0 msgr2=0x7f291c10ca70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:34:13.181 INFO:teuthology.orchestra.run.smithi169.stderr:2023-12-16T21:34:13.179+0000 7f29225c9700 1 --2- 172.21.15.169:0/3927930416 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f291c0fd3d0 0x7f291c10ca70 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7f290c00b690 tx=0x7f290c009e30 comp rx=0 tx=0).stop 2023-12-16T21:34:13.181 INFO:teuthology.orchestra.run.smithi169.stderr:2023-12-16T21:34:13.179+0000 7f29225c9700 1 -- 172.21.15.169:0/3927930416 shutdown_connections 2023-12-16T21:34:13.181 INFO:teuthology.orchestra.run.smithi169.stderr:2023-12-16T21:34:13.179+0000 7f29225c9700 1 --2- 172.21.15.169:0/3927930416 >> [v2:172.21.15.45:6800/2215417738,v1:172.21.15.45:6801/2215417738] conn(0x7f29040366d0 0x7f2904038b80 unknown :-1 s=CLOSED pgs=23 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:34:13.181 INFO:teuthology.orchestra.run.smithi169.stderr:2023-12-16T21:34:13.179+0000 7f29225c9700 1 --2- 172.21.15.169:0/3927930416 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f291c0fd3d0 0x7f291c10ca70 unknown :-1 s=CLOSED pgs=113 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:34:13.181 INFO:teuthology.orchestra.run.smithi169.stderr:2023-12-16T21:34:13.179+0000 7f29225c9700 1 -- 172.21.15.169:0/3927930416 >> 172.21.15.169:0/3927930416 conn(0x7f291c0f6e50 msgr2=0x7f291c0f7be0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-16T21:34:13.182 INFO:teuthology.orchestra.run.smithi169.stderr:2023-12-16T21:34:13.179+0000 7f29225c9700 1 -- 172.21.15.169:0/3927930416 shutdown_connections 2023-12-16T21:34:13.182 INFO:teuthology.orchestra.run.smithi169.stderr:2023-12-16T21:34:13.179+0000 7f29225c9700 1 -- 172.21.15.169:0/3927930416 wait complete. 2023-12-16T21:34:13.182 INFO:teuthology.orchestra.run.smithi169.stderr:dumped monmap epoch 1 2023-12-16T21:34:13.566 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Dec 16 21:34:13 smithi045 ceph-mon[80763]: pgmap v11: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-16T21:34:13.566 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Dec 16 21:34:13 smithi045 ceph-mon[80763]: from='mgr.14162 172.21.15.45:0/142909557' entity='mgr.smithi045.nhtmwc' 2023-12-16T21:34:13.566 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Dec 16 21:34:13 smithi045 ceph-mon[80763]: from='client.? 172.21.15.169:0/3927930416' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-12-16T21:34:14.430 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-12-16T21:34:14.430 DEBUG:teuthology.orchestra.run.smithi169:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 88ff363c-9c5a-11ee-95a5-87774f69a715 -- ceph mon dump -f json 2023-12-16T21:34:14.681 INFO:teuthology.orchestra.run.smithi169.stderr:Inferring config /etc/ceph/ceph.conf 2023-12-16T21:34:14.726 INFO:teuthology.orchestra.run.smithi169.stderr:Inferring config /etc/ceph/ceph.conf 2023-12-16T21:34:15.455 INFO:teuthology.orchestra.run.smithi169.stderr:2023-12-16T21:34:15.452+0000 7f23a0b52700 1 -- 172.21.15.169:0/2919703890 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f239c0fd3b0 msgr2=0x7f239c0ff790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:34:15.456 INFO:teuthology.orchestra.run.smithi169.stderr:2023-12-16T21:34:15.452+0000 7f23a0b52700 1 --2- 172.21.15.169:0/2919703890 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f239c0fd3b0 0x7f239c0ff790 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7f23900048c0 tx=0x7f2390009e30 comp rx=0 tx=0).stop 2023-12-16T21:34:15.456 INFO:teuthology.orchestra.run.smithi169.stderr:2023-12-16T21:34:15.452+0000 7f23a0b52700 1 -- 172.21.15.169:0/2919703890 shutdown_connections 2023-12-16T21:34:15.456 INFO:teuthology.orchestra.run.smithi169.stderr:2023-12-16T21:34:15.452+0000 7f23a0b52700 1 --2- 172.21.15.169:0/2919703890 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f239c0fd3b0 0x7f239c0ff790 unknown :-1 s=CLOSED pgs=114 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:34:15.456 INFO:teuthology.orchestra.run.smithi169.stderr:2023-12-16T21:34:15.452+0000 7f23a0b52700 1 -- 172.21.15.169:0/2919703890 >> 172.21.15.169:0/2919703890 conn(0x7f239c0f6e50 msgr2=0x7f239c0f9260 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-16T21:34:15.456 INFO:teuthology.orchestra.run.smithi169.stderr:2023-12-16T21:34:15.452+0000 7f23a0b52700 1 -- 172.21.15.169:0/2919703890 shutdown_connections 2023-12-16T21:34:15.456 INFO:teuthology.orchestra.run.smithi169.stderr:2023-12-16T21:34:15.452+0000 7f23a0b52700 1 -- 172.21.15.169:0/2919703890 wait complete. 2023-12-16T21:34:15.456 INFO:teuthology.orchestra.run.smithi169.stderr:2023-12-16T21:34:15.453+0000 7f23a0b52700 1 Processor -- start 2023-12-16T21:34:15.456 INFO:teuthology.orchestra.run.smithi169.stderr:2023-12-16T21:34:15.453+0000 7f23a0b52700 1 -- start start 2023-12-16T21:34:15.456 INFO:teuthology.orchestra.run.smithi169.stderr:2023-12-16T21:34:15.453+0000 7f23a0b52700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f239c0fd3b0 0x7f239c10ed30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:34:15.456 INFO:teuthology.orchestra.run.smithi169.stderr:2023-12-16T21:34:15.453+0000 7f23a0b52700 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f239c10f270 con 0x7f239c0fd3b0 2023-12-16T21:34:15.456 INFO:teuthology.orchestra.run.smithi169.stderr:2023-12-16T21:34:15.453+0000 7f239a59c700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f239c0fd3b0 0x7f239c10ed30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:34:15.456 INFO:teuthology.orchestra.run.smithi169.stderr:2023-12-16T21:34:15.453+0000 7f239a59c700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f239c0fd3b0 0x7f239c10ed30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.169:52172/0 (socket says 172.21.15.169:52172) 2023-12-16T21:34:15.456 INFO:teuthology.orchestra.run.smithi169.stderr:2023-12-16T21:34:15.453+0000 7f239a59c700 1 -- 172.21.15.169:0/2637764047 learned_addr learned my addr 172.21.15.169:0/2637764047 (peer_addr_for_me v2:172.21.15.169:0/0) 2023-12-16T21:34:15.457 INFO:teuthology.orchestra.run.smithi169.stderr:2023-12-16T21:34:15.454+0000 7f239a59c700 1 -- 172.21.15.169:0/2637764047 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2390005040 con 0x7f239c0fd3b0 2023-12-16T21:34:15.458 INFO:teuthology.orchestra.run.smithi169.stderr:2023-12-16T21:34:15.454+0000 7f239a59c700 1 --2- 172.21.15.169:0/2637764047 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f239c0fd3b0 0x7f239c10ed30 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7f239000b690 tx=0x7f2390009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:34:15.458 INFO:teuthology.orchestra.run.smithi169.stderr:2023-12-16T21:34:15.454+0000 7f238affd700 1 -- 172.21.15.169:0/2637764047 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f2390009650 con 0x7f239c0fd3b0 2023-12-16T21:34:15.458 INFO:teuthology.orchestra.run.smithi169.stderr:2023-12-16T21:34:15.454+0000 7f238affd700 1 -- 172.21.15.169:0/2637764047 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f23900097b0 con 0x7f239c0fd3b0 2023-12-16T21:34:15.458 INFO:teuthology.orchestra.run.smithi169.stderr:2023-12-16T21:34:15.454+0000 7f238affd700 1 -- 172.21.15.169:0/2637764047 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f239001c5b0 con 0x7f239c0fd3b0 2023-12-16T21:34:15.458 INFO:teuthology.orchestra.run.smithi169.stderr:2023-12-16T21:34:15.454+0000 7f23a0b52700 1 -- 172.21.15.169:0/2637764047 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f239c10f470 con 0x7f239c0fd3b0 2023-12-16T21:34:15.459 INFO:teuthology.orchestra.run.smithi169.stderr:2023-12-16T21:34:15.454+0000 7f23a0b52700 1 -- 172.21.15.169:0/2637764047 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f239c10ffd0 con 0x7f239c0fd3b0 2023-12-16T21:34:15.459 INFO:teuthology.orchestra.run.smithi169.stderr:2023-12-16T21:34:15.455+0000 7f238affd700 1 -- 172.21.15.169:0/2637764047 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 13) v1 ==== 43337+0+0 (secure 0 0 0) 0x7f2390013420 con 0x7f239c0fd3b0 2023-12-16T21:34:15.459 INFO:teuthology.orchestra.run.smithi169.stderr:2023-12-16T21:34:15.455+0000 7f23a0b52700 1 -- 172.21.15.169:0/2637764047 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f239c04bb80 con 0x7f239c0fd3b0 2023-12-16T21:34:15.459 INFO:teuthology.orchestra.run.smithi169.stderr:2023-12-16T21:34:15.455+0000 7f238affd700 1 --2- 172.21.15.169:0/2637764047 >> [v2:172.21.15.45:6800/2215417738,v1:172.21.15.45:6801/2215417738] conn(0x7f2384036720 0x7f2384038bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:34:15.459 INFO:teuthology.orchestra.run.smithi169.stderr:2023-12-16T21:34:15.455+0000 7f238affd700 1 -- 172.21.15.169:0/2637764047 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1151+0+0 (secure 0 0 0) 0x7f2390049830 con 0x7f239c0fd3b0 2023-12-16T21:34:15.459 INFO:teuthology.orchestra.run.smithi169.stderr:2023-12-16T21:34:15.455+0000 7f2399d9b700 1 --2- 172.21.15.169:0/2637764047 >> [v2:172.21.15.45:6800/2215417738,v1:172.21.15.45:6801/2215417738] conn(0x7f2384036720 0x7f2384038bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:34:15.459 INFO:teuthology.orchestra.run.smithi169.stderr:2023-12-16T21:34:15.456+0000 7f2399d9b700 1 --2- 172.21.15.169:0/2637764047 >> [v2:172.21.15.45:6800/2215417738,v1:172.21.15.45:6801/2215417738] conn(0x7f2384036720 0x7f2384038bd0 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7f238c009990 tx=0x7f238c006e40 comp rx=0 tx=0).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:34:15.462 INFO:teuthology.orchestra.run.smithi169.stderr:2023-12-16T21:34:15.460+0000 7f238affd700 1 -- 172.21.15.169:0/2637764047 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+174570 (secure 0 0 0) 0x7f2390011550 con 0x7f239c0fd3b0 2023-12-16T21:34:15.566 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Dec 16 21:34:15 smithi045 ceph-mon[80763]: pgmap v12: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-16T21:34:15.735 INFO:teuthology.orchestra.run.smithi169.stderr:2023-12-16T21:34:15.732+0000 7f23a0b52700 1 -- 172.21.15.169:0/2637764047 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f239c110160 con 0x7f239c0fd3b0 2023-12-16T21:34:15.735 INFO:teuthology.orchestra.run.smithi169.stderr:2023-12-16T21:34:15.732+0000 7f238affd700 1 -- 172.21.15.169:0/2637764047 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+739 (secure 0 0 0) 0x7f2390011370 con 0x7f239c0fd3b0 2023-12-16T21:34:15.735 INFO:teuthology.orchestra.run.smithi169.stdout: 2023-12-16T21:34:15.735 INFO:teuthology.orchestra.run.smithi169.stdout:{"epoch":1,"fsid":"88ff363c-9c5a-11ee-95a5-87774f69a715","modified":"2023-12-16T21:32:46.744221Z","created":"2023-12-16T21:32:46.744221Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"smithi045","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:3300","nonce":0},{"type":"v1","addr":"172.21.15.45:6789","nonce":0}]},"addr":"172.21.15.45:6789/0","public_addr":"172.21.15.45:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-12-16T21:34:15.737 INFO:teuthology.orchestra.run.smithi169.stderr:2023-12-16T21:34:15.735+0000 7f23a0b52700 1 -- 172.21.15.169:0/2637764047 >> [v2:172.21.15.45:6800/2215417738,v1:172.21.15.45:6801/2215417738] conn(0x7f2384036720 msgr2=0x7f2384038bd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:34:15.737 INFO:teuthology.orchestra.run.smithi169.stderr:2023-12-16T21:34:15.735+0000 7f23a0b52700 1 --2- 172.21.15.169:0/2637764047 >> [v2:172.21.15.45:6800/2215417738,v1:172.21.15.45:6801/2215417738] conn(0x7f2384036720 0x7f2384038bd0 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7f238c009990 tx=0x7f238c006e40 comp rx=0 tx=0).stop 2023-12-16T21:34:15.737 INFO:teuthology.orchestra.run.smithi169.stderr:2023-12-16T21:34:15.735+0000 7f23a0b52700 1 -- 172.21.15.169:0/2637764047 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f239c0fd3b0 msgr2=0x7f239c10ed30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:34:15.737 INFO:teuthology.orchestra.run.smithi169.stderr:2023-12-16T21:34:15.735+0000 7f23a0b52700 1 --2- 172.21.15.169:0/2637764047 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f239c0fd3b0 0x7f239c10ed30 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7f239000b690 tx=0x7f2390009e30 comp rx=0 tx=0).stop 2023-12-16T21:34:15.737 INFO:teuthology.orchestra.run.smithi169.stderr:2023-12-16T21:34:15.735+0000 7f23a0b52700 1 -- 172.21.15.169:0/2637764047 shutdown_connections 2023-12-16T21:34:15.737 INFO:teuthology.orchestra.run.smithi169.stderr:2023-12-16T21:34:15.735+0000 7f23a0b52700 1 --2- 172.21.15.169:0/2637764047 >> [v2:172.21.15.45:6800/2215417738,v1:172.21.15.45:6801/2215417738] conn(0x7f2384036720 0x7f2384038bd0 unknown :-1 s=CLOSED pgs=24 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:34:15.737 INFO:teuthology.orchestra.run.smithi169.stderr:2023-12-16T21:34:15.735+0000 7f23a0b52700 1 --2- 172.21.15.169:0/2637764047 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f239c0fd3b0 0x7f239c10ed30 unknown :-1 s=CLOSED pgs=115 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:34:15.737 INFO:teuthology.orchestra.run.smithi169.stderr:2023-12-16T21:34:15.735+0000 7f23a0b52700 1 -- 172.21.15.169:0/2637764047 >> 172.21.15.169:0/2637764047 conn(0x7f239c0f6e50 msgr2=0x7f239c0fb1a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-16T21:34:15.737 INFO:teuthology.orchestra.run.smithi169.stderr:2023-12-16T21:34:15.735+0000 7f23a0b52700 1 -- 172.21.15.169:0/2637764047 shutdown_connections 2023-12-16T21:34:15.737 INFO:teuthology.orchestra.run.smithi169.stderr:2023-12-16T21:34:15.735+0000 7f23a0b52700 1 -- 172.21.15.169:0/2637764047 wait complete. 2023-12-16T21:34:15.738 INFO:teuthology.orchestra.run.smithi169.stderr:dumped monmap epoch 1 2023-12-16T21:34:16.566 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Dec 16 21:34:16 smithi045 ceph-mon[80763]: from='client.? 172.21.15.169:0/2637764047' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-12-16T21:34:17.041 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-12-16T21:34:17.041 DEBUG:teuthology.orchestra.run.smithi169:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 88ff363c-9c5a-11ee-95a5-87774f69a715 -- ceph mon dump -f json 2023-12-16T21:34:17.283 INFO:teuthology.orchestra.run.smithi169.stderr:Inferring config /etc/ceph/ceph.conf 2023-12-16T21:34:17.330 INFO:teuthology.orchestra.run.smithi169.stderr:Inferring config /etc/ceph/ceph.conf 2023-12-16T21:34:17.565 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Dec 16 21:34:17 smithi045 ceph-mon[80763]: pgmap v13: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-16T21:34:18.129 INFO:teuthology.orchestra.run.smithi169.stderr:2023-12-16T21:34:18.122+0000 7fccc6ae1700 1 -- 172.21.15.169:0/2944612568 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fccc00fb770 msgr2=0x7fccc00fbb80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:34:18.129 INFO:teuthology.orchestra.run.smithi169.stderr:2023-12-16T21:34:18.122+0000 7fccc6ae1700 1 --2- 172.21.15.169:0/2944612568 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fccc00fb770 0x7fccc00fbb80 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7fccb40048c0 tx=0x7fccb4009e30 comp rx=0 tx=0).stop 2023-12-16T21:34:18.129 INFO:teuthology.orchestra.run.smithi169.stderr:2023-12-16T21:34:18.123+0000 7fccc6ae1700 1 -- 172.21.15.169:0/2944612568 shutdown_connections 2023-12-16T21:34:18.129 INFO:teuthology.orchestra.run.smithi169.stderr:2023-12-16T21:34:18.123+0000 7fccc6ae1700 1 --2- 172.21.15.169:0/2944612568 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fccc00fb770 0x7fccc00fbb80 unknown :-1 s=CLOSED pgs=116 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:34:18.129 INFO:teuthology.orchestra.run.smithi169.stderr:2023-12-16T21:34:18.123+0000 7fccc6ae1700 1 -- 172.21.15.169:0/2944612568 >> 172.21.15.169:0/2944612568 conn(0x7fccc00f6e50 msgr2=0x7fccc00f9260 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-16T21:34:18.130 INFO:teuthology.orchestra.run.smithi169.stderr:2023-12-16T21:34:18.123+0000 7fccc6ae1700 1 -- 172.21.15.169:0/2944612568 shutdown_connections 2023-12-16T21:34:18.130 INFO:teuthology.orchestra.run.smithi169.stderr:2023-12-16T21:34:18.123+0000 7fccc6ae1700 1 -- 172.21.15.169:0/2944612568 wait complete. 2023-12-16T21:34:18.130 INFO:teuthology.orchestra.run.smithi169.stderr:2023-12-16T21:34:18.123+0000 7fccc6ae1700 1 Processor -- start 2023-12-16T21:34:18.130 INFO:teuthology.orchestra.run.smithi169.stderr:2023-12-16T21:34:18.123+0000 7fccc6ae1700 1 -- start start 2023-12-16T21:34:18.130 INFO:teuthology.orchestra.run.smithi169.stderr:2023-12-16T21:34:18.124+0000 7fccc6ae1700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fccc00fb770 0x7fccc010cab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:34:18.130 INFO:teuthology.orchestra.run.smithi169.stderr:2023-12-16T21:34:18.124+0000 7fccc6ae1700 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fccc010cff0 con 0x7fccc00fb770 2023-12-16T21:34:18.130 INFO:teuthology.orchestra.run.smithi169.stderr:2023-12-16T21:34:18.124+0000 7fccc487d700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fccc00fb770 0x7fccc010cab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:34:18.131 INFO:teuthology.orchestra.run.smithi169.stderr:2023-12-16T21:34:18.124+0000 7fccc487d700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fccc00fb770 0x7fccc010cab0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.169:52184/0 (socket says 172.21.15.169:52184) 2023-12-16T21:34:18.131 INFO:teuthology.orchestra.run.smithi169.stderr:2023-12-16T21:34:18.124+0000 7fccc487d700 1 -- 172.21.15.169:0/345235710 learned_addr learned my addr 172.21.15.169:0/345235710 (peer_addr_for_me v2:172.21.15.169:0/0) 2023-12-16T21:34:18.131 INFO:teuthology.orchestra.run.smithi169.stderr:2023-12-16T21:34:18.124+0000 7fccc487d700 1 -- 172.21.15.169:0/345235710 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fccb4005040 con 0x7fccc00fb770 2023-12-16T21:34:18.131 INFO:teuthology.orchestra.run.smithi169.stderr:2023-12-16T21:34:18.124+0000 7fccc487d700 1 --2- 172.21.15.169:0/345235710 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fccc00fb770 0x7fccc010cab0 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7fccb400b690 tx=0x7fccb4009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:34:18.131 INFO:teuthology.orchestra.run.smithi169.stderr:2023-12-16T21:34:18.124+0000 7fccbd7fa700 1 -- 172.21.15.169:0/345235710 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fccb4009650 con 0x7fccc00fb770 2023-12-16T21:34:18.131 INFO:teuthology.orchestra.run.smithi169.stderr:2023-12-16T21:34:18.124+0000 7fccbd7fa700 1 -- 172.21.15.169:0/345235710 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fccb40097b0 con 0x7fccc00fb770 2023-12-16T21:34:18.132 INFO:teuthology.orchestra.run.smithi169.stderr:2023-12-16T21:34:18.124+0000 7fccc6ae1700 1 -- 172.21.15.169:0/345235710 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fccc010d1f0 con 0x7fccc00fb770 2023-12-16T21:34:18.132 INFO:teuthology.orchestra.run.smithi169.stderr:2023-12-16T21:34:18.124+0000 7fccbd7fa700 1 -- 172.21.15.169:0/345235710 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fccb401c5d0 con 0x7fccc00fb770 2023-12-16T21:34:18.132 INFO:teuthology.orchestra.run.smithi169.stderr:2023-12-16T21:34:18.125+0000 7fccc6ae1700 1 -- 172.21.15.169:0/345235710 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fccc010de90 con 0x7fccc00fb770 2023-12-16T21:34:18.132 INFO:teuthology.orchestra.run.smithi169.stderr:2023-12-16T21:34:18.125+0000 7fccbd7fa700 1 -- 172.21.15.169:0/345235710 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 13) v1 ==== 43337+0+0 (secure 0 0 0) 0x7fccb401c840 con 0x7fccc00fb770 2023-12-16T21:34:18.132 INFO:teuthology.orchestra.run.smithi169.stderr:2023-12-16T21:34:18.126+0000 7fccbd7fa700 1 --2- 172.21.15.169:0/345235710 >> [v2:172.21.15.45:6800/2215417738,v1:172.21.15.45:6801/2215417738] conn(0x7fccb00366d0 0x7fccb0038b80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:34:18.132 INFO:teuthology.orchestra.run.smithi169.stderr:2023-12-16T21:34:18.126+0000 7fccbd7fa700 1 -- 172.21.15.169:0/345235710 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1151+0+0 (secure 0 0 0) 0x7fccb4049a60 con 0x7fccc00fb770 2023-12-16T21:34:18.133 INFO:teuthology.orchestra.run.smithi169.stderr:2023-12-16T21:34:18.126+0000 7fccc6ae1700 1 -- 172.21.15.169:0/345235710 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fccc005f270 con 0x7fccc00fb770 2023-12-16T21:34:18.133 INFO:teuthology.orchestra.run.smithi169.stderr:2023-12-16T21:34:18.126+0000 7fccbffff700 1 --2- 172.21.15.169:0/345235710 >> [v2:172.21.15.45:6800/2215417738,v1:172.21.15.45:6801/2215417738] conn(0x7fccb00366d0 0x7fccb0038b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:34:18.133 INFO:teuthology.orchestra.run.smithi169.stderr:2023-12-16T21:34:18.126+0000 7fccbffff700 1 --2- 172.21.15.169:0/345235710 >> [v2:172.21.15.45:6800/2215417738,v1:172.21.15.45:6801/2215417738] conn(0x7fccb00366d0 0x7fccb0038b80 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7fccac009990 tx=0x7fccac006e40 comp rx=0 tx=0).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:34:18.134 INFO:teuthology.orchestra.run.smithi169.stderr:2023-12-16T21:34:18.131+0000 7fccbd7fa700 1 -- 172.21.15.169:0/345235710 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+174570 (secure 0 0 0) 0x7fccb4023950 con 0x7fccc00fb770 2023-12-16T21:34:18.396 INFO:teuthology.orchestra.run.smithi169.stderr:2023-12-16T21:34:18.394+0000 7fccc6ae1700 1 -- 172.21.15.169:0/345235710 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fccc010e940 con 0x7fccc00fb770 2023-12-16T21:34:18.396 INFO:teuthology.orchestra.run.smithi169.stderr:2023-12-16T21:34:18.394+0000 7fccbd7fa700 1 -- 172.21.15.169:0/345235710 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+739 (secure 0 0 0) 0x7fccb4023300 con 0x7fccc00fb770 2023-12-16T21:34:18.397 INFO:teuthology.orchestra.run.smithi169.stdout: 2023-12-16T21:34:18.397 INFO:teuthology.orchestra.run.smithi169.stdout:{"epoch":1,"fsid":"88ff363c-9c5a-11ee-95a5-87774f69a715","modified":"2023-12-16T21:32:46.744221Z","created":"2023-12-16T21:32:46.744221Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"smithi045","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:3300","nonce":0},{"type":"v1","addr":"172.21.15.45:6789","nonce":0}]},"addr":"172.21.15.45:6789/0","public_addr":"172.21.15.45:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-12-16T21:34:18.399 INFO:teuthology.orchestra.run.smithi169.stderr:2023-12-16T21:34:18.396+0000 7fccc6ae1700 1 -- 172.21.15.169:0/345235710 >> [v2:172.21.15.45:6800/2215417738,v1:172.21.15.45:6801/2215417738] conn(0x7fccb00366d0 msgr2=0x7fccb0038b80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:34:18.399 INFO:teuthology.orchestra.run.smithi169.stderr:2023-12-16T21:34:18.396+0000 7fccc6ae1700 1 --2- 172.21.15.169:0/345235710 >> [v2:172.21.15.45:6800/2215417738,v1:172.21.15.45:6801/2215417738] conn(0x7fccb00366d0 0x7fccb0038b80 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7fccac009990 tx=0x7fccac006e40 comp rx=0 tx=0).stop 2023-12-16T21:34:18.399 INFO:teuthology.orchestra.run.smithi169.stderr:2023-12-16T21:34:18.396+0000 7fccc6ae1700 1 -- 172.21.15.169:0/345235710 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fccc00fb770 msgr2=0x7fccc010cab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:34:18.399 INFO:teuthology.orchestra.run.smithi169.stderr:2023-12-16T21:34:18.396+0000 7fccc6ae1700 1 --2- 172.21.15.169:0/345235710 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fccc00fb770 0x7fccc010cab0 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7fccb400b690 tx=0x7fccb4009e30 comp rx=0 tx=0).stop 2023-12-16T21:34:18.400 INFO:teuthology.orchestra.run.smithi169.stderr:2023-12-16T21:34:18.396+0000 7fccc6ae1700 1 -- 172.21.15.169:0/345235710 shutdown_connections 2023-12-16T21:34:18.400 INFO:teuthology.orchestra.run.smithi169.stderr:2023-12-16T21:34:18.396+0000 7fccc6ae1700 1 --2- 172.21.15.169:0/345235710 >> [v2:172.21.15.45:6800/2215417738,v1:172.21.15.45:6801/2215417738] conn(0x7fccb00366d0 0x7fccb0038b80 unknown :-1 s=CLOSED pgs=25 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:34:18.400 INFO:teuthology.orchestra.run.smithi169.stderr:2023-12-16T21:34:18.396+0000 7fccc6ae1700 1 --2- 172.21.15.169:0/345235710 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fccc00fb770 0x7fccc010cab0 unknown :-1 s=CLOSED pgs=117 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:34:18.400 INFO:teuthology.orchestra.run.smithi169.stderr:2023-12-16T21:34:18.396+0000 7fccc6ae1700 1 -- 172.21.15.169:0/345235710 >> 172.21.15.169:0/345235710 conn(0x7fccc00f6e50 msgr2=0x7fccc00660a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-16T21:34:18.400 INFO:teuthology.orchestra.run.smithi169.stderr:2023-12-16T21:34:18.396+0000 7fccc6ae1700 1 -- 172.21.15.169:0/345235710 shutdown_connections 2023-12-16T21:34:18.400 INFO:teuthology.orchestra.run.smithi169.stderr:2023-12-16T21:34:18.396+0000 7fccc6ae1700 1 -- 172.21.15.169:0/345235710 wait complete. 2023-12-16T21:34:18.401 INFO:teuthology.orchestra.run.smithi169.stderr:dumped monmap epoch 1 2023-12-16T21:34:19.545 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Dec 16 21:34:19 smithi045 ceph-mon[80763]: pgmap v14: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-16T21:34:19.545 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Dec 16 21:34:19 smithi045 ceph-mon[80763]: from='client.? 172.21.15.169:0/345235710' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-12-16T21:34:19.656 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-12-16T21:34:19.656 DEBUG:teuthology.orchestra.run.smithi169:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 88ff363c-9c5a-11ee-95a5-87774f69a715 -- ceph mon dump -f json 2023-12-16T21:34:19.907 INFO:teuthology.orchestra.run.smithi169.stderr:Inferring config /etc/ceph/ceph.conf 2023-12-16T21:34:19.955 INFO:teuthology.orchestra.run.smithi169.stderr:Inferring config /etc/ceph/ceph.conf 2023-12-16T21:34:20.702 INFO:teuthology.orchestra.run.smithi169.stderr:2023-12-16T21:34:20.700+0000 7f171f707700 1 -- 172.21.15.169:0/1276477334 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f17180fd3b0 msgr2=0x7f17180ff790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:34:20.705 INFO:teuthology.orchestra.run.smithi169.stderr:2023-12-16T21:34:20.700+0000 7f171f707700 1 --2- 172.21.15.169:0/1276477334 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f17180fd3b0 0x7f17180ff790 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7f17140048c0 tx=0x7f1714009e30 comp rx=0 tx=0).stop 2023-12-16T21:34:20.705 INFO:teuthology.orchestra.run.smithi169.stderr:2023-12-16T21:34:20.700+0000 7f171f707700 1 -- 172.21.15.169:0/1276477334 shutdown_connections 2023-12-16T21:34:20.705 INFO:teuthology.orchestra.run.smithi169.stderr:2023-12-16T21:34:20.700+0000 7f171f707700 1 --2- 172.21.15.169:0/1276477334 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f17180fd3b0 0x7f17180ff790 unknown :-1 s=CLOSED pgs=118 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:34:20.705 INFO:teuthology.orchestra.run.smithi169.stderr:2023-12-16T21:34:20.700+0000 7f171f707700 1 -- 172.21.15.169:0/1276477334 >> 172.21.15.169:0/1276477334 conn(0x7f17180f6e50 msgr2=0x7f17180f9260 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-16T21:34:20.705 INFO:teuthology.orchestra.run.smithi169.stderr:2023-12-16T21:34:20.700+0000 7f171f707700 1 -- 172.21.15.169:0/1276477334 shutdown_connections 2023-12-16T21:34:20.707 INFO:teuthology.orchestra.run.smithi169.stderr:2023-12-16T21:34:20.700+0000 7f171f707700 1 -- 172.21.15.169:0/1276477334 wait complete. 2023-12-16T21:34:20.707 INFO:teuthology.orchestra.run.smithi169.stderr:2023-12-16T21:34:20.700+0000 7f171f707700 1 Processor -- start 2023-12-16T21:34:20.707 INFO:teuthology.orchestra.run.smithi169.stderr:2023-12-16T21:34:20.700+0000 7f171f707700 1 -- start start 2023-12-16T21:34:20.707 INFO:teuthology.orchestra.run.smithi169.stderr:2023-12-16T21:34:20.701+0000 7f171f707700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f17180fd3b0 0x7f171810ca90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:34:20.707 INFO:teuthology.orchestra.run.smithi169.stderr:2023-12-16T21:34:20.701+0000 7f171f707700 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f171810cfd0 con 0x7f17180fd3b0 2023-12-16T21:34:20.707 INFO:teuthology.orchestra.run.smithi169.stderr:2023-12-16T21:34:20.701+0000 7f171d4a3700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f17180fd3b0 0x7f171810ca90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:34:20.707 INFO:teuthology.orchestra.run.smithi169.stderr:2023-12-16T21:34:20.701+0000 7f171d4a3700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f17180fd3b0 0x7f171810ca90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.169:52214/0 (socket says 172.21.15.169:52214) 2023-12-16T21:34:20.707 INFO:teuthology.orchestra.run.smithi169.stderr:2023-12-16T21:34:20.701+0000 7f171d4a3700 1 -- 172.21.15.169:0/2072130761 learned_addr learned my addr 172.21.15.169:0/2072130761 (peer_addr_for_me v2:172.21.15.169:0/0) 2023-12-16T21:34:20.707 INFO:teuthology.orchestra.run.smithi169.stderr:2023-12-16T21:34:20.701+0000 7f171d4a3700 1 -- 172.21.15.169:0/2072130761 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1714005040 con 0x7f17180fd3b0 2023-12-16T21:34:20.708 INFO:teuthology.orchestra.run.smithi169.stderr:2023-12-16T21:34:20.701+0000 7f171d4a3700 1 --2- 172.21.15.169:0/2072130761 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f17180fd3b0 0x7f171810ca90 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7f171400b690 tx=0x7f1714009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:34:20.708 INFO:teuthology.orchestra.run.smithi169.stderr:2023-12-16T21:34:20.701+0000 7f170dffb700 1 -- 172.21.15.169:0/2072130761 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f1714009650 con 0x7f17180fd3b0 2023-12-16T21:34:20.708 INFO:teuthology.orchestra.run.smithi169.stderr:2023-12-16T21:34:20.701+0000 7f170dffb700 1 -- 172.21.15.169:0/2072130761 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f17140097b0 con 0x7f17180fd3b0 2023-12-16T21:34:20.708 INFO:teuthology.orchestra.run.smithi169.stderr:2023-12-16T21:34:20.701+0000 7f171f707700 1 -- 172.21.15.169:0/2072130761 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f171810d1d0 con 0x7f17180fd3b0 2023-12-16T21:34:20.708 INFO:teuthology.orchestra.run.smithi169.stderr:2023-12-16T21:34:20.701+0000 7f170dffb700 1 -- 172.21.15.169:0/2072130761 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f171401c5b0 con 0x7f17180fd3b0 2023-12-16T21:34:20.708 INFO:teuthology.orchestra.run.smithi169.stderr:2023-12-16T21:34:20.702+0000 7f171f707700 1 -- 172.21.15.169:0/2072130761 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f171810de70 con 0x7f17180fd3b0 2023-12-16T21:34:20.708 INFO:teuthology.orchestra.run.smithi169.stderr:2023-12-16T21:34:20.702+0000 7f170dffb700 1 -- 172.21.15.169:0/2072130761 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 13) v1 ==== 43337+0+0 (secure 0 0 0) 0x7f171401c820 con 0x7f17180fd3b0 2023-12-16T21:34:20.708 INFO:teuthology.orchestra.run.smithi169.stderr:2023-12-16T21:34:20.702+0000 7f170dffb700 1 --2- 172.21.15.169:0/2072130761 >> [v2:172.21.15.45:6800/2215417738,v1:172.21.15.45:6801/2215417738] conn(0x7f1704036720 0x7f1704038bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:34:20.708 INFO:teuthology.orchestra.run.smithi169.stderr:2023-12-16T21:34:20.703+0000 7f170dffb700 1 -- 172.21.15.169:0/2072130761 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1151+0+0 (secure 0 0 0) 0x7f1714049820 con 0x7f17180fd3b0 2023-12-16T21:34:20.708 INFO:teuthology.orchestra.run.smithi169.stderr:2023-12-16T21:34:20.703+0000 7f171cca2700 1 --2- 172.21.15.169:0/2072130761 >> [v2:172.21.15.45:6800/2215417738,v1:172.21.15.45:6801/2215417738] conn(0x7f1704036720 0x7f1704038bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:34:20.708 INFO:teuthology.orchestra.run.smithi169.stderr:2023-12-16T21:34:20.703+0000 7f171f707700 1 -- 172.21.15.169:0/2072130761 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f171805f270 con 0x7f17180fd3b0 2023-12-16T21:34:20.708 INFO:teuthology.orchestra.run.smithi169.stderr:2023-12-16T21:34:20.703+0000 7f171cca2700 1 --2- 172.21.15.169:0/2072130761 >> [v2:172.21.15.45:6800/2215417738,v1:172.21.15.45:6801/2215417738] conn(0x7f1704036720 0x7f1704038bd0 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f1708009990 tx=0x7f1708006e40 comp rx=0 tx=0).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:34:20.710 INFO:teuthology.orchestra.run.smithi169.stderr:2023-12-16T21:34:20.708+0000 7f170dffb700 1 -- 172.21.15.169:0/2072130761 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+174570 (secure 0 0 0) 0x7f1714023300 con 0x7f17180fd3b0 2023-12-16T21:34:20.971 INFO:teuthology.orchestra.run.smithi169.stderr:2023-12-16T21:34:20.968+0000 7f171f707700 1 -- 172.21.15.169:0/2072130761 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f171810e880 con 0x7f17180fd3b0 2023-12-16T21:34:20.971 INFO:teuthology.orchestra.run.smithi169.stderr:2023-12-16T21:34:20.969+0000 7f170dffb700 1 -- 172.21.15.169:0/2072130761 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+739 (secure 0 0 0) 0x7f1714034090 con 0x7f17180fd3b0 2023-12-16T21:34:20.971 INFO:teuthology.orchestra.run.smithi169.stdout: 2023-12-16T21:34:20.971 INFO:teuthology.orchestra.run.smithi169.stdout:{"epoch":1,"fsid":"88ff363c-9c5a-11ee-95a5-87774f69a715","modified":"2023-12-16T21:32:46.744221Z","created":"2023-12-16T21:32:46.744221Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"smithi045","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:3300","nonce":0},{"type":"v1","addr":"172.21.15.45:6789","nonce":0}]},"addr":"172.21.15.45:6789/0","public_addr":"172.21.15.45:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-12-16T21:34:20.973 INFO:teuthology.orchestra.run.smithi169.stderr:2023-12-16T21:34:20.971+0000 7f171f707700 1 -- 172.21.15.169:0/2072130761 >> [v2:172.21.15.45:6800/2215417738,v1:172.21.15.45:6801/2215417738] conn(0x7f1704036720 msgr2=0x7f1704038bd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:34:20.973 INFO:teuthology.orchestra.run.smithi169.stderr:2023-12-16T21:34:20.971+0000 7f171f707700 1 --2- 172.21.15.169:0/2072130761 >> [v2:172.21.15.45:6800/2215417738,v1:172.21.15.45:6801/2215417738] conn(0x7f1704036720 0x7f1704038bd0 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f1708009990 tx=0x7f1708006e40 comp rx=0 tx=0).stop 2023-12-16T21:34:20.973 INFO:teuthology.orchestra.run.smithi169.stderr:2023-12-16T21:34:20.971+0000 7f171f707700 1 -- 172.21.15.169:0/2072130761 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f17180fd3b0 msgr2=0x7f171810ca90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:34:20.973 INFO:teuthology.orchestra.run.smithi169.stderr:2023-12-16T21:34:20.971+0000 7f171f707700 1 --2- 172.21.15.169:0/2072130761 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f17180fd3b0 0x7f171810ca90 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7f171400b690 tx=0x7f1714009e30 comp rx=0 tx=0).stop 2023-12-16T21:34:20.973 INFO:teuthology.orchestra.run.smithi169.stderr:2023-12-16T21:34:20.971+0000 7f171f707700 1 -- 172.21.15.169:0/2072130761 shutdown_connections 2023-12-16T21:34:20.973 INFO:teuthology.orchestra.run.smithi169.stderr:2023-12-16T21:34:20.971+0000 7f171f707700 1 --2- 172.21.15.169:0/2072130761 >> [v2:172.21.15.45:6800/2215417738,v1:172.21.15.45:6801/2215417738] conn(0x7f1704036720 0x7f1704038bd0 unknown :-1 s=CLOSED pgs=26 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:34:20.973 INFO:teuthology.orchestra.run.smithi169.stderr:2023-12-16T21:34:20.971+0000 7f171f707700 1 --2- 172.21.15.169:0/2072130761 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f17180fd3b0 0x7f171810ca90 unknown :-1 s=CLOSED pgs=119 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:34:20.973 INFO:teuthology.orchestra.run.smithi169.stderr:2023-12-16T21:34:20.971+0000 7f171f707700 1 -- 172.21.15.169:0/2072130761 >> 172.21.15.169:0/2072130761 conn(0x7f17180f6e50 msgr2=0x7f17180f7bf0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-16T21:34:20.973 INFO:teuthology.orchestra.run.smithi169.stderr:2023-12-16T21:34:20.971+0000 7f171f707700 1 -- 172.21.15.169:0/2072130761 shutdown_connections 2023-12-16T21:34:20.974 INFO:teuthology.orchestra.run.smithi169.stderr:2023-12-16T21:34:20.971+0000 7f171f707700 1 -- 172.21.15.169:0/2072130761 wait complete. 2023-12-16T21:34:20.974 INFO:teuthology.orchestra.run.smithi169.stderr:dumped monmap epoch 1 2023-12-16T21:34:21.566 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Dec 16 21:34:21 smithi045 ceph-mon[80763]: pgmap v15: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-16T21:34:21.566 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Dec 16 21:34:21 smithi045 ceph-mon[80763]: from='mgr.14162 172.21.15.45:0/142909557' entity='mgr.smithi045.nhtmwc' 2023-12-16T21:34:21.566 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Dec 16 21:34:21 smithi045 ceph-mon[80763]: from='mgr.14162 172.21.15.45:0/142909557' entity='mgr.smithi045.nhtmwc' 2023-12-16T21:34:21.566 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Dec 16 21:34:21 smithi045 ceph-mon[80763]: from='mgr.14162 172.21.15.45:0/142909557' entity='mgr.smithi045.nhtmwc' 2023-12-16T21:34:21.566 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Dec 16 21:34:21 smithi045 ceph-mon[80763]: from='mgr.14162 172.21.15.45:0/142909557' entity='mgr.smithi045.nhtmwc' cmd=[{"prefix": "mgr module enable", "module": "prometheus"}]: dispatch 2023-12-16T21:34:21.566 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Dec 16 21:34:21 smithi045 ceph-mon[80763]: from='client.? 172.21.15.169:0/2072130761' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-12-16T21:34:22.279 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-12-16T21:34:22.279 DEBUG:teuthology.orchestra.run.smithi169:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 88ff363c-9c5a-11ee-95a5-87774f69a715 -- ceph mon dump -f json 2023-12-16T21:34:22.523 INFO:teuthology.orchestra.run.smithi169.stderr:Inferring config /etc/ceph/ceph.conf 2023-12-16T21:34:22.570 INFO:teuthology.orchestra.run.smithi169.stderr:Inferring config /etc/ceph/ceph.conf 2023-12-16T21:34:23.062 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Dec 16 21:34:22 smithi045 ceph-mon[80763]: from='mgr.14162 172.21.15.45:0/142909557' entity='mgr.smithi045.nhtmwc' cmd='[{"prefix": "mgr module enable", "module": "prometheus"}]': finished 2023-12-16T21:34:23.062 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Dec 16 21:34:22 smithi045 ceph-mon[80763]: mgrmap e14: smithi045.nhtmwc(active, since 43s) 2023-12-16T21:34:23.318 INFO:teuthology.orchestra.run.smithi169.stderr:2023-12-16T21:34:23.314+0000 7f131220b700 1 -- 172.21.15.169:0/590982028 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f130c0fb770 msgr2=0x7f130c0fbb80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:34:23.318 INFO:teuthology.orchestra.run.smithi169.stderr:2023-12-16T21:34:23.314+0000 7f131220b700 1 --2- 172.21.15.169:0/590982028 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f130c0fb770 0x7f130c0fbb80 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7f12fc0048c0 tx=0x7f12fc009e30 comp rx=0 tx=0).stop 2023-12-16T21:34:23.318 INFO:teuthology.orchestra.run.smithi169.stderr:2023-12-16T21:34:23.314+0000 7f131220b700 1 -- 172.21.15.169:0/590982028 shutdown_connections 2023-12-16T21:34:23.318 INFO:teuthology.orchestra.run.smithi169.stderr:2023-12-16T21:34:23.314+0000 7f131220b700 1 --2- 172.21.15.169:0/590982028 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f130c0fb770 0x7f130c0fbb80 unknown :-1 s=CLOSED pgs=122 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:34:23.318 INFO:teuthology.orchestra.run.smithi169.stderr:2023-12-16T21:34:23.314+0000 7f131220b700 1 -- 172.21.15.169:0/590982028 >> 172.21.15.169:0/590982028 conn(0x7f130c0f6e50 msgr2=0x7f130c0f9260 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-16T21:34:23.318 INFO:teuthology.orchestra.run.smithi169.stderr:2023-12-16T21:34:23.314+0000 7f131220b700 1 -- 172.21.15.169:0/590982028 shutdown_connections 2023-12-16T21:34:23.319 INFO:teuthology.orchestra.run.smithi169.stderr:2023-12-16T21:34:23.314+0000 7f131220b700 1 -- 172.21.15.169:0/590982028 wait complete. 2023-12-16T21:34:23.319 INFO:teuthology.orchestra.run.smithi169.stderr:2023-12-16T21:34:23.315+0000 7f131220b700 1 Processor -- start 2023-12-16T21:34:23.319 INFO:teuthology.orchestra.run.smithi169.stderr:2023-12-16T21:34:23.315+0000 7f131220b700 1 -- start start 2023-12-16T21:34:23.319 INFO:teuthology.orchestra.run.smithi169.stderr:2023-12-16T21:34:23.315+0000 7f131220b700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f130c0fb770 0x7f130c10f780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:34:23.319 INFO:teuthology.orchestra.run.smithi169.stderr:2023-12-16T21:34:23.315+0000 7f131220b700 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f130c066520 con 0x7f130c0fb770 2023-12-16T21:34:23.319 INFO:teuthology.orchestra.run.smithi169.stderr:2023-12-16T21:34:23.315+0000 7f130b7fe700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f130c0fb770 0x7f130c10f780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:34:23.319 INFO:teuthology.orchestra.run.smithi169.stderr:2023-12-16T21:34:23.315+0000 7f130b7fe700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f130c0fb770 0x7f130c10f780 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.169:55080/0 (socket says 172.21.15.169:55080) 2023-12-16T21:34:23.319 INFO:teuthology.orchestra.run.smithi169.stderr:2023-12-16T21:34:23.315+0000 7f130b7fe700 1 -- 172.21.15.169:0/1981716866 learned_addr learned my addr 172.21.15.169:0/1981716866 (peer_addr_for_me v2:172.21.15.169:0/0) 2023-12-16T21:34:23.319 INFO:teuthology.orchestra.run.smithi169.stderr:2023-12-16T21:34:23.315+0000 7f130b7fe700 1 -- 172.21.15.169:0/1981716866 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f12fc005040 con 0x7f130c0fb770 2023-12-16T21:34:23.319 INFO:teuthology.orchestra.run.smithi169.stderr:2023-12-16T21:34:23.316+0000 7f130b7fe700 1 --2- 172.21.15.169:0/1981716866 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f130c0fb770 0x7f130c10f780 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7f12fc009330 tx=0x7f12fc009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:34:23.319 INFO:teuthology.orchestra.run.smithi169.stderr:2023-12-16T21:34:23.316+0000 7f1308ff9700 1 -- 172.21.15.169:0/1981716866 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f12fc0096b0 con 0x7f130c0fb770 2023-12-16T21:34:23.319 INFO:teuthology.orchestra.run.smithi169.stderr:2023-12-16T21:34:23.316+0000 7f1308ff9700 1 -- 172.21.15.169:0/1981716866 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f12fc009810 con 0x7f130c0fb770 2023-12-16T21:34:23.319 INFO:teuthology.orchestra.run.smithi169.stderr:2023-12-16T21:34:23.316+0000 7f131220b700 1 -- 172.21.15.169:0/1981716866 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f130c066720 con 0x7f130c0fb770 2023-12-16T21:34:23.319 INFO:teuthology.orchestra.run.smithi169.stderr:2023-12-16T21:34:23.316+0000 7f1308ff9700 1 -- 172.21.15.169:0/1981716866 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f12fc01c5b0 con 0x7f130c0fb770 2023-12-16T21:34:23.320 INFO:teuthology.orchestra.run.smithi169.stderr:2023-12-16T21:34:23.316+0000 7f131220b700 1 -- 172.21.15.169:0/1981716866 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f130c10ff00 con 0x7f130c0fb770 2023-12-16T21:34:23.320 INFO:teuthology.orchestra.run.smithi169.stderr:2023-12-16T21:34:23.317+0000 7f1308ff9700 1 -- 172.21.15.169:0/1981716866 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 14) v1 ==== 43351+0+0 (secure 0 0 0) 0x7f12fc013420 con 0x7f130c0fb770 2023-12-16T21:34:23.320 INFO:teuthology.orchestra.run.smithi169.stderr:2023-12-16T21:34:23.317+0000 7f1308ff9700 1 --2- 172.21.15.169:0/1981716866 >> [v2:172.21.15.45:6800/2215417738,v1:172.21.15.45:6801/2215417738] conn(0x7f12f8036720 0x7f12f8038bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:34:23.321 INFO:teuthology.orchestra.run.smithi169.stderr:2023-12-16T21:34:23.317+0000 7f131220b700 1 -- 172.21.15.169:0/1981716866 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f130c105400 con 0x7f130c0fb770 2023-12-16T21:34:23.321 INFO:teuthology.orchestra.run.smithi169.stderr:2023-12-16T21:34:23.317+0000 7f1308ff9700 1 -- 172.21.15.169:0/1981716866 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1151+0+0 (secure 0 0 0) 0x7f12fc049b60 con 0x7f130c0fb770 2023-12-16T21:34:23.321 INFO:teuthology.orchestra.run.smithi169.stderr:2023-12-16T21:34:23.317+0000 7f1303fff700 1 -- 172.21.15.169:0/1981716866 >> [v2:172.21.15.45:6800/2215417738,v1:172.21.15.45:6801/2215417738] conn(0x7f12f8036720 msgr2=0x7f12f8038bd0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.45:6800/2215417738 2023-12-16T21:34:23.321 INFO:teuthology.orchestra.run.smithi169.stderr:2023-12-16T21:34:23.317+0000 7f1303fff700 1 --2- 172.21.15.169:0/1981716866 >> [v2:172.21.15.45:6800/2215417738,v1:172.21.15.45:6801/2215417738] conn(0x7f12f8036720 0x7f12f8038bd0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-12-16T21:34:23.324 INFO:teuthology.orchestra.run.smithi169.stderr:2023-12-16T21:34:23.322+0000 7f1308ff9700 1 -- 172.21.15.169:0/1981716866 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+174570 (secure 0 0 0) 0x7f12fc0119c0 con 0x7f130c0fb770 2023-12-16T21:34:23.520 INFO:teuthology.orchestra.run.smithi169.stderr:2023-12-16T21:34:23.518+0000 7f1303fff700 1 -- 172.21.15.169:0/1981716866 >> [v2:172.21.15.45:6800/2215417738,v1:172.21.15.45:6801/2215417738] conn(0x7f12f8036720 msgr2=0x7f12f8038bd0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.45:6800/2215417738 2023-12-16T21:34:23.520 INFO:teuthology.orchestra.run.smithi169.stderr:2023-12-16T21:34:23.518+0000 7f1303fff700 1 --2- 172.21.15.169:0/1981716866 >> [v2:172.21.15.45:6800/2215417738,v1:172.21.15.45:6801/2215417738] conn(0x7f12f8036720 0x7f12f8038bd0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-12-16T21:34:23.583 INFO:teuthology.orchestra.run.smithi169.stderr:2023-12-16T21:34:23.581+0000 7f131220b700 1 -- 172.21.15.169:0/1981716866 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f130c04bb80 con 0x7f130c0fb770 2023-12-16T21:34:23.583 INFO:teuthology.orchestra.run.smithi169.stderr:2023-12-16T21:34:23.581+0000 7f1308ff9700 1 -- 172.21.15.169:0/1981716866 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+739 (secure 0 0 0) 0x7f12fc011370 con 0x7f130c0fb770 2023-12-16T21:34:23.583 INFO:teuthology.orchestra.run.smithi169.stdout: 2023-12-16T21:34:23.584 INFO:teuthology.orchestra.run.smithi169.stdout:{"epoch":1,"fsid":"88ff363c-9c5a-11ee-95a5-87774f69a715","modified":"2023-12-16T21:32:46.744221Z","created":"2023-12-16T21:32:46.744221Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"smithi045","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:3300","nonce":0},{"type":"v1","addr":"172.21.15.45:6789","nonce":0}]},"addr":"172.21.15.45:6789/0","public_addr":"172.21.15.45:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-12-16T21:34:23.586 INFO:teuthology.orchestra.run.smithi169.stderr:2023-12-16T21:34:23.583+0000 7f131220b700 1 -- 172.21.15.169:0/1981716866 >> [v2:172.21.15.45:6800/2215417738,v1:172.21.15.45:6801/2215417738] conn(0x7f12f8036720 msgr2=0x7f12f8038bd0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-12-16T21:34:23.586 INFO:teuthology.orchestra.run.smithi169.stderr:2023-12-16T21:34:23.583+0000 7f131220b700 1 --2- 172.21.15.169:0/1981716866 >> [v2:172.21.15.45:6800/2215417738,v1:172.21.15.45:6801/2215417738] conn(0x7f12f8036720 0x7f12f8038bd0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:34:23.586 INFO:teuthology.orchestra.run.smithi169.stderr:2023-12-16T21:34:23.583+0000 7f131220b700 1 -- 172.21.15.169:0/1981716866 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f130c0fb770 msgr2=0x7f130c10f780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:34:23.586 INFO:teuthology.orchestra.run.smithi169.stderr:2023-12-16T21:34:23.583+0000 7f131220b700 1 --2- 172.21.15.169:0/1981716866 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f130c0fb770 0x7f130c10f780 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7f12fc009330 tx=0x7f12fc009e30 comp rx=0 tx=0).stop 2023-12-16T21:34:23.586 INFO:teuthology.orchestra.run.smithi169.stderr:2023-12-16T21:34:23.583+0000 7f131220b700 1 -- 172.21.15.169:0/1981716866 shutdown_connections 2023-12-16T21:34:23.586 INFO:teuthology.orchestra.run.smithi169.stderr:2023-12-16T21:34:23.583+0000 7f131220b700 1 --2- 172.21.15.169:0/1981716866 >> [v2:172.21.15.45:6800/2215417738,v1:172.21.15.45:6801/2215417738] conn(0x7f12f8036720 0x7f12f8038bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:34:23.586 INFO:teuthology.orchestra.run.smithi169.stderr:2023-12-16T21:34:23.583+0000 7f131220b700 1 --2- 172.21.15.169:0/1981716866 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f130c0fb770 0x7f130c10f780 unknown :-1 s=CLOSED pgs=123 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:34:23.586 INFO:teuthology.orchestra.run.smithi169.stderr:2023-12-16T21:34:23.583+0000 7f131220b700 1 -- 172.21.15.169:0/1981716866 >> 172.21.15.169:0/1981716866 conn(0x7f130c0f6e50 msgr2=0x7f130c0f7bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-16T21:34:23.586 INFO:teuthology.orchestra.run.smithi169.stderr:2023-12-16T21:34:23.583+0000 7f131220b700 1 -- 172.21.15.169:0/1981716866 shutdown_connections 2023-12-16T21:34:23.586 INFO:teuthology.orchestra.run.smithi169.stderr:2023-12-16T21:34:23.583+0000 7f131220b700 1 -- 172.21.15.169:0/1981716866 wait complete. 2023-12-16T21:34:23.587 INFO:teuthology.orchestra.run.smithi169.stderr:dumped monmap epoch 1 2023-12-16T21:34:23.931 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Dec 16 21:34:23 smithi045 ceph-mon[80763]: from='client.? 172.21.15.169:0/1981716866' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-12-16T21:34:24.836 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-12-16T21:34:24.837 DEBUG:teuthology.orchestra.run.smithi169:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 88ff363c-9c5a-11ee-95a5-87774f69a715 -- ceph mon dump -f json 2023-12-16T21:34:25.085 INFO:teuthology.orchestra.run.smithi169.stderr:Inferring config /etc/ceph/ceph.conf 2023-12-16T21:34:25.131 INFO:teuthology.orchestra.run.smithi169.stderr:Inferring config /etc/ceph/ceph.conf 2023-12-16T21:34:25.883 INFO:teuthology.orchestra.run.smithi169.stderr:2023-12-16T21:34:25.881+0000 7f04c9f58700 1 -- 172.21.15.169:0/3674195755 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f04c4066530 msgr2=0x7f04c40ff450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:34:25.884 INFO:teuthology.orchestra.run.smithi169.stderr:2023-12-16T21:34:25.881+0000 7f04c9f58700 1 --2- 172.21.15.169:0/3674195755 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f04c4066530 0x7f04c40ff450 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7f04b4009c60 tx=0x7f04b4004d10 comp rx=0 tx=0).stop 2023-12-16T21:34:25.884 INFO:teuthology.orchestra.run.smithi169.stderr:2023-12-16T21:34:25.881+0000 7f04c9f58700 1 -- 172.21.15.169:0/3674195755 shutdown_connections 2023-12-16T21:34:25.884 INFO:teuthology.orchestra.run.smithi169.stderr:2023-12-16T21:34:25.881+0000 7f04c9f58700 1 --2- 172.21.15.169:0/3674195755 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f04c4066530 0x7f04c40ff450 unknown :-1 s=CLOSED pgs=124 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:34:25.884 INFO:teuthology.orchestra.run.smithi169.stderr:2023-12-16T21:34:25.881+0000 7f04c9f58700 1 -- 172.21.15.169:0/3674195755 >> 172.21.15.169:0/3674195755 conn(0x7f04c40f6ec0 msgr2=0x7f04c40f92f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-16T21:34:25.884 INFO:teuthology.orchestra.run.smithi169.stderr:2023-12-16T21:34:25.881+0000 7f04c9f58700 1 -- 172.21.15.169:0/3674195755 shutdown_connections 2023-12-16T21:34:25.884 INFO:teuthology.orchestra.run.smithi169.stderr:2023-12-16T21:34:25.881+0000 7f04c9f58700 1 -- 172.21.15.169:0/3674195755 wait complete. 2023-12-16T21:34:25.884 INFO:teuthology.orchestra.run.smithi169.stderr:2023-12-16T21:34:25.881+0000 7f04c9f58700 1 Processor -- start 2023-12-16T21:34:25.884 INFO:teuthology.orchestra.run.smithi169.stderr:2023-12-16T21:34:25.881+0000 7f04c9f58700 1 -- start start 2023-12-16T21:34:25.886 INFO:teuthology.orchestra.run.smithi169.stderr:2023-12-16T21:34:25.882+0000 7f04c9f58700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f04c4066530 0x7f04c40fced0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:34:25.886 INFO:teuthology.orchestra.run.smithi169.stderr:2023-12-16T21:34:25.882+0000 7f04c9f58700 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f04c40fb350 con 0x7f04c4066530 2023-12-16T21:34:25.886 INFO:teuthology.orchestra.run.smithi169.stderr:2023-12-16T21:34:25.882+0000 7f04c37fe700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f04c4066530 0x7f04c40fced0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:34:25.886 INFO:teuthology.orchestra.run.smithi169.stderr:2023-12-16T21:34:25.882+0000 7f04c37fe700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f04c4066530 0x7f04c40fced0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.169:55106/0 (socket says 172.21.15.169:55106) 2023-12-16T21:34:25.886 INFO:teuthology.orchestra.run.smithi169.stderr:2023-12-16T21:34:25.882+0000 7f04c37fe700 1 -- 172.21.15.169:0/519641825 learned_addr learned my addr 172.21.15.169:0/519641825 (peer_addr_for_me v2:172.21.15.169:0/0) 2023-12-16T21:34:25.886 INFO:teuthology.orchestra.run.smithi169.stderr:2023-12-16T21:34:25.882+0000 7f04c37fe700 1 -- 172.21.15.169:0/519641825 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f04b4009940 con 0x7f04c4066530 2023-12-16T21:34:25.886 INFO:teuthology.orchestra.run.smithi169.stderr:2023-12-16T21:34:25.882+0000 7f04c37fe700 1 --2- 172.21.15.169:0/519641825 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f04c4066530 0x7f04c40fced0 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7f04b4000f80 tx=0x7f04b4004d10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:34:25.886 INFO:teuthology.orchestra.run.smithi169.stderr:2023-12-16T21:34:25.882+0000 7f04abfff700 1 -- 172.21.15.169:0/519641825 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f04b400bec0 con 0x7f04c4066530 2023-12-16T21:34:25.886 INFO:teuthology.orchestra.run.smithi169.stderr:2023-12-16T21:34:25.883+0000 7f04abfff700 1 -- 172.21.15.169:0/519641825 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f04b401d8f0 con 0x7f04c4066530 2023-12-16T21:34:25.886 INFO:teuthology.orchestra.run.smithi169.stderr:2023-12-16T21:34:25.883+0000 7f04c9f58700 1 -- 172.21.15.169:0/519641825 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f04c40fb550 con 0x7f04c4066530 2023-12-16T21:34:25.887 INFO:teuthology.orchestra.run.smithi169.stderr:2023-12-16T21:34:25.883+0000 7f04abfff700 1 -- 172.21.15.169:0/519641825 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f04b4012890 con 0x7f04c4066530 2023-12-16T21:34:25.887 INFO:teuthology.orchestra.run.smithi169.stderr:2023-12-16T21:34:25.883+0000 7f04c9f58700 1 -- 172.21.15.169:0/519641825 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f04c40fba40 con 0x7f04c4066530 2023-12-16T21:34:25.887 INFO:teuthology.orchestra.run.smithi169.stderr:2023-12-16T21:34:25.883+0000 7f04abfff700 1 -- 172.21.15.169:0/519641825 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 14) v1 ==== 43351+0+0 (secure 0 0 0) 0x7f04b4019040 con 0x7f04c4066530 2023-12-16T21:34:25.887 INFO:teuthology.orchestra.run.smithi169.stderr:2023-12-16T21:34:25.884+0000 7f04abfff700 1 --2- 172.21.15.169:0/519641825 >> [v2:172.21.15.45:6800/2215417738,v1:172.21.15.45:6801/2215417738] conn(0x7f04ac036770 0x7f04ac038c20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:34:25.887 INFO:teuthology.orchestra.run.smithi169.stderr:2023-12-16T21:34:25.884+0000 7f04abfff700 1 -- 172.21.15.169:0/519641825 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1151+0+0 (secure 0 0 0) 0x7f04b4049670 con 0x7f04c4066530 2023-12-16T21:34:25.887 INFO:teuthology.orchestra.run.smithi169.stderr:2023-12-16T21:34:25.884+0000 7f04c2ffd700 1 -- 172.21.15.169:0/519641825 >> [v2:172.21.15.45:6800/2215417738,v1:172.21.15.45:6801/2215417738] conn(0x7f04ac036770 msgr2=0x7f04ac038c20 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.45:6800/2215417738 2023-12-16T21:34:25.887 INFO:teuthology.orchestra.run.smithi169.stderr:2023-12-16T21:34:25.884+0000 7f04c2ffd700 1 --2- 172.21.15.169:0/519641825 >> [v2:172.21.15.45:6800/2215417738,v1:172.21.15.45:6801/2215417738] conn(0x7f04ac036770 0x7f04ac038c20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-12-16T21:34:25.887 INFO:teuthology.orchestra.run.smithi169.stderr:2023-12-16T21:34:25.884+0000 7f04c9f58700 1 -- 172.21.15.169:0/519641825 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f04c4106460 con 0x7f04c4066530 2023-12-16T21:34:25.891 INFO:teuthology.orchestra.run.smithi169.stderr:2023-12-16T21:34:25.889+0000 7f04abfff700 1 -- 172.21.15.169:0/519641825 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+174570 (secure 0 0 0) 0x7f04b4021020 con 0x7f04c4066530 2023-12-16T21:34:26.087 INFO:teuthology.orchestra.run.smithi169.stderr:2023-12-16T21:34:26.085+0000 7f04c2ffd700 1 -- 172.21.15.169:0/519641825 >> [v2:172.21.15.45:6800/2215417738,v1:172.21.15.45:6801/2215417738] conn(0x7f04ac036770 msgr2=0x7f04ac038c20 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.45:6800/2215417738 2023-12-16T21:34:26.087 INFO:teuthology.orchestra.run.smithi169.stderr:2023-12-16T21:34:26.085+0000 7f04c2ffd700 1 --2- 172.21.15.169:0/519641825 >> [v2:172.21.15.45:6800/2215417738,v1:172.21.15.45:6801/2215417738] conn(0x7f04ac036770 0x7f04ac038c20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-12-16T21:34:26.175 INFO:teuthology.orchestra.run.smithi169.stderr:2023-12-16T21:34:26.173+0000 7f04c9f58700 1 -- 172.21.15.169:0/519641825 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f04c405f270 con 0x7f04c4066530 2023-12-16T21:34:26.175 INFO:teuthology.orchestra.run.smithi169.stderr:2023-12-16T21:34:26.173+0000 7f04abfff700 1 -- 172.21.15.169:0/519641825 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+739 (secure 0 0 0) 0x7f04b4017070 con 0x7f04c4066530 2023-12-16T21:34:26.176 INFO:teuthology.orchestra.run.smithi169.stdout: 2023-12-16T21:34:26.176 INFO:teuthology.orchestra.run.smithi169.stdout:{"epoch":1,"fsid":"88ff363c-9c5a-11ee-95a5-87774f69a715","modified":"2023-12-16T21:32:46.744221Z","created":"2023-12-16T21:32:46.744221Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"smithi045","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:3300","nonce":0},{"type":"v1","addr":"172.21.15.45:6789","nonce":0}]},"addr":"172.21.15.45:6789/0","public_addr":"172.21.15.45:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-12-16T21:34:26.178 INFO:teuthology.orchestra.run.smithi169.stderr:2023-12-16T21:34:26.175+0000 7f04c9f58700 1 -- 172.21.15.169:0/519641825 >> [v2:172.21.15.45:6800/2215417738,v1:172.21.15.45:6801/2215417738] conn(0x7f04ac036770 msgr2=0x7f04ac038c20 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-12-16T21:34:26.178 INFO:teuthology.orchestra.run.smithi169.stderr:2023-12-16T21:34:26.175+0000 7f04c9f58700 1 --2- 172.21.15.169:0/519641825 >> [v2:172.21.15.45:6800/2215417738,v1:172.21.15.45:6801/2215417738] conn(0x7f04ac036770 0x7f04ac038c20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:34:26.178 INFO:teuthology.orchestra.run.smithi169.stderr:2023-12-16T21:34:26.175+0000 7f04c9f58700 1 -- 172.21.15.169:0/519641825 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f04c4066530 msgr2=0x7f04c40fced0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:34:26.178 INFO:teuthology.orchestra.run.smithi169.stderr:2023-12-16T21:34:26.175+0000 7f04c9f58700 1 --2- 172.21.15.169:0/519641825 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f04c4066530 0x7f04c40fced0 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7f04b4000f80 tx=0x7f04b4004d10 comp rx=0 tx=0).stop 2023-12-16T21:34:26.178 INFO:teuthology.orchestra.run.smithi169.stderr:2023-12-16T21:34:26.175+0000 7f04c9f58700 1 -- 172.21.15.169:0/519641825 shutdown_connections 2023-12-16T21:34:26.178 INFO:teuthology.orchestra.run.smithi169.stderr:2023-12-16T21:34:26.175+0000 7f04c9f58700 1 --2- 172.21.15.169:0/519641825 >> [v2:172.21.15.45:6800/2215417738,v1:172.21.15.45:6801/2215417738] conn(0x7f04ac036770 0x7f04ac038c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:34:26.178 INFO:teuthology.orchestra.run.smithi169.stderr:2023-12-16T21:34:26.175+0000 7f04c9f58700 1 --2- 172.21.15.169:0/519641825 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f04c4066530 0x7f04c40fced0 unknown :-1 s=CLOSED pgs=125 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:34:26.178 INFO:teuthology.orchestra.run.smithi169.stderr:2023-12-16T21:34:26.175+0000 7f04c9f58700 1 -- 172.21.15.169:0/519641825 >> 172.21.15.169:0/519641825 conn(0x7f04c40f6ec0 msgr2=0x7f04c40f7da0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-16T21:34:26.178 INFO:teuthology.orchestra.run.smithi169.stderr:2023-12-16T21:34:26.175+0000 7f04c9f58700 1 -- 172.21.15.169:0/519641825 shutdown_connections 2023-12-16T21:34:26.178 INFO:teuthology.orchestra.run.smithi169.stderr:2023-12-16T21:34:26.175+0000 7f04c9f58700 1 -- 172.21.15.169:0/519641825 wait complete. 2023-12-16T21:34:26.179 INFO:teuthology.orchestra.run.smithi169.stderr:dumped monmap epoch 1 2023-12-16T21:34:26.512 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Dec 16 21:34:26 smithi045 ceph-mon[80763]: from='client.? 172.21.15.169:0/519641825' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-12-16T21:34:27.436 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-12-16T21:34:27.437 DEBUG:teuthology.orchestra.run.smithi169:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 88ff363c-9c5a-11ee-95a5-87774f69a715 -- ceph mon dump -f json 2023-12-16T21:34:27.678 INFO:teuthology.orchestra.run.smithi169.stderr:Inferring config /etc/ceph/ceph.conf 2023-12-16T21:34:27.725 INFO:teuthology.orchestra.run.smithi169.stderr:Inferring config /etc/ceph/ceph.conf 2023-12-16T21:34:28.468 INFO:teuthology.orchestra.run.smithi169.stderr:2023-12-16T21:34:28.464+0000 7fb0e961d700 1 -- 172.21.15.169:0/1417428391 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fb0e40fc880 msgr2=0x7fb0e40fcc90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:34:28.468 INFO:teuthology.orchestra.run.smithi169.stderr:2023-12-16T21:34:28.464+0000 7fb0e961d700 1 --2- 172.21.15.169:0/1417428391 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fb0e40fc880 0x7fb0e40fcc90 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7fb0d40048c0 tx=0x7fb0d4009e30 comp rx=0 tx=0).stop 2023-12-16T21:34:28.468 INFO:teuthology.orchestra.run.smithi169.stderr:2023-12-16T21:34:28.464+0000 7fb0e961d700 1 -- 172.21.15.169:0/1417428391 shutdown_connections 2023-12-16T21:34:28.468 INFO:teuthology.orchestra.run.smithi169.stderr:2023-12-16T21:34:28.464+0000 7fb0e961d700 1 --2- 172.21.15.169:0/1417428391 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fb0e40fc880 0x7fb0e40fcc90 unknown :-1 s=CLOSED pgs=126 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:34:28.469 INFO:teuthology.orchestra.run.smithi169.stderr:2023-12-16T21:34:28.464+0000 7fb0e961d700 1 -- 172.21.15.169:0/1417428391 >> 172.21.15.169:0/1417428391 conn(0x7fb0e40f7f40 msgr2=0x7fb0e40fa370 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-16T21:34:28.469 INFO:teuthology.orchestra.run.smithi169.stderr:2023-12-16T21:34:28.464+0000 7fb0e961d700 1 -- 172.21.15.169:0/1417428391 shutdown_connections 2023-12-16T21:34:28.469 INFO:teuthology.orchestra.run.smithi169.stderr:2023-12-16T21:34:28.464+0000 7fb0e961d700 1 -- 172.21.15.169:0/1417428391 wait complete. 2023-12-16T21:34:28.469 INFO:teuthology.orchestra.run.smithi169.stderr:2023-12-16T21:34:28.464+0000 7fb0e961d700 1 Processor -- start 2023-12-16T21:34:28.469 INFO:teuthology.orchestra.run.smithi169.stderr:2023-12-16T21:34:28.465+0000 7fb0e961d700 1 -- start start 2023-12-16T21:34:28.469 INFO:teuthology.orchestra.run.smithi169.stderr:2023-12-16T21:34:28.465+0000 7fb0e961d700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fb0e40fc880 0x7fb0e410cac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:34:28.469 INFO:teuthology.orchestra.run.smithi169.stderr:2023-12-16T21:34:28.465+0000 7fb0e961d700 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb0e410d000 con 0x7fb0e40fc880 2023-12-16T21:34:28.469 INFO:teuthology.orchestra.run.smithi169.stderr:2023-12-16T21:34:28.465+0000 7fb0e2ffd700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fb0e40fc880 0x7fb0e410cac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:34:28.469 INFO:teuthology.orchestra.run.smithi169.stderr:2023-12-16T21:34:28.465+0000 7fb0e2ffd700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fb0e40fc880 0x7fb0e410cac0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.169:55124/0 (socket says 172.21.15.169:55124) 2023-12-16T21:34:28.469 INFO:teuthology.orchestra.run.smithi169.stderr:2023-12-16T21:34:28.465+0000 7fb0e2ffd700 1 -- 172.21.15.169:0/2386149533 learned_addr learned my addr 172.21.15.169:0/2386149533 (peer_addr_for_me v2:172.21.15.169:0/0) 2023-12-16T21:34:28.469 INFO:teuthology.orchestra.run.smithi169.stderr:2023-12-16T21:34:28.465+0000 7fb0e2ffd700 1 -- 172.21.15.169:0/2386149533 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb0d4005040 con 0x7fb0e40fc880 2023-12-16T21:34:28.470 INFO:teuthology.orchestra.run.smithi169.stderr:2023-12-16T21:34:28.465+0000 7fb0e2ffd700 1 --2- 172.21.15.169:0/2386149533 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fb0e40fc880 0x7fb0e410cac0 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7fb0d4009330 tx=0x7fb0d4009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:34:28.470 INFO:teuthology.orchestra.run.smithi169.stderr:2023-12-16T21:34:28.466+0000 7fb0dbfff700 1 -- 172.21.15.169:0/2386149533 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fb0d40096b0 con 0x7fb0e40fc880 2023-12-16T21:34:28.470 INFO:teuthology.orchestra.run.smithi169.stderr:2023-12-16T21:34:28.466+0000 7fb0dbfff700 1 -- 172.21.15.169:0/2386149533 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fb0d4009810 con 0x7fb0e40fc880 2023-12-16T21:34:28.470 INFO:teuthology.orchestra.run.smithi169.stderr:2023-12-16T21:34:28.466+0000 7fb0e961d700 1 -- 172.21.15.169:0/2386149533 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb0e410d200 con 0x7fb0e40fc880 2023-12-16T21:34:28.470 INFO:teuthology.orchestra.run.smithi169.stderr:2023-12-16T21:34:28.466+0000 7fb0dbfff700 1 -- 172.21.15.169:0/2386149533 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fb0d401c5b0 con 0x7fb0e40fc880 2023-12-16T21:34:28.473 INFO:teuthology.orchestra.run.smithi169.stderr:2023-12-16T21:34:28.466+0000 7fb0e961d700 1 -- 172.21.15.169:0/2386149533 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb0e410dea0 con 0x7fb0e40fc880 2023-12-16T21:34:28.474 INFO:teuthology.orchestra.run.smithi169.stderr:2023-12-16T21:34:28.467+0000 7fb0dbfff700 1 -- 172.21.15.169:0/2386149533 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 14) v1 ==== 43351+0+0 (secure 0 0 0) 0x7fb0d4013420 con 0x7fb0e40fc880 2023-12-16T21:34:28.474 INFO:teuthology.orchestra.run.smithi169.stderr:2023-12-16T21:34:28.467+0000 7fb0e961d700 1 -- 172.21.15.169:0/2386149533 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb0e405f270 con 0x7fb0e40fc880 2023-12-16T21:34:28.474 INFO:teuthology.orchestra.run.smithi169.stderr:2023-12-16T21:34:28.467+0000 7fb0dbfff700 1 --2- 172.21.15.169:0/2386149533 >> [v2:172.21.15.45:6800/2215417738,v1:172.21.15.45:6801/2215417738] conn(0x7fb0d0036720 0x7fb0d0038bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:34:28.474 INFO:teuthology.orchestra.run.smithi169.stderr:2023-12-16T21:34:28.467+0000 7fb0dbfff700 1 -- 172.21.15.169:0/2386149533 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1151+0+0 (secure 0 0 0) 0x7fb0d4049aa0 con 0x7fb0e40fc880 2023-12-16T21:34:28.474 INFO:teuthology.orchestra.run.smithi169.stderr:2023-12-16T21:34:28.467+0000 7fb0da7fc700 1 -- 172.21.15.169:0/2386149533 >> [v2:172.21.15.45:6800/2215417738,v1:172.21.15.45:6801/2215417738] conn(0x7fb0d0036720 msgr2=0x7fb0d0038bd0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.45:6800/2215417738 2023-12-16T21:34:28.474 INFO:teuthology.orchestra.run.smithi169.stderr:2023-12-16T21:34:28.467+0000 7fb0da7fc700 1 --2- 172.21.15.169:0/2386149533 >> [v2:172.21.15.45:6800/2215417738,v1:172.21.15.45:6801/2215417738] conn(0x7fb0d0036720 0x7fb0d0038bd0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-12-16T21:34:28.474 INFO:teuthology.orchestra.run.smithi169.stderr:2023-12-16T21:34:28.472+0000 7fb0dbfff700 1 -- 172.21.15.169:0/2386149533 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+174570 (secure 0 0 0) 0x7fb0d4025400 con 0x7fb0e40fc880 2023-12-16T21:34:28.586 INFO:teuthology.orchestra.run.smithi169.stderr:2023-12-16T21:34:28.584+0000 7fb0dbfff700 1 -- 172.21.15.169:0/2386149533 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mgrmap(e 15) v1 ==== 43075+0+0 (secure 0 0 0) 0x7fb0d4025980 con 0x7fb0e40fc880 2023-12-16T21:34:28.586 INFO:teuthology.orchestra.run.smithi169.stderr:2023-12-16T21:34:28.584+0000 7fb0dbfff700 1 -- 172.21.15.169:0/2386149533 >> [v2:172.21.15.45:6800/2215417738,v1:172.21.15.45:6801/2215417738] conn(0x7fb0d0036720 msgr2=0x7fb0d0038bd0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-12-16T21:34:28.586 INFO:teuthology.orchestra.run.smithi169.stderr:2023-12-16T21:34:28.584+0000 7fb0dbfff700 1 --2- 172.21.15.169:0/2386149533 >> [v2:172.21.15.45:6800/2215417738,v1:172.21.15.45:6801/2215417738] conn(0x7fb0d0036720 0x7fb0d0038bd0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:34:28.740 INFO:teuthology.orchestra.run.smithi169.stderr:2023-12-16T21:34:28.737+0000 7fb0e961d700 1 -- 172.21.15.169:0/2386149533 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fb0e410e9f0 con 0x7fb0e40fc880 2023-12-16T21:34:28.740 INFO:teuthology.orchestra.run.smithi169.stderr:2023-12-16T21:34:28.738+0000 7fb0dbfff700 1 -- 172.21.15.169:0/2386149533 <== mon.0 v2:172.21.15.45:3300/0 8 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+739 (secure 0 0 0) 0x7fb0d4011db0 con 0x7fb0e40fc880 2023-12-16T21:34:28.740 INFO:teuthology.orchestra.run.smithi169.stdout: 2023-12-16T21:34:28.740 INFO:teuthology.orchestra.run.smithi169.stdout:{"epoch":1,"fsid":"88ff363c-9c5a-11ee-95a5-87774f69a715","modified":"2023-12-16T21:32:46.744221Z","created":"2023-12-16T21:32:46.744221Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"smithi045","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:3300","nonce":0},{"type":"v1","addr":"172.21.15.45:6789","nonce":0}]},"addr":"172.21.15.45:6789/0","public_addr":"172.21.15.45:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-12-16T21:34:28.742 INFO:teuthology.orchestra.run.smithi169.stderr:2023-12-16T21:34:28.740+0000 7fb0e961d700 1 -- 172.21.15.169:0/2386149533 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fb0e40fc880 msgr2=0x7fb0e410cac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:34:28.742 INFO:teuthology.orchestra.run.smithi169.stderr:2023-12-16T21:34:28.740+0000 7fb0e961d700 1 --2- 172.21.15.169:0/2386149533 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fb0e40fc880 0x7fb0e410cac0 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7fb0d4009330 tx=0x7fb0d4009e30 comp rx=0 tx=0).stop 2023-12-16T21:34:28.742 INFO:teuthology.orchestra.run.smithi169.stderr:2023-12-16T21:34:28.740+0000 7fb0e961d700 1 -- 172.21.15.169:0/2386149533 shutdown_connections 2023-12-16T21:34:28.742 INFO:teuthology.orchestra.run.smithi169.stderr:2023-12-16T21:34:28.740+0000 7fb0e961d700 1 --2- 172.21.15.169:0/2386149533 >> [v2:172.21.15.45:6800/2215417738,v1:172.21.15.45:6801/2215417738] conn(0x7fb0d0036720 0x7fb0d0038bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:34:28.742 INFO:teuthology.orchestra.run.smithi169.stderr:2023-12-16T21:34:28.740+0000 7fb0e961d700 1 --2- 172.21.15.169:0/2386149533 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fb0e40fc880 0x7fb0e410cac0 unknown :-1 s=CLOSED pgs=127 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:34:28.742 INFO:teuthology.orchestra.run.smithi169.stderr:2023-12-16T21:34:28.740+0000 7fb0e961d700 1 -- 172.21.15.169:0/2386149533 >> 172.21.15.169:0/2386149533 conn(0x7fb0e40f7f40 msgr2=0x7fb0e40f8d70 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-16T21:34:28.742 INFO:teuthology.orchestra.run.smithi169.stderr:2023-12-16T21:34:28.740+0000 7fb0e961d700 1 -- 172.21.15.169:0/2386149533 shutdown_connections 2023-12-16T21:34:28.743 INFO:teuthology.orchestra.run.smithi169.stderr:2023-12-16T21:34:28.740+0000 7fb0e961d700 1 -- 172.21.15.169:0/2386149533 wait complete. 2023-12-16T21:34:28.743 INFO:teuthology.orchestra.run.smithi169.stderr:dumped monmap epoch 1 2023-12-16T21:34:29.066 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Dec 16 21:34:28 smithi045 ceph-mon[80763]: Active manager daemon smithi045.nhtmwc restarted 2023-12-16T21:34:29.066 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Dec 16 21:34:28 smithi045 ceph-mon[80763]: Activating manager daemon smithi045.nhtmwc 2023-12-16T21:34:29.066 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Dec 16 21:34:28 smithi045 ceph-mon[80763]: osdmap e5: 0 total, 0 up, 0 in 2023-12-16T21:34:29.066 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Dec 16 21:34:28 smithi045 ceph-mon[80763]: mgrmap e15: smithi045.nhtmwc(active, starting, since 0.00294731s) 2023-12-16T21:34:29.066 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Dec 16 21:34:28 smithi045 ceph-mon[80763]: from='mgr.14214 172.21.15.45:0/1227909437' entity='mgr.smithi045.nhtmwc' cmd=[{"prefix": "mon metadata", "id": "smithi045"}]: dispatch 2023-12-16T21:34:29.066 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Dec 16 21:34:28 smithi045 ceph-mon[80763]: from='mgr.14214 172.21.15.45:0/1227909437' entity='mgr.smithi045.nhtmwc' cmd=[{"prefix": "mgr metadata", "who": "smithi045.nhtmwc", "id": "smithi045.nhtmwc"}]: dispatch 2023-12-16T21:34:29.066 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Dec 16 21:34:28 smithi045 ceph-mon[80763]: from='mgr.14214 172.21.15.45:0/1227909437' entity='mgr.smithi045.nhtmwc' cmd=[{"prefix": "mds metadata"}]: dispatch 2023-12-16T21:34:29.066 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Dec 16 21:34:28 smithi045 ceph-mon[80763]: from='mgr.14214 172.21.15.45:0/1227909437' entity='mgr.smithi045.nhtmwc' cmd=[{"prefix": "osd metadata"}]: dispatch 2023-12-16T21:34:29.066 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Dec 16 21:34:28 smithi045 ceph-mon[80763]: from='mgr.14214 172.21.15.45:0/1227909437' entity='mgr.smithi045.nhtmwc' cmd=[{"prefix": "mon metadata"}]: dispatch 2023-12-16T21:34:29.067 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Dec 16 21:34:28 smithi045 ceph-mon[80763]: Manager daemon smithi045.nhtmwc is now available 2023-12-16T21:34:29.067 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Dec 16 21:34:28 smithi045 ceph-mon[80763]: from='mgr.14214 172.21.15.45:0/1227909437' entity='mgr.smithi045.nhtmwc' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-12-16T21:34:29.067 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Dec 16 21:34:28 smithi045 ceph-mon[80763]: from='mgr.14214 172.21.15.45:0/1227909437' entity='mgr.smithi045.nhtmwc' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-12-16T21:34:29.816 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Dec 16 21:34:29 smithi045 ceph-mon[80763]: from='mgr.14214 172.21.15.45:0/1227909437' entity='mgr.smithi045.nhtmwc' 2023-12-16T21:34:29.816 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Dec 16 21:34:29 smithi045 ceph-mon[80763]: from='mgr.14214 172.21.15.45:0/1227909437' entity='mgr.smithi045.nhtmwc' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2023-12-16T21:34:29.816 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Dec 16 21:34:29 smithi045 ceph-mon[80763]: from='mgr.14214 172.21.15.45:0/1227909437' entity='mgr.smithi045.nhtmwc' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi045.nhtmwc/mirror_snapshot_schedule"}]: dispatch 2023-12-16T21:34:29.816 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Dec 16 21:34:29 smithi045 ceph-mon[80763]: from='mgr.14214 172.21.15.45:0/1227909437' entity='mgr.smithi045.nhtmwc' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi045.nhtmwc/trash_purge_schedule"}]: dispatch 2023-12-16T21:34:29.816 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Dec 16 21:34:29 smithi045 ceph-mon[80763]: from='client.? 172.21.15.169:0/2386149533' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-12-16T21:34:29.816 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Dec 16 21:34:29 smithi045 ceph-mon[80763]: from='mgr.14214 172.21.15.45:0/1227909437' entity='mgr.smithi045.nhtmwc' 2023-12-16T21:34:29.816 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Dec 16 21:34:29 smithi045 ceph-mon[80763]: from='mgr.14214 172.21.15.45:0/1227909437' entity='mgr.smithi045.nhtmwc' 2023-12-16T21:34:29.816 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Dec 16 21:34:29 smithi045 ceph-mon[80763]: mgrmap e16: smithi045.nhtmwc(active, since 1.00562s) 2023-12-16T21:34:30.002 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-12-16T21:34:30.003 DEBUG:teuthology.orchestra.run.smithi169:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 88ff363c-9c5a-11ee-95a5-87774f69a715 -- ceph mon dump -f json 2023-12-16T21:34:30.250 INFO:teuthology.orchestra.run.smithi169.stderr:Inferring config /etc/ceph/ceph.conf 2023-12-16T21:34:30.297 INFO:teuthology.orchestra.run.smithi169.stderr:Inferring config /etc/ceph/ceph.conf 2023-12-16T21:34:31.066 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Dec 16 21:34:30 smithi045 ceph-mon[80763]: [16/Dec/2023:21:34:29] ENGINE Bus STARTING 2023-12-16T21:34:31.066 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Dec 16 21:34:30 smithi045 ceph-mon[80763]: [16/Dec/2023:21:34:29] ENGINE Serving on https://172.21.15.45:7150 2023-12-16T21:34:31.066 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Dec 16 21:34:30 smithi045 ceph-mon[80763]: [16/Dec/2023:21:34:29] ENGINE Bus STARTED 2023-12-16T21:34:31.066 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Dec 16 21:34:30 smithi045 ceph-mon[80763]: from='mgr.14214 172.21.15.45:0/1227909437' entity='mgr.smithi045.nhtmwc' 2023-12-16T21:34:31.066 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Dec 16 21:34:30 smithi045 ceph-mon[80763]: from='mgr.14214 172.21.15.45:0/1227909437' entity='mgr.smithi045.nhtmwc' 2023-12-16T21:34:31.585 INFO:teuthology.orchestra.run.smithi169.stderr:2023-12-16T21:34:31.582+0000 7ff2daab4700 1 -- 172.21.15.169:0/1309718099 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff2d40fd3c0 msgr2=0x7ff2d40ff7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:34:31.585 INFO:teuthology.orchestra.run.smithi169.stderr:2023-12-16T21:34:31.582+0000 7ff2daab4700 1 --2- 172.21.15.169:0/1309718099 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff2d40fd3c0 0x7ff2d40ff7a0 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7ff2c80048c0 tx=0x7ff2c8009e30 comp rx=0 tx=0).stop 2023-12-16T21:34:31.585 INFO:teuthology.orchestra.run.smithi169.stderr:2023-12-16T21:34:31.582+0000 7ff2daab4700 1 -- 172.21.15.169:0/1309718099 shutdown_connections 2023-12-16T21:34:31.585 INFO:teuthology.orchestra.run.smithi169.stderr:2023-12-16T21:34:31.582+0000 7ff2daab4700 1 --2- 172.21.15.169:0/1309718099 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff2d40fd3c0 0x7ff2d40ff7a0 unknown :-1 s=CLOSED pgs=135 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:34:31.586 INFO:teuthology.orchestra.run.smithi169.stderr:2023-12-16T21:34:31.582+0000 7ff2daab4700 1 -- 172.21.15.169:0/1309718099 >> 172.21.15.169:0/1309718099 conn(0x7ff2d40f6e40 msgr2=0x7ff2d40f9270 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-16T21:34:31.586 INFO:teuthology.orchestra.run.smithi169.stderr:2023-12-16T21:34:31.582+0000 7ff2daab4700 1 -- 172.21.15.169:0/1309718099 shutdown_connections 2023-12-16T21:34:31.586 INFO:teuthology.orchestra.run.smithi169.stderr:2023-12-16T21:34:31.582+0000 7ff2daab4700 1 -- 172.21.15.169:0/1309718099 wait complete. 2023-12-16T21:34:31.586 INFO:teuthology.orchestra.run.smithi169.stderr:2023-12-16T21:34:31.582+0000 7ff2daab4700 1 Processor -- start 2023-12-16T21:34:31.586 INFO:teuthology.orchestra.run.smithi169.stderr:2023-12-16T21:34:31.582+0000 7ff2daab4700 1 -- start start 2023-12-16T21:34:31.586 INFO:teuthology.orchestra.run.smithi169.stderr:2023-12-16T21:34:31.583+0000 7ff2daab4700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff2d40fd3c0 0x7ff2d410ca60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:34:31.586 INFO:teuthology.orchestra.run.smithi169.stderr:2023-12-16T21:34:31.583+0000 7ff2daab4700 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff2d410cfa0 con 0x7ff2d40fd3c0 2023-12-16T21:34:31.586 INFO:teuthology.orchestra.run.smithi169.stderr:2023-12-16T21:34:31.583+0000 7ff2d8850700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff2d40fd3c0 0x7ff2d410ca60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:34:31.586 INFO:teuthology.orchestra.run.smithi169.stderr:2023-12-16T21:34:31.583+0000 7ff2d8850700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff2d40fd3c0 0x7ff2d410ca60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.169:43656/0 (socket says 172.21.15.169:43656) 2023-12-16T21:34:31.586 INFO:teuthology.orchestra.run.smithi169.stderr:2023-12-16T21:34:31.583+0000 7ff2d8850700 1 -- 172.21.15.169:0/4011290899 learned_addr learned my addr 172.21.15.169:0/4011290899 (peer_addr_for_me v2:172.21.15.169:0/0) 2023-12-16T21:34:31.586 INFO:teuthology.orchestra.run.smithi169.stderr:2023-12-16T21:34:31.583+0000 7ff2d8850700 1 -- 172.21.15.169:0/4011290899 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff2c8005040 con 0x7ff2d40fd3c0 2023-12-16T21:34:31.586 INFO:teuthology.orchestra.run.smithi169.stderr:2023-12-16T21:34:31.583+0000 7ff2d8850700 1 --2- 172.21.15.169:0/4011290899 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff2d40fd3c0 0x7ff2d410ca60 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7ff2c8009040 tx=0x7ff2c8009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:34:31.587 INFO:teuthology.orchestra.run.smithi169.stderr:2023-12-16T21:34:31.583+0000 7ff2d17fa700 1 -- 172.21.15.169:0/4011290899 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7ff2c8009620 con 0x7ff2d40fd3c0 2023-12-16T21:34:31.588 INFO:teuthology.orchestra.run.smithi169.stderr:2023-12-16T21:34:31.583+0000 7ff2daab4700 1 -- 172.21.15.169:0/4011290899 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff2d410d1a0 con 0x7ff2d40fd3c0 2023-12-16T21:34:31.588 INFO:teuthology.orchestra.run.smithi169.stderr:2023-12-16T21:34:31.583+0000 7ff2d17fa700 1 -- 172.21.15.169:0/4011290899 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7ff2c8009780 con 0x7ff2d40fd3c0 2023-12-16T21:34:31.588 INFO:teuthology.orchestra.run.smithi169.stderr:2023-12-16T21:34:31.583+0000 7ff2d17fa700 1 -- 172.21.15.169:0/4011290899 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7ff2c801c5b0 con 0x7ff2d40fd3c0 2023-12-16T21:34:31.588 INFO:teuthology.orchestra.run.smithi169.stderr:2023-12-16T21:34:31.584+0000 7ff2daab4700 1 -- 172.21.15.169:0/4011290899 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff2d410de40 con 0x7ff2d40fd3c0 2023-12-16T21:34:31.588 INFO:teuthology.orchestra.run.smithi169.stderr:2023-12-16T21:34:31.584+0000 7ff2daab4700 1 -- 172.21.15.169:0/4011290899 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff2d405f270 con 0x7ff2d40fd3c0 2023-12-16T21:34:31.588 INFO:teuthology.orchestra.run.smithi169.stderr:2023-12-16T21:34:31.584+0000 7ff2d17fa700 1 -- 172.21.15.169:0/4011290899 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 17) v1 ==== 43394+0+0 (secure 0 0 0) 0x7ff2c801c870 con 0x7ff2d40fd3c0 2023-12-16T21:34:31.588 INFO:teuthology.orchestra.run.smithi169.stderr:2023-12-16T21:34:31.585+0000 7ff2d17fa700 1 --2- 172.21.15.169:0/4011290899 >> [v2:172.21.15.45:6800/2012105536,v1:172.21.15.45:6801/2012105536] conn(0x7ff2bc0367c0 0x7ff2bc038c70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:34:31.588 INFO:teuthology.orchestra.run.smithi169.stderr:2023-12-16T21:34:31.585+0000 7ff2d17fa700 1 -- 172.21.15.169:0/4011290899 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1366+0+0 (secure 0 0 0) 0x7ff2c8049a70 con 0x7ff2d40fd3c0 2023-12-16T21:34:31.588 INFO:teuthology.orchestra.run.smithi169.stderr:2023-12-16T21:34:31.585+0000 7ff2d3fff700 1 --2- 172.21.15.169:0/4011290899 >> [v2:172.21.15.45:6800/2012105536,v1:172.21.15.45:6801/2012105536] conn(0x7ff2bc0367c0 0x7ff2bc038c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:34:31.588 INFO:teuthology.orchestra.run.smithi169.stderr:2023-12-16T21:34:31.585+0000 7ff2d3fff700 1 --2- 172.21.15.169:0/4011290899 >> [v2:172.21.15.45:6800/2012105536,v1:172.21.15.45:6801/2012105536] conn(0x7ff2bc0367c0 0x7ff2bc038c70 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7ff2c4009990 tx=0x7ff2c4006e40 comp rx=0 tx=0).ready entity=mgr.14214 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:34:31.592 INFO:teuthology.orchestra.run.smithi169.stderr:2023-12-16T21:34:31.590+0000 7ff2d17fa700 1 -- 172.21.15.169:0/4011290899 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+174570 (secure 0 0 0) 0x7ff2c8011550 con 0x7ff2d40fd3c0 2023-12-16T21:34:31.862 INFO:teuthology.orchestra.run.smithi169.stderr:2023-12-16T21:34:31.859+0000 7ff2daab4700 1 -- 172.21.15.169:0/4011290899 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7ff2d410e990 con 0x7ff2d40fd3c0 2023-12-16T21:34:31.862 INFO:teuthology.orchestra.run.smithi169.stderr:2023-12-16T21:34:31.859+0000 7ff2d17fa700 1 -- 172.21.15.169:0/4011290899 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+739 (secure 0 0 0) 0x7ff2c8011370 con 0x7ff2d40fd3c0 2023-12-16T21:34:31.862 INFO:teuthology.orchestra.run.smithi169.stdout: 2023-12-16T21:34:31.862 INFO:teuthology.orchestra.run.smithi169.stdout:{"epoch":1,"fsid":"88ff363c-9c5a-11ee-95a5-87774f69a715","modified":"2023-12-16T21:32:46.744221Z","created":"2023-12-16T21:32:46.744221Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"smithi045","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:3300","nonce":0},{"type":"v1","addr":"172.21.15.45:6789","nonce":0}]},"addr":"172.21.15.45:6789/0","public_addr":"172.21.15.45:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-12-16T21:34:31.864 INFO:teuthology.orchestra.run.smithi169.stderr:2023-12-16T21:34:31.862+0000 7ff2daab4700 1 -- 172.21.15.169:0/4011290899 >> [v2:172.21.15.45:6800/2012105536,v1:172.21.15.45:6801/2012105536] conn(0x7ff2bc0367c0 msgr2=0x7ff2bc038c70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:34:31.864 INFO:teuthology.orchestra.run.smithi169.stderr:2023-12-16T21:34:31.862+0000 7ff2daab4700 1 --2- 172.21.15.169:0/4011290899 >> [v2:172.21.15.45:6800/2012105536,v1:172.21.15.45:6801/2012105536] conn(0x7ff2bc0367c0 0x7ff2bc038c70 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7ff2c4009990 tx=0x7ff2c4006e40 comp rx=0 tx=0).stop 2023-12-16T21:34:31.864 INFO:teuthology.orchestra.run.smithi169.stderr:2023-12-16T21:34:31.862+0000 7ff2daab4700 1 -- 172.21.15.169:0/4011290899 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff2d40fd3c0 msgr2=0x7ff2d410ca60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:34:31.864 INFO:teuthology.orchestra.run.smithi169.stderr:2023-12-16T21:34:31.862+0000 7ff2daab4700 1 --2- 172.21.15.169:0/4011290899 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff2d40fd3c0 0x7ff2d410ca60 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7ff2c8009040 tx=0x7ff2c8009e30 comp rx=0 tx=0).stop 2023-12-16T21:34:31.864 INFO:teuthology.orchestra.run.smithi169.stderr:2023-12-16T21:34:31.862+0000 7ff2daab4700 1 -- 172.21.15.169:0/4011290899 shutdown_connections 2023-12-16T21:34:31.864 INFO:teuthology.orchestra.run.smithi169.stderr:2023-12-16T21:34:31.862+0000 7ff2daab4700 1 --2- 172.21.15.169:0/4011290899 >> [v2:172.21.15.45:6800/2012105536,v1:172.21.15.45:6801/2012105536] conn(0x7ff2bc0367c0 0x7ff2bc038c70 unknown :-1 s=CLOSED pgs=6 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:34:31.864 INFO:teuthology.orchestra.run.smithi169.stderr:2023-12-16T21:34:31.862+0000 7ff2daab4700 1 --2- 172.21.15.169:0/4011290899 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff2d40fd3c0 0x7ff2d410ca60 unknown :-1 s=CLOSED pgs=136 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:34:31.864 INFO:teuthology.orchestra.run.smithi169.stderr:2023-12-16T21:34:31.862+0000 7ff2daab4700 1 -- 172.21.15.169:0/4011290899 >> 172.21.15.169:0/4011290899 conn(0x7ff2d40f6e40 msgr2=0x7ff2d40f7b50 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-16T21:34:31.865 INFO:teuthology.orchestra.run.smithi169.stderr:2023-12-16T21:34:31.862+0000 7ff2daab4700 1 -- 172.21.15.169:0/4011290899 shutdown_connections 2023-12-16T21:34:31.865 INFO:teuthology.orchestra.run.smithi169.stderr:2023-12-16T21:34:31.862+0000 7ff2daab4700 1 -- 172.21.15.169:0/4011290899 wait complete. 2023-12-16T21:34:31.865 INFO:teuthology.orchestra.run.smithi169.stderr:dumped monmap epoch 1 2023-12-16T21:34:32.566 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Dec 16 21:34:32 smithi045 ceph-mon[80763]: mgrmap e17: smithi045.nhtmwc(active, since 2s) 2023-12-16T21:34:32.566 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Dec 16 21:34:32 smithi045 ceph-mon[80763]: from='mgr.14214 172.21.15.45:0/1227909437' entity='mgr.smithi045.nhtmwc' 2023-12-16T21:34:32.566 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Dec 16 21:34:32 smithi045 ceph-mon[80763]: from='mgr.14214 172.21.15.45:0/1227909437' entity='mgr.smithi045.nhtmwc' 2023-12-16T21:34:32.566 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Dec 16 21:34:32 smithi045 ceph-mon[80763]: from='client.? 172.21.15.169:0/4011290899' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-12-16T21:34:32.566 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Dec 16 21:34:32 smithi045 ceph-mon[80763]: from='mgr.14214 172.21.15.45:0/1227909437' entity='mgr.smithi045.nhtmwc' 2023-12-16T21:34:32.566 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Dec 16 21:34:32 smithi045 ceph-mon[80763]: from='mgr.14214 172.21.15.45:0/1227909437' entity='mgr.smithi045.nhtmwc' 2023-12-16T21:34:32.566 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Dec 16 21:34:32 smithi045 ceph-mon[80763]: from='mgr.14214 172.21.15.45:0/1227909437' entity='mgr.smithi045.nhtmwc' cmd=[{"prefix": "config rm", "who": "osd/host:smithi045", "name": "osd_memory_target"}]: dispatch 2023-12-16T21:34:33.138 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-12-16T21:34:33.139 DEBUG:teuthology.orchestra.run.smithi169:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 88ff363c-9c5a-11ee-95a5-87774f69a715 -- ceph mon dump -f json 2023-12-16T21:34:33.384 INFO:teuthology.orchestra.run.smithi169.stderr:Inferring config /var/lib/ceph/88ff363c-9c5a-11ee-95a5-87774f69a715/config/ceph.conf 2023-12-16T21:34:33.816 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Dec 16 21:34:33 smithi045 ceph-mon[80763]: from='mgr.14214 172.21.15.45:0/1227909437' entity='mgr.smithi045.nhtmwc' 2023-12-16T21:34:33.816 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Dec 16 21:34:33 smithi045 ceph-mon[80763]: from='mgr.14214 172.21.15.45:0/1227909437' entity='mgr.smithi045.nhtmwc' 2023-12-16T21:34:33.816 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Dec 16 21:34:33 smithi045 ceph-mon[80763]: from='mgr.14214 172.21.15.45:0/1227909437' entity='mgr.smithi045.nhtmwc' 2023-12-16T21:34:33.816 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Dec 16 21:34:33 smithi045 ceph-mon[80763]: from='mgr.14214 172.21.15.45:0/1227909437' entity='mgr.smithi045.nhtmwc' 2023-12-16T21:34:33.816 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Dec 16 21:34:33 smithi045 ceph-mon[80763]: from='mgr.14214 172.21.15.45:0/1227909437' entity='mgr.smithi045.nhtmwc' cmd=[{"prefix": "config rm", "who": "osd/host:smithi169", "name": "osd_memory_target"}]: dispatch 2023-12-16T21:34:33.817 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Dec 16 21:34:33 smithi045 ceph-mon[80763]: from='mgr.14214 172.21.15.45:0/1227909437' entity='mgr.smithi045.nhtmwc' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-12-16T21:34:33.817 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Dec 16 21:34:33 smithi045 ceph-mon[80763]: from='mgr.14214 172.21.15.45:0/1227909437' entity='mgr.smithi045.nhtmwc' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-12-16T21:34:33.817 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Dec 16 21:34:33 smithi045 ceph-mon[80763]: Updating smithi045:/etc/ceph/ceph.conf 2023-12-16T21:34:33.817 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Dec 16 21:34:33 smithi045 ceph-mon[80763]: Updating smithi169:/etc/ceph/ceph.conf 2023-12-16T21:34:33.817 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Dec 16 21:34:33 smithi045 ceph-mon[80763]: Updating smithi169:/var/lib/ceph/88ff363c-9c5a-11ee-95a5-87774f69a715/config/ceph.conf 2023-12-16T21:34:33.817 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Dec 16 21:34:33 smithi045 ceph-mon[80763]: Updating smithi045:/var/lib/ceph/88ff363c-9c5a-11ee-95a5-87774f69a715/config/ceph.conf 2023-12-16T21:34:33.817 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Dec 16 21:34:33 smithi045 ceph-mon[80763]: from='mgr.14214 172.21.15.45:0/1227909437' entity='mgr.smithi045.nhtmwc' 2023-12-16T21:34:33.817 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Dec 16 21:34:33 smithi045 ceph-mon[80763]: from='mgr.14214 172.21.15.45:0/1227909437' entity='mgr.smithi045.nhtmwc' 2023-12-16T21:34:33.817 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Dec 16 21:34:33 smithi045 ceph-mon[80763]: from='mgr.14214 172.21.15.45:0/1227909437' entity='mgr.smithi045.nhtmwc' 2023-12-16T21:34:33.817 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Dec 16 21:34:33 smithi045 ceph-mon[80763]: from='mgr.14214 172.21.15.45:0/1227909437' entity='mgr.smithi045.nhtmwc' 2023-12-16T21:34:33.817 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Dec 16 21:34:33 smithi045 ceph-mon[80763]: from='mgr.14214 172.21.15.45:0/1227909437' entity='mgr.smithi045.nhtmwc' 2023-12-16T21:34:33.817 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Dec 16 21:34:33 smithi045 ceph-mon[80763]: from='mgr.14214 172.21.15.45:0/1227909437' entity='mgr.smithi045.nhtmwc' cmd=[{"prefix": "auth get-or-create", "entity": "client.crash.smithi169", "caps": ["mon", "profile crash", "mgr", "profile crash"]}]: dispatch 2023-12-16T21:34:33.817 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Dec 16 21:34:33 smithi045 ceph-mon[80763]: from='mgr.14214 172.21.15.45:0/1227909437' entity='mgr.smithi045.nhtmwc' cmd='[{"prefix": "auth get-or-create", "entity": "client.crash.smithi169", "caps": ["mon", "profile crash", "mgr", "profile crash"]}]': finished 2023-12-16T21:34:33.817 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Dec 16 21:34:33 smithi045 ceph-mon[80763]: from='mgr.14214 172.21.15.45:0/1227909437' entity='mgr.smithi045.nhtmwc' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-12-16T21:34:34.727 INFO:teuthology.orchestra.run.smithi169.stderr:2023-12-16T21:34:34.724+0000 7f1411aca700 1 -- 172.21.15.169:0/65458120 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f140c1017d0 msgr2=0x7f140c101ba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:34:34.728 INFO:teuthology.orchestra.run.smithi169.stderr:2023-12-16T21:34:34.724+0000 7f1411aca700 1 --2- 172.21.15.169:0/65458120 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f140c1017d0 0x7f140c101ba0 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7f13fc0048c0 tx=0x7f13fc009e30 comp rx=0 tx=0).stop 2023-12-16T21:34:34.728 INFO:teuthology.orchestra.run.smithi169.stderr:2023-12-16T21:34:34.724+0000 7f1411aca700 1 -- 172.21.15.169:0/65458120 shutdown_connections 2023-12-16T21:34:34.728 INFO:teuthology.orchestra.run.smithi169.stderr:2023-12-16T21:34:34.724+0000 7f1411aca700 1 --2- 172.21.15.169:0/65458120 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f140c1017d0 0x7f140c101ba0 unknown :-1 s=CLOSED pgs=137 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:34:34.728 INFO:teuthology.orchestra.run.smithi169.stderr:2023-12-16T21:34:34.724+0000 7f1411aca700 1 -- 172.21.15.169:0/65458120 >> 172.21.15.169:0/65458120 conn(0x7f140c0f7310 msgr2=0x7f140c0f9720 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-16T21:34:34.728 INFO:teuthology.orchestra.run.smithi169.stderr:2023-12-16T21:34:34.724+0000 7f1411aca700 1 -- 172.21.15.169:0/65458120 shutdown_connections 2023-12-16T21:34:34.728 INFO:teuthology.orchestra.run.smithi169.stderr:2023-12-16T21:34:34.724+0000 7f1411aca700 1 -- 172.21.15.169:0/65458120 wait complete. 2023-12-16T21:34:34.728 INFO:teuthology.orchestra.run.smithi169.stderr:2023-12-16T21:34:34.725+0000 7f1411aca700 1 Processor -- start 2023-12-16T21:34:34.728 INFO:teuthology.orchestra.run.smithi169.stderr:2023-12-16T21:34:34.725+0000 7f1411aca700 1 -- start start 2023-12-16T21:34:34.728 INFO:teuthology.orchestra.run.smithi169.stderr:2023-12-16T21:34:34.725+0000 7f1411aca700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f140c1017d0 0x7f140c10dbb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:34:34.728 INFO:teuthology.orchestra.run.smithi169.stderr:2023-12-16T21:34:34.725+0000 7f1411aca700 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f140c10e0f0 con 0x7f140c1017d0 2023-12-16T21:34:34.728 INFO:teuthology.orchestra.run.smithi169.stderr:2023-12-16T21:34:34.725+0000 7f140b7fe700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f140c1017d0 0x7f140c10dbb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:34:34.728 INFO:teuthology.orchestra.run.smithi169.stderr:2023-12-16T21:34:34.725+0000 7f140b7fe700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f140c1017d0 0x7f140c10dbb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.169:43684/0 (socket says 172.21.15.169:43684) 2023-12-16T21:34:34.728 INFO:teuthology.orchestra.run.smithi169.stderr:2023-12-16T21:34:34.725+0000 7f140b7fe700 1 -- 172.21.15.169:0/64450196 learned_addr learned my addr 172.21.15.169:0/64450196 (peer_addr_for_me v2:172.21.15.169:0/0) 2023-12-16T21:34:34.729 INFO:teuthology.orchestra.run.smithi169.stderr:2023-12-16T21:34:34.726+0000 7f140b7fe700 1 -- 172.21.15.169:0/64450196 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f13fc005040 con 0x7f140c1017d0 2023-12-16T21:34:34.729 INFO:teuthology.orchestra.run.smithi169.stderr:2023-12-16T21:34:34.726+0000 7f140b7fe700 1 --2- 172.21.15.169:0/64450196 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f140c1017d0 0x7f140c10dbb0 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7f13fc009040 tx=0x7f13fc009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:34:34.730 INFO:teuthology.orchestra.run.smithi169.stderr:2023-12-16T21:34:34.726+0000 7f13f3fff700 1 -- 172.21.15.169:0/64450196 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f13fc004e60 con 0x7f140c1017d0 2023-12-16T21:34:34.730 INFO:teuthology.orchestra.run.smithi169.stderr:2023-12-16T21:34:34.726+0000 7f13f3fff700 1 -- 172.21.15.169:0/64450196 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f13fc009650 con 0x7f140c1017d0 2023-12-16T21:34:34.730 INFO:teuthology.orchestra.run.smithi169.stderr:2023-12-16T21:34:34.726+0000 7f13f3fff700 1 -- 172.21.15.169:0/64450196 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f13fc01c3f0 con 0x7f140c1017d0 2023-12-16T21:34:34.730 INFO:teuthology.orchestra.run.smithi169.stderr:2023-12-16T21:34:34.726+0000 7f1411aca700 1 -- 172.21.15.169:0/64450196 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f140c10e2f0 con 0x7f140c1017d0 2023-12-16T21:34:34.730 INFO:teuthology.orchestra.run.smithi169.stderr:2023-12-16T21:34:34.726+0000 7f1411aca700 1 -- 172.21.15.169:0/64450196 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f140c10e7e0 con 0x7f140c1017d0 2023-12-16T21:34:34.730 INFO:teuthology.orchestra.run.smithi169.stderr:2023-12-16T21:34:34.727+0000 7f1411aca700 1 -- 172.21.15.169:0/64450196 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f140c066680 con 0x7f140c1017d0 2023-12-16T21:34:34.730 INFO:teuthology.orchestra.run.smithi169.stderr:2023-12-16T21:34:34.728+0000 7f13f3fff700 1 -- 172.21.15.169:0/64450196 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 17) v1 ==== 43394+0+0 (secure 0 0 0) 0x7f13fc0097c0 con 0x7f140c1017d0 2023-12-16T21:34:34.731 INFO:teuthology.orchestra.run.smithi169.stderr:2023-12-16T21:34:34.728+0000 7f13f3fff700 1 --2- 172.21.15.169:0/64450196 >> [v2:172.21.15.45:6800/2012105536,v1:172.21.15.45:6801/2012105536] conn(0x7f13f4036810 0x7f13f4038cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:34:34.731 INFO:teuthology.orchestra.run.smithi169.stderr:2023-12-16T21:34:34.728+0000 7f13f3fff700 1 -- 172.21.15.169:0/64450196 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1366+0+0 (secure 0 0 0) 0x7f13fc04a3e0 con 0x7f140c1017d0 2023-12-16T21:34:34.731 INFO:teuthology.orchestra.run.smithi169.stderr:2023-12-16T21:34:34.729+0000 7f140affd700 1 --2- 172.21.15.169:0/64450196 >> [v2:172.21.15.45:6800/2012105536,v1:172.21.15.45:6801/2012105536] conn(0x7f13f4036810 0x7f13f4038cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:34:34.731 INFO:teuthology.orchestra.run.smithi169.stderr:2023-12-16T21:34:34.729+0000 7f140affd700 1 --2- 172.21.15.169:0/64450196 >> [v2:172.21.15.45:6800/2012105536,v1:172.21.15.45:6801/2012105536] conn(0x7f13f4036810 0x7f13f4038cc0 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7f1400009990 tx=0x7f1400006e40 comp rx=0 tx=0).ready entity=mgr.14214 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:34:34.735 INFO:teuthology.orchestra.run.smithi169.stderr:2023-12-16T21:34:34.733+0000 7f13f3fff700 1 -- 172.21.15.169:0/64450196 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+174570 (secure 0 0 0) 0x7f13fc01bba0 con 0x7f140c1017d0 2023-12-16T21:34:34.816 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Dec 16 21:34:34 smithi045 ceph-mon[80763]: Updating smithi169:/etc/ceph/ceph.client.admin.keyring 2023-12-16T21:34:34.816 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Dec 16 21:34:34 smithi045 ceph-mon[80763]: Updating smithi045:/etc/ceph/ceph.client.admin.keyring 2023-12-16T21:34:34.816 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Dec 16 21:34:34 smithi045 ceph-mon[80763]: Updating smithi169:/var/lib/ceph/88ff363c-9c5a-11ee-95a5-87774f69a715/config/ceph.client.admin.keyring 2023-12-16T21:34:34.816 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Dec 16 21:34:34 smithi045 ceph-mon[80763]: Updating smithi045:/var/lib/ceph/88ff363c-9c5a-11ee-95a5-87774f69a715/config/ceph.client.admin.keyring 2023-12-16T21:34:34.816 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Dec 16 21:34:34 smithi045 ceph-mon[80763]: Deploying daemon crash.smithi169 on smithi169 2023-12-16T21:34:35.027 INFO:teuthology.orchestra.run.smithi169.stderr:2023-12-16T21:34:35.024+0000 7f1411aca700 1 -- 172.21.15.169:0/64450196 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f140c100a40 con 0x7f140c1017d0 2023-12-16T21:34:35.027 INFO:teuthology.orchestra.run.smithi169.stderr:2023-12-16T21:34:35.025+0000 7f13f3fff700 1 -- 172.21.15.169:0/64450196 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+739 (secure 0 0 0) 0x7f13fc01b9a0 con 0x7f140c1017d0 2023-12-16T21:34:35.027 INFO:teuthology.orchestra.run.smithi169.stdout: 2023-12-16T21:34:35.027 INFO:teuthology.orchestra.run.smithi169.stdout:{"epoch":1,"fsid":"88ff363c-9c5a-11ee-95a5-87774f69a715","modified":"2023-12-16T21:32:46.744221Z","created":"2023-12-16T21:32:46.744221Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"smithi045","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:3300","nonce":0},{"type":"v1","addr":"172.21.15.45:6789","nonce":0}]},"addr":"172.21.15.45:6789/0","public_addr":"172.21.15.45:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-12-16T21:34:35.029 INFO:teuthology.orchestra.run.smithi169.stderr:2023-12-16T21:34:35.026+0000 7f1411aca700 1 -- 172.21.15.169:0/64450196 >> [v2:172.21.15.45:6800/2012105536,v1:172.21.15.45:6801/2012105536] conn(0x7f13f4036810 msgr2=0x7f13f4038cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:34:35.029 INFO:teuthology.orchestra.run.smithi169.stderr:2023-12-16T21:34:35.026+0000 7f1411aca700 1 --2- 172.21.15.169:0/64450196 >> [v2:172.21.15.45:6800/2012105536,v1:172.21.15.45:6801/2012105536] conn(0x7f13f4036810 0x7f13f4038cc0 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7f1400009990 tx=0x7f1400006e40 comp rx=0 tx=0).stop 2023-12-16T21:34:35.029 INFO:teuthology.orchestra.run.smithi169.stderr:2023-12-16T21:34:35.026+0000 7f1411aca700 1 -- 172.21.15.169:0/64450196 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f140c1017d0 msgr2=0x7f140c10dbb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:34:35.029 INFO:teuthology.orchestra.run.smithi169.stderr:2023-12-16T21:34:35.026+0000 7f1411aca700 1 --2- 172.21.15.169:0/64450196 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f140c1017d0 0x7f140c10dbb0 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7f13fc009040 tx=0x7f13fc009e30 comp rx=0 tx=0).stop 2023-12-16T21:34:35.029 INFO:teuthology.orchestra.run.smithi169.stderr:2023-12-16T21:34:35.027+0000 7f1411aca700 1 -- 172.21.15.169:0/64450196 shutdown_connections 2023-12-16T21:34:35.030 INFO:teuthology.orchestra.run.smithi169.stderr:2023-12-16T21:34:35.027+0000 7f1411aca700 1 --2- 172.21.15.169:0/64450196 >> [v2:172.21.15.45:6800/2012105536,v1:172.21.15.45:6801/2012105536] conn(0x7f13f4036810 0x7f13f4038cc0 unknown :-1 s=CLOSED pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:34:35.030 INFO:teuthology.orchestra.run.smithi169.stderr:2023-12-16T21:34:35.027+0000 7f1411aca700 1 --2- 172.21.15.169:0/64450196 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f140c1017d0 0x7f140c10dbb0 unknown :-1 s=CLOSED pgs=138 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:34:35.030 INFO:teuthology.orchestra.run.smithi169.stderr:2023-12-16T21:34:35.027+0000 7f1411aca700 1 -- 172.21.15.169:0/64450196 >> 172.21.15.169:0/64450196 conn(0x7f140c0f7310 msgr2=0x7f140c0f92a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-16T21:34:35.030 INFO:teuthology.orchestra.run.smithi169.stderr:2023-12-16T21:34:35.027+0000 7f1411aca700 1 -- 172.21.15.169:0/64450196 shutdown_connections 2023-12-16T21:34:35.030 INFO:teuthology.orchestra.run.smithi169.stderr:2023-12-16T21:34:35.027+0000 7f1411aca700 1 -- 172.21.15.169:0/64450196 wait complete. 2023-12-16T21:34:35.030 INFO:teuthology.orchestra.run.smithi169.stderr:dumped monmap epoch 1 2023-12-16T21:34:35.816 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Dec 16 21:34:35 smithi045 ceph-mon[80763]: from='client.? 172.21.15.169:0/64450196' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-12-16T21:34:36.276 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-12-16T21:34:36.277 DEBUG:teuthology.orchestra.run.smithi169:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 88ff363c-9c5a-11ee-95a5-87774f69a715 -- ceph mon dump -f json 2023-12-16T21:34:36.565 INFO:teuthology.orchestra.run.smithi169.stderr:Inferring config /var/lib/ceph/88ff363c-9c5a-11ee-95a5-87774f69a715/config/ceph.conf 2023-12-16T21:34:37.066 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Dec 16 21:34:36 smithi045 ceph-mon[80763]: from='mgr.14214 172.21.15.45:0/1227909437' entity='mgr.smithi045.nhtmwc' 2023-12-16T21:34:37.066 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Dec 16 21:34:36 smithi045 ceph-mon[80763]: from='mgr.14214 172.21.15.45:0/1227909437' entity='mgr.smithi045.nhtmwc' 2023-12-16T21:34:37.066 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Dec 16 21:34:36 smithi045 ceph-mon[80763]: from='mgr.14214 172.21.15.45:0/1227909437' entity='mgr.smithi045.nhtmwc' 2023-12-16T21:34:37.066 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Dec 16 21:34:36 smithi045 ceph-mon[80763]: from='mgr.14214 172.21.15.45:0/1227909437' entity='mgr.smithi045.nhtmwc' 2023-12-16T21:34:37.066 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Dec 16 21:34:36 smithi045 ceph-mon[80763]: Deploying daemon node-exporter.smithi169 on smithi169 2023-12-16T21:34:37.437 INFO:teuthology.orchestra.run.smithi169.stderr:2023-12-16T21:34:37.434+0000 7f06907ed700 1 -- 172.21.15.169:0/797978852 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f06881028f0 msgr2=0x7f0688102cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:34:37.438 INFO:teuthology.orchestra.run.smithi169.stderr:2023-12-16T21:34:37.434+0000 7f06907ed700 1 --2- 172.21.15.169:0/797978852 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f06881028f0 0x7f0688102cc0 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7f06840048c0 tx=0x7f0684009e30 comp rx=0 tx=0).stop 2023-12-16T21:34:37.438 INFO:teuthology.orchestra.run.smithi169.stderr:2023-12-16T21:34:37.434+0000 7f06907ed700 1 -- 172.21.15.169:0/797978852 shutdown_connections 2023-12-16T21:34:37.438 INFO:teuthology.orchestra.run.smithi169.stderr:2023-12-16T21:34:37.434+0000 7f06907ed700 1 --2- 172.21.15.169:0/797978852 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f06881028f0 0x7f0688102cc0 unknown :-1 s=CLOSED pgs=139 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:34:37.438 INFO:teuthology.orchestra.run.smithi169.stderr:2023-12-16T21:34:37.434+0000 7f06907ed700 1 -- 172.21.15.169:0/797978852 >> 172.21.15.169:0/797978852 conn(0x7f0688071d60 msgr2=0x7f0688072160 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-16T21:34:37.438 INFO:teuthology.orchestra.run.smithi169.stderr:2023-12-16T21:34:37.434+0000 7f06907ed700 1 -- 172.21.15.169:0/797978852 shutdown_connections 2023-12-16T21:34:37.438 INFO:teuthology.orchestra.run.smithi169.stderr:2023-12-16T21:34:37.434+0000 7f06907ed700 1 -- 172.21.15.169:0/797978852 wait complete. 2023-12-16T21:34:37.438 INFO:teuthology.orchestra.run.smithi169.stderr:2023-12-16T21:34:37.435+0000 7f06907ed700 1 Processor -- start 2023-12-16T21:34:37.438 INFO:teuthology.orchestra.run.smithi169.stderr:2023-12-16T21:34:37.435+0000 7f06907ed700 1 -- start start 2023-12-16T21:34:37.438 INFO:teuthology.orchestra.run.smithi169.stderr:2023-12-16T21:34:37.435+0000 7f06907ed700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f06881028f0 0x7f0688112be0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:34:37.438 INFO:teuthology.orchestra.run.smithi169.stderr:2023-12-16T21:34:37.435+0000 7f06907ed700 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0688113120 con 0x7f06881028f0 2023-12-16T21:34:37.439 INFO:teuthology.orchestra.run.smithi169.stderr:2023-12-16T21:34:37.435+0000 7f068e589700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f06881028f0 0x7f0688112be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:34:37.439 INFO:teuthology.orchestra.run.smithi169.stderr:2023-12-16T21:34:37.435+0000 7f068e589700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f06881028f0 0x7f0688112be0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.169:43718/0 (socket says 172.21.15.169:43718) 2023-12-16T21:34:37.439 INFO:teuthology.orchestra.run.smithi169.stderr:2023-12-16T21:34:37.435+0000 7f068e589700 1 -- 172.21.15.169:0/2062887851 learned_addr learned my addr 172.21.15.169:0/2062887851 (peer_addr_for_me v2:172.21.15.169:0/0) 2023-12-16T21:34:37.440 INFO:teuthology.orchestra.run.smithi169.stderr:2023-12-16T21:34:37.436+0000 7f068e589700 1 -- 172.21.15.169:0/2062887851 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0684005040 con 0x7f06881028f0 2023-12-16T21:34:37.440 INFO:teuthology.orchestra.run.smithi169.stderr:2023-12-16T21:34:37.436+0000 7f068e589700 1 --2- 172.21.15.169:0/2062887851 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f06881028f0 0x7f0688112be0 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7f0684009300 tx=0x7f0684009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:34:37.440 INFO:teuthology.orchestra.run.smithi169.stderr:2023-12-16T21:34:37.436+0000 7f067effd700 1 -- 172.21.15.169:0/2062887851 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f0684004e60 con 0x7f06881028f0 2023-12-16T21:34:37.441 INFO:teuthology.orchestra.run.smithi169.stderr:2023-12-16T21:34:37.436+0000 7f06907ed700 1 -- 172.21.15.169:0/2062887851 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0688113320 con 0x7f06881028f0 2023-12-16T21:34:37.441 INFO:teuthology.orchestra.run.smithi169.stderr:2023-12-16T21:34:37.436+0000 7f067effd700 1 -- 172.21.15.169:0/2062887851 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f0684013420 con 0x7f06881028f0 2023-12-16T21:34:37.441 INFO:teuthology.orchestra.run.smithi169.stderr:2023-12-16T21:34:37.436+0000 7f067effd700 1 -- 172.21.15.169:0/2062887851 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f068401c5b0 con 0x7f06881028f0 2023-12-16T21:34:37.441 INFO:teuthology.orchestra.run.smithi169.stderr:2023-12-16T21:34:37.436+0000 7f06907ed700 1 -- 172.21.15.169:0/2062887851 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0688113790 con 0x7f06881028f0 2023-12-16T21:34:37.441 INFO:teuthology.orchestra.run.smithi169.stderr:2023-12-16T21:34:37.437+0000 7f067effd700 1 -- 172.21.15.169:0/2062887851 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 17) v1 ==== 43394+0+0 (secure 0 0 0) 0x7f068401c7f0 con 0x7f06881028f0 2023-12-16T21:34:37.441 INFO:teuthology.orchestra.run.smithi169.stderr:2023-12-16T21:34:37.437+0000 7f067effd700 1 --2- 172.21.15.169:0/2062887851 >> [v2:172.21.15.45:6800/2012105536,v1:172.21.15.45:6801/2012105536] conn(0x7f06740367c0 0x7f0674038c70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:34:37.441 INFO:teuthology.orchestra.run.smithi169.stderr:2023-12-16T21:34:37.437+0000 7f067effd700 1 -- 172.21.15.169:0/2062887851 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1366+0+0 (secure 0 0 0) 0x7f0684049aa0 con 0x7f06881028f0 2023-12-16T21:34:37.441 INFO:teuthology.orchestra.run.smithi169.stderr:2023-12-16T21:34:37.437+0000 7f06907ed700 1 -- 172.21.15.169:0/2062887851 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f068805f270 con 0x7f06881028f0 2023-12-16T21:34:37.441 INFO:teuthology.orchestra.run.smithi169.stderr:2023-12-16T21:34:37.437+0000 7f068dd88700 1 --2- 172.21.15.169:0/2062887851 >> [v2:172.21.15.45:6800/2012105536,v1:172.21.15.45:6801/2012105536] conn(0x7f06740367c0 0x7f0674038c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:34:37.441 INFO:teuthology.orchestra.run.smithi169.stderr:2023-12-16T21:34:37.438+0000 7f068dd88700 1 --2- 172.21.15.169:0/2062887851 >> [v2:172.21.15.45:6800/2012105536,v1:172.21.15.45:6801/2012105536] conn(0x7f06740367c0 0x7f0674038c70 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f0678009990 tx=0x7f0678006e40 comp rx=0 tx=0).ready entity=mgr.14214 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:34:37.444 INFO:teuthology.orchestra.run.smithi169.stderr:2023-12-16T21:34:37.442+0000 7f067effd700 1 -- 172.21.15.169:0/2062887851 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+174570 (secure 0 0 0) 0x7f0684023300 con 0x7f06881028f0 2023-12-16T21:34:37.729 INFO:teuthology.orchestra.run.smithi169.stderr:2023-12-16T21:34:37.727+0000 7f06907ed700 1 -- 172.21.15.169:0/2062887851 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f068810e1c0 con 0x7f06881028f0 2023-12-16T21:34:37.729 INFO:teuthology.orchestra.run.smithi169.stderr:2023-12-16T21:34:37.727+0000 7f067effd700 1 -- 172.21.15.169:0/2062887851 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+739 (secure 0 0 0) 0x7f0684011b60 con 0x7f06881028f0 2023-12-16T21:34:37.730 INFO:teuthology.orchestra.run.smithi169.stdout: 2023-12-16T21:34:37.730 INFO:teuthology.orchestra.run.smithi169.stdout:{"epoch":1,"fsid":"88ff363c-9c5a-11ee-95a5-87774f69a715","modified":"2023-12-16T21:32:46.744221Z","created":"2023-12-16T21:32:46.744221Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"smithi045","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:3300","nonce":0},{"type":"v1","addr":"172.21.15.45:6789","nonce":0}]},"addr":"172.21.15.45:6789/0","public_addr":"172.21.15.45:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-12-16T21:34:37.732 INFO:teuthology.orchestra.run.smithi169.stderr:2023-12-16T21:34:37.730+0000 7f06907ed700 1 -- 172.21.15.169:0/2062887851 >> [v2:172.21.15.45:6800/2012105536,v1:172.21.15.45:6801/2012105536] conn(0x7f06740367c0 msgr2=0x7f0674038c70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:34:37.732 INFO:teuthology.orchestra.run.smithi169.stderr:2023-12-16T21:34:37.730+0000 7f06907ed700 1 --2- 172.21.15.169:0/2062887851 >> [v2:172.21.15.45:6800/2012105536,v1:172.21.15.45:6801/2012105536] conn(0x7f06740367c0 0x7f0674038c70 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f0678009990 tx=0x7f0678006e40 comp rx=0 tx=0).stop 2023-12-16T21:34:37.732 INFO:teuthology.orchestra.run.smithi169.stderr:2023-12-16T21:34:37.730+0000 7f06907ed700 1 -- 172.21.15.169:0/2062887851 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f06881028f0 msgr2=0x7f0688112be0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:34:37.732 INFO:teuthology.orchestra.run.smithi169.stderr:2023-12-16T21:34:37.730+0000 7f06907ed700 1 --2- 172.21.15.169:0/2062887851 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f06881028f0 0x7f0688112be0 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7f0684009300 tx=0x7f0684009e30 comp rx=0 tx=0).stop 2023-12-16T21:34:37.732 INFO:teuthology.orchestra.run.smithi169.stderr:2023-12-16T21:34:37.730+0000 7f06907ed700 1 -- 172.21.15.169:0/2062887851 shutdown_connections 2023-12-16T21:34:37.732 INFO:teuthology.orchestra.run.smithi169.stderr:2023-12-16T21:34:37.730+0000 7f06907ed700 1 --2- 172.21.15.169:0/2062887851 >> [v2:172.21.15.45:6800/2012105536,v1:172.21.15.45:6801/2012105536] conn(0x7f06740367c0 0x7f0674038c70 unknown :-1 s=CLOSED pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:34:37.733 INFO:teuthology.orchestra.run.smithi169.stderr:2023-12-16T21:34:37.730+0000 7f06907ed700 1 --2- 172.21.15.169:0/2062887851 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f06881028f0 0x7f0688112be0 unknown :-1 s=CLOSED pgs=140 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:34:37.733 INFO:teuthology.orchestra.run.smithi169.stderr:2023-12-16T21:34:37.730+0000 7f06907ed700 1 -- 172.21.15.169:0/2062887851 >> 172.21.15.169:0/2062887851 conn(0x7f0688071d60 msgr2=0x7f0688103eb0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-16T21:34:37.733 INFO:teuthology.orchestra.run.smithi169.stderr:2023-12-16T21:34:37.730+0000 7f06907ed700 1 -- 172.21.15.169:0/2062887851 shutdown_connections 2023-12-16T21:34:37.733 INFO:teuthology.orchestra.run.smithi169.stderr:2023-12-16T21:34:37.730+0000 7f06907ed700 1 -- 172.21.15.169:0/2062887851 wait complete. 2023-12-16T21:34:37.733 INFO:teuthology.orchestra.run.smithi169.stderr:dumped monmap epoch 1 2023-12-16T21:34:38.066 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Dec 16 21:34:37 smithi045 ceph-mon[80763]: from='client.? 172.21.15.169:0/2062887851' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-12-16T21:34:39.804 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-12-16T21:34:39.804 DEBUG:teuthology.orchestra.run.smithi169:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 88ff363c-9c5a-11ee-95a5-87774f69a715 -- ceph mon dump -f json 2023-12-16T21:34:40.066 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Dec 16 21:34:39 smithi045 ceph-mon[80763]: from='mgr.14214 172.21.15.45:0/1227909437' entity='mgr.smithi045.nhtmwc' 2023-12-16T21:34:40.067 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Dec 16 21:34:39 smithi045 ceph-mon[80763]: from='mgr.14214 172.21.15.45:0/1227909437' entity='mgr.smithi045.nhtmwc' 2023-12-16T21:34:40.067 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Dec 16 21:34:39 smithi045 ceph-mon[80763]: from='mgr.14214 172.21.15.45:0/1227909437' entity='mgr.smithi045.nhtmwc' 2023-12-16T21:34:40.067 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Dec 16 21:34:39 smithi045 ceph-mon[80763]: from='mgr.14214 172.21.15.45:0/1227909437' entity='mgr.smithi045.nhtmwc' 2023-12-16T21:34:40.067 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Dec 16 21:34:39 smithi045 ceph-mon[80763]: from='mgr.14214 172.21.15.45:0/1227909437' entity='mgr.smithi045.nhtmwc' 2023-12-16T21:34:40.067 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Dec 16 21:34:39 smithi045 ceph-mon[80763]: from='mgr.14214 172.21.15.45:0/1227909437' entity='mgr.smithi045.nhtmwc' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.smithi169.cncbgh", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2023-12-16T21:34:40.067 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Dec 16 21:34:39 smithi045 ceph-mon[80763]: from='mgr.14214 172.21.15.45:0/1227909437' entity='mgr.smithi045.nhtmwc' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.smithi169.cncbgh", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2023-12-16T21:34:40.067 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Dec 16 21:34:39 smithi045 ceph-mon[80763]: from='mgr.14214 172.21.15.45:0/1227909437' entity='mgr.smithi045.nhtmwc' cmd=[{"prefix": "mgr services"}]: dispatch 2023-12-16T21:34:40.067 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Dec 16 21:34:39 smithi045 ceph-mon[80763]: from='mgr.14214 172.21.15.45:0/1227909437' entity='mgr.smithi045.nhtmwc' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-12-16T21:34:40.220 INFO:teuthology.orchestra.run.smithi169.stderr:Inferring config /var/lib/ceph/88ff363c-9c5a-11ee-95a5-87774f69a715/config/ceph.conf 2023-12-16T21:34:41.019 INFO:teuthology.orchestra.run.smithi169.stderr:2023-12-16T21:34:41.013+0000 7f70cc951700 1 -- 172.21.15.169:0/3466950024 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f70c81028b0 msgr2=0x7f70c8102c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:34:41.019 INFO:teuthology.orchestra.run.smithi169.stderr:2023-12-16T21:34:41.013+0000 7f70cc951700 1 --2- 172.21.15.169:0/3466950024 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f70c81028b0 0x7f70c8102c80 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7f70b80048c0 tx=0x7f70b8009e30 comp rx=0 tx=0).stop 2023-12-16T21:34:41.019 INFO:teuthology.orchestra.run.smithi169.stderr:2023-12-16T21:34:41.013+0000 7f70cc951700 1 -- 172.21.15.169:0/3466950024 shutdown_connections 2023-12-16T21:34:41.019 INFO:teuthology.orchestra.run.smithi169.stderr:2023-12-16T21:34:41.013+0000 7f70cc951700 1 --2- 172.21.15.169:0/3466950024 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f70c81028b0 0x7f70c8102c80 unknown :-1 s=CLOSED pgs=141 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:34:41.019 INFO:teuthology.orchestra.run.smithi169.stderr:2023-12-16T21:34:41.013+0000 7f70cc951700 1 -- 172.21.15.169:0/3466950024 >> 172.21.15.169:0/3466950024 conn(0x7f70c80f8410 msgr2=0x7f70c80fa820 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-16T21:34:41.019 INFO:teuthology.orchestra.run.smithi169.stderr:2023-12-16T21:34:41.013+0000 7f70cc951700 1 -- 172.21.15.169:0/3466950024 shutdown_connections 2023-12-16T21:34:41.019 INFO:teuthology.orchestra.run.smithi169.stderr:2023-12-16T21:34:41.013+0000 7f70cc951700 1 -- 172.21.15.169:0/3466950024 wait complete. 2023-12-16T21:34:41.019 INFO:teuthology.orchestra.run.smithi169.stderr:2023-12-16T21:34:41.014+0000 7f70cc951700 1 Processor -- start 2023-12-16T21:34:41.019 INFO:teuthology.orchestra.run.smithi169.stderr:2023-12-16T21:34:41.014+0000 7f70cc951700 1 -- start start 2023-12-16T21:34:41.019 INFO:teuthology.orchestra.run.smithi169.stderr:2023-12-16T21:34:41.014+0000 7f70cc951700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f70c81028b0 0x7f70c8112b60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:34:41.020 INFO:teuthology.orchestra.run.smithi169.stderr:2023-12-16T21:34:41.014+0000 7f70cc951700 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f70c81130a0 con 0x7f70c81028b0 2023-12-16T21:34:41.020 INFO:teuthology.orchestra.run.smithi169.stderr:2023-12-16T21:34:41.014+0000 7f70c659c700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f70c81028b0 0x7f70c8112b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:34:41.020 INFO:teuthology.orchestra.run.smithi169.stderr:2023-12-16T21:34:41.015+0000 7f70c659c700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f70c81028b0 0x7f70c8112b60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.169:43750/0 (socket says 172.21.15.169:43750) 2023-12-16T21:34:41.020 INFO:teuthology.orchestra.run.smithi169.stderr:2023-12-16T21:34:41.015+0000 7f70c659c700 1 -- 172.21.15.169:0/1002873366 learned_addr learned my addr 172.21.15.169:0/1002873366 (peer_addr_for_me v2:172.21.15.169:0/0) 2023-12-16T21:34:41.020 INFO:teuthology.orchestra.run.smithi169.stderr:2023-12-16T21:34:41.015+0000 7f70c659c700 1 -- 172.21.15.169:0/1002873366 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f70b8005040 con 0x7f70c81028b0 2023-12-16T21:34:41.020 INFO:teuthology.orchestra.run.smithi169.stderr:2023-12-16T21:34:41.015+0000 7f70c659c700 1 --2- 172.21.15.169:0/1002873366 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f70c81028b0 0x7f70c8112b60 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7f70b8009040 tx=0x7f70b8009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:34:41.020 INFO:teuthology.orchestra.run.smithi169.stderr:2023-12-16T21:34:41.015+0000 7f70bf7fe700 1 -- 172.21.15.169:0/1002873366 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f70b8004e60 con 0x7f70c81028b0 2023-12-16T21:34:41.020 INFO:teuthology.orchestra.run.smithi169.stderr:2023-12-16T21:34:41.015+0000 7f70cc951700 1 -- 172.21.15.169:0/1002873366 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f70c81132a0 con 0x7f70c81028b0 2023-12-16T21:34:41.020 INFO:teuthology.orchestra.run.smithi169.stderr:2023-12-16T21:34:41.015+0000 7f70bf7fe700 1 -- 172.21.15.169:0/1002873366 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f70b8009650 con 0x7f70c81028b0 2023-12-16T21:34:41.020 INFO:teuthology.orchestra.run.smithi169.stderr:2023-12-16T21:34:41.015+0000 7f70bf7fe700 1 -- 172.21.15.169:0/1002873366 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f70b801c3f0 con 0x7f70c81028b0 2023-12-16T21:34:41.020 INFO:teuthology.orchestra.run.smithi169.stderr:2023-12-16T21:34:41.015+0000 7f70cc951700 1 -- 172.21.15.169:0/1002873366 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f70c810ca20 con 0x7f70c81028b0 2023-12-16T21:34:41.021 INFO:teuthology.orchestra.run.smithi169.stderr:2023-12-16T21:34:41.016+0000 7f70bf7fe700 1 -- 172.21.15.169:0/1002873366 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 17) v1 ==== 43394+0+0 (secure 0 0 0) 0x7f70b80097c0 con 0x7f70c81028b0 2023-12-16T21:34:41.021 INFO:teuthology.orchestra.run.smithi169.stderr:2023-12-16T21:34:41.016+0000 7f70cc951700 1 -- 172.21.15.169:0/1002873366 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f70c805f270 con 0x7f70c81028b0 2023-12-16T21:34:41.021 INFO:teuthology.orchestra.run.smithi169.stderr:2023-12-16T21:34:41.016+0000 7f70bf7fe700 1 --2- 172.21.15.169:0/1002873366 >> [v2:172.21.15.45:6800/2012105536,v1:172.21.15.45:6801/2012105536] conn(0x7f70b40367c0 0x7f70b4038c70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:34:41.021 INFO:teuthology.orchestra.run.smithi169.stderr:2023-12-16T21:34:41.016+0000 7f70bf7fe700 1 -- 172.21.15.169:0/1002873366 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1366+0+0 (secure 0 0 0) 0x7f70b8049a60 con 0x7f70c81028b0 2023-12-16T21:34:41.021 INFO:teuthology.orchestra.run.smithi169.stderr:2023-12-16T21:34:41.017+0000 7f70bdd9b700 1 --2- 172.21.15.169:0/1002873366 >> [v2:172.21.15.45:6800/2012105536,v1:172.21.15.45:6801/2012105536] conn(0x7f70b40367c0 0x7f70b4038c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:34:41.021 INFO:teuthology.orchestra.run.smithi169.stderr:2023-12-16T21:34:41.017+0000 7f70bdd9b700 1 --2- 172.21.15.169:0/1002873366 >> [v2:172.21.15.45:6800/2012105536,v1:172.21.15.45:6801/2012105536] conn(0x7f70b40367c0 0x7f70b4038c70 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7f70b0009990 tx=0x7f70b0006e40 comp rx=0 tx=0).ready entity=mgr.14214 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:34:41.024 INFO:teuthology.orchestra.run.smithi169.stderr:2023-12-16T21:34:41.021+0000 7f70bf7fe700 1 -- 172.21.15.169:0/1002873366 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+174570 (secure 0 0 0) 0x7f70b8011550 con 0x7f70c81028b0 2023-12-16T21:34:41.066 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Dec 16 21:34:40 smithi045 ceph-mon[80763]: Deploying daemon mgr.smithi169.cncbgh on smithi169 2023-12-16T21:34:41.320 INFO:teuthology.orchestra.run.smithi169.stderr:2023-12-16T21:34:41.315+0000 7f70cc951700 1 -- 172.21.15.169:0/1002873366 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f70c810e140 con 0x7f70c81028b0 2023-12-16T21:34:41.320 INFO:teuthology.orchestra.run.smithi169.stderr:2023-12-16T21:34:41.316+0000 7f70bf7fe700 1 -- 172.21.15.169:0/1002873366 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+739 (secure 0 0 0) 0x7f70b8011370 con 0x7f70c81028b0 2023-12-16T21:34:41.320 INFO:teuthology.orchestra.run.smithi169.stdout: 2023-12-16T21:34:41.320 INFO:teuthology.orchestra.run.smithi169.stdout:{"epoch":1,"fsid":"88ff363c-9c5a-11ee-95a5-87774f69a715","modified":"2023-12-16T21:32:46.744221Z","created":"2023-12-16T21:32:46.744221Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"smithi045","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:3300","nonce":0},{"type":"v1","addr":"172.21.15.45:6789","nonce":0}]},"addr":"172.21.15.45:6789/0","public_addr":"172.21.15.45:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-12-16T21:34:41.322 INFO:teuthology.orchestra.run.smithi169.stderr:2023-12-16T21:34:41.318+0000 7f70cc951700 1 -- 172.21.15.169:0/1002873366 >> [v2:172.21.15.45:6800/2012105536,v1:172.21.15.45:6801/2012105536] conn(0x7f70b40367c0 msgr2=0x7f70b4038c70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:34:41.322 INFO:teuthology.orchestra.run.smithi169.stderr:2023-12-16T21:34:41.318+0000 7f70cc951700 1 --2- 172.21.15.169:0/1002873366 >> [v2:172.21.15.45:6800/2012105536,v1:172.21.15.45:6801/2012105536] conn(0x7f70b40367c0 0x7f70b4038c70 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7f70b0009990 tx=0x7f70b0006e40 comp rx=0 tx=0).stop 2023-12-16T21:34:41.322 INFO:teuthology.orchestra.run.smithi169.stderr:2023-12-16T21:34:41.318+0000 7f70cc951700 1 -- 172.21.15.169:0/1002873366 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f70c81028b0 msgr2=0x7f70c8112b60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:34:41.322 INFO:teuthology.orchestra.run.smithi169.stderr:2023-12-16T21:34:41.318+0000 7f70cc951700 1 --2- 172.21.15.169:0/1002873366 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f70c81028b0 0x7f70c8112b60 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7f70b8009040 tx=0x7f70b8009e30 comp rx=0 tx=0).stop 2023-12-16T21:34:41.322 INFO:teuthology.orchestra.run.smithi169.stderr:2023-12-16T21:34:41.318+0000 7f70cc951700 1 -- 172.21.15.169:0/1002873366 shutdown_connections 2023-12-16T21:34:41.322 INFO:teuthology.orchestra.run.smithi169.stderr:2023-12-16T21:34:41.318+0000 7f70cc951700 1 --2- 172.21.15.169:0/1002873366 >> [v2:172.21.15.45:6800/2012105536,v1:172.21.15.45:6801/2012105536] conn(0x7f70b40367c0 0x7f70b4038c70 unknown :-1 s=CLOSED pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:34:41.322 INFO:teuthology.orchestra.run.smithi169.stderr:2023-12-16T21:34:41.318+0000 7f70cc951700 1 --2- 172.21.15.169:0/1002873366 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f70c81028b0 0x7f70c8112b60 unknown :-1 s=CLOSED pgs=142 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:34:41.322 INFO:teuthology.orchestra.run.smithi169.stderr:2023-12-16T21:34:41.318+0000 7f70cc951700 1 -- 172.21.15.169:0/1002873366 >> 172.21.15.169:0/1002873366 conn(0x7f70c80f8410 msgr2=0x7f70c80f9070 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-16T21:34:41.322 INFO:teuthology.orchestra.run.smithi169.stderr:2023-12-16T21:34:41.318+0000 7f70cc951700 1 -- 172.21.15.169:0/1002873366 shutdown_connections 2023-12-16T21:34:41.322 INFO:teuthology.orchestra.run.smithi169.stderr:2023-12-16T21:34:41.319+0000 7f70cc951700 1 -- 172.21.15.169:0/1002873366 wait complete. 2023-12-16T21:34:41.323 INFO:teuthology.orchestra.run.smithi169.stderr:dumped monmap epoch 1 2023-12-16T21:34:42.566 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Dec 16 21:34:42 smithi045 ceph-mon[80763]: from='mgr.14214 172.21.15.45:0/1227909437' entity='mgr.smithi045.nhtmwc' 2023-12-16T21:34:42.566 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Dec 16 21:34:42 smithi045 ceph-mon[80763]: from='mgr.14214 172.21.15.45:0/1227909437' entity='mgr.smithi045.nhtmwc' 2023-12-16T21:34:42.566 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Dec 16 21:34:42 smithi045 ceph-mon[80763]: from='mgr.14214 172.21.15.45:0/1227909437' entity='mgr.smithi045.nhtmwc' 2023-12-16T21:34:42.566 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Dec 16 21:34:42 smithi045 ceph-mon[80763]: from='mgr.14214 172.21.15.45:0/1227909437' entity='mgr.smithi045.nhtmwc' 2023-12-16T21:34:42.566 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Dec 16 21:34:42 smithi045 ceph-mon[80763]: from='mgr.14214 172.21.15.45:0/1227909437' entity='mgr.smithi045.nhtmwc' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2023-12-16T21:34:42.566 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Dec 16 21:34:42 smithi045 ceph-mon[80763]: from='mgr.14214 172.21.15.45:0/1227909437' entity='mgr.smithi045.nhtmwc' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-12-16T21:34:42.567 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Dec 16 21:34:42 smithi045 ceph-mon[80763]: Deploying daemon mon.smithi169 on smithi169 2023-12-16T21:34:42.567 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Dec 16 21:34:42 smithi045 ceph-mon[80763]: from='client.? 172.21.15.169:0/1002873366' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-12-16T21:34:42.634 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-12-16T21:34:42.634 DEBUG:teuthology.orchestra.run.smithi169:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 88ff363c-9c5a-11ee-95a5-87774f69a715 -- ceph mon dump -f json 2023-12-16T21:34:43.003 INFO:teuthology.orchestra.run.smithi169.stderr:Inferring config /var/lib/ceph/88ff363c-9c5a-11ee-95a5-87774f69a715/mon.smithi169/config 2023-12-16T21:34:43.083 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Dec 16 21:34:42 smithi169 systemd[1]: Starting Ceph mon.smithi169 for 88ff363c-9c5a-11ee-95a5-87774f69a715... 2023-12-16T21:34:43.633 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Dec 16 21:34:43 smithi169 podman[88275]: 2023-12-16 21:34:43.310660495 +0000 UTC m=+0.018752183 image pull 60cc83a9c8f10fdde41a08d59bce905c1a01fc7389b91726758c2505caa3f0c5 quay.ceph.io/ceph-ci/ceph@sha256:7a1644eb327b921af6e6cf30118d696e8368bd632346b48049fba15207e60112 2023-12-16T21:34:43.634 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Dec 16 21:34:43 smithi169 podman[88275]: 2023-12-16 21:34:43.539948835 +0000 UTC m=+0.248040508 container create f964ea9d81bd83e299c42fa160b98c92a27c2fe641b5a44a8e73f0df2942ce38 (image=quay.ceph.io/ceph-ci/ceph@sha256:7a1644eb327b921af6e6cf30118d696e8368bd632346b48049fba15207e60112, name=ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-mon-smithi169, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=0396eef90bef641b676c164ec7a3876f45010308, RELEASE=quincy-1f2f551, GIT_BRANCH=HEAD, io.buildah.version=1.31.3, org.label-schema.build-date=20231212, CEPH_POINT_RELEASE=, GIT_CLEAN=True, ceph=True, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image) 2023-12-16T21:34:43.834 INFO:teuthology.orchestra.run.smithi169.stderr:2023-12-16T21:34:43.830+0000 7f6a6e00c700 1 -- 172.21.15.169:0/1673122442 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f6a681029e0 msgr2=0x7f6a68102db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:34:43.834 INFO:teuthology.orchestra.run.smithi169.stderr:2023-12-16T21:34:43.830+0000 7f6a6e00c700 1 --2- 172.21.15.169:0/1673122442 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f6a681029e0 0x7f6a68102db0 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7f6a580048c0 tx=0x7f6a58009e30 comp rx=0 tx=0).stop 2023-12-16T21:34:43.835 INFO:teuthology.orchestra.run.smithi169.stderr:2023-12-16T21:34:43.831+0000 7f6a6e00c700 1 -- 172.21.15.169:0/1673122442 shutdown_connections 2023-12-16T21:34:43.835 INFO:teuthology.orchestra.run.smithi169.stderr:2023-12-16T21:34:43.831+0000 7f6a6e00c700 1 --2- 172.21.15.169:0/1673122442 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f6a681029e0 0x7f6a68102db0 unknown :-1 s=CLOSED pgs=145 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:34:43.835 INFO:teuthology.orchestra.run.smithi169.stderr:2023-12-16T21:34:43.831+0000 7f6a6e00c700 1 -- 172.21.15.169:0/1673122442 >> 172.21.15.169:0/1673122442 conn(0x7f6a68071760 msgr2=0x7f6a68071b60 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-16T21:34:43.835 INFO:teuthology.orchestra.run.smithi169.stderr:2023-12-16T21:34:43.831+0000 7f6a6e00c700 1 -- 172.21.15.169:0/1673122442 shutdown_connections 2023-12-16T21:34:43.835 INFO:teuthology.orchestra.run.smithi169.stderr:2023-12-16T21:34:43.831+0000 7f6a6e00c700 1 -- 172.21.15.169:0/1673122442 wait complete. 2023-12-16T21:34:43.836 INFO:teuthology.orchestra.run.smithi169.stderr:2023-12-16T21:34:43.832+0000 7f6a6e00c700 1 Processor -- start 2023-12-16T21:34:43.836 INFO:teuthology.orchestra.run.smithi169.stderr:2023-12-16T21:34:43.832+0000 7f6a6e00c700 1 -- start start 2023-12-16T21:34:43.837 INFO:teuthology.orchestra.run.smithi169.stderr:2023-12-16T21:34:43.832+0000 7f6a6e00c700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f6a681029e0 0x7f6a68112ba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:34:43.837 INFO:teuthology.orchestra.run.smithi169.stderr:2023-12-16T21:34:43.832+0000 7f6a6e00c700 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6a681130e0 con 0x7f6a681029e0 2023-12-16T21:34:43.837 INFO:teuthology.orchestra.run.smithi169.stderr:2023-12-16T21:34:43.832+0000 7f6a677fe700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f6a681029e0 0x7f6a68112ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:34:43.837 INFO:teuthology.orchestra.run.smithi169.stderr:2023-12-16T21:34:43.832+0000 7f6a677fe700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f6a681029e0 0x7f6a68112ba0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.169:48894/0 (socket says 172.21.15.169:48894) 2023-12-16T21:34:43.837 INFO:teuthology.orchestra.run.smithi169.stderr:2023-12-16T21:34:43.832+0000 7f6a677fe700 1 -- 172.21.15.169:0/1223880013 learned_addr learned my addr 172.21.15.169:0/1223880013 (peer_addr_for_me v2:172.21.15.169:0/0) 2023-12-16T21:34:43.837 INFO:teuthology.orchestra.run.smithi169.stderr:2023-12-16T21:34:43.833+0000 7f6a677fe700 1 -- 172.21.15.169:0/1223880013 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6a58005040 con 0x7f6a681029e0 2023-12-16T21:34:43.837 INFO:teuthology.orchestra.run.smithi169.stderr:2023-12-16T21:34:43.833+0000 7f6a677fe700 1 --2- 172.21.15.169:0/1223880013 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f6a681029e0 0x7f6a68112ba0 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7f6a58009300 tx=0x7f6a58009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:34:43.837 INFO:teuthology.orchestra.run.smithi169.stderr:2023-12-16T21:34:43.833+0000 7f6a4ffff700 1 -- 172.21.15.169:0/1223880013 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f6a58004e60 con 0x7f6a681029e0 2023-12-16T21:34:43.837 INFO:teuthology.orchestra.run.smithi169.stderr:2023-12-16T21:34:43.833+0000 7f6a4ffff700 1 -- 172.21.15.169:0/1223880013 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f6a58009680 con 0x7f6a681029e0 2023-12-16T21:34:43.837 INFO:teuthology.orchestra.run.smithi169.stderr:2023-12-16T21:34:43.833+0000 7f6a4ffff700 1 -- 172.21.15.169:0/1223880013 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f6a5801c3f0 con 0x7f6a681029e0 2023-12-16T21:34:43.837 INFO:teuthology.orchestra.run.smithi169.stderr:2023-12-16T21:34:43.833+0000 7f6a6e00c700 1 -- 172.21.15.169:0/1223880013 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6a68113340 con 0x7f6a681029e0 2023-12-16T21:34:43.837 INFO:teuthology.orchestra.run.smithi169.stderr:2023-12-16T21:34:43.833+0000 7f6a6e00c700 1 -- 172.21.15.169:0/1223880013 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6a6810ca60 con 0x7f6a681029e0 2023-12-16T21:34:43.838 INFO:teuthology.orchestra.run.smithi169.stderr:2023-12-16T21:34:43.834+0000 7f6a4ffff700 1 -- 172.21.15.169:0/1223880013 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 17) v1 ==== 43394+0+0 (secure 0 0 0) 0x7f6a5801c670 con 0x7f6a681029e0 2023-12-16T21:34:43.838 INFO:teuthology.orchestra.run.smithi169.stderr:2023-12-16T21:34:43.834+0000 7f6a4ffff700 1 --2- 172.21.15.169:0/1223880013 >> [v2:172.21.15.45:6800/2012105536,v1:172.21.15.45:6801/2012105536] conn(0x7f6a500367d0 0x7f6a50038c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:34:43.839 INFO:teuthology.orchestra.run.smithi169.stderr:2023-12-16T21:34:43.834+0000 7f6a4ffff700 1 -- 172.21.15.169:0/1223880013 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1366+0+0 (secure 0 0 0) 0x7f6a58049b70 con 0x7f6a681029e0 2023-12-16T21:34:43.839 INFO:teuthology.orchestra.run.smithi169.stderr:2023-12-16T21:34:43.834+0000 7f6a6e00c700 1 -- 172.21.15.169:0/1223880013 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6a6805f270 con 0x7f6a681029e0 2023-12-16T21:34:43.839 INFO:teuthology.orchestra.run.smithi169.stderr:2023-12-16T21:34:43.835+0000 7f6a66ffd700 1 --2- 172.21.15.169:0/1223880013 >> [v2:172.21.15.45:6800/2012105536,v1:172.21.15.45:6801/2012105536] conn(0x7f6a500367d0 0x7f6a50038c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:34:43.839 INFO:teuthology.orchestra.run.smithi169.stderr:2023-12-16T21:34:43.835+0000 7f6a66ffd700 1 --2- 172.21.15.169:0/1223880013 >> [v2:172.21.15.45:6800/2012105536,v1:172.21.15.45:6801/2012105536] conn(0x7f6a500367d0 0x7f6a50038c80 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7f6a5c009990 tx=0x7f6a5c006e40 comp rx=0 tx=0).ready entity=mgr.14214 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:34:43.843 INFO:teuthology.orchestra.run.smithi169.stderr:2023-12-16T21:34:43.840+0000 7f6a4ffff700 1 -- 172.21.15.169:0/1223880013 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+174570 (secure 0 0 0) 0x7f6a58025400 con 0x7f6a681029e0 2023-12-16T21:34:43.891 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Dec 16 21:34:43 smithi169 podman[88275]: 2023-12-16 21:34:43.832443811 +0000 UTC m=+0.540535484 container init f964ea9d81bd83e299c42fa160b98c92a27c2fe641b5a44a8e73f0df2942ce38 (image=quay.ceph.io/ceph-ci/ceph@sha256:7a1644eb327b921af6e6cf30118d696e8368bd632346b48049fba15207e60112, name=ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-mon-smithi169, org.label-schema.license=GPLv2, org.label-schema.build-date=20231212, CEPH_POINT_RELEASE=, ceph=True, GIT_BRANCH=HEAD, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=0396eef90bef641b676c164ec7a3876f45010308, io.buildah.version=1.31.3, org.label-schema.vendor=CentOS, RELEASE=quincy-1f2f551) 2023-12-16T21:34:43.891 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Dec 16 21:34:43 smithi169 podman[88275]: 2023-12-16 21:34:43.837339535 +0000 UTC m=+0.545431200 container start f964ea9d81bd83e299c42fa160b98c92a27c2fe641b5a44a8e73f0df2942ce38 (image=quay.ceph.io/ceph-ci/ceph@sha256:7a1644eb327b921af6e6cf30118d696e8368bd632346b48049fba15207e60112, name=ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-mon-smithi169, GIT_COMMIT=0396eef90bef641b676c164ec7a3876f45010308, RELEASE=quincy-1f2f551, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, io.buildah.version=1.31.3, org.label-schema.license=GPLv2, org.label-schema.build-date=20231212, org.label-schema.schema-version=1.0, GIT_CLEAN=True, ceph=True, org.label-schema.vendor=CentOS) 2023-12-16T21:34:43.891 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Dec 16 21:34:43 smithi169 ceph-mon[88350]: set uid:gid to 167:167 (ceph:ceph) 2023-12-16T21:34:43.891 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Dec 16 21:34:43 smithi169 ceph-mon[88350]: ceph version 17.2.7-125-g1f2f5510 (1f2f55109dc1ed2dc2dda9de1db3c70923ba0dfd) quincy (stable), process ceph-mon, pid 2 2023-12-16T21:34:43.891 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Dec 16 21:34:43 smithi169 ceph-mon[88350]: pidfile_write: ignore empty --pid-file 2023-12-16T21:34:43.891 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Dec 16 21:34:43 smithi169 ceph-mon[88350]: load: jerasure load: lrc 2023-12-16T21:34:43.891 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Dec 16 21:34:43 smithi169 ceph-mon[88350]: rocksdb: RocksDB version: 6.15.5 2023-12-16T21:34:43.891 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Dec 16 21:34:43 smithi169 ceph-mon[88350]: rocksdb: Git sha rocksdb_build_git_sha:@0@ 2023-12-16T21:34:43.891 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Dec 16 21:34:43 smithi169 ceph-mon[88350]: rocksdb: Compile date Dec 13 2023 2023-12-16T21:34:43.891 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Dec 16 21:34:43 smithi169 ceph-mon[88350]: rocksdb: DB SUMMARY 2023-12-16T21:34:43.891 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Dec 16 21:34:43 smithi169 ceph-mon[88350]: rocksdb: DB Session ID: S3I0I5CVFTX9XKWB5P69 2023-12-16T21:34:43.892 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Dec 16 21:34:43 smithi169 ceph-mon[88350]: rocksdb: CURRENT file: CURRENT 2023-12-16T21:34:43.892 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Dec 16 21:34:43 smithi169 ceph-mon[88350]: rocksdb: IDENTITY file: IDENTITY 2023-12-16T21:34:43.892 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Dec 16 21:34:43 smithi169 ceph-mon[88350]: rocksdb: MANIFEST file: MANIFEST-000003 size: 57 Bytes 2023-12-16T21:34:43.892 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Dec 16 21:34:43 smithi169 ceph-mon[88350]: rocksdb: SST files in /var/lib/ceph/mon/ceph-smithi169/store.db dir, Total Num: 0, files: 2023-12-16T21:34:43.892 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Dec 16 21:34:43 smithi169 ceph-mon[88350]: rocksdb: Write Ahead Log file in /var/lib/ceph/mon/ceph-smithi169/store.db: 000004.log size: 511 ; 2023-12-16T21:34:43.892 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Dec 16 21:34:43 smithi169 ceph-mon[88350]: rocksdb: Options.error_if_exists: 0 2023-12-16T21:34:43.892 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Dec 16 21:34:43 smithi169 ceph-mon[88350]: rocksdb: Options.create_if_missing: 0 2023-12-16T21:34:43.892 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Dec 16 21:34:43 smithi169 ceph-mon[88350]: rocksdb: Options.paranoid_checks: 1 2023-12-16T21:34:43.892 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Dec 16 21:34:43 smithi169 ceph-mon[88350]: rocksdb: Options.track_and_verify_wals_in_manifest: 0 2023-12-16T21:34:43.892 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Dec 16 21:34:43 smithi169 ceph-mon[88350]: rocksdb: Options.env: 0x55fd2f04aa80 2023-12-16T21:34:43.892 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Dec 16 21:34:43 smithi169 ceph-mon[88350]: rocksdb: Options.fs: Posix File System 2023-12-16T21:34:43.892 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Dec 16 21:34:43 smithi169 ceph-mon[88350]: rocksdb: Options.info_log: 0x55fd317ef000 2023-12-16T21:34:43.892 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Dec 16 21:34:43 smithi169 ceph-mon[88350]: rocksdb: Options.max_file_opening_threads: 16 2023-12-16T21:34:43.892 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Dec 16 21:34:43 smithi169 ceph-mon[88350]: rocksdb: Options.statistics: (nil) 2023-12-16T21:34:43.892 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Dec 16 21:34:43 smithi169 ceph-mon[88350]: rocksdb: Options.use_fsync: 0 2023-12-16T21:34:43.894 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Dec 16 21:34:43 smithi169 ceph-mon[88350]: rocksdb: Options.max_log_file_size: 0 2023-12-16T21:34:43.894 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Dec 16 21:34:43 smithi169 ceph-mon[88350]: rocksdb: Options.max_manifest_file_size: 1073741824 2023-12-16T21:34:43.894 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Dec 16 21:34:43 smithi169 ceph-mon[88350]: rocksdb: Options.log_file_time_to_roll: 0 2023-12-16T21:34:43.894 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Dec 16 21:34:43 smithi169 ceph-mon[88350]: rocksdb: Options.keep_log_file_num: 1000 2023-12-16T21:34:43.894 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Dec 16 21:34:43 smithi169 ceph-mon[88350]: rocksdb: Options.recycle_log_file_num: 0 2023-12-16T21:34:43.894 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Dec 16 21:34:43 smithi169 ceph-mon[88350]: rocksdb: Options.allow_fallocate: 1 2023-12-16T21:34:43.894 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Dec 16 21:34:43 smithi169 ceph-mon[88350]: rocksdb: Options.allow_mmap_reads: 0 2023-12-16T21:34:43.894 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Dec 16 21:34:43 smithi169 ceph-mon[88350]: rocksdb: Options.allow_mmap_writes: 0 2023-12-16T21:34:43.894 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Dec 16 21:34:43 smithi169 ceph-mon[88350]: rocksdb: Options.use_direct_reads: 0 2023-12-16T21:34:43.895 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Dec 16 21:34:43 smithi169 ceph-mon[88350]: rocksdb: Options.use_direct_io_for_flush_and_compaction: 0 2023-12-16T21:34:43.895 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Dec 16 21:34:43 smithi169 ceph-mon[88350]: rocksdb: Options.create_missing_column_families: 0 2023-12-16T21:34:43.895 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Dec 16 21:34:43 smithi169 ceph-mon[88350]: rocksdb: Options.db_log_dir: 2023-12-16T21:34:43.895 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Dec 16 21:34:43 smithi169 ceph-mon[88350]: rocksdb: Options.wal_dir: /var/lib/ceph/mon/ceph-smithi169/store.db 2023-12-16T21:34:43.895 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Dec 16 21:34:43 smithi169 ceph-mon[88350]: rocksdb: Options.table_cache_numshardbits: 6 2023-12-16T21:34:43.895 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Dec 16 21:34:43 smithi169 ceph-mon[88350]: rocksdb: Options.WAL_ttl_seconds: 0 2023-12-16T21:34:43.895 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Dec 16 21:34:43 smithi169 ceph-mon[88350]: rocksdb: Options.WAL_size_limit_MB: 0 2023-12-16T21:34:43.895 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Dec 16 21:34:43 smithi169 ceph-mon[88350]: rocksdb: Options.max_write_batch_group_size_bytes: 1048576 2023-12-16T21:34:43.895 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Dec 16 21:34:43 smithi169 ceph-mon[88350]: rocksdb: Options.manifest_preallocation_size: 4194304 2023-12-16T21:34:43.895 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Dec 16 21:34:43 smithi169 ceph-mon[88350]: rocksdb: Options.is_fd_close_on_exec: 1 2023-12-16T21:34:43.895 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Dec 16 21:34:43 smithi169 ceph-mon[88350]: rocksdb: Options.advise_random_on_open: 1 2023-12-16T21:34:43.895 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Dec 16 21:34:43 smithi169 ceph-mon[88350]: rocksdb: Options.db_write_buffer_size: 0 2023-12-16T21:34:43.895 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Dec 16 21:34:43 smithi169 ceph-mon[88350]: rocksdb: Options.write_buffer_manager: 0x55fd325649f0 2023-12-16T21:34:43.895 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Dec 16 21:34:43 smithi169 ceph-mon[88350]: rocksdb: Options.access_hint_on_compaction_start: 1 2023-12-16T21:34:43.895 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Dec 16 21:34:43 smithi169 ceph-mon[88350]: rocksdb: Options.new_table_reader_for_compaction_inputs: 0 2023-12-16T21:34:43.895 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Dec 16 21:34:43 smithi169 ceph-mon[88350]: rocksdb: Options.random_access_max_buffer_size: 1048576 2023-12-16T21:34:43.896 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Dec 16 21:34:43 smithi169 ceph-mon[88350]: rocksdb: Options.use_adaptive_mutex: 0 2023-12-16T21:34:43.896 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Dec 16 21:34:43 smithi169 ceph-mon[88350]: rocksdb: Options.rate_limiter: (nil) 2023-12-16T21:34:43.896 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Dec 16 21:34:43 smithi169 ceph-mon[88350]: rocksdb: Options.sst_file_manager.rate_bytes_per_sec: 0 2023-12-16T21:34:43.896 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Dec 16 21:34:43 smithi169 ceph-mon[88350]: rocksdb: Options.wal_recovery_mode: 2 2023-12-16T21:34:43.896 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Dec 16 21:34:43 smithi169 ceph-mon[88350]: rocksdb: Options.enable_thread_tracking: 0 2023-12-16T21:34:43.896 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Dec 16 21:34:43 smithi169 ceph-mon[88350]: rocksdb: Options.enable_pipelined_write: 0 2023-12-16T21:34:43.896 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Dec 16 21:34:43 smithi169 ceph-mon[88350]: rocksdb: Options.unordered_write: 0 2023-12-16T21:34:43.896 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Dec 16 21:34:43 smithi169 ceph-mon[88350]: rocksdb: Options.allow_concurrent_memtable_write: 1 2023-12-16T21:34:43.896 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Dec 16 21:34:43 smithi169 ceph-mon[88350]: rocksdb: Options.enable_write_thread_adaptive_yield: 1 2023-12-16T21:34:43.896 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Dec 16 21:34:43 smithi169 ceph-mon[88350]: rocksdb: Options.write_thread_max_yield_usec: 100 2023-12-16T21:34:43.896 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Dec 16 21:34:43 smithi169 ceph-mon[88350]: rocksdb: Options.write_thread_slow_yield_usec: 3 2023-12-16T21:34:43.896 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Dec 16 21:34:43 smithi169 ceph-mon[88350]: rocksdb: Options.row_cache: None 2023-12-16T21:34:43.896 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Dec 16 21:34:43 smithi169 ceph-mon[88350]: rocksdb: Options.wal_filter: None 2023-12-16T21:34:43.897 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Dec 16 21:34:43 smithi169 ceph-mon[88350]: rocksdb: Options.avoid_flush_during_recovery: 0 2023-12-16T21:34:43.897 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Dec 16 21:34:43 smithi169 ceph-mon[88350]: rocksdb: Options.allow_ingest_behind: 0 2023-12-16T21:34:43.897 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Dec 16 21:34:43 smithi169 ceph-mon[88350]: rocksdb: Options.preserve_deletes: 0 2023-12-16T21:34:43.897 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Dec 16 21:34:43 smithi169 ceph-mon[88350]: rocksdb: Options.two_write_queues: 0 2023-12-16T21:34:43.897 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Dec 16 21:34:43 smithi169 ceph-mon[88350]: rocksdb: Options.manual_wal_flush: 0 2023-12-16T21:34:43.897 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Dec 16 21:34:43 smithi169 ceph-mon[88350]: rocksdb: Options.atomic_flush: 0 2023-12-16T21:34:43.897 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Dec 16 21:34:43 smithi169 ceph-mon[88350]: rocksdb: Options.avoid_unnecessary_blocking_io: 0 2023-12-16T21:34:43.897 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Dec 16 21:34:43 smithi169 ceph-mon[88350]: rocksdb: Options.persist_stats_to_disk: 0 2023-12-16T21:34:43.897 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Dec 16 21:34:43 smithi169 ceph-mon[88350]: rocksdb: Options.write_dbid_to_manifest: 0 2023-12-16T21:34:43.897 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Dec 16 21:34:43 smithi169 ceph-mon[88350]: rocksdb: Options.log_readahead_size: 0 2023-12-16T21:34:43.897 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Dec 16 21:34:43 smithi169 ceph-mon[88350]: rocksdb: Options.file_checksum_gen_factory: Unknown 2023-12-16T21:34:43.897 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Dec 16 21:34:43 smithi169 ceph-mon[88350]: rocksdb: Options.best_efforts_recovery: 0 2023-12-16T21:34:43.897 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Dec 16 21:34:43 smithi169 ceph-mon[88350]: rocksdb: Options.max_bgerror_resume_count: 2147483647 2023-12-16T21:34:43.897 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Dec 16 21:34:43 smithi169 ceph-mon[88350]: rocksdb: Options.bgerror_resume_retry_interval: 1000000 2023-12-16T21:34:43.898 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Dec 16 21:34:43 smithi169 ceph-mon[88350]: rocksdb: Options.allow_data_in_errors: 0 2023-12-16T21:34:43.898 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Dec 16 21:34:43 smithi169 bash[88275]: f964ea9d81bd83e299c42fa160b98c92a27c2fe641b5a44a8e73f0df2942ce38 2023-12-16T21:34:43.898 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Dec 16 21:34:43 smithi169 ceph-mon[88350]: rocksdb: Options.db_host_id: __hostname__ 2023-12-16T21:34:43.898 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Dec 16 21:34:43 smithi169 ceph-mon[88350]: rocksdb: Options.max_background_jobs: 2 2023-12-16T21:34:43.898 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Dec 16 21:34:43 smithi169 ceph-mon[88350]: rocksdb: Options.max_background_compactions: -1 2023-12-16T21:34:43.898 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Dec 16 21:34:43 smithi169 ceph-mon[88350]: rocksdb: Options.max_subcompactions: 1 2023-12-16T21:34:43.898 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Dec 16 21:34:43 smithi169 ceph-mon[88350]: rocksdb: Options.avoid_flush_during_shutdown: 0 2023-12-16T21:34:43.898 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Dec 16 21:34:43 smithi169 ceph-mon[88350]: rocksdb: Options.writable_file_max_buffer_size: 1048576 2023-12-16T21:34:43.898 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Dec 16 21:34:43 smithi169 ceph-mon[88350]: rocksdb: Options.delayed_write_rate : 16777216 2023-12-16T21:34:43.898 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Dec 16 21:34:43 smithi169 ceph-mon[88350]: rocksdb: Options.max_total_wal_size: 0 2023-12-16T21:34:43.898 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Dec 16 21:34:43 smithi169 ceph-mon[88350]: rocksdb: Options.delete_obsolete_files_period_micros: 21600000000 2023-12-16T21:34:43.898 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Dec 16 21:34:43 smithi169 ceph-mon[88350]: rocksdb: Options.stats_dump_period_sec: 600 2023-12-16T21:34:43.898 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Dec 16 21:34:43 smithi169 ceph-mon[88350]: rocksdb: Options.stats_persist_period_sec: 600 2023-12-16T21:34:43.898 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Dec 16 21:34:43 smithi169 ceph-mon[88350]: rocksdb: Options.stats_history_buffer_size: 1048576 2023-12-16T21:34:43.898 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Dec 16 21:34:43 smithi169 ceph-mon[88350]: rocksdb: Options.max_open_files: -1 2023-12-16T21:34:43.899 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Dec 16 21:34:43 smithi169 ceph-mon[88350]: rocksdb: Options.bytes_per_sync: 0 2023-12-16T21:34:43.899 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Dec 16 21:34:43 smithi169 ceph-mon[88350]: rocksdb: Options.wal_bytes_per_sync: 0 2023-12-16T21:34:43.899 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Dec 16 21:34:43 smithi169 ceph-mon[88350]: rocksdb: Options.strict_bytes_per_sync: 0 2023-12-16T21:34:43.899 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Dec 16 21:34:43 smithi169 ceph-mon[88350]: rocksdb: Options.compaction_readahead_size: 0 2023-12-16T21:34:43.899 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Dec 16 21:34:43 smithi169 ceph-mon[88350]: rocksdb: Options.max_background_flushes: -1 2023-12-16T21:34:43.899 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Dec 16 21:34:43 smithi169 ceph-mon[88350]: rocksdb: Compression algorithms supported: 2023-12-16T21:34:43.899 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Dec 16 21:34:43 smithi169 ceph-mon[88350]: rocksdb: kZSTDNotFinalCompression supported: 0 2023-12-16T21:34:43.899 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Dec 16 21:34:43 smithi169 ceph-mon[88350]: rocksdb: kZSTD supported: 0 2023-12-16T21:34:43.899 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Dec 16 21:34:43 smithi169 ceph-mon[88350]: rocksdb: kXpressCompression supported: 0 2023-12-16T21:34:43.899 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Dec 16 21:34:43 smithi169 ceph-mon[88350]: rocksdb: kLZ4HCCompression supported: 1 2023-12-16T21:34:43.899 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Dec 16 21:34:43 smithi169 ceph-mon[88350]: rocksdb: kZlibCompression supported: 1 2023-12-16T21:34:43.899 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Dec 16 21:34:43 smithi169 ceph-mon[88350]: rocksdb: kSnappyCompression supported: 1 2023-12-16T21:34:43.899 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Dec 16 21:34:43 smithi169 ceph-mon[88350]: rocksdb: kLZ4Compression supported: 1 2023-12-16T21:34:43.899 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Dec 16 21:34:43 smithi169 ceph-mon[88350]: rocksdb: kBZip2Compression supported: 0 2023-12-16T21:34:43.900 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Dec 16 21:34:43 smithi169 ceph-mon[88350]: rocksdb: Fast CRC32 supported: Supported on x86 2023-12-16T21:34:43.900 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Dec 16 21:34:43 smithi169 ceph-mon[88350]: rocksdb: [db/version_set.cc:4724] Recovering from manifest file: /var/lib/ceph/mon/ceph-smithi169/store.db/MANIFEST-000003 2023-12-16T21:34:43.900 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Dec 16 21:34:43 smithi169 ceph-mon[88350]: rocksdb: [db/column_family.cc:595] --------------- Options for column family [default]: 2023-12-16T21:34:43.900 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Dec 16 21:34:43 smithi169 ceph-mon[88350]: rocksdb: Options.comparator: leveldb.BytewiseComparator 2023-12-16T21:34:43.900 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Dec 16 21:34:43 smithi169 ceph-mon[88350]: rocksdb: Options.merge_operator: 2023-12-16T21:34:43.900 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Dec 16 21:34:43 smithi169 ceph-mon[88350]: rocksdb: Options.compaction_filter: None 2023-12-16T21:34:43.900 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Dec 16 21:34:43 smithi169 ceph-mon[88350]: rocksdb: Options.compaction_filter_factory: None 2023-12-16T21:34:43.900 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Dec 16 21:34:43 smithi169 ceph-mon[88350]: rocksdb: Options.sst_partitioner_factory: None 2023-12-16T21:34:43.900 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Dec 16 21:34:43 smithi169 ceph-mon[88350]: rocksdb: Options.memtable_factory: SkipListFactory 2023-12-16T21:34:43.900 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Dec 16 21:34:43 smithi169 ceph-mon[88350]: rocksdb: Options.table_factory: BlockBasedTable 2023-12-16T21:34:43.900 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Dec 16 21:34:43 smithi169 ceph-mon[88350]: rocksdb: table_factory options: flush_block_policy_factory: FlushBlockBySizePolicyFactory (0x55fd31844f90) 2023-12-16T21:34:43.900 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout: cache_index_and_filter_blocks: 1 2023-12-16T21:34:43.900 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout: cache_index_and_filter_blocks_with_high_priority: 0 2023-12-16T21:34:43.900 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout: pin_l0_filter_and_index_blocks_in_cache: 0 2023-12-16T21:34:43.901 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout: pin_top_level_index_and_filter: 1 2023-12-16T21:34:43.901 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout: index_type: 0 2023-12-16T21:34:43.901 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout: data_block_index_type: 0 2023-12-16T21:34:43.901 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout: index_shortening: 1 2023-12-16T21:34:43.901 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout: data_block_hash_table_util_ratio: 0.750000 2023-12-16T21:34:43.901 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout: hash_index_allow_collision: 1 2023-12-16T21:34:43.901 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout: checksum: 1 2023-12-16T21:34:43.901 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout: no_block_cache: 0 2023-12-16T21:34:43.901 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout: block_cache: 0x55fd31877090 2023-12-16T21:34:43.901 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout: block_cache_name: BinnedLRUCache 2023-12-16T21:34:43.901 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout: block_cache_options: 2023-12-16T21:34:43.901 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout: capacity : 536870912 2023-12-16T21:34:43.901 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout: num_shard_bits : 4 2023-12-16T21:34:43.901 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout: strict_capacity_limit : 0 2023-12-16T21:34:43.902 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout: high_pri_pool_ratio: 0.000 2023-12-16T21:34:43.902 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout: block_cache_compressed: (nil) 2023-12-16T21:34:43.902 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout: persistent_cache: (nil) 2023-12-16T21:34:43.902 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout: block_size: 4096 2023-12-16T21:34:43.902 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout: block_size_deviation: 10 2023-12-16T21:34:43.902 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout: block_restart_interval: 16 2023-12-16T21:34:43.902 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout: index_block_restart_interval: 1 2023-12-16T21:34:43.902 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout: metadata_block_size: 4096 2023-12-16T21:34:43.902 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout: partition_filters: 0 2023-12-16T21:34:43.902 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout: use_delta_encoding: 1 2023-12-16T21:34:43.902 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout: filter_policy: rocksdb.BuiltinBloomFilter 2023-12-16T21:34:43.902 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout: whole_key_filtering: 1 2023-12-16T21:34:43.902 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout: verify_compression: 0 2023-12-16T21:34:43.902 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout: read_amp_bytes_per_bit: 0 2023-12-16T21:34:43.903 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout: format_version: 4 2023-12-16T21:34:43.903 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout: enable_index_compression: 1 2023-12-16T21:34:43.903 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout: block_align: 0 2023-12-16T21:34:43.903 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Dec 16 21:34:43 smithi169 ceph-mon[88350]: rocksdb: Options.write_buffer_size: 33554432 2023-12-16T21:34:43.903 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Dec 16 21:34:43 smithi169 ceph-mon[88350]: rocksdb: Options.max_write_buffer_number: 2 2023-12-16T21:34:43.903 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Dec 16 21:34:43 smithi169 ceph-mon[88350]: rocksdb: Options.compression: NoCompression 2023-12-16T21:34:43.903 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Dec 16 21:34:43 smithi169 ceph-mon[88350]: rocksdb: Options.bottommost_compression: Disabled 2023-12-16T21:34:43.903 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Dec 16 21:34:43 smithi169 ceph-mon[88350]: rocksdb: Options.prefix_extractor: nullptr 2023-12-16T21:34:43.903 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Dec 16 21:34:43 smithi169 ceph-mon[88350]: rocksdb: Options.memtable_insert_with_hint_prefix_extractor: nullptr 2023-12-16T21:34:43.903 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Dec 16 21:34:43 smithi169 ceph-mon[88350]: rocksdb: Options.num_levels: 7 2023-12-16T21:34:43.903 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Dec 16 21:34:43 smithi169 ceph-mon[88350]: rocksdb: Options.min_write_buffer_number_to_merge: 1 2023-12-16T21:34:43.903 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Dec 16 21:34:43 smithi169 ceph-mon[88350]: rocksdb: Options.max_write_buffer_number_to_maintain: 0 2023-12-16T21:34:43.903 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Dec 16 21:34:43 smithi169 ceph-mon[88350]: rocksdb: Options.max_write_buffer_size_to_maintain: 0 2023-12-16T21:34:43.904 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Dec 16 21:34:43 smithi169 ceph-mon[88350]: rocksdb: Options.bottommost_compression_opts.window_bits: -14 2023-12-16T21:34:43.904 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Dec 16 21:34:43 smithi169 ceph-mon[88350]: rocksdb: Options.bottommost_compression_opts.level: 32767 2023-12-16T21:34:43.904 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Dec 16 21:34:43 smithi169 ceph-mon[88350]: rocksdb: Options.bottommost_compression_opts.strategy: 0 2023-12-16T21:34:43.904 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Dec 16 21:34:43 smithi169 ceph-mon[88350]: rocksdb: Options.bottommost_compression_opts.max_dict_bytes: 0 2023-12-16T21:34:43.904 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Dec 16 21:34:43 smithi169 ceph-mon[88350]: rocksdb: Options.bottommost_compression_opts.zstd_max_train_bytes: 0 2023-12-16T21:34:43.904 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Dec 16 21:34:43 smithi169 ceph-mon[88350]: rocksdb: Options.bottommost_compression_opts.parallel_threads: 1 2023-12-16T21:34:43.904 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Dec 16 21:34:43 smithi169 ceph-mon[88350]: rocksdb: Options.bottommost_compression_opts.enabled: false 2023-12-16T21:34:43.904 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Dec 16 21:34:43 smithi169 ceph-mon[88350]: rocksdb: Options.compression_opts.window_bits: -14 2023-12-16T21:34:43.904 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Dec 16 21:34:43 smithi169 ceph-mon[88350]: rocksdb: Options.compression_opts.level: 32767 2023-12-16T21:34:43.904 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Dec 16 21:34:43 smithi169 ceph-mon[88350]: rocksdb: Options.compression_opts.strategy: 0 2023-12-16T21:34:43.904 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Dec 16 21:34:43 smithi169 ceph-mon[88350]: rocksdb: Options.compression_opts.max_dict_bytes: 0 2023-12-16T21:34:43.904 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Dec 16 21:34:43 smithi169 ceph-mon[88350]: rocksdb: Options.compression_opts.zstd_max_train_bytes: 0 2023-12-16T21:34:43.905 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Dec 16 21:34:43 smithi169 ceph-mon[88350]: rocksdb: Options.compression_opts.parallel_threads: 1 2023-12-16T21:34:43.905 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Dec 16 21:34:43 smithi169 ceph-mon[88350]: rocksdb: Options.compression_opts.enabled: false 2023-12-16T21:34:43.905 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Dec 16 21:34:43 smithi169 ceph-mon[88350]: rocksdb: Options.level0_file_num_compaction_trigger: 4 2023-12-16T21:34:43.905 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Dec 16 21:34:43 smithi169 ceph-mon[88350]: rocksdb: Options.level0_slowdown_writes_trigger: 20 2023-12-16T21:34:43.905 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Dec 16 21:34:43 smithi169 ceph-mon[88350]: rocksdb: Options.level0_stop_writes_trigger: 36 2023-12-16T21:34:43.905 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Dec 16 21:34:43 smithi169 ceph-mon[88350]: rocksdb: Options.target_file_size_base: 67108864 2023-12-16T21:34:43.905 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Dec 16 21:34:43 smithi169 ceph-mon[88350]: rocksdb: Options.target_file_size_multiplier: 1 2023-12-16T21:34:43.905 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Dec 16 21:34:43 smithi169 ceph-mon[88350]: rocksdb: Options.max_bytes_for_level_base: 268435456 2023-12-16T21:34:43.905 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Dec 16 21:34:43 smithi169 ceph-mon[88350]: rocksdb: Options.level_compaction_dynamic_level_bytes: 1 2023-12-16T21:34:43.905 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Dec 16 21:34:43 smithi169 ceph-mon[88350]: rocksdb: Options.max_bytes_for_level_multiplier: 10.000000 2023-12-16T21:34:43.905 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Dec 16 21:34:43 smithi169 ceph-mon[88350]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[0]: 1 2023-12-16T21:34:43.905 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Dec 16 21:34:43 smithi169 ceph-mon[88350]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[1]: 1 2023-12-16T21:34:43.905 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Dec 16 21:34:43 smithi169 ceph-mon[88350]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[2]: 1 2023-12-16T21:34:43.905 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Dec 16 21:34:43 smithi169 ceph-mon[88350]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[3]: 1 2023-12-16T21:34:43.906 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Dec 16 21:34:43 smithi169 ceph-mon[88350]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[4]: 1 2023-12-16T21:34:43.906 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Dec 16 21:34:43 smithi169 ceph-mon[88350]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[5]: 1 2023-12-16T21:34:43.906 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Dec 16 21:34:43 smithi169 ceph-mon[88350]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[6]: 1 2023-12-16T21:34:43.906 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Dec 16 21:34:43 smithi169 ceph-mon[88350]: rocksdb: Options.max_sequential_skip_in_iterations: 8 2023-12-16T21:34:43.906 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Dec 16 21:34:43 smithi169 ceph-mon[88350]: rocksdb: Options.max_compaction_bytes: 1677721600 2023-12-16T21:34:43.906 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Dec 16 21:34:43 smithi169 ceph-mon[88350]: rocksdb: Options.arena_block_size: 4194304 2023-12-16T21:34:43.906 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Dec 16 21:34:43 smithi169 ceph-mon[88350]: rocksdb: Options.soft_pending_compaction_bytes_limit: 68719476736 2023-12-16T21:34:43.907 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Dec 16 21:34:43 smithi169 ceph-mon[88350]: rocksdb: Options.hard_pending_compaction_bytes_limit: 274877906944 2023-12-16T21:34:43.907 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Dec 16 21:34:43 smithi169 ceph-mon[88350]: rocksdb: Options.rate_limit_delay_max_milliseconds: 100 2023-12-16T21:34:43.907 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Dec 16 21:34:43 smithi169 ceph-mon[88350]: rocksdb: Options.disable_auto_compactions: 0 2023-12-16T21:34:43.907 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Dec 16 21:34:43 smithi169 ceph-mon[88350]: rocksdb: Options.compaction_style: kCompactionStyleLevel 2023-12-16T21:34:43.907 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Dec 16 21:34:43 smithi169 ceph-mon[88350]: rocksdb: Options.compaction_pri: kMinOverlappingRatio 2023-12-16T21:34:43.907 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Dec 16 21:34:43 smithi169 ceph-mon[88350]: rocksdb: Options.compaction_options_universal.size_ratio: 1 2023-12-16T21:34:43.907 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Dec 16 21:34:43 smithi169 ceph-mon[88350]: rocksdb: Options.compaction_options_universal.min_merge_width: 2 2023-12-16T21:34:43.907 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Dec 16 21:34:43 smithi169 ceph-mon[88350]: rocksdb: Options.compaction_options_universal.max_merge_width: 4294967295 2023-12-16T21:34:43.907 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Dec 16 21:34:43 smithi169 ceph-mon[88350]: rocksdb: Options.compaction_options_universal.max_size_amplification_percent: 200 2023-12-16T21:34:43.907 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Dec 16 21:34:43 smithi169 ceph-mon[88350]: rocksdb: Options.compaction_options_universal.compression_size_percent: -1 2023-12-16T21:34:43.907 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Dec 16 21:34:43 smithi169 ceph-mon[88350]: rocksdb: Options.compaction_options_universal.stop_style: kCompactionStopStyleTotalSize 2023-12-16T21:34:43.907 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Dec 16 21:34:43 smithi169 ceph-mon[88350]: rocksdb: Options.compaction_options_fifo.max_table_files_size: 1073741824 2023-12-16T21:34:43.907 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Dec 16 21:34:43 smithi169 ceph-mon[88350]: rocksdb: Options.compaction_options_fifo.allow_compaction: 0 2023-12-16T21:34:43.907 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Dec 16 21:34:43 smithi169 ceph-mon[88350]: rocksdb: Options.table_properties_collectors: 2023-12-16T21:34:43.907 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Dec 16 21:34:43 smithi169 ceph-mon[88350]: rocksdb: Options.inplace_update_support: 0 2023-12-16T21:34:43.908 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Dec 16 21:34:43 smithi169 ceph-mon[88350]: rocksdb: Options.inplace_update_num_locks: 10000 2023-12-16T21:34:43.908 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Dec 16 21:34:43 smithi169 ceph-mon[88350]: rocksdb: Options.memtable_prefix_bloom_size_ratio: 0.000000 2023-12-16T21:34:43.908 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Dec 16 21:34:43 smithi169 ceph-mon[88350]: rocksdb: Options.memtable_whole_key_filtering: 0 2023-12-16T21:34:43.908 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Dec 16 21:34:43 smithi169 ceph-mon[88350]: rocksdb: Options.memtable_huge_page_size: 0 2023-12-16T21:34:43.908 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Dec 16 21:34:43 smithi169 ceph-mon[88350]: rocksdb: Options.bloom_locality: 0 2023-12-16T21:34:43.908 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Dec 16 21:34:43 smithi169 ceph-mon[88350]: rocksdb: Options.max_successive_merges: 0 2023-12-16T21:34:43.908 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Dec 16 21:34:43 smithi169 ceph-mon[88350]: rocksdb: Options.optimize_filters_for_hits: 0 2023-12-16T21:34:43.908 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Dec 16 21:34:43 smithi169 ceph-mon[88350]: rocksdb: Options.paranoid_file_checks: 0 2023-12-16T21:34:43.908 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Dec 16 21:34:43 smithi169 ceph-mon[88350]: rocksdb: Options.force_consistency_checks: 1 2023-12-16T21:34:43.908 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Dec 16 21:34:43 smithi169 ceph-mon[88350]: rocksdb: Options.report_bg_io_stats: 0 2023-12-16T21:34:43.908 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Dec 16 21:34:43 smithi169 ceph-mon[88350]: rocksdb: Options.ttl: 2592000 2023-12-16T21:34:43.908 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Dec 16 21:34:43 smithi169 ceph-mon[88350]: rocksdb: Options.periodic_compaction_seconds: 0 2023-12-16T21:34:43.908 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Dec 16 21:34:43 smithi169 ceph-mon[88350]: rocksdb: Options.enable_blob_files: false 2023-12-16T21:34:43.908 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Dec 16 21:34:43 smithi169 ceph-mon[88350]: rocksdb: Options.min_blob_size: 0 2023-12-16T21:34:43.909 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Dec 16 21:34:43 smithi169 ceph-mon[88350]: rocksdb: Options.blob_file_size: 268435456 2023-12-16T21:34:43.909 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Dec 16 21:34:43 smithi169 ceph-mon[88350]: rocksdb: Options.blob_compression_type: NoCompression 2023-12-16T21:34:43.909 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Dec 16 21:34:43 smithi169 ceph-mon[88350]: rocksdb: Options.enable_blob_garbage_collection: false 2023-12-16T21:34:43.909 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Dec 16 21:34:43 smithi169 ceph-mon[88350]: rocksdb: Options.blob_garbage_collection_age_cutoff: 0.250000 2023-12-16T21:34:43.909 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Dec 16 21:34:43 smithi169 ceph-mon[88350]: rocksdb: [db/version_set.cc:4764] Recovered from manifest file:/var/lib/ceph/mon/ceph-smithi169/store.db/MANIFEST-000003 succeeded,manifest_file_number is 3, next_file_number is 5, last_sequence is 0, log_number is 0,prev_log_number is 0,max_column_family is 0,min_log_number_to_keep is 0 2023-12-16T21:34:43.909 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Dec 16 21:34:43 smithi169 ceph-mon[88350]: rocksdb: [db/version_set.cc:4779] Column family [default] (ID 0), log number is 0 2023-12-16T21:34:43.922 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Dec 16 21:34:43 smithi169 ceph-mon[88350]: rocksdb: [db/version_set.cc:4082] Creating manifest 7 2023-12-16T21:34:43.922 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Dec 16 21:34:43 smithi169 ceph-mon[88350]: rocksdb: EVENT_LOG_v1 {"time_micros": 1702762483864393, "job": 1, "event": "recovery_started", "wal_files": [4]} 2023-12-16T21:34:43.922 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Dec 16 21:34:43 smithi169 ceph-mon[88350]: rocksdb: [db/db_impl/db_impl_open.cc:845] Recovering log #4 mode 2 2023-12-16T21:34:43.922 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Dec 16 21:34:43 smithi169 ceph-mon[88350]: rocksdb: [table/block_based/filter_policy.cc:991] Using legacy Bloom filter with high (20) bits/key. Dramatic filter space and/or accuracy improvement is available with format_version>=5. 2023-12-16T21:34:43.922 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Dec 16 21:34:43 smithi169 ceph-mon[88350]: rocksdb: EVENT_LOG_v1 {"time_micros": 1702762483864830, "cf_name": "default", "job": 1, "event": "table_file_creation", "file_number": 8, "file_size": 1545, "file_checksum": "", "file_checksum_func_name": "Unknown", "table_properties": {"data_size": 523, "index_size": 31, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 1, "index_value_is_delta_encoded": 1, "filter_size": 69, "raw_key_size": 115, "raw_average_key_size": 23, "raw_value_size": 401, "raw_average_value_size": 80, "num_data_blocks": 1, "num_entries": 5, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 0, "filter_policy": "rocksdb.BuiltinBloomFilter", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "", "prefix_extractor_name": "nullptr", "property_collectors": "[]", "compression": "NoCompression", "compression_options": "window_bits=-14; level=32767; strategy=0; max_dict_bytes=0; zstd_max_train_bytes=0; enabled=0; ", "creation_time": 1702762483, "oldest_key_time": 0, "file_creation_time": 0, "db_id": "f0254ebc-c99f-4a26-8fd3-cca78ed42c65", "db_session_id": "S3I0I5CVFTX9XKWB5P69"}} 2023-12-16T21:34:43.922 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Dec 16 21:34:43 smithi169 ceph-mon[88350]: rocksdb: [db/version_set.cc:4082] Creating manifest 9 2023-12-16T21:34:43.922 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Dec 16 21:34:43 smithi169 ceph-mon[88350]: rocksdb: EVENT_LOG_v1 {"time_micros": 1702762483865342, "job": 1, "event": "recovery_finished"} 2023-12-16T21:34:43.922 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Dec 16 21:34:43 smithi169 ceph-mon[88350]: rocksdb: [file/delete_scheduler.cc:69] Deleted file /var/lib/ceph/mon/ceph-smithi169/store.db/000004.log immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2023-12-16T21:34:43.922 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Dec 16 21:34:43 smithi169 ceph-mon[88350]: rocksdb: [db/db_impl/db_impl_open.cc:1700] SstFileManager instance 0x55fd3180b180 2023-12-16T21:34:43.922 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Dec 16 21:34:43 smithi169 ceph-mon[88350]: rocksdb: DB pointer 0x55fd318e0000 2023-12-16T21:34:43.922 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Dec 16 21:34:43 smithi169 ceph-mon[88350]: rocksdb: [db/db_impl/db_impl.cc:901] ------- DUMPING STATS ------- 2023-12-16T21:34:43.922 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Dec 16 21:34:43 smithi169 ceph-mon[88350]: rocksdb: [db/db_impl/db_impl.cc:903] 2023-12-16T21:34:43.922 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout: ** DB Stats ** 2023-12-16T21:34:43.922 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout: Uptime(secs): 0.0 total, 0.0 interval 2023-12-16T21:34:43.923 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout: Cumulative writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 GB, 0.00 MB/s 2023-12-16T21:34:43.923 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout: Cumulative WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2023-12-16T21:34:43.923 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout: Cumulative stall: 00:00:0.000 H:M:S, 0.0 percent 2023-12-16T21:34:43.923 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout: Interval writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 MB, 0.00 MB/s 2023-12-16T21:34:43.923 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout: Interval WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 MB, 0.00 MB/s 2023-12-16T21:34:43.923 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout: Interval stall: 00:00:0.000 H:M:S, 0.0 percent 2023-12-16T21:34:43.923 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout: 2023-12-16T21:34:43.923 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout: ** Compaction Stats [default] ** 2023-12-16T21:34:43.923 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout: Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop 2023-12-16T21:34:43.923 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout: ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2023-12-16T21:34:43.923 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout: L0 1/0 1.51 KB 0.2 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 3.6 0.00 0.00 1 0.000 0 0 2023-12-16T21:34:43.924 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout: Sum 1/0 1.51 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 3.6 0.00 0.00 1 0.000 0 0 2023-12-16T21:34:43.924 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout: Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 3.6 0.00 0.00 1 0.000 0 0 2023-12-16T21:34:43.924 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout: 2023-12-16T21:34:43.924 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout: ** Compaction Stats [default] ** 2023-12-16T21:34:43.924 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout: Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop 2023-12-16T21:34:43.924 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout: ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2023-12-16T21:34:43.924 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout: User 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 3.6 0.00 0.00 1 0.000 0 0 2023-12-16T21:34:43.924 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout: Uptime(secs): 0.0 total, 0.0 interval 2023-12-16T21:34:43.924 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout: Flush(GB): cumulative 0.000, interval 0.000 2023-12-16T21:34:43.925 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout: AddFile(GB): cumulative 0.000, interval 0.000 2023-12-16T21:34:43.925 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout: AddFile(Total Files): cumulative 0, interval 0 2023-12-16T21:34:43.925 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout: AddFile(L0 Files): cumulative 0, interval 0 2023-12-16T21:34:43.925 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout: AddFile(Keys): cumulative 0, interval 0 2023-12-16T21:34:43.925 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout: Cumulative compaction: 0.00 GB write, 0.39 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2023-12-16T21:34:43.925 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout: Interval compaction: 0.00 GB write, 0.39 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2023-12-16T21:34:43.925 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout: Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count 2023-12-16T21:34:43.925 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout: 2023-12-16T21:34:43.925 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout: ** File Read Latency Histogram By Level [default] ** 2023-12-16T21:34:43.925 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout: 2023-12-16T21:34:43.925 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout: ** Compaction Stats [default] ** 2023-12-16T21:34:43.925 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout: Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop 2023-12-16T21:34:43.925 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout: ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2023-12-16T21:34:43.925 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout: L0 1/0 1.51 KB 0.2 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 3.6 0.00 0.00 1 0.000 0 0 2023-12-16T21:34:43.926 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout: Sum 1/0 1.51 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 3.6 0.00 0.00 1 0.000 0 0 2023-12-16T21:34:43.926 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout: Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.00 0.00 0 0.000 0 0 2023-12-16T21:34:43.926 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout: 2023-12-16T21:34:43.926 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout: ** Compaction Stats [default] ** 2023-12-16T21:34:43.926 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout: Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop 2023-12-16T21:34:43.926 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout: ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2023-12-16T21:34:43.926 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout: User 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 3.6 0.00 0.00 1 0.000 0 0 2023-12-16T21:34:43.926 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout: Uptime(secs): 0.0 total, 0.0 interval 2023-12-16T21:34:43.926 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout: Flush(GB): cumulative 0.000, interval 0.000 2023-12-16T21:34:43.926 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout: AddFile(GB): cumulative 0.000, interval 0.000 2023-12-16T21:34:43.926 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout: AddFile(Total Files): cumulative 0, interval 0 2023-12-16T21:34:43.927 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout: AddFile(L0 Files): cumulative 0, interval 0 2023-12-16T21:34:43.927 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout: AddFile(Keys): cumulative 0, interval 0 2023-12-16T21:34:43.927 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout: Cumulative compaction: 0.00 GB write, 0.39 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2023-12-16T21:34:43.927 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout: Interval compaction: 0.00 GB write, 0.00 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2023-12-16T21:34:43.927 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout: Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count 2023-12-16T21:34:43.927 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout: 2023-12-16T21:34:43.927 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout: ** File Read Latency Histogram By Level [default] ** 2023-12-16T21:34:43.927 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Dec 16 21:34:43 smithi169 ceph-mon[88350]: mon.smithi169 does not exist in monmap, will attempt to join an existing cluster 2023-12-16T21:34:43.927 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Dec 16 21:34:43 smithi169 ceph-mon[88350]: using public_addr v2:172.21.15.169:0/0 -> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] 2023-12-16T21:34:43.927 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Dec 16 21:34:43 smithi169 systemd[1]: Started Ceph mon.smithi169 for 88ff363c-9c5a-11ee-95a5-87774f69a715. 2023-12-16T21:34:43.927 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Dec 16 21:34:43 smithi169 ceph-mon[88350]: starting mon.smithi169 rank -1 at public addrs [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] at bind addrs [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] mon_data /var/lib/ceph/mon/ceph-smithi169 fsid 88ff363c-9c5a-11ee-95a5-87774f69a715 2023-12-16T21:34:43.927 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Dec 16 21:34:43 smithi169 ceph-mon[88350]: mon.smithi169@-1(???) e0 preinit fsid 88ff363c-9c5a-11ee-95a5-87774f69a715 2023-12-16T21:34:43.927 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Dec 16 21:34:43 smithi169 ceph-mon[88350]: mon.smithi169@-1(synchronizing).mds e1 new map 2023-12-16T21:34:43.927 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Dec 16 21:34:43 smithi169 ceph-mon[88350]: mon.smithi169@-1(synchronizing).mds e1 print_map 2023-12-16T21:34:43.928 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout: e1 2023-12-16T21:34:43.928 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout: enable_multiple, ever_enabled_multiple: 1,1 2023-12-16T21:34:43.928 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout: default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2023-12-16T21:34:43.928 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout: legacy client fscid: -1 2023-12-16T21:34:43.928 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout: 2023-12-16T21:34:43.928 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout: No filesystems configured 2023-12-16T21:34:43.928 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Dec 16 21:34:43 smithi169 ceph-mon[88350]: mon.smithi169@-1(synchronizing).osd e0 _set_cache_ratios kv ratio 0.25 inc ratio 0.375 full ratio 0.375 2023-12-16T21:34:43.928 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Dec 16 21:34:43 smithi169 ceph-mon[88350]: mon.smithi169@-1(synchronizing).osd e0 register_cache_with_pcm pcm target: 2147483648 pcm max: 1020054732 pcm min: 134217728 inc_osd_cache size: 1 2023-12-16T21:34:43.928 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Dec 16 21:34:43 smithi169 ceph-mon[88350]: mon.smithi169@-1(synchronizing).osd e1 e1: 0 total, 0 up, 0 in 2023-12-16T21:34:43.928 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Dec 16 21:34:43 smithi169 ceph-mon[88350]: mon.smithi169@-1(synchronizing).osd e2 e2: 0 total, 0 up, 0 in 2023-12-16T21:34:43.928 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Dec 16 21:34:43 smithi169 ceph-mon[88350]: mon.smithi169@-1(synchronizing).osd e3 e3: 0 total, 0 up, 0 in 2023-12-16T21:34:43.928 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Dec 16 21:34:43 smithi169 ceph-mon[88350]: mon.smithi169@-1(synchronizing).osd e4 e4: 0 total, 0 up, 0 in 2023-12-16T21:34:43.928 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Dec 16 21:34:43 smithi169 ceph-mon[88350]: mon.smithi169@-1(synchronizing).osd e5 e5: 0 total, 0 up, 0 in 2023-12-16T21:34:43.929 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Dec 16 21:34:43 smithi169 ceph-mon[88350]: mon.smithi169@-1(synchronizing).osd e5 crush map has features 3314932999778484224, adjusting msgr requires 2023-12-16T21:34:43.929 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Dec 16 21:34:43 smithi169 ceph-mon[88350]: mon.smithi169@-1(synchronizing).osd e5 crush map has features 288514050185494528, adjusting msgr requires 2023-12-16T21:34:43.929 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Dec 16 21:34:43 smithi169 ceph-mon[88350]: mon.smithi169@-1(synchronizing).osd e5 crush map has features 288514050185494528, adjusting msgr requires 2023-12-16T21:34:43.929 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Dec 16 21:34:43 smithi169 ceph-mon[88350]: mon.smithi169@-1(synchronizing).osd e5 crush map has features 288514050185494528, adjusting msgr requires 2023-12-16T21:34:43.929 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Dec 16 21:34:43 smithi169 ceph-mon[88350]: pgmap v7: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-16T21:34:43.929 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Dec 16 21:34:43 smithi169 ceph-mon[88350]: from='client.? 172.21.15.169:0/1852407003' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-12-16T21:34:43.929 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Dec 16 21:34:43 smithi169 ceph-mon[88350]: pgmap v8: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-16T21:34:43.929 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Dec 16 21:34:43 smithi169 ceph-mon[88350]: from='client.? 172.21.15.169:0/161799855' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-12-16T21:34:43.929 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Dec 16 21:34:43 smithi169 ceph-mon[88350]: pgmap v9: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-16T21:34:43.929 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Dec 16 21:34:43 smithi169 ceph-mon[88350]: pgmap v10: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-16T21:34:43.929 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Dec 16 21:34:43 smithi169 ceph-mon[88350]: from='mgr.14162 172.21.15.45:0/142909557' entity='mgr.smithi045.nhtmwc' 2023-12-16T21:34:43.929 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Dec 16 21:34:43 smithi169 ceph-mon[88350]: from='mgr.14162 172.21.15.45:0/142909557' entity='mgr.smithi045.nhtmwc' 2023-12-16T21:34:43.929 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Dec 16 21:34:43 smithi169 ceph-mon[88350]: from='mgr.14162 172.21.15.45:0/142909557' entity='mgr.smithi045.nhtmwc' 2023-12-16T21:34:43.929 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Dec 16 21:34:43 smithi169 ceph-mon[88350]: from='mgr.14162 172.21.15.45:0/142909557' entity='mgr.smithi045.nhtmwc' 2023-12-16T21:34:43.930 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Dec 16 21:34:43 smithi169 ceph-mon[88350]: from='mgr.14162 172.21.15.45:0/142909557' entity='mgr.smithi045.nhtmwc' 2023-12-16T21:34:43.930 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Dec 16 21:34:43 smithi169 ceph-mon[88350]: from='mgr.14162 172.21.15.45:0/142909557' entity='mgr.smithi045.nhtmwc' 2023-12-16T21:34:43.930 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Dec 16 21:34:43 smithi169 ceph-mon[88350]: Deploying daemon prometheus.smithi045 on smithi045 2023-12-16T21:34:43.930 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Dec 16 21:34:43 smithi169 ceph-mon[88350]: from='client.? 172.21.15.169:0/1754624243' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-12-16T21:34:43.930 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Dec 16 21:34:43 smithi169 ceph-mon[88350]: pgmap v11: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-16T21:34:43.930 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Dec 16 21:34:43 smithi169 ceph-mon[88350]: from='mgr.14162 172.21.15.45:0/142909557' entity='mgr.smithi045.nhtmwc' 2023-12-16T21:34:43.930 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Dec 16 21:34:43 smithi169 ceph-mon[88350]: from='client.? 172.21.15.169:0/3927930416' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-12-16T21:34:43.930 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Dec 16 21:34:43 smithi169 ceph-mon[88350]: pgmap v12: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-16T21:34:43.930 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Dec 16 21:34:43 smithi169 ceph-mon[88350]: from='client.? 172.21.15.169:0/2637764047' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-12-16T21:34:43.930 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Dec 16 21:34:43 smithi169 ceph-mon[88350]: pgmap v13: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-16T21:34:43.930 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Dec 16 21:34:43 smithi169 ceph-mon[88350]: pgmap v14: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-16T21:34:43.930 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Dec 16 21:34:43 smithi169 ceph-mon[88350]: from='client.? 172.21.15.169:0/345235710' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-12-16T21:34:43.930 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Dec 16 21:34:43 smithi169 ceph-mon[88350]: pgmap v15: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-16T21:34:43.930 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Dec 16 21:34:43 smithi169 ceph-mon[88350]: from='mgr.14162 172.21.15.45:0/142909557' entity='mgr.smithi045.nhtmwc' 2023-12-16T21:34:43.930 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Dec 16 21:34:43 smithi169 ceph-mon[88350]: from='mgr.14162 172.21.15.45:0/142909557' entity='mgr.smithi045.nhtmwc' 2023-12-16T21:34:43.930 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Dec 16 21:34:43 smithi169 ceph-mon[88350]: from='mgr.14162 172.21.15.45:0/142909557' entity='mgr.smithi045.nhtmwc' 2023-12-16T21:34:43.931 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Dec 16 21:34:43 smithi169 ceph-mon[88350]: from='mgr.14162 172.21.15.45:0/142909557' entity='mgr.smithi045.nhtmwc' cmd=[{"prefix": "mgr module enable", "module": "prometheus"}]: dispatch 2023-12-16T21:34:43.931 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Dec 16 21:34:43 smithi169 ceph-mon[88350]: from='client.? 172.21.15.169:0/2072130761' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-12-16T21:34:43.931 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Dec 16 21:34:43 smithi169 ceph-mon[88350]: from='mgr.14162 172.21.15.45:0/142909557' entity='mgr.smithi045.nhtmwc' cmd='[{"prefix": "mgr module enable", "module": "prometheus"}]': finished 2023-12-16T21:34:43.932 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Dec 16 21:34:43 smithi169 ceph-mon[88350]: mgrmap e14: smithi045.nhtmwc(active, since 43s) 2023-12-16T21:34:43.932 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Dec 16 21:34:43 smithi169 ceph-mon[88350]: from='client.? 172.21.15.169:0/1981716866' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-12-16T21:34:43.932 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Dec 16 21:34:43 smithi169 ceph-mon[88350]: from='client.? 172.21.15.169:0/519641825' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-12-16T21:34:43.932 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Dec 16 21:34:43 smithi169 ceph-mon[88350]: Active manager daemon smithi045.nhtmwc restarted 2023-12-16T21:34:43.932 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Dec 16 21:34:43 smithi169 ceph-mon[88350]: Activating manager daemon smithi045.nhtmwc 2023-12-16T21:34:43.932 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Dec 16 21:34:43 smithi169 ceph-mon[88350]: osdmap e5: 0 total, 0 up, 0 in 2023-12-16T21:34:43.932 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Dec 16 21:34:43 smithi169 ceph-mon[88350]: mgrmap e15: smithi045.nhtmwc(active, starting, since 0.00294731s) 2023-12-16T21:34:43.932 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Dec 16 21:34:43 smithi169 ceph-mon[88350]: from='mgr.14214 172.21.15.45:0/1227909437' entity='mgr.smithi045.nhtmwc' cmd=[{"prefix": "mon metadata", "id": "smithi045"}]: dispatch 2023-12-16T21:34:43.932 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Dec 16 21:34:43 smithi169 ceph-mon[88350]: from='mgr.14214 172.21.15.45:0/1227909437' entity='mgr.smithi045.nhtmwc' cmd=[{"prefix": "mgr metadata", "who": "smithi045.nhtmwc", "id": "smithi045.nhtmwc"}]: dispatch 2023-12-16T21:34:43.932 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Dec 16 21:34:43 smithi169 ceph-mon[88350]: from='mgr.14214 172.21.15.45:0/1227909437' entity='mgr.smithi045.nhtmwc' cmd=[{"prefix": "mds metadata"}]: dispatch 2023-12-16T21:34:43.932 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Dec 16 21:34:43 smithi169 ceph-mon[88350]: from='mgr.14214 172.21.15.45:0/1227909437' entity='mgr.smithi045.nhtmwc' cmd=[{"prefix": "osd metadata"}]: dispatch 2023-12-16T21:34:43.932 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Dec 16 21:34:43 smithi169 ceph-mon[88350]: from='mgr.14214 172.21.15.45:0/1227909437' entity='mgr.smithi045.nhtmwc' cmd=[{"prefix": "mon metadata"}]: dispatch 2023-12-16T21:34:43.932 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Dec 16 21:34:43 smithi169 ceph-mon[88350]: Manager daemon smithi045.nhtmwc is now available 2023-12-16T21:34:43.932 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Dec 16 21:34:43 smithi169 ceph-mon[88350]: from='mgr.14214 172.21.15.45:0/1227909437' entity='mgr.smithi045.nhtmwc' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-12-16T21:34:43.933 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Dec 16 21:34:43 smithi169 ceph-mon[88350]: from='mgr.14214 172.21.15.45:0/1227909437' entity='mgr.smithi045.nhtmwc' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-12-16T21:34:43.933 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Dec 16 21:34:43 smithi169 ceph-mon[88350]: from='mgr.14214 172.21.15.45:0/1227909437' entity='mgr.smithi045.nhtmwc' 2023-12-16T21:34:43.933 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Dec 16 21:34:43 smithi169 ceph-mon[88350]: from='mgr.14214 172.21.15.45:0/1227909437' entity='mgr.smithi045.nhtmwc' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2023-12-16T21:34:43.933 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Dec 16 21:34:43 smithi169 ceph-mon[88350]: from='mgr.14214 172.21.15.45:0/1227909437' entity='mgr.smithi045.nhtmwc' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi045.nhtmwc/mirror_snapshot_schedule"}]: dispatch 2023-12-16T21:34:43.933 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Dec 16 21:34:43 smithi169 ceph-mon[88350]: from='mgr.14214 172.21.15.45:0/1227909437' entity='mgr.smithi045.nhtmwc' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi045.nhtmwc/trash_purge_schedule"}]: dispatch 2023-12-16T21:34:43.933 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Dec 16 21:34:43 smithi169 ceph-mon[88350]: from='client.? 172.21.15.169:0/2386149533' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-12-16T21:34:43.933 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Dec 16 21:34:43 smithi169 ceph-mon[88350]: from='mgr.14214 172.21.15.45:0/1227909437' entity='mgr.smithi045.nhtmwc' 2023-12-16T21:34:43.933 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Dec 16 21:34:43 smithi169 ceph-mon[88350]: from='mgr.14214 172.21.15.45:0/1227909437' entity='mgr.smithi045.nhtmwc' 2023-12-16T21:34:43.933 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Dec 16 21:34:43 smithi169 ceph-mon[88350]: mgrmap e16: smithi045.nhtmwc(active, since 1.00562s) 2023-12-16T21:34:43.933 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Dec 16 21:34:43 smithi169 ceph-mon[88350]: [16/Dec/2023:21:34:29] ENGINE Bus STARTING 2023-12-16T21:34:43.934 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Dec 16 21:34:43 smithi169 ceph-mon[88350]: [16/Dec/2023:21:34:29] ENGINE Serving on https://172.21.15.45:7150 2023-12-16T21:34:43.934 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Dec 16 21:34:43 smithi169 ceph-mon[88350]: [16/Dec/2023:21:34:29] ENGINE Bus STARTED 2023-12-16T21:34:43.934 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Dec 16 21:34:43 smithi169 ceph-mon[88350]: from='mgr.14214 172.21.15.45:0/1227909437' entity='mgr.smithi045.nhtmwc' 2023-12-16T21:34:43.934 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Dec 16 21:34:43 smithi169 ceph-mon[88350]: from='mgr.14214 172.21.15.45:0/1227909437' entity='mgr.smithi045.nhtmwc' 2023-12-16T21:34:43.934 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Dec 16 21:34:43 smithi169 ceph-mon[88350]: mgrmap e17: smithi045.nhtmwc(active, since 2s) 2023-12-16T21:34:43.934 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Dec 16 21:34:43 smithi169 ceph-mon[88350]: from='mgr.14214 172.21.15.45:0/1227909437' entity='mgr.smithi045.nhtmwc' 2023-12-16T21:34:43.934 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Dec 16 21:34:43 smithi169 ceph-mon[88350]: from='mgr.14214 172.21.15.45:0/1227909437' entity='mgr.smithi045.nhtmwc' 2023-12-16T21:34:43.934 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Dec 16 21:34:43 smithi169 ceph-mon[88350]: from='client.? 172.21.15.169:0/4011290899' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-12-16T21:34:43.934 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Dec 16 21:34:43 smithi169 ceph-mon[88350]: from='mgr.14214 172.21.15.45:0/1227909437' entity='mgr.smithi045.nhtmwc' 2023-12-16T21:34:43.934 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Dec 16 21:34:43 smithi169 ceph-mon[88350]: from='mgr.14214 172.21.15.45:0/1227909437' entity='mgr.smithi045.nhtmwc' 2023-12-16T21:34:43.934 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Dec 16 21:34:43 smithi169 ceph-mon[88350]: from='mgr.14214 172.21.15.45:0/1227909437' entity='mgr.smithi045.nhtmwc' cmd=[{"prefix": "config rm", "who": "osd/host:smithi045", "name": "osd_memory_target"}]: dispatch 2023-12-16T21:34:43.934 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Dec 16 21:34:43 smithi169 ceph-mon[88350]: from='mgr.14214 172.21.15.45:0/1227909437' entity='mgr.smithi045.nhtmwc' 2023-12-16T21:34:43.934 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Dec 16 21:34:43 smithi169 ceph-mon[88350]: from='mgr.14214 172.21.15.45:0/1227909437' entity='mgr.smithi045.nhtmwc' 2023-12-16T21:34:43.934 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Dec 16 21:34:43 smithi169 ceph-mon[88350]: from='mgr.14214 172.21.15.45:0/1227909437' entity='mgr.smithi045.nhtmwc' 2023-12-16T21:34:43.934 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Dec 16 21:34:43 smithi169 ceph-mon[88350]: from='mgr.14214 172.21.15.45:0/1227909437' entity='mgr.smithi045.nhtmwc' 2023-12-16T21:34:43.935 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Dec 16 21:34:43 smithi169 ceph-mon[88350]: from='mgr.14214 172.21.15.45:0/1227909437' entity='mgr.smithi045.nhtmwc' cmd=[{"prefix": "config rm", "who": "osd/host:smithi169", "name": "osd_memory_target"}]: dispatch 2023-12-16T21:34:43.935 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Dec 16 21:34:43 smithi169 ceph-mon[88350]: from='mgr.14214 172.21.15.45:0/1227909437' entity='mgr.smithi045.nhtmwc' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-12-16T21:34:43.935 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Dec 16 21:34:43 smithi169 ceph-mon[88350]: from='mgr.14214 172.21.15.45:0/1227909437' entity='mgr.smithi045.nhtmwc' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-12-16T21:34:43.935 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Dec 16 21:34:43 smithi169 ceph-mon[88350]: Updating smithi045:/etc/ceph/ceph.conf 2023-12-16T21:34:43.935 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Dec 16 21:34:43 smithi169 ceph-mon[88350]: Updating smithi169:/etc/ceph/ceph.conf 2023-12-16T21:34:43.935 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Dec 16 21:34:43 smithi169 ceph-mon[88350]: Updating smithi169:/var/lib/ceph/88ff363c-9c5a-11ee-95a5-87774f69a715/config/ceph.conf 2023-12-16T21:34:43.935 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Dec 16 21:34:43 smithi169 ceph-mon[88350]: Updating smithi045:/var/lib/ceph/88ff363c-9c5a-11ee-95a5-87774f69a715/config/ceph.conf 2023-12-16T21:34:43.935 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Dec 16 21:34:43 smithi169 ceph-mon[88350]: from='mgr.14214 172.21.15.45:0/1227909437' entity='mgr.smithi045.nhtmwc' 2023-12-16T21:34:43.935 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Dec 16 21:34:43 smithi169 ceph-mon[88350]: from='mgr.14214 172.21.15.45:0/1227909437' entity='mgr.smithi045.nhtmwc' 2023-12-16T21:34:43.935 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Dec 16 21:34:43 smithi169 ceph-mon[88350]: from='mgr.14214 172.21.15.45:0/1227909437' entity='mgr.smithi045.nhtmwc' 2023-12-16T21:34:43.935 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Dec 16 21:34:43 smithi169 ceph-mon[88350]: from='mgr.14214 172.21.15.45:0/1227909437' entity='mgr.smithi045.nhtmwc' 2023-12-16T21:34:43.935 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Dec 16 21:34:43 smithi169 ceph-mon[88350]: from='mgr.14214 172.21.15.45:0/1227909437' entity='mgr.smithi045.nhtmwc' 2023-12-16T21:34:43.935 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Dec 16 21:34:43 smithi169 ceph-mon[88350]: from='mgr.14214 172.21.15.45:0/1227909437' entity='mgr.smithi045.nhtmwc' cmd=[{"prefix": "auth get-or-create", "entity": "client.crash.smithi169", "caps": ["mon", "profile crash", "mgr", "profile crash"]}]: dispatch 2023-12-16T21:34:43.935 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Dec 16 21:34:43 smithi169 ceph-mon[88350]: from='mgr.14214 172.21.15.45:0/1227909437' entity='mgr.smithi045.nhtmwc' cmd='[{"prefix": "auth get-or-create", "entity": "client.crash.smithi169", "caps": ["mon", "profile crash", "mgr", "profile crash"]}]': finished 2023-12-16T21:34:43.936 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Dec 16 21:34:43 smithi169 ceph-mon[88350]: from='mgr.14214 172.21.15.45:0/1227909437' entity='mgr.smithi045.nhtmwc' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-12-16T21:34:43.936 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Dec 16 21:34:43 smithi169 ceph-mon[88350]: Updating smithi169:/etc/ceph/ceph.client.admin.keyring 2023-12-16T21:34:43.936 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Dec 16 21:34:43 smithi169 ceph-mon[88350]: Updating smithi045:/etc/ceph/ceph.client.admin.keyring 2023-12-16T21:34:43.936 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Dec 16 21:34:43 smithi169 ceph-mon[88350]: Updating smithi169:/var/lib/ceph/88ff363c-9c5a-11ee-95a5-87774f69a715/config/ceph.client.admin.keyring 2023-12-16T21:34:43.936 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Dec 16 21:34:43 smithi169 ceph-mon[88350]: Updating smithi045:/var/lib/ceph/88ff363c-9c5a-11ee-95a5-87774f69a715/config/ceph.client.admin.keyring 2023-12-16T21:34:43.936 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Dec 16 21:34:43 smithi169 ceph-mon[88350]: Deploying daemon crash.smithi169 on smithi169 2023-12-16T21:34:43.936 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Dec 16 21:34:43 smithi169 ceph-mon[88350]: from='client.? 172.21.15.169:0/64450196' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-12-16T21:34:43.936 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Dec 16 21:34:43 smithi169 ceph-mon[88350]: from='mgr.14214 172.21.15.45:0/1227909437' entity='mgr.smithi045.nhtmwc' 2023-12-16T21:34:43.936 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Dec 16 21:34:43 smithi169 ceph-mon[88350]: from='mgr.14214 172.21.15.45:0/1227909437' entity='mgr.smithi045.nhtmwc' 2023-12-16T21:34:43.936 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Dec 16 21:34:43 smithi169 ceph-mon[88350]: from='mgr.14214 172.21.15.45:0/1227909437' entity='mgr.smithi045.nhtmwc' 2023-12-16T21:34:43.936 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Dec 16 21:34:43 smithi169 ceph-mon[88350]: from='mgr.14214 172.21.15.45:0/1227909437' entity='mgr.smithi045.nhtmwc' 2023-12-16T21:34:43.937 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Dec 16 21:34:43 smithi169 ceph-mon[88350]: Deploying daemon node-exporter.smithi169 on smithi169 2023-12-16T21:34:43.937 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Dec 16 21:34:43 smithi169 ceph-mon[88350]: from='client.? 172.21.15.169:0/2062887851' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-12-16T21:34:43.937 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Dec 16 21:34:43 smithi169 ceph-mon[88350]: from='mgr.14214 172.21.15.45:0/1227909437' entity='mgr.smithi045.nhtmwc' 2023-12-16T21:34:43.937 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Dec 16 21:34:43 smithi169 ceph-mon[88350]: from='mgr.14214 172.21.15.45:0/1227909437' entity='mgr.smithi045.nhtmwc' 2023-12-16T21:34:43.937 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Dec 16 21:34:43 smithi169 ceph-mon[88350]: from='mgr.14214 172.21.15.45:0/1227909437' entity='mgr.smithi045.nhtmwc' 2023-12-16T21:34:43.937 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Dec 16 21:34:43 smithi169 ceph-mon[88350]: from='mgr.14214 172.21.15.45:0/1227909437' entity='mgr.smithi045.nhtmwc' 2023-12-16T21:34:43.937 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Dec 16 21:34:43 smithi169 ceph-mon[88350]: from='mgr.14214 172.21.15.45:0/1227909437' entity='mgr.smithi045.nhtmwc' 2023-12-16T21:34:43.937 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Dec 16 21:34:43 smithi169 ceph-mon[88350]: from='mgr.14214 172.21.15.45:0/1227909437' entity='mgr.smithi045.nhtmwc' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.smithi169.cncbgh", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2023-12-16T21:34:43.937 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Dec 16 21:34:43 smithi169 ceph-mon[88350]: from='mgr.14214 172.21.15.45:0/1227909437' entity='mgr.smithi045.nhtmwc' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.smithi169.cncbgh", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2023-12-16T21:34:43.937 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Dec 16 21:34:43 smithi169 ceph-mon[88350]: from='mgr.14214 172.21.15.45:0/1227909437' entity='mgr.smithi045.nhtmwc' cmd=[{"prefix": "mgr services"}]: dispatch 2023-12-16T21:34:43.937 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Dec 16 21:34:43 smithi169 ceph-mon[88350]: from='mgr.14214 172.21.15.45:0/1227909437' entity='mgr.smithi045.nhtmwc' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-12-16T21:34:43.937 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Dec 16 21:34:43 smithi169 ceph-mon[88350]: Deploying daemon mgr.smithi169.cncbgh on smithi169 2023-12-16T21:34:43.938 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Dec 16 21:34:43 smithi169 ceph-mon[88350]: from='mgr.14214 172.21.15.45:0/1227909437' entity='mgr.smithi045.nhtmwc' 2023-12-16T21:34:43.938 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Dec 16 21:34:43 smithi169 ceph-mon[88350]: from='mgr.14214 172.21.15.45:0/1227909437' entity='mgr.smithi045.nhtmwc' 2023-12-16T21:34:43.938 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Dec 16 21:34:43 smithi169 ceph-mon[88350]: from='mgr.14214 172.21.15.45:0/1227909437' entity='mgr.smithi045.nhtmwc' 2023-12-16T21:34:43.938 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Dec 16 21:34:43 smithi169 ceph-mon[88350]: from='mgr.14214 172.21.15.45:0/1227909437' entity='mgr.smithi045.nhtmwc' 2023-12-16T21:34:43.938 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Dec 16 21:34:43 smithi169 ceph-mon[88350]: from='mgr.14214 172.21.15.45:0/1227909437' entity='mgr.smithi045.nhtmwc' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2023-12-16T21:34:43.938 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Dec 16 21:34:43 smithi169 ceph-mon[88350]: from='mgr.14214 172.21.15.45:0/1227909437' entity='mgr.smithi045.nhtmwc' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-12-16T21:34:43.938 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Dec 16 21:34:43 smithi169 ceph-mon[88350]: Deploying daemon mon.smithi169 on smithi169 2023-12-16T21:34:43.938 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Dec 16 21:34:43 smithi169 ceph-mon[88350]: from='client.? 172.21.15.169:0/1002873366' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-12-16T21:34:43.939 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Dec 16 21:34:43 smithi169 ceph-mon[88350]: from='mgr.14214 172.21.15.45:0/1227909437' entity='mgr.smithi045.nhtmwc' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2023-12-16T21:34:43.939 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Dec 16 21:34:43 smithi169 ceph-mon[88350]: from='mgr.14214 172.21.15.45:0/1227909437' entity='mgr.smithi045.nhtmwc' 2023-12-16T21:34:43.939 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Dec 16 21:34:43 smithi169 ceph-mon[88350]: mon.smithi169@-1(synchronizing).paxosservice(auth 1..6) refresh upgraded, format 0 -> 3 2023-12-16T21:34:43.939 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Dec 16 21:34:43 smithi169 ceph-mon[88350]: mon.smithi169@-1(synchronizing) e1 handle_conf_change mon_allow_pool_delete 2023-12-16T21:34:43.939 INFO:teuthology.orchestra.run.smithi169.stderr:2023-12-16T21:34:43.904+0000 7f6a4ffff700 1 -- 172.21.15.169:0/1223880013 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f6a580259a0 con 0x7f6a681029e0 2023-12-16T21:34:44.066 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Dec 16 21:34:43 smithi045 ceph-mon[80763]: from='mgr.14214 172.21.15.45:0/1227909437' entity='mgr.smithi045.nhtmwc' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2023-12-16T21:34:44.066 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Dec 16 21:34:43 smithi045 ceph-mon[80763]: from='mgr.14214 172.21.15.45:0/1227909437' entity='mgr.smithi045.nhtmwc' 2023-12-16T21:34:44.172 INFO:teuthology.orchestra.run.smithi169.stderr:2023-12-16T21:34:44.169+0000 7f6a6e00c700 1 -- 172.21.15.169:0/1223880013 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f6a6810e180 con 0x7f6a681029e0 2023-12-16T21:34:48.954 INFO:teuthology.orchestra.run.smithi169.stderr:2023-12-16T21:34:48.952+0000 7f6a4ffff700 1 -- 172.21.15.169:0/1223880013 <== mon.0 v2:172.21.15.45:3300/0 8 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 2 v2) v1 ==== 95+0+1017 (secure 0 0 0) 0x7f6a58049380 con 0x7f6a681029e0 2023-12-16T21:34:48.955 INFO:teuthology.orchestra.run.smithi169.stdout: 2023-12-16T21:34:48.955 INFO:teuthology.orchestra.run.smithi169.stdout:{"epoch":2,"fsid":"88ff363c-9c5a-11ee-95a5-87774f69a715","modified":"2023-12-16T21:34:43.905979Z","created":"2023-12-16T21:32:46.744221Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"smithi045","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:3300","nonce":0},{"type":"v1","addr":"172.21.15.45:6789","nonce":0}]},"addr":"172.21.15.45:6789/0","public_addr":"172.21.15.45:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"smithi169","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:3300","nonce":0},{"type":"v1","addr":"172.21.15.169:6789","nonce":0}]},"addr":"172.21.15.169:6789/0","public_addr":"172.21.15.169:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0,1]} 2023-12-16T21:34:48.957 INFO:teuthology.orchestra.run.smithi169.stderr:2023-12-16T21:34:48.955+0000 7f6a6e00c700 1 -- 172.21.15.169:0/1223880013 >> [v2:172.21.15.45:6800/2012105536,v1:172.21.15.45:6801/2012105536] conn(0x7f6a500367d0 msgr2=0x7f6a50038c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:34:48.957 INFO:teuthology.orchestra.run.smithi169.stderr:2023-12-16T21:34:48.955+0000 7f6a6e00c700 1 --2- 172.21.15.169:0/1223880013 >> [v2:172.21.15.45:6800/2012105536,v1:172.21.15.45:6801/2012105536] conn(0x7f6a500367d0 0x7f6a50038c80 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7f6a5c009990 tx=0x7f6a5c006e40 comp rx=0 tx=0).stop 2023-12-16T21:34:48.957 INFO:teuthology.orchestra.run.smithi169.stderr:2023-12-16T21:34:48.955+0000 7f6a6e00c700 1 -- 172.21.15.169:0/1223880013 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f6a681029e0 msgr2=0x7f6a68112ba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:34:48.957 INFO:teuthology.orchestra.run.smithi169.stderr:2023-12-16T21:34:48.955+0000 7f6a6e00c700 1 --2- 172.21.15.169:0/1223880013 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f6a681029e0 0x7f6a68112ba0 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7f6a58009300 tx=0x7f6a58009e30 comp rx=0 tx=0).stop 2023-12-16T21:34:48.957 INFO:teuthology.orchestra.run.smithi169.stderr:2023-12-16T21:34:48.955+0000 7f6a6e00c700 1 -- 172.21.15.169:0/1223880013 shutdown_connections 2023-12-16T21:34:48.957 INFO:teuthology.orchestra.run.smithi169.stderr:2023-12-16T21:34:48.955+0000 7f6a6e00c700 1 --2- 172.21.15.169:0/1223880013 >> [v2:172.21.15.45:6800/2012105536,v1:172.21.15.45:6801/2012105536] conn(0x7f6a500367d0 0x7f6a50038c80 unknown :-1 s=CLOSED pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:34:48.957 INFO:teuthology.orchestra.run.smithi169.stderr:2023-12-16T21:34:48.955+0000 7f6a6e00c700 1 --2- 172.21.15.169:0/1223880013 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f6a681029e0 0x7f6a68112ba0 unknown :-1 s=CLOSED pgs=146 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:34:48.957 INFO:teuthology.orchestra.run.smithi169.stderr:2023-12-16T21:34:48.955+0000 7f6a6e00c700 1 -- 172.21.15.169:0/1223880013 >> 172.21.15.169:0/1223880013 conn(0x7f6a68071760 msgr2=0x7f6a68103f30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-16T21:34:48.958 INFO:teuthology.orchestra.run.smithi169.stderr:2023-12-16T21:34:48.955+0000 7f6a6e00c700 1 -- 172.21.15.169:0/1223880013 shutdown_connections 2023-12-16T21:34:48.958 INFO:teuthology.orchestra.run.smithi169.stderr:2023-12-16T21:34:48.955+0000 7f6a6e00c700 1 -- 172.21.15.169:0/1223880013 wait complete. 2023-12-16T21:34:48.958 INFO:teuthology.orchestra.run.smithi169.stderr:dumped monmap epoch 2 2023-12-16T21:34:49.215 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Dec 16 21:34:49 smithi169 ceph-mon[88350]: from='mgr.14214 172.21.15.45:0/1227909437' entity='mgr.smithi045.nhtmwc' cmd=[{"prefix": "mon metadata", "id": "smithi045"}]: dispatch 2023-12-16T21:34:49.215 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Dec 16 21:34:49 smithi169 ceph-mon[88350]: from='mgr.14214 172.21.15.45:0/1227909437' entity='mgr.smithi045.nhtmwc' cmd=[{"prefix": "mon metadata", "id": "smithi169"}]: dispatch 2023-12-16T21:34:49.215 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Dec 16 21:34:49 smithi169 ceph-mon[88350]: mon.smithi045 calling monitor election 2023-12-16T21:34:49.215 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Dec 16 21:34:49 smithi169 ceph-mon[88350]: from='client.? 172.21.15.169:0/1223880013' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-12-16T21:34:49.215 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Dec 16 21:34:49 smithi169 ceph-mon[88350]: from='mgr.14214 172.21.15.45:0/1227909437' entity='mgr.smithi045.nhtmwc' cmd=[{"prefix": "mon metadata", "id": "smithi169"}]: dispatch 2023-12-16T21:34:49.215 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Dec 16 21:34:49 smithi169 ceph-mon[88350]: from='mgr.14214 172.21.15.45:0/1227909437' entity='mgr.smithi045.nhtmwc' cmd=[{"prefix": "mon metadata", "id": "smithi169"}]: dispatch 2023-12-16T21:34:49.215 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Dec 16 21:34:49 smithi169 ceph-mon[88350]: mon.smithi169 calling monitor election 2023-12-16T21:34:49.215 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Dec 16 21:34:49 smithi169 ceph-mon[88350]: from='mgr.14214 172.21.15.45:0/1227909437' entity='mgr.smithi045.nhtmwc' cmd=[{"prefix": "mon metadata", "id": "smithi169"}]: dispatch 2023-12-16T21:34:49.216 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Dec 16 21:34:49 smithi169 ceph-mon[88350]: from='mgr.? 172.21.15.169:0/1013414716' entity='mgr.smithi169.cncbgh' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/smithi169.cncbgh/crt"}]: dispatch 2023-12-16T21:34:49.216 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Dec 16 21:34:49 smithi169 ceph-mon[88350]: from='mgr.14214 172.21.15.45:0/1227909437' entity='mgr.smithi045.nhtmwc' cmd=[{"prefix": "mon metadata", "id": "smithi169"}]: dispatch 2023-12-16T21:34:49.216 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Dec 16 21:34:49 smithi169 ceph-mon[88350]: pgmap v3: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-16T21:34:49.216 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Dec 16 21:34:49 smithi169 ceph-mon[88350]: from='mgr.14214 172.21.15.45:0/1227909437' entity='mgr.smithi045.nhtmwc' cmd=[{"prefix": "mon metadata", "id": "smithi169"}]: dispatch 2023-12-16T21:34:49.216 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Dec 16 21:34:49 smithi169 ceph-mon[88350]: mon.smithi045 is new leader, mons smithi045,smithi169 in quorum (ranks 0,1) 2023-12-16T21:34:49.216 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Dec 16 21:34:49 smithi169 ceph-mon[88350]: monmap e2: 2 mons at {smithi045=[v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0],smithi169=[v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0]} removed_ranks: {} disallowed_leaders: {} 2023-12-16T21:34:49.216 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Dec 16 21:34:49 smithi169 ceph-mon[88350]: fsmap 2023-12-16T21:34:49.216 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Dec 16 21:34:49 smithi169 ceph-mon[88350]: osdmap e5: 0 total, 0 up, 0 in 2023-12-16T21:34:49.216 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Dec 16 21:34:49 smithi169 ceph-mon[88350]: mgrmap e17: smithi045.nhtmwc(active, since 20s) 2023-12-16T21:34:49.216 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Dec 16 21:34:49 smithi169 ceph-mon[88350]: Standby manager daemon smithi169.cncbgh started 2023-12-16T21:34:49.216 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Dec 16 21:34:49 smithi169 ceph-mon[88350]: from='mgr.? 172.21.15.169:0/1013414716' entity='mgr.smithi169.cncbgh' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2023-12-16T21:34:49.216 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Dec 16 21:34:49 smithi169 ceph-mon[88350]: from='mgr.? 172.21.15.169:0/1013414716' entity='mgr.smithi169.cncbgh' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/smithi169.cncbgh/key"}]: dispatch 2023-12-16T21:34:49.216 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Dec 16 21:34:49 smithi169 ceph-mon[88350]: from='mgr.? 172.21.15.169:0/1013414716' entity='mgr.smithi169.cncbgh' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2023-12-16T21:34:49.216 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Dec 16 21:34:49 smithi169 ceph-mon[88350]: overall HEALTH_OK 2023-12-16T21:34:49.217 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Dec 16 21:34:49 smithi169 ceph-mon[88350]: from='mgr.14214 172.21.15.45:0/1227909437' entity='mgr.smithi045.nhtmwc' 2023-12-16T21:34:49.217 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Dec 16 21:34:49 smithi169 ceph-mon[88350]: from='mgr.14214 172.21.15.45:0/1227909437' entity='mgr.smithi045.nhtmwc' 2023-12-16T21:34:49.217 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Dec 16 21:34:49 smithi169 ceph-mon[88350]: from='mgr.14214 172.21.15.45:0/1227909437' entity='mgr.smithi045.nhtmwc' 2023-12-16T21:34:49.217 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Dec 16 21:34:49 smithi169 ceph-mon[88350]: from='mgr.14214 172.21.15.45:0/1227909437' entity='mgr.smithi045.nhtmwc' 2023-12-16T21:34:49.217 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Dec 16 21:34:49 smithi169 ceph-mon[88350]: from='mgr.14214 172.21.15.45:0/1227909437' entity='mgr.smithi045.nhtmwc' 2023-12-16T21:34:49.218 INFO:tasks.cephadm:Generating final ceph.conf file... 2023-12-16T21:34:49.218 DEBUG:teuthology.orchestra.run.smithi045:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 88ff363c-9c5a-11ee-95a5-87774f69a715 -- ceph config generate-minimal-conf 2023-12-16T21:34:49.316 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Dec 16 21:34:49 smithi045 ceph-mon[80763]: from='mgr.14214 172.21.15.45:0/1227909437' entity='mgr.smithi045.nhtmwc' cmd=[{"prefix": "mon metadata", "id": "smithi045"}]: dispatch 2023-12-16T21:34:49.316 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Dec 16 21:34:49 smithi045 ceph-mon[80763]: from='mgr.14214 172.21.15.45:0/1227909437' entity='mgr.smithi045.nhtmwc' cmd=[{"prefix": "mon metadata", "id": "smithi169"}]: dispatch 2023-12-16T21:34:49.317 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Dec 16 21:34:49 smithi045 ceph-mon[80763]: mon.smithi045 calling monitor election 2023-12-16T21:34:49.317 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Dec 16 21:34:49 smithi045 ceph-mon[80763]: from='client.? 172.21.15.169:0/1223880013' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-12-16T21:34:49.317 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Dec 16 21:34:49 smithi045 ceph-mon[80763]: from='mgr.14214 172.21.15.45:0/1227909437' entity='mgr.smithi045.nhtmwc' cmd=[{"prefix": "mon metadata", "id": "smithi169"}]: dispatch 2023-12-16T21:34:49.317 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Dec 16 21:34:49 smithi045 ceph-mon[80763]: from='mgr.14214 172.21.15.45:0/1227909437' entity='mgr.smithi045.nhtmwc' cmd=[{"prefix": "mon metadata", "id": "smithi169"}]: dispatch 2023-12-16T21:34:49.317 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Dec 16 21:34:49 smithi045 ceph-mon[80763]: mon.smithi169 calling monitor election 2023-12-16T21:34:49.317 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Dec 16 21:34:49 smithi045 ceph-mon[80763]: from='mgr.14214 172.21.15.45:0/1227909437' entity='mgr.smithi045.nhtmwc' cmd=[{"prefix": "mon metadata", "id": "smithi169"}]: dispatch 2023-12-16T21:34:49.317 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Dec 16 21:34:49 smithi045 ceph-mon[80763]: from='mgr.? 172.21.15.169:0/1013414716' entity='mgr.smithi169.cncbgh' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/smithi169.cncbgh/crt"}]: dispatch 2023-12-16T21:34:49.317 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Dec 16 21:34:49 smithi045 ceph-mon[80763]: from='mgr.14214 172.21.15.45:0/1227909437' entity='mgr.smithi045.nhtmwc' cmd=[{"prefix": "mon metadata", "id": "smithi169"}]: dispatch 2023-12-16T21:34:49.317 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Dec 16 21:34:49 smithi045 ceph-mon[80763]: pgmap v3: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-16T21:34:49.317 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Dec 16 21:34:49 smithi045 ceph-mon[80763]: from='mgr.14214 172.21.15.45:0/1227909437' entity='mgr.smithi045.nhtmwc' cmd=[{"prefix": "mon metadata", "id": "smithi169"}]: dispatch 2023-12-16T21:34:49.317 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Dec 16 21:34:49 smithi045 ceph-mon[80763]: mon.smithi045 is new leader, mons smithi045,smithi169 in quorum (ranks 0,1) 2023-12-16T21:34:49.317 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Dec 16 21:34:49 smithi045 ceph-mon[80763]: monmap e2: 2 mons at {smithi045=[v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0],smithi169=[v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0]} removed_ranks: {} disallowed_leaders: {} 2023-12-16T21:34:49.317 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Dec 16 21:34:49 smithi045 ceph-mon[80763]: fsmap 2023-12-16T21:34:49.317 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Dec 16 21:34:49 smithi045 ceph-mon[80763]: osdmap e5: 0 total, 0 up, 0 in 2023-12-16T21:34:49.318 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Dec 16 21:34:49 smithi045 ceph-mon[80763]: mgrmap e17: smithi045.nhtmwc(active, since 20s) 2023-12-16T21:34:49.318 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Dec 16 21:34:49 smithi045 ceph-mon[80763]: Standby manager daemon smithi169.cncbgh started 2023-12-16T21:34:49.318 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Dec 16 21:34:49 smithi045 ceph-mon[80763]: from='mgr.? 172.21.15.169:0/1013414716' entity='mgr.smithi169.cncbgh' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2023-12-16T21:34:49.318 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Dec 16 21:34:49 smithi045 ceph-mon[80763]: from='mgr.? 172.21.15.169:0/1013414716' entity='mgr.smithi169.cncbgh' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/smithi169.cncbgh/key"}]: dispatch 2023-12-16T21:34:49.318 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Dec 16 21:34:49 smithi045 ceph-mon[80763]: from='mgr.? 172.21.15.169:0/1013414716' entity='mgr.smithi169.cncbgh' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2023-12-16T21:34:49.318 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Dec 16 21:34:49 smithi045 ceph-mon[80763]: overall HEALTH_OK 2023-12-16T21:34:49.318 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Dec 16 21:34:49 smithi045 ceph-mon[80763]: from='mgr.14214 172.21.15.45:0/1227909437' entity='mgr.smithi045.nhtmwc' 2023-12-16T21:34:49.318 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Dec 16 21:34:49 smithi045 ceph-mon[80763]: from='mgr.14214 172.21.15.45:0/1227909437' entity='mgr.smithi045.nhtmwc' 2023-12-16T21:34:49.318 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Dec 16 21:34:49 smithi045 ceph-mon[80763]: from='mgr.14214 172.21.15.45:0/1227909437' entity='mgr.smithi045.nhtmwc' 2023-12-16T21:34:49.318 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Dec 16 21:34:49 smithi045 ceph-mon[80763]: from='mgr.14214 172.21.15.45:0/1227909437' entity='mgr.smithi045.nhtmwc' 2023-12-16T21:34:49.318 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Dec 16 21:34:49 smithi045 ceph-mon[80763]: from='mgr.14214 172.21.15.45:0/1227909437' entity='mgr.smithi045.nhtmwc' 2023-12-16T21:34:49.492 INFO:teuthology.orchestra.run.smithi045.stderr:Inferring config /var/lib/ceph/88ff363c-9c5a-11ee-95a5-87774f69a715/mon.smithi045/config 2023-12-16T21:34:50.127 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Dec 16 21:34:50 smithi045 ceph-mon[80763]: mgrmap e18: smithi045.nhtmwc(active, since 20s), standbys: smithi169.cncbgh 2023-12-16T21:34:50.127 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Dec 16 21:34:50 smithi045 ceph-mon[80763]: from='mgr.14214 172.21.15.45:0/1227909437' entity='mgr.smithi045.nhtmwc' cmd=[{"prefix": "mgr metadata", "who": "smithi169.cncbgh", "id": "smithi169.cncbgh"}]: dispatch 2023-12-16T21:34:50.127 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Dec 16 21:34:50 smithi045 ceph-mon[80763]: from='mgr.14214 172.21.15.45:0/1227909437' entity='mgr.smithi045.nhtmwc' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-12-16T21:34:50.127 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Dec 16 21:34:50 smithi045 ceph-mon[80763]: from='mgr.14214 172.21.15.45:0/1227909437' entity='mgr.smithi045.nhtmwc' cmd=[{"prefix": "mon metadata", "id": "smithi169"}]: dispatch 2023-12-16T21:34:50.228 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:34:50.222+0000 7fc1d03e1700 1 -- 172.21.15.45:0/2577045694 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fc1c81028f0 msgr2=0x7fc1c8102cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:34:50.228 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:34:50.222+0000 7fc1d03e1700 1 --2- 172.21.15.45:0/2577045694 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fc1c81028f0 0x7fc1c8102cc0 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7fc1c40048c0 tx=0x7fc1c4009e30 comp rx=0 tx=0).stop 2023-12-16T21:34:50.228 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:34:50.223+0000 7fc1d03e1700 1 -- 172.21.15.45:0/2577045694 shutdown_connections 2023-12-16T21:34:50.228 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:34:50.223+0000 7fc1d03e1700 1 --2- 172.21.15.45:0/2577045694 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fc1c81028f0 0x7fc1c8102cc0 unknown :-1 s=CLOSED pgs=149 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:34:50.228 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:34:50.223+0000 7fc1d03e1700 1 -- 172.21.15.45:0/2577045694 >> 172.21.15.45:0/2577045694 conn(0x7fc1c80715e0 msgr2=0x7fc1c80719e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-16T21:34:50.228 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:34:50.223+0000 7fc1d03e1700 1 -- 172.21.15.45:0/2577045694 shutdown_connections 2023-12-16T21:34:50.228 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:34:50.223+0000 7fc1d03e1700 1 -- 172.21.15.45:0/2577045694 wait complete. 2023-12-16T21:34:50.228 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:34:50.223+0000 7fc1d03e1700 1 Processor -- start 2023-12-16T21:34:50.228 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:34:50.223+0000 7fc1d03e1700 1 -- start start 2023-12-16T21:34:50.228 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:34:50.223+0000 7fc1d03e1700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fc1c81028f0 0x7fc1c8112ca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:34:50.228 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:34:50.223+0000 7fc1d03e1700 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fc1c81131e0 0x7fc1c810cb40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:34:50.228 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:34:50.223+0000 7fc1d03e1700 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc1c81137f0 con 0x7fc1c81028f0 2023-12-16T21:34:50.228 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:34:50.223+0000 7fc1d03e1700 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc1c810d080 con 0x7fc1c81131e0 2023-12-16T21:34:50.229 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:34:50.224+0000 7fc1ce17d700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fc1c81028f0 0x7fc1c8112ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:34:50.229 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:34:50.224+0000 7fc1ce17d700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fc1c81028f0 0x7fc1c8112ca0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:44646/0 (socket says 172.21.15.45:44646) 2023-12-16T21:34:50.229 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:34:50.224+0000 7fc1ce17d700 1 -- 172.21.15.45:0/794869141 learned_addr learned my addr 172.21.15.45:0/794869141 (peer_addr_for_me v2:172.21.15.45:0/0) 2023-12-16T21:34:50.229 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:34:50.224+0000 7fc1cd97c700 1 --2- 172.21.15.45:0/794869141 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fc1c81131e0 0x7fc1c810cb40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:34:50.229 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:34:50.224+0000 7fc1cd97c700 1 -- 172.21.15.45:0/794869141 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fc1c81131e0 msgr2=0x7fc1c810cb40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 13 2023-12-16T21:34:50.229 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:34:50.224+0000 7fc1cd97c700 1 -- 172.21.15.45:0/794869141 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fc1c81131e0 msgr2=0x7fc1c810cb40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2023-12-16T21:34:50.229 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:34:50.224+0000 7fc1cd97c700 1 --2- 172.21.15.45:0/794869141 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fc1c81131e0 0x7fc1c810cb40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2023-12-16T21:34:50.229 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:34:50.224+0000 7fc1ce17d700 1 -- 172.21.15.45:0/794869141 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fc1c81131e0 msgr2=0x7fc1c810cb40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:34:50.229 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:34:50.224+0000 7fc1cd97c700 1 --2- 172.21.15.45:0/794869141 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fc1c81131e0 0x7fc1c810cb40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-12-16T21:34:50.229 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:34:50.224+0000 7fc1ce17d700 1 --2- 172.21.15.45:0/794869141 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fc1c81131e0 0x7fc1c810cb40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:34:50.229 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:34:50.224+0000 7fc1ce17d700 1 -- 172.21.15.45:0/794869141 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc1c4005040 con 0x7fc1c81028f0 2023-12-16T21:34:50.229 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:34:50.224+0000 7fc1ce17d700 1 --2- 172.21.15.45:0/794869141 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fc1c81028f0 0x7fc1c8112ca0 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7fc1c400bfe0 tx=0x7fc1c4009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:34:50.229 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:34:50.224+0000 7fc1beffd700 1 -- 172.21.15.45:0/794869141 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fc1c400e040 con 0x7fc1c81028f0 2023-12-16T21:34:50.229 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:34:50.224+0000 7fc1beffd700 1 -- 172.21.15.45:0/794869141 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fc1c4013420 con 0x7fc1c81028f0 2023-12-16T21:34:50.230 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:34:50.224+0000 7fc1d03e1700 1 -- 172.21.15.45:0/794869141 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc1c810d300 con 0x7fc1c81028f0 2023-12-16T21:34:50.230 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:34:50.224+0000 7fc1beffd700 1 -- 172.21.15.45:0/794869141 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fc1c401c630 con 0x7fc1c81028f0 2023-12-16T21:34:50.230 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:34:50.225+0000 7fc1d03e1700 1 -- 172.21.15.45:0/794869141 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc1c810d8c0 con 0x7fc1c81028f0 2023-12-16T21:34:50.231 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:34:50.226+0000 7fc1d03e1700 1 -- 172.21.15.45:0/794869141 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc1c805f270 con 0x7fc1c81028f0 2023-12-16T21:34:50.231 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:34:50.226+0000 7fc1beffd700 1 -- 172.21.15.45:0/794869141 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 18) v1 ==== 86391+0+0 (secure 0 0 0) 0x7fc1c40139e0 con 0x7fc1c81028f0 2023-12-16T21:34:50.231 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:34:50.226+0000 7fc1beffd700 1 --2- 172.21.15.45:0/794869141 >> [v2:172.21.15.45:6800/2012105536,v1:172.21.15.45:6801/2012105536] conn(0x7fc1b4069030 0x7fc1b406b4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:34:50.231 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:34:50.226+0000 7fc1beffd700 1 -- 172.21.15.45:0/794869141 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1366+0+0 (secure 0 0 0) 0x7fc1c4086f80 con 0x7fc1c81028f0 2023-12-16T21:34:50.231 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:34:50.227+0000 7fc1cd97c700 1 --2- 172.21.15.45:0/794869141 >> [v2:172.21.15.45:6800/2012105536,v1:172.21.15.45:6801/2012105536] conn(0x7fc1b4069030 0x7fc1b406b4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:34:50.231 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:34:50.227+0000 7fc1cd97c700 1 --2- 172.21.15.45:0/794869141 >> [v2:172.21.15.45:6800/2012105536,v1:172.21.15.45:6801/2012105536] conn(0x7fc1b4069030 0x7fc1b406b4e0 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7fc1b8006f30 tx=0x7fc1b8008040 comp rx=0 tx=0).ready entity=mgr.14214 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:34:50.234 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:34:50.231+0000 7fc1beffd700 1 -- 172.21.15.45:0/794869141 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+174570 (secure 0 0 0) 0x7fc1c40b8a00 con 0x7fc1c81028f0 2023-12-16T21:34:50.383 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Dec 16 21:34:50 smithi169 ceph-mon[88350]: mgrmap e18: smithi045.nhtmwc(active, since 20s), standbys: smithi169.cncbgh 2023-12-16T21:34:50.383 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Dec 16 21:34:50 smithi169 ceph-mon[88350]: from='mgr.14214 172.21.15.45:0/1227909437' entity='mgr.smithi045.nhtmwc' cmd=[{"prefix": "mgr metadata", "who": "smithi169.cncbgh", "id": "smithi169.cncbgh"}]: dispatch 2023-12-16T21:34:50.383 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Dec 16 21:34:50 smithi169 ceph-mon[88350]: from='mgr.14214 172.21.15.45:0/1227909437' entity='mgr.smithi045.nhtmwc' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-12-16T21:34:50.384 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Dec 16 21:34:50 smithi169 ceph-mon[88350]: from='mgr.14214 172.21.15.45:0/1227909437' entity='mgr.smithi045.nhtmwc' cmd=[{"prefix": "mon metadata", "id": "smithi169"}]: dispatch 2023-12-16T21:34:50.438 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:34:50.434+0000 7fc1d03e1700 1 -- 172.21.15.45:0/794869141 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "config generate-minimal-conf"} v 0) v1 -- 0x7fc1c810e280 con 0x7fc1c81028f0 2023-12-16T21:34:50.438 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:34:50.435+0000 7fc1beffd700 1 -- 172.21.15.45:0/794869141 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mon_command_ack([{"prefix": "config generate-minimal-conf"}]=0 v10) v1 ==== 76+0+225 (secure 0 0 0) 0x7fc1c4058060 con 0x7fc1c81028f0 2023-12-16T21:34:50.439 INFO:teuthology.orchestra.run.smithi045.stdout:# minimal ceph.conf for 88ff363c-9c5a-11ee-95a5-87774f69a715 2023-12-16T21:34:50.439 INFO:teuthology.orchestra.run.smithi045.stdout:[global] 2023-12-16T21:34:50.439 INFO:teuthology.orchestra.run.smithi045.stdout: fsid = 88ff363c-9c5a-11ee-95a5-87774f69a715 2023-12-16T21:34:50.439 INFO:teuthology.orchestra.run.smithi045.stdout: mon_host = [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] 2023-12-16T21:34:50.442 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:34:50.438+0000 7fc1d03e1700 1 -- 172.21.15.45:0/794869141 >> [v2:172.21.15.45:6800/2012105536,v1:172.21.15.45:6801/2012105536] conn(0x7fc1b4069030 msgr2=0x7fc1b406b4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:34:50.442 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:34:50.438+0000 7fc1d03e1700 1 --2- 172.21.15.45:0/794869141 >> [v2:172.21.15.45:6800/2012105536,v1:172.21.15.45:6801/2012105536] conn(0x7fc1b4069030 0x7fc1b406b4e0 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7fc1b8006f30 tx=0x7fc1b8008040 comp rx=0 tx=0).stop 2023-12-16T21:34:50.442 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:34:50.438+0000 7fc1d03e1700 1 -- 172.21.15.45:0/794869141 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fc1c81028f0 msgr2=0x7fc1c8112ca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:34:50.442 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:34:50.438+0000 7fc1d03e1700 1 --2- 172.21.15.45:0/794869141 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fc1c81028f0 0x7fc1c8112ca0 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7fc1c400bfe0 tx=0x7fc1c4009e30 comp rx=0 tx=0).stop 2023-12-16T21:34:50.442 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:34:50.438+0000 7fc1d03e1700 1 -- 172.21.15.45:0/794869141 shutdown_connections 2023-12-16T21:34:50.442 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:34:50.438+0000 7fc1d03e1700 1 --2- 172.21.15.45:0/794869141 >> [v2:172.21.15.45:6800/2012105536,v1:172.21.15.45:6801/2012105536] conn(0x7fc1b4069030 0x7fc1b406b4e0 unknown :-1 s=CLOSED pgs=21 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:34:50.442 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:34:50.438+0000 7fc1d03e1700 1 --2- 172.21.15.45:0/794869141 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fc1c81028f0 0x7fc1c8112ca0 unknown :-1 s=CLOSED pgs=150 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:34:50.442 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:34:50.438+0000 7fc1d03e1700 1 --2- 172.21.15.45:0/794869141 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fc1c81131e0 0x7fc1c810cb40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:34:50.442 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:34:50.438+0000 7fc1d03e1700 1 -- 172.21.15.45:0/794869141 >> 172.21.15.45:0/794869141 conn(0x7fc1c80715e0 msgr2=0x7fc1c8103f60 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-16T21:34:50.442 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:34:50.438+0000 7fc1d03e1700 1 -- 172.21.15.45:0/794869141 shutdown_connections 2023-12-16T21:34:50.442 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:34:50.438+0000 7fc1d03e1700 1 -- 172.21.15.45:0/794869141 wait complete. 2023-12-16T21:34:50.830 INFO:tasks.cephadm:Distributing (final) config and client.admin keyring... 2023-12-16T21:34:50.830 DEBUG:teuthology.orchestra.run.smithi045:> set -ex 2023-12-16T21:34:50.830 DEBUG:teuthology.orchestra.run.smithi045:> sudo dd of=/etc/ceph/ceph.conf 2023-12-16T21:34:50.863 DEBUG:teuthology.orchestra.run.smithi045:> set -ex 2023-12-16T21:34:50.863 DEBUG:teuthology.orchestra.run.smithi045:> sudo dd of=/etc/ceph/ceph.client.admin.keyring 2023-12-16T21:34:50.935 DEBUG:teuthology.orchestra.run.smithi169:> set -ex 2023-12-16T21:34:50.935 DEBUG:teuthology.orchestra.run.smithi169:> sudo dd of=/etc/ceph/ceph.conf 2023-12-16T21:34:50.966 DEBUG:teuthology.orchestra.run.smithi169:> set -ex 2023-12-16T21:34:50.966 DEBUG:teuthology.orchestra.run.smithi169:> sudo dd of=/etc/ceph/ceph.client.admin.keyring 2023-12-16T21:34:51.033 INFO:tasks.cephadm:Deploying OSDs... 2023-12-16T21:34:51.033 DEBUG:teuthology.orchestra.run.smithi045:> set -ex 2023-12-16T21:34:51.033 DEBUG:teuthology.orchestra.run.smithi045:> dd if=/scratch_devs of=/dev/stdout 2023-12-16T21:34:51.049 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2023-12-16T21:34:51.049 DEBUG:teuthology.orchestra.run.smithi045:> stat /dev/vg_nvme/lv_1 2023-12-16T21:34:51.104 INFO:teuthology.orchestra.run.smithi045.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2023-12-16T21:34:51.104 INFO:teuthology.orchestra.run.smithi045.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2023-12-16T21:34:51.104 INFO:teuthology.orchestra.run.smithi045.stdout:Device: 5h/5d Inode: 664 Links: 1 2023-12-16T21:34:51.105 INFO:teuthology.orchestra.run.smithi045.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2023-12-16T21:34:51.105 INFO:teuthology.orchestra.run.smithi045.stdout:Context: system_u:object_r:device_t:s0 2023-12-16T21:34:51.105 INFO:teuthology.orchestra.run.smithi045.stdout:Access: 2023-12-16 21:34:20.315449830 +0000 2023-12-16T21:34:51.105 INFO:teuthology.orchestra.run.smithi045.stdout:Modify: 2023-12-16 21:29:07.041063628 +0000 2023-12-16T21:34:51.105 INFO:teuthology.orchestra.run.smithi045.stdout:Change: 2023-12-16 21:29:07.041063628 +0000 2023-12-16T21:34:51.105 INFO:teuthology.orchestra.run.smithi045.stdout: Birth: 2023-12-16 21:29:07.041063628 +0000 2023-12-16T21:34:51.105 DEBUG:teuthology.orchestra.run.smithi045:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2023-12-16T21:34:51.178 INFO:teuthology.orchestra.run.smithi045.stderr:1+0 records in 2023-12-16T21:34:51.178 INFO:teuthology.orchestra.run.smithi045.stderr:1+0 records out 2023-12-16T21:34:51.178 INFO:teuthology.orchestra.run.smithi045.stderr:512 bytes copied, 0.000109345 s, 4.7 MB/s 2023-12-16T21:34:51.179 DEBUG:teuthology.orchestra.run.smithi045:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2023-12-16T21:34:51.234 DEBUG:teuthology.orchestra.run.smithi045:> stat /dev/vg_nvme/lv_2 2023-12-16T21:34:51.290 INFO:teuthology.orchestra.run.smithi045.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2023-12-16T21:34:51.290 INFO:teuthology.orchestra.run.smithi045.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2023-12-16T21:34:51.290 INFO:teuthology.orchestra.run.smithi045.stdout:Device: 5h/5d Inode: 681 Links: 1 2023-12-16T21:34:51.290 INFO:teuthology.orchestra.run.smithi045.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2023-12-16T21:34:51.290 INFO:teuthology.orchestra.run.smithi045.stdout:Context: system_u:object_r:device_t:s0 2023-12-16T21:34:51.290 INFO:teuthology.orchestra.run.smithi045.stdout:Access: 2023-12-16 21:34:20.315449830 +0000 2023-12-16T21:34:51.290 INFO:teuthology.orchestra.run.smithi045.stdout:Modify: 2023-12-16 21:29:07.564055393 +0000 2023-12-16T21:34:51.290 INFO:teuthology.orchestra.run.smithi045.stdout:Change: 2023-12-16 21:29:07.564055393 +0000 2023-12-16T21:34:51.291 INFO:teuthology.orchestra.run.smithi045.stdout: Birth: 2023-12-16 21:29:07.564055393 +0000 2023-12-16T21:34:51.291 DEBUG:teuthology.orchestra.run.smithi045:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2023-12-16T21:34:51.305 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Dec 16 21:34:51 smithi045 ceph-mon[80763]: from='mgr.14214 172.21.15.45:0/1227909437' entity='mgr.smithi045.nhtmwc' 2023-12-16T21:34:51.305 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Dec 16 21:34:51 smithi045 ceph-mon[80763]: from='mgr.14214 172.21.15.45:0/1227909437' entity='mgr.smithi045.nhtmwc' 2023-12-16T21:34:51.305 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Dec 16 21:34:51 smithi045 ceph-mon[80763]: from='mgr.14214 172.21.15.45:0/1227909437' entity='mgr.smithi045.nhtmwc' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-12-16T21:34:51.305 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Dec 16 21:34:51 smithi045 ceph-mon[80763]: from='mgr.14214 172.21.15.45:0/1227909437' entity='mgr.smithi045.nhtmwc' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-12-16T21:34:51.305 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Dec 16 21:34:51 smithi045 ceph-mon[80763]: Updating smithi045:/etc/ceph/ceph.conf 2023-12-16T21:34:51.305 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Dec 16 21:34:51 smithi045 ceph-mon[80763]: Updating smithi169:/etc/ceph/ceph.conf 2023-12-16T21:34:51.305 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Dec 16 21:34:51 smithi045 ceph-mon[80763]: from='client.? 172.21.15.45:0/794869141' entity='client.admin' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-12-16T21:34:51.305 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Dec 16 21:34:51 smithi045 ceph-mon[80763]: Updating smithi169:/var/lib/ceph/88ff363c-9c5a-11ee-95a5-87774f69a715/config/ceph.conf 2023-12-16T21:34:51.306 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Dec 16 21:34:51 smithi045 ceph-mon[80763]: Updating smithi045:/var/lib/ceph/88ff363c-9c5a-11ee-95a5-87774f69a715/config/ceph.conf 2023-12-16T21:34:51.306 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Dec 16 21:34:51 smithi045 ceph-mon[80763]: pgmap v4: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-16T21:34:51.306 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Dec 16 21:34:51 smithi045 ceph-mon[80763]: from='mgr.14214 172.21.15.45:0/1227909437' entity='mgr.smithi045.nhtmwc' 2023-12-16T21:34:51.306 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Dec 16 21:34:51 smithi045 ceph-mon[80763]: from='mgr.14214 172.21.15.45:0/1227909437' entity='mgr.smithi045.nhtmwc' 2023-12-16T21:34:51.306 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Dec 16 21:34:51 smithi045 ceph-mon[80763]: from='mgr.14214 172.21.15.45:0/1227909437' entity='mgr.smithi045.nhtmwc' 2023-12-16T21:34:51.306 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Dec 16 21:34:51 smithi045 ceph-mon[80763]: from='mgr.14214 172.21.15.45:0/1227909437' entity='mgr.smithi045.nhtmwc' 2023-12-16T21:34:51.306 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Dec 16 21:34:51 smithi045 ceph-mon[80763]: from='mgr.14214 172.21.15.45:0/1227909437' entity='mgr.smithi045.nhtmwc' 2023-12-16T21:34:51.306 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Dec 16 21:34:51 smithi045 ceph-mon[80763]: from='mgr.14214 172.21.15.45:0/1227909437' entity='mgr.smithi045.nhtmwc' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2023-12-16T21:34:51.306 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Dec 16 21:34:51 smithi045 ceph-mon[80763]: from='mgr.14214 172.21.15.45:0/1227909437' entity='mgr.smithi045.nhtmwc' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2023-12-16T21:34:51.306 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Dec 16 21:34:51 smithi045 ceph-mon[80763]: from='mgr.14214 172.21.15.45:0/1227909437' entity='mgr.smithi045.nhtmwc' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-12-16T21:34:51.336 INFO:teuthology.orchestra.run.smithi045.stderr:1+0 records in 2023-12-16T21:34:51.336 INFO:teuthology.orchestra.run.smithi045.stderr:1+0 records out 2023-12-16T21:34:51.337 INFO:teuthology.orchestra.run.smithi045.stderr:512 bytes copied, 0.000102489 s, 5.0 MB/s 2023-12-16T21:34:51.338 DEBUG:teuthology.orchestra.run.smithi045:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2023-12-16T21:34:51.394 DEBUG:teuthology.orchestra.run.smithi045:> stat /dev/vg_nvme/lv_3 2023-12-16T21:34:51.450 INFO:teuthology.orchestra.run.smithi045.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2023-12-16T21:34:51.451 INFO:teuthology.orchestra.run.smithi045.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2023-12-16T21:34:51.451 INFO:teuthology.orchestra.run.smithi045.stdout:Device: 5h/5d Inode: 703 Links: 1 2023-12-16T21:34:51.451 INFO:teuthology.orchestra.run.smithi045.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2023-12-16T21:34:51.451 INFO:teuthology.orchestra.run.smithi045.stdout:Context: system_u:object_r:device_t:s0 2023-12-16T21:34:51.451 INFO:teuthology.orchestra.run.smithi045.stdout:Access: 2023-12-16 21:34:20.316449814 +0000 2023-12-16T21:34:51.451 INFO:teuthology.orchestra.run.smithi045.stdout:Modify: 2023-12-16 21:29:08.060047584 +0000 2023-12-16T21:34:51.451 INFO:teuthology.orchestra.run.smithi045.stdout:Change: 2023-12-16 21:29:08.060047584 +0000 2023-12-16T21:34:51.451 INFO:teuthology.orchestra.run.smithi045.stdout: Birth: 2023-12-16 21:29:08.060047584 +0000 2023-12-16T21:34:51.451 DEBUG:teuthology.orchestra.run.smithi045:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2023-12-16T21:34:51.521 INFO:teuthology.orchestra.run.smithi045.stderr:1+0 records in 2023-12-16T21:34:51.521 INFO:teuthology.orchestra.run.smithi045.stderr:1+0 records out 2023-12-16T21:34:51.521 INFO:teuthology.orchestra.run.smithi045.stderr:512 bytes copied, 8.7436e-05 s, 5.9 MB/s 2023-12-16T21:34:51.523 DEBUG:teuthology.orchestra.run.smithi045:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2023-12-16T21:34:51.579 DEBUG:teuthology.orchestra.run.smithi045:> stat /dev/vg_nvme/lv_4 2023-12-16T21:34:51.633 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Dec 16 21:34:51 smithi169 ceph-mon[88350]: from='mgr.14214 172.21.15.45:0/1227909437' entity='mgr.smithi045.nhtmwc' 2023-12-16T21:34:51.634 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Dec 16 21:34:51 smithi169 ceph-mon[88350]: from='mgr.14214 172.21.15.45:0/1227909437' entity='mgr.smithi045.nhtmwc' 2023-12-16T21:34:51.634 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Dec 16 21:34:51 smithi169 ceph-mon[88350]: from='mgr.14214 172.21.15.45:0/1227909437' entity='mgr.smithi045.nhtmwc' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-12-16T21:34:51.634 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Dec 16 21:34:51 smithi169 ceph-mon[88350]: from='mgr.14214 172.21.15.45:0/1227909437' entity='mgr.smithi045.nhtmwc' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-12-16T21:34:51.634 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Dec 16 21:34:51 smithi169 ceph-mon[88350]: Updating smithi045:/etc/ceph/ceph.conf 2023-12-16T21:34:51.634 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Dec 16 21:34:51 smithi169 ceph-mon[88350]: Updating smithi169:/etc/ceph/ceph.conf 2023-12-16T21:34:51.634 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Dec 16 21:34:51 smithi169 ceph-mon[88350]: from='client.? 172.21.15.45:0/794869141' entity='client.admin' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-12-16T21:34:51.634 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Dec 16 21:34:51 smithi169 ceph-mon[88350]: Updating smithi169:/var/lib/ceph/88ff363c-9c5a-11ee-95a5-87774f69a715/config/ceph.conf 2023-12-16T21:34:51.634 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Dec 16 21:34:51 smithi169 ceph-mon[88350]: Updating smithi045:/var/lib/ceph/88ff363c-9c5a-11ee-95a5-87774f69a715/config/ceph.conf 2023-12-16T21:34:51.634 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Dec 16 21:34:51 smithi169 ceph-mon[88350]: pgmap v4: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-16T21:34:51.634 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Dec 16 21:34:51 smithi169 ceph-mon[88350]: from='mgr.14214 172.21.15.45:0/1227909437' entity='mgr.smithi045.nhtmwc' 2023-12-16T21:34:51.634 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Dec 16 21:34:51 smithi169 ceph-mon[88350]: from='mgr.14214 172.21.15.45:0/1227909437' entity='mgr.smithi045.nhtmwc' 2023-12-16T21:34:51.634 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Dec 16 21:34:51 smithi169 ceph-mon[88350]: from='mgr.14214 172.21.15.45:0/1227909437' entity='mgr.smithi045.nhtmwc' 2023-12-16T21:34:51.634 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Dec 16 21:34:51 smithi169 ceph-mon[88350]: from='mgr.14214 172.21.15.45:0/1227909437' entity='mgr.smithi045.nhtmwc' 2023-12-16T21:34:51.634 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Dec 16 21:34:51 smithi169 ceph-mon[88350]: from='mgr.14214 172.21.15.45:0/1227909437' entity='mgr.smithi045.nhtmwc' 2023-12-16T21:34:51.634 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Dec 16 21:34:51 smithi169 ceph-mon[88350]: from='mgr.14214 172.21.15.45:0/1227909437' entity='mgr.smithi045.nhtmwc' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2023-12-16T21:34:51.635 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Dec 16 21:34:51 smithi169 ceph-mon[88350]: from='mgr.14214 172.21.15.45:0/1227909437' entity='mgr.smithi045.nhtmwc' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2023-12-16T21:34:51.635 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Dec 16 21:34:51 smithi169 ceph-mon[88350]: from='mgr.14214 172.21.15.45:0/1227909437' entity='mgr.smithi045.nhtmwc' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-12-16T21:34:51.635 INFO:teuthology.orchestra.run.smithi045.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2023-12-16T21:34:51.635 INFO:teuthology.orchestra.run.smithi045.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2023-12-16T21:34:51.636 INFO:teuthology.orchestra.run.smithi045.stdout:Device: 5h/5d Inode: 719 Links: 1 2023-12-16T21:34:51.636 INFO:teuthology.orchestra.run.smithi045.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2023-12-16T21:34:51.636 INFO:teuthology.orchestra.run.smithi045.stdout:Context: system_u:object_r:device_t:s0 2023-12-16T21:34:51.636 INFO:teuthology.orchestra.run.smithi045.stdout:Access: 2023-12-16 21:34:20.316449814 +0000 2023-12-16T21:34:51.636 INFO:teuthology.orchestra.run.smithi045.stdout:Modify: 2023-12-16 21:29:08.576039459 +0000 2023-12-16T21:34:51.636 INFO:teuthology.orchestra.run.smithi045.stdout:Change: 2023-12-16 21:29:08.576039459 +0000 2023-12-16T21:34:51.636 INFO:teuthology.orchestra.run.smithi045.stdout: Birth: 2023-12-16 21:29:08.576039459 +0000 2023-12-16T21:34:51.636 DEBUG:teuthology.orchestra.run.smithi045:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2023-12-16T21:34:51.706 INFO:teuthology.orchestra.run.smithi045.stderr:1+0 records in 2023-12-16T21:34:51.707 INFO:teuthology.orchestra.run.smithi045.stderr:1+0 records out 2023-12-16T21:34:51.707 INFO:teuthology.orchestra.run.smithi045.stderr:512 bytes copied, 0.000105797 s, 4.8 MB/s 2023-12-16T21:34:51.708 DEBUG:teuthology.orchestra.run.smithi045:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2023-12-16T21:34:51.765 DEBUG:teuthology.orchestra.run.smithi169:> set -ex 2023-12-16T21:34:51.765 DEBUG:teuthology.orchestra.run.smithi169:> dd if=/scratch_devs of=/dev/stdout 2023-12-16T21:34:51.781 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2023-12-16T21:34:51.782 DEBUG:teuthology.orchestra.run.smithi169:> stat /dev/vg_nvme/lv_1 2023-12-16T21:34:51.838 INFO:teuthology.orchestra.run.smithi169.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2023-12-16T21:34:51.838 INFO:teuthology.orchestra.run.smithi169.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2023-12-16T21:34:51.838 INFO:teuthology.orchestra.run.smithi169.stdout:Device: 5h/5d Inode: 663 Links: 1 2023-12-16T21:34:51.838 INFO:teuthology.orchestra.run.smithi169.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2023-12-16T21:34:51.838 INFO:teuthology.orchestra.run.smithi169.stdout:Context: system_u:object_r:device_t:s0 2023-12-16T21:34:51.838 INFO:teuthology.orchestra.run.smithi169.stdout:Access: 2023-12-16 21:34:42.920918047 +0000 2023-12-16T21:34:51.838 INFO:teuthology.orchestra.run.smithi169.stdout:Modify: 2023-12-16 21:28:56.661669170 +0000 2023-12-16T21:34:51.838 INFO:teuthology.orchestra.run.smithi169.stdout:Change: 2023-12-16 21:28:56.661669170 +0000 2023-12-16T21:34:51.838 INFO:teuthology.orchestra.run.smithi169.stdout: Birth: 2023-12-16 21:28:56.661669170 +0000 2023-12-16T21:34:51.839 DEBUG:teuthology.orchestra.run.smithi169:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2023-12-16T21:34:51.905 INFO:teuthology.orchestra.run.smithi169.stderr:1+0 records in 2023-12-16T21:34:51.905 INFO:teuthology.orchestra.run.smithi169.stderr:1+0 records out 2023-12-16T21:34:51.905 INFO:teuthology.orchestra.run.smithi169.stderr:512 bytes copied, 0.000243186 s, 2.1 MB/s 2023-12-16T21:34:51.906 DEBUG:teuthology.orchestra.run.smithi169:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2023-12-16T21:34:51.963 DEBUG:teuthology.orchestra.run.smithi169:> stat /dev/vg_nvme/lv_2 2023-12-16T21:34:52.019 INFO:teuthology.orchestra.run.smithi169.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2023-12-16T21:34:52.019 INFO:teuthology.orchestra.run.smithi169.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2023-12-16T21:34:52.019 INFO:teuthology.orchestra.run.smithi169.stdout:Device: 5h/5d Inode: 681 Links: 1 2023-12-16T21:34:52.019 INFO:teuthology.orchestra.run.smithi169.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2023-12-16T21:34:52.019 INFO:teuthology.orchestra.run.smithi169.stdout:Context: system_u:object_r:device_t:s0 2023-12-16T21:34:52.019 INFO:teuthology.orchestra.run.smithi169.stdout:Access: 2023-12-16 21:34:42.920918047 +0000 2023-12-16T21:34:52.019 INFO:teuthology.orchestra.run.smithi169.stdout:Modify: 2023-12-16 21:28:57.159658712 +0000 2023-12-16T21:34:52.020 INFO:teuthology.orchestra.run.smithi169.stdout:Change: 2023-12-16 21:28:57.159658712 +0000 2023-12-16T21:34:52.020 INFO:teuthology.orchestra.run.smithi169.stdout: Birth: 2023-12-16 21:28:57.159658712 +0000 2023-12-16T21:34:52.020 DEBUG:teuthology.orchestra.run.smithi169:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2023-12-16T21:34:52.085 INFO:teuthology.orchestra.run.smithi169.stderr:1+0 records in 2023-12-16T21:34:52.085 INFO:teuthology.orchestra.run.smithi169.stderr:1+0 records out 2023-12-16T21:34:52.085 INFO:teuthology.orchestra.run.smithi169.stderr:512 bytes copied, 0.000188519 s, 2.7 MB/s 2023-12-16T21:34:52.086 DEBUG:teuthology.orchestra.run.smithi169:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2023-12-16T21:34:52.142 DEBUG:teuthology.orchestra.run.smithi169:> stat /dev/vg_nvme/lv_3 2023-12-16T21:34:52.196 INFO:teuthology.orchestra.run.smithi169.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2023-12-16T21:34:52.197 INFO:teuthology.orchestra.run.smithi169.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2023-12-16T21:34:52.197 INFO:teuthology.orchestra.run.smithi169.stdout:Device: 5h/5d Inode: 700 Links: 1 2023-12-16T21:34:52.197 INFO:teuthology.orchestra.run.smithi169.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2023-12-16T21:34:52.197 INFO:teuthology.orchestra.run.smithi169.stdout:Context: system_u:object_r:device_t:s0 2023-12-16T21:34:52.197 INFO:teuthology.orchestra.run.smithi169.stdout:Access: 2023-12-16 21:34:42.921918026 +0000 2023-12-16T21:34:52.197 INFO:teuthology.orchestra.run.smithi169.stdout:Modify: 2023-12-16 21:28:57.649648422 +0000 2023-12-16T21:34:52.197 INFO:teuthology.orchestra.run.smithi169.stdout:Change: 2023-12-16 21:28:57.649648422 +0000 2023-12-16T21:34:52.197 INFO:teuthology.orchestra.run.smithi169.stdout: Birth: 2023-12-16 21:28:57.649648422 +0000 2023-12-16T21:34:52.197 DEBUG:teuthology.orchestra.run.smithi169:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2023-12-16T21:34:52.262 INFO:teuthology.orchestra.run.smithi169.stderr:1+0 records in 2023-12-16T21:34:52.263 INFO:teuthology.orchestra.run.smithi169.stderr:1+0 records out 2023-12-16T21:34:52.263 INFO:teuthology.orchestra.run.smithi169.stderr:512 bytes copied, 0.000205385 s, 2.5 MB/s 2023-12-16T21:34:52.264 DEBUG:teuthology.orchestra.run.smithi169:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2023-12-16T21:34:52.319 DEBUG:teuthology.orchestra.run.smithi169:> stat /dev/vg_nvme/lv_4 2023-12-16T21:34:52.374 INFO:teuthology.orchestra.run.smithi169.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2023-12-16T21:34:52.374 INFO:teuthology.orchestra.run.smithi169.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2023-12-16T21:34:52.375 INFO:teuthology.orchestra.run.smithi169.stdout:Device: 5h/5d Inode: 719 Links: 1 2023-12-16T21:34:52.375 INFO:teuthology.orchestra.run.smithi169.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2023-12-16T21:34:52.375 INFO:teuthology.orchestra.run.smithi169.stdout:Context: system_u:object_r:device_t:s0 2023-12-16T21:34:52.375 INFO:teuthology.orchestra.run.smithi169.stdout:Access: 2023-12-16 21:34:42.921918026 +0000 2023-12-16T21:34:52.375 INFO:teuthology.orchestra.run.smithi169.stdout:Modify: 2023-12-16 21:28:58.152637860 +0000 2023-12-16T21:34:52.375 INFO:teuthology.orchestra.run.smithi169.stdout:Change: 2023-12-16 21:28:58.152637860 +0000 2023-12-16T21:34:52.375 INFO:teuthology.orchestra.run.smithi169.stdout: Birth: 2023-12-16 21:28:58.152637860 +0000 2023-12-16T21:34:52.375 DEBUG:teuthology.orchestra.run.smithi169:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2023-12-16T21:34:52.440 INFO:teuthology.orchestra.run.smithi169.stderr:1+0 records in 2023-12-16T21:34:52.441 INFO:teuthology.orchestra.run.smithi169.stderr:1+0 records out 2023-12-16T21:34:52.441 INFO:teuthology.orchestra.run.smithi169.stderr:512 bytes copied, 0.000183064 s, 2.8 MB/s 2023-12-16T21:34:52.442 DEBUG:teuthology.orchestra.run.smithi169:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2023-12-16T21:34:52.497 INFO:tasks.cephadm:Deploying osd.0 on smithi045 with /dev/vg_nvme/lv_4... 2023-12-16T21:34:52.497 DEBUG:teuthology.orchestra.run.smithi045:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 88ff363c-9c5a-11ee-95a5-87774f69a715 -- ceph-volume lvm zap /dev/vg_nvme/lv_4 2023-12-16T21:34:52.769 INFO:teuthology.orchestra.run.smithi045.stderr:Inferring config /var/lib/ceph/88ff363c-9c5a-11ee-95a5-87774f69a715/mon.smithi045/config 2023-12-16T21:34:52.948 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Dec 16 21:34:52 smithi045 ceph-mon[80763]: Reconfiguring mon.smithi045 (unknown last config time)... 2023-12-16T21:34:52.949 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Dec 16 21:34:52 smithi045 ceph-mon[80763]: Reconfiguring daemon mon.smithi045 on smithi045 2023-12-16T21:34:52.949 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Dec 16 21:34:52 smithi045 ceph-mon[80763]: from='mgr.14214 172.21.15.45:0/1227909437' entity='mgr.smithi045.nhtmwc' 2023-12-16T21:34:52.949 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Dec 16 21:34:52 smithi045 ceph-mon[80763]: from='mgr.14214 172.21.15.45:0/1227909437' entity='mgr.smithi045.nhtmwc' 2023-12-16T21:34:52.949 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Dec 16 21:34:52 smithi045 ceph-mon[80763]: Reconfiguring mgr.smithi045.nhtmwc (unknown last config time)... 2023-12-16T21:34:52.949 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Dec 16 21:34:52 smithi045 ceph-mon[80763]: from='mgr.14214 172.21.15.45:0/1227909437' entity='mgr.smithi045.nhtmwc' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.smithi045.nhtmwc", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2023-12-16T21:34:52.949 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Dec 16 21:34:52 smithi045 ceph-mon[80763]: from='mgr.14214 172.21.15.45:0/1227909437' entity='mgr.smithi045.nhtmwc' cmd=[{"prefix": "mgr services"}]: dispatch 2023-12-16T21:34:52.949 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Dec 16 21:34:52 smithi045 ceph-mon[80763]: from='mgr.14214 172.21.15.45:0/1227909437' entity='mgr.smithi045.nhtmwc' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-12-16T21:34:52.949 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Dec 16 21:34:52 smithi045 ceph-mon[80763]: Reconfiguring daemon mgr.smithi045.nhtmwc on smithi045 2023-12-16T21:34:52.949 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Dec 16 21:34:52 smithi045 ceph-mon[80763]: from='mgr.14214 172.21.15.45:0/1227909437' entity='mgr.smithi045.nhtmwc' 2023-12-16T21:34:52.949 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Dec 16 21:34:52 smithi045 ceph-mon[80763]: from='mgr.14214 172.21.15.45:0/1227909437' entity='mgr.smithi045.nhtmwc' 2023-12-16T21:34:52.949 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Dec 16 21:34:52 smithi045 ceph-mon[80763]: from='mgr.14214 172.21.15.45:0/1227909437' entity='mgr.smithi045.nhtmwc' cmd=[{"prefix": "auth get-or-create", "entity": "client.crash.smithi045", "caps": ["mon", "profile crash", "mgr", "profile crash"]}]: dispatch 2023-12-16T21:34:52.950 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Dec 16 21:34:52 smithi045 ceph-mon[80763]: from='mgr.14214 172.21.15.45:0/1227909437' entity='mgr.smithi045.nhtmwc' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-12-16T21:34:53.133 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Dec 16 21:34:52 smithi169 ceph-mon[88350]: Reconfiguring mon.smithi045 (unknown last config time)... 2023-12-16T21:34:53.133 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Dec 16 21:34:52 smithi169 ceph-mon[88350]: Reconfiguring daemon mon.smithi045 on smithi045 2023-12-16T21:34:53.133 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Dec 16 21:34:52 smithi169 ceph-mon[88350]: from='mgr.14214 172.21.15.45:0/1227909437' entity='mgr.smithi045.nhtmwc' 2023-12-16T21:34:53.133 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Dec 16 21:34:52 smithi169 ceph-mon[88350]: from='mgr.14214 172.21.15.45:0/1227909437' entity='mgr.smithi045.nhtmwc' 2023-12-16T21:34:53.134 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Dec 16 21:34:52 smithi169 ceph-mon[88350]: Reconfiguring mgr.smithi045.nhtmwc (unknown last config time)... 2023-12-16T21:34:53.134 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Dec 16 21:34:52 smithi169 ceph-mon[88350]: from='mgr.14214 172.21.15.45:0/1227909437' entity='mgr.smithi045.nhtmwc' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.smithi045.nhtmwc", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2023-12-16T21:34:53.134 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Dec 16 21:34:52 smithi169 ceph-mon[88350]: from='mgr.14214 172.21.15.45:0/1227909437' entity='mgr.smithi045.nhtmwc' cmd=[{"prefix": "mgr services"}]: dispatch 2023-12-16T21:34:53.134 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Dec 16 21:34:52 smithi169 ceph-mon[88350]: from='mgr.14214 172.21.15.45:0/1227909437' entity='mgr.smithi045.nhtmwc' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-12-16T21:34:53.134 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Dec 16 21:34:52 smithi169 ceph-mon[88350]: Reconfiguring daemon mgr.smithi045.nhtmwc on smithi045 2023-12-16T21:34:53.134 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Dec 16 21:34:52 smithi169 ceph-mon[88350]: from='mgr.14214 172.21.15.45:0/1227909437' entity='mgr.smithi045.nhtmwc' 2023-12-16T21:34:53.134 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Dec 16 21:34:52 smithi169 ceph-mon[88350]: from='mgr.14214 172.21.15.45:0/1227909437' entity='mgr.smithi045.nhtmwc' 2023-12-16T21:34:53.134 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Dec 16 21:34:52 smithi169 ceph-mon[88350]: from='mgr.14214 172.21.15.45:0/1227909437' entity='mgr.smithi045.nhtmwc' cmd=[{"prefix": "auth get-or-create", "entity": "client.crash.smithi045", "caps": ["mon", "profile crash", "mgr", "profile crash"]}]: dispatch 2023-12-16T21:34:53.134 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Dec 16 21:34:52 smithi169 ceph-mon[88350]: from='mgr.14214 172.21.15.45:0/1227909437' entity='mgr.smithi045.nhtmwc' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-12-16T21:34:53.748 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Dec 16 21:34:53 smithi045 ceph-mon[80763]: Reconfiguring crash.smithi045 (monmap changed)... 2023-12-16T21:34:53.748 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Dec 16 21:34:53 smithi045 ceph-mon[80763]: Reconfiguring daemon crash.smithi045 on smithi045 2023-12-16T21:34:53.748 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Dec 16 21:34:53 smithi045 ceph-mon[80763]: pgmap v5: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-16T21:34:53.748 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Dec 16 21:34:53 smithi045 ceph-mon[80763]: from='mgr.14214 172.21.15.45:0/1227909437' entity='mgr.smithi045.nhtmwc' 2023-12-16T21:34:54.133 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Dec 16 21:34:53 smithi169 ceph-mon[88350]: Reconfiguring crash.smithi045 (monmap changed)... 2023-12-16T21:34:54.133 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Dec 16 21:34:53 smithi169 ceph-mon[88350]: Reconfiguring daemon crash.smithi045 on smithi045 2023-12-16T21:34:54.133 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Dec 16 21:34:53 smithi169 ceph-mon[88350]: pgmap v5: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-16T21:34:54.133 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Dec 16 21:34:53 smithi169 ceph-mon[88350]: from='mgr.14214 172.21.15.45:0/1227909437' entity='mgr.smithi045.nhtmwc' 2023-12-16T21:34:54.263 INFO:teuthology.orchestra.run.smithi045.stderr:--> Zapping: /dev/vg_nvme/lv_4 2023-12-16T21:34:54.263 INFO:teuthology.orchestra.run.smithi045.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_4 bs=1M count=10 conv=fsync 2023-12-16T21:34:54.263 INFO:teuthology.orchestra.run.smithi045.stderr: stderr: 10+0 records in 2023-12-16T21:34:54.263 INFO:teuthology.orchestra.run.smithi045.stderr:10+0 records out 2023-12-16T21:34:54.263 INFO:teuthology.orchestra.run.smithi045.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.00864469 s, 1.2 GB/s 2023-12-16T21:34:54.264 INFO:teuthology.orchestra.run.smithi045.stderr:--> Zapping successful for: 2023-12-16T21:34:54.891 DEBUG:teuthology.orchestra.run.smithi045:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 88ff363c-9c5a-11ee-95a5-87774f69a715 -- ceph orch daemon add osd smithi045:vg_nvme/lv_4 2023-12-16T21:34:54.950 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Dec 16 21:34:54 smithi045 ceph-mon[80763]: from='mgr.14214 172.21.15.45:0/1227909437' entity='mgr.smithi045.nhtmwc' 2023-12-16T21:34:54.951 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Dec 16 21:34:54 smithi045 ceph-mon[80763]: from='mgr.14214 172.21.15.45:0/1227909437' entity='mgr.smithi045.nhtmwc' 2023-12-16T21:34:54.951 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Dec 16 21:34:54 smithi045 ceph-mon[80763]: Reconfiguring alertmanager.smithi045 (dependencies changed)... 2023-12-16T21:34:55.133 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Dec 16 21:34:54 smithi169 ceph-mon[88350]: from='mgr.14214 172.21.15.45:0/1227909437' entity='mgr.smithi045.nhtmwc' 2023-12-16T21:34:55.134 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Dec 16 21:34:54 smithi169 ceph-mon[88350]: from='mgr.14214 172.21.15.45:0/1227909437' entity='mgr.smithi045.nhtmwc' 2023-12-16T21:34:55.134 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Dec 16 21:34:54 smithi169 ceph-mon[88350]: Reconfiguring alertmanager.smithi045 (dependencies changed)... 2023-12-16T21:34:55.419 INFO:teuthology.orchestra.run.smithi045.stderr:Inferring config /var/lib/ceph/88ff363c-9c5a-11ee-95a5-87774f69a715/mon.smithi045/config 2023-12-16T21:34:55.856 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Dec 16 21:34:55 smithi045 ceph-mon[80763]: Reconfiguring daemon alertmanager.smithi045 on smithi045 2023-12-16T21:34:55.856 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Dec 16 21:34:55 smithi045 ceph-mon[80763]: pgmap v6: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-16T21:34:56.133 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Dec 16 21:34:55 smithi169 ceph-mon[88350]: Reconfiguring daemon alertmanager.smithi045 on smithi045 2023-12-16T21:34:56.133 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Dec 16 21:34:55 smithi169 ceph-mon[88350]: pgmap v6: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-16T21:34:56.179 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:34:56.175+0000 7f6292f83700 1 -- 172.21.15.45:0/1604128485 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f628c0ffd70 msgr2=0x7f628c1021d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:34:56.179 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:34:56.175+0000 7f6292f83700 1 --2- 172.21.15.45:0/1604128485 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f628c0ffd70 0x7f628c1021d0 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7f6278004940 tx=0x7f6278009e30 comp rx=0 tx=0).stop 2023-12-16T21:34:56.179 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:34:56.175+0000 7f6292f83700 1 -- 172.21.15.45:0/1604128485 shutdown_connections 2023-12-16T21:34:56.179 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:34:56.175+0000 7f6292f83700 1 --2- 172.21.15.45:0/1604128485 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f628c0ffd70 0x7f628c1021d0 unknown :-1 s=CLOSED pgs=151 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:34:56.179 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:34:56.175+0000 7f6292f83700 1 --2- 172.21.15.45:0/1604128485 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f628c066920 0x7f628c0ff7a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:34:56.179 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:34:56.175+0000 7f6292f83700 1 -- 172.21.15.45:0/1604128485 >> 172.21.15.45:0/1604128485 conn(0x7f628c0f71b0 msgr2=0x7f628c0f95e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-16T21:34:56.179 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:34:56.175+0000 7f6292f83700 1 -- 172.21.15.45:0/1604128485 shutdown_connections 2023-12-16T21:34:56.179 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:34:56.176+0000 7f6292f83700 1 -- 172.21.15.45:0/1604128485 wait complete. 2023-12-16T21:34:56.179 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:34:56.176+0000 7f6292f83700 1 Processor -- start 2023-12-16T21:34:56.179 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:34:56.176+0000 7f6292f83700 1 -- start start 2023-12-16T21:34:56.179 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:34:56.176+0000 7f6292f83700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f628c066920 0x7f628c10d580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:34:56.180 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:34:56.176+0000 7f6292f83700 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f628c0ffd70 0x7f628c10dac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:34:56.180 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:34:56.176+0000 7f6292f83700 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f628c10e150 con 0x7f628c066920 2023-12-16T21:34:56.180 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:34:56.176+0000 7f6292f83700 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f628c10e990 con 0x7f628c0ffd70 2023-12-16T21:34:56.180 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:34:56.177+0000 7f6290d1f700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f628c066920 0x7f628c10d580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:34:56.180 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:34:56.177+0000 7f6290d1f700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f628c066920 0x7f628c10d580 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:38360/0 (socket says 172.21.15.45:38360) 2023-12-16T21:34:56.180 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:34:56.177+0000 7f6290d1f700 1 -- 172.21.15.45:0/3186521198 learned_addr learned my addr 172.21.15.45:0/3186521198 (peer_addr_for_me v2:172.21.15.45:0/0) 2023-12-16T21:34:56.180 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:34:56.177+0000 7f628bfff700 1 --2- 172.21.15.45:0/3186521198 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f628c0ffd70 0x7f628c10dac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:34:56.181 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:34:56.177+0000 7f6290d1f700 1 -- 172.21.15.45:0/3186521198 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f628c0ffd70 msgr2=0x7f628c10dac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:34:56.181 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:34:56.177+0000 7f6290d1f700 1 --2- 172.21.15.45:0/3186521198 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f628c0ffd70 0x7f628c10dac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:34:56.181 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:34:56.177+0000 7f6290d1f700 1 -- 172.21.15.45:0/3186521198 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6278005040 con 0x7f628c066920 2023-12-16T21:34:56.181 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:34:56.177+0000 7f6290d1f700 1 --2- 172.21.15.45:0/3186521198 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f628c066920 0x7f628c10d580 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7f628000b790 tx=0x7f6280009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:34:56.181 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:34:56.177+0000 7f62897fa700 1 -- 172.21.15.45:0/3186521198 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f628000f6f0 con 0x7f628c066920 2023-12-16T21:34:56.181 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:34:56.177+0000 7f62897fa700 1 -- 172.21.15.45:0/3186521198 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f628000fd00 con 0x7f628c066920 2023-12-16T21:34:56.181 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:34:56.177+0000 7f6292f83700 1 -- 172.21.15.45:0/3186521198 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f628c10ec70 con 0x7f628c066920 2023-12-16T21:34:56.182 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:34:56.177+0000 7f62897fa700 1 -- 172.21.15.45:0/3186521198 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f62800103a0 con 0x7f628c066920 2023-12-16T21:34:56.182 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:34:56.178+0000 7f6292f83700 1 -- 172.21.15.45:0/3186521198 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f628c10f1b0 con 0x7f628c066920 2023-12-16T21:34:56.182 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:34:56.179+0000 7f6292f83700 1 -- 172.21.15.45:0/3186521198 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f628c04bb80 con 0x7f628c066920 2023-12-16T21:34:56.182 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:34:56.179+0000 7f62897fa700 1 -- 172.21.15.45:0/3186521198 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 18) v1 ==== 86391+0+0 (secure 0 0 0) 0x7f6280010500 con 0x7f628c066920 2023-12-16T21:34:56.182 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:34:56.179+0000 7f62897fa700 1 --2- 172.21.15.45:0/3186521198 >> [v2:172.21.15.45:6800/2012105536,v1:172.21.15.45:6801/2012105536] conn(0x7f627c068fa0 0x7f627c06b450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:34:56.182 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:34:56.180+0000 7f62897fa700 1 -- 172.21.15.45:0/3186521198 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1366+0+0 (secure 0 0 0) 0x7f6280085960 con 0x7f628c066920 2023-12-16T21:34:56.182 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:34:56.180+0000 7f628bfff700 1 --2- 172.21.15.45:0/3186521198 >> [v2:172.21.15.45:6800/2012105536,v1:172.21.15.45:6801/2012105536] conn(0x7f627c068fa0 0x7f627c06b450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:34:56.183 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:34:56.180+0000 7f628bfff700 1 --2- 172.21.15.45:0/3186521198 >> [v2:172.21.15.45:6800/2012105536,v1:172.21.15.45:6801/2012105536] conn(0x7f627c068fa0 0x7f627c06b450 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7f627800b3b0 tx=0x7f6278009e30 comp rx=0 tx=0).ready entity=mgr.14214 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:34:56.187 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:34:56.184+0000 7f62897fa700 1 -- 172.21.15.45:0/3186521198 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+174570 (secure 0 0 0) 0x7f6280056a40 con 0x7f628c066920 2023-12-16T21:34:56.406 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:34:56.403+0000 7f6292f83700 1 -- 172.21.15.45:0/3186521198 --> [v2:172.21.15.45:6800/2012105536,v1:172.21.15.45:6801/2012105536] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi045:vg_nvme/lv_4", "target": ["mon-mgr", ""]}) v1 -- 0x7f628c100a10 con 0x7f627c068fa0 2023-12-16T21:34:57.316 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Dec 16 21:34:57 smithi045 ceph-mon[80763]: from='client.14246 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi045:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2023-12-16T21:34:57.316 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Dec 16 21:34:57 smithi045 ceph-mon[80763]: from='mgr.14214 172.21.15.45:0/1227909437' entity='mgr.smithi045.nhtmwc' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2023-12-16T21:34:57.316 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Dec 16 21:34:57 smithi045 ceph-mon[80763]: from='mgr.14214 172.21.15.45:0/1227909437' entity='mgr.smithi045.nhtmwc' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2023-12-16T21:34:57.316 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Dec 16 21:34:57 smithi045 ceph-mon[80763]: from='mgr.14214 172.21.15.45:0/1227909437' entity='mgr.smithi045.nhtmwc' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-12-16T21:34:57.316 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Dec 16 21:34:57 smithi045 ceph-mon[80763]: pgmap v7: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-16T21:34:57.317 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Dec 16 21:34:57 smithi045 ceph-mon[80763]: from='mgr.14214 172.21.15.45:0/1227909437' entity='mgr.smithi045.nhtmwc' 2023-12-16T21:34:57.317 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Dec 16 21:34:57 smithi045 ceph-mon[80763]: from='mgr.14214 172.21.15.45:0/1227909437' entity='mgr.smithi045.nhtmwc' 2023-12-16T21:34:57.633 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Dec 16 21:34:57 smithi169 ceph-mon[88350]: from='client.14246 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi045:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2023-12-16T21:34:57.633 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Dec 16 21:34:57 smithi169 ceph-mon[88350]: from='mgr.14214 172.21.15.45:0/1227909437' entity='mgr.smithi045.nhtmwc' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2023-12-16T21:34:57.633 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Dec 16 21:34:57 smithi169 ceph-mon[88350]: from='mgr.14214 172.21.15.45:0/1227909437' entity='mgr.smithi045.nhtmwc' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2023-12-16T21:34:57.634 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Dec 16 21:34:57 smithi169 ceph-mon[88350]: from='mgr.14214 172.21.15.45:0/1227909437' entity='mgr.smithi045.nhtmwc' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-12-16T21:34:57.634 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Dec 16 21:34:57 smithi169 ceph-mon[88350]: pgmap v7: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-16T21:34:57.634 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Dec 16 21:34:57 smithi169 ceph-mon[88350]: from='mgr.14214 172.21.15.45:0/1227909437' entity='mgr.smithi045.nhtmwc' 2023-12-16T21:34:57.634 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Dec 16 21:34:57 smithi169 ceph-mon[88350]: from='mgr.14214 172.21.15.45:0/1227909437' entity='mgr.smithi045.nhtmwc' 2023-12-16T21:34:58.566 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Dec 16 21:34:58 smithi045 ceph-mon[80763]: Reconfiguring grafana.smithi045 (dependencies changed)... 2023-12-16T21:34:58.566 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Dec 16 21:34:58 smithi045 ceph-mon[80763]: Reconfiguring daemon grafana.smithi045 on smithi045 2023-12-16T21:34:58.566 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Dec 16 21:34:58 smithi045 ceph-mon[80763]: from='client.? 172.21.15.45:0/3972331822' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "bc814a3e-c753-4948-b9eb-0d9d2d30904c"}]: dispatch 2023-12-16T21:34:58.566 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Dec 16 21:34:58 smithi045 ceph-mon[80763]: from='client.? 172.21.15.45:0/3972331822' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "bc814a3e-c753-4948-b9eb-0d9d2d30904c"}]': finished 2023-12-16T21:34:58.567 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Dec 16 21:34:58 smithi045 ceph-mon[80763]: osdmap e6: 1 total, 0 up, 1 in 2023-12-16T21:34:58.567 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Dec 16 21:34:58 smithi045 ceph-mon[80763]: from='mgr.14214 172.21.15.45:0/1227909437' entity='mgr.smithi045.nhtmwc' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2023-12-16T21:34:58.633 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Dec 16 21:34:58 smithi169 ceph-mon[88350]: Reconfiguring grafana.smithi045 (dependencies changed)... 2023-12-16T21:34:58.633 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Dec 16 21:34:58 smithi169 ceph-mon[88350]: Reconfiguring daemon grafana.smithi045 on smithi045 2023-12-16T21:34:58.633 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Dec 16 21:34:58 smithi169 ceph-mon[88350]: from='client.? 172.21.15.45:0/3972331822' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "bc814a3e-c753-4948-b9eb-0d9d2d30904c"}]: dispatch 2023-12-16T21:34:58.633 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Dec 16 21:34:58 smithi169 ceph-mon[88350]: from='client.? 172.21.15.45:0/3972331822' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "bc814a3e-c753-4948-b9eb-0d9d2d30904c"}]': finished 2023-12-16T21:34:58.633 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Dec 16 21:34:58 smithi169 ceph-mon[88350]: osdmap e6: 1 total, 0 up, 1 in 2023-12-16T21:34:58.634 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Dec 16 21:34:58 smithi169 ceph-mon[88350]: from='mgr.14214 172.21.15.45:0/1227909437' entity='mgr.smithi045.nhtmwc' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2023-12-16T21:34:59.568 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Dec 16 21:34:59 smithi045 ceph-mon[80763]: pgmap v9: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-16T21:34:59.568 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Dec 16 21:34:59 smithi045 ceph-mon[80763]: from='mgr.14214 172.21.15.45:0/1227909437' entity='mgr.smithi045.nhtmwc' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2023-12-16T21:34:59.569 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Dec 16 21:34:59 smithi045 ceph-mon[80763]: from='client.? 172.21.15.45:0/1586378786' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2023-12-16T21:34:59.633 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Dec 16 21:34:59 smithi169 ceph-mon[88350]: pgmap v9: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-16T21:34:59.633 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Dec 16 21:34:59 smithi169 ceph-mon[88350]: from='mgr.14214 172.21.15.45:0/1227909437' entity='mgr.smithi045.nhtmwc' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2023-12-16T21:34:59.633 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Dec 16 21:34:59 smithi169 ceph-mon[88350]: from='client.? 172.21.15.45:0/1586378786' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2023-12-16T21:35:02.007 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Dec 16 21:35:01 smithi045 ceph-mon[80763]: pgmap v10: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-16T21:35:02.133 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Dec 16 21:35:01 smithi169 ceph-mon[88350]: pgmap v10: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-16T21:35:03.905 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Dec 16 21:35:03 smithi045 ceph-mon[80763]: pgmap v11: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-16T21:35:04.133 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Dec 16 21:35:03 smithi169 ceph-mon[88350]: pgmap v11: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-16T21:35:05.688 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Dec 16 21:35:05 smithi045 ceph-mon[80763]: pgmap v12: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-16T21:35:05.688 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Dec 16 21:35:05 smithi045 ceph-mon[80763]: from='mgr.14214 172.21.15.45:0/1227909437' entity='mgr.smithi045.nhtmwc' 2023-12-16T21:35:05.689 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Dec 16 21:35:05 smithi045 ceph-mon[80763]: from='mgr.14214 172.21.15.45:0/1227909437' entity='mgr.smithi045.nhtmwc' 2023-12-16T21:35:05.689 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Dec 16 21:35:05 smithi045 ceph-mon[80763]: Reconfiguring prometheus.smithi045 (dependencies changed)... 2023-12-16T21:35:05.689 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Dec 16 21:35:05 smithi045 ceph-mon[80763]: Reconfiguring daemon prometheus.smithi045 on smithi045 2023-12-16T21:35:05.689 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Dec 16 21:35:05 smithi045 ceph-mon[80763]: from='mgr.14214 172.21.15.45:0/1227909437' entity='mgr.smithi045.nhtmwc' cmd=[{"prefix": "auth get", "entity": "osd.0"}]: dispatch 2023-12-16T21:35:05.689 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Dec 16 21:35:05 smithi045 ceph-mon[80763]: from='mgr.14214 172.21.15.45:0/1227909437' entity='mgr.smithi045.nhtmwc' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-12-16T21:35:05.883 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Dec 16 21:35:05 smithi169 ceph-mon[88350]: pgmap v12: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-16T21:35:05.883 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Dec 16 21:35:05 smithi169 ceph-mon[88350]: from='mgr.14214 172.21.15.45:0/1227909437' entity='mgr.smithi045.nhtmwc' 2023-12-16T21:35:05.883 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Dec 16 21:35:05 smithi169 ceph-mon[88350]: from='mgr.14214 172.21.15.45:0/1227909437' entity='mgr.smithi045.nhtmwc' 2023-12-16T21:35:05.884 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Dec 16 21:35:05 smithi169 ceph-mon[88350]: Reconfiguring prometheus.smithi045 (dependencies changed)... 2023-12-16T21:35:05.884 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Dec 16 21:35:05 smithi169 ceph-mon[88350]: Reconfiguring daemon prometheus.smithi045 on smithi045 2023-12-16T21:35:05.884 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Dec 16 21:35:05 smithi169 ceph-mon[88350]: from='mgr.14214 172.21.15.45:0/1227909437' entity='mgr.smithi045.nhtmwc' cmd=[{"prefix": "auth get", "entity": "osd.0"}]: dispatch 2023-12-16T21:35:05.884 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Dec 16 21:35:05 smithi169 ceph-mon[88350]: from='mgr.14214 172.21.15.45:0/1227909437' entity='mgr.smithi045.nhtmwc' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-12-16T21:35:06.757 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Dec 16 21:35:06 smithi045 ceph-mon[80763]: Deploying daemon osd.0 on smithi045 2023-12-16T21:35:06.883 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Dec 16 21:35:06 smithi169 ceph-mon[88350]: Deploying daemon osd.0 on smithi045 2023-12-16T21:35:07.621 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Dec 16 21:35:07 smithi045 ceph-mon[80763]: pgmap v13: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-16T21:35:07.621 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Dec 16 21:35:07 smithi045 ceph-mon[80763]: from='mgr.14214 172.21.15.45:0/1227909437' entity='mgr.smithi045.nhtmwc' 2023-12-16T21:35:07.621 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Dec 16 21:35:07 smithi045 ceph-mon[80763]: from='mgr.14214 172.21.15.45:0/1227909437' entity='mgr.smithi045.nhtmwc' 2023-12-16T21:35:07.621 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Dec 16 21:35:07 smithi045 ceph-mon[80763]: from='mgr.14214 172.21.15.45:0/1227909437' entity='mgr.smithi045.nhtmwc' cmd=[{"prefix": "auth get-or-create", "entity": "client.crash.smithi169", "caps": ["mon", "profile crash", "mgr", "profile crash"]}]: dispatch 2023-12-16T21:35:07.621 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Dec 16 21:35:07 smithi045 ceph-mon[80763]: from='mgr.14214 172.21.15.45:0/1227909437' entity='mgr.smithi045.nhtmwc' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-12-16T21:35:07.868 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Dec 16 21:35:07 smithi169 ceph-mon[88350]: pgmap v13: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-16T21:35:07.868 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Dec 16 21:35:07 smithi169 ceph-mon[88350]: from='mgr.14214 172.21.15.45:0/1227909437' entity='mgr.smithi045.nhtmwc' 2023-12-16T21:35:07.869 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Dec 16 21:35:07 smithi169 ceph-mon[88350]: from='mgr.14214 172.21.15.45:0/1227909437' entity='mgr.smithi045.nhtmwc' 2023-12-16T21:35:07.869 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Dec 16 21:35:07 smithi169 ceph-mon[88350]: from='mgr.14214 172.21.15.45:0/1227909437' entity='mgr.smithi045.nhtmwc' cmd=[{"prefix": "auth get-or-create", "entity": "client.crash.smithi169", "caps": ["mon", "profile crash", "mgr", "profile crash"]}]: dispatch 2023-12-16T21:35:07.869 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Dec 16 21:35:07 smithi169 ceph-mon[88350]: from='mgr.14214 172.21.15.45:0/1227909437' entity='mgr.smithi045.nhtmwc' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-12-16T21:35:08.772 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Dec 16 21:35:08 smithi045 ceph-mon[80763]: Reconfiguring crash.smithi169 (monmap changed)... 2023-12-16T21:35:08.772 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Dec 16 21:35:08 smithi045 ceph-mon[80763]: Reconfiguring daemon crash.smithi169 on smithi169 2023-12-16T21:35:08.773 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Dec 16 21:35:08 smithi045 ceph-mon[80763]: from='mgr.14214 172.21.15.45:0/1227909437' entity='mgr.smithi045.nhtmwc' 2023-12-16T21:35:08.773 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Dec 16 21:35:08 smithi045 ceph-mon[80763]: from='mgr.14214 172.21.15.45:0/1227909437' entity='mgr.smithi045.nhtmwc' 2023-12-16T21:35:08.773 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Dec 16 21:35:08 smithi045 ceph-mon[80763]: from='mgr.14214 172.21.15.45:0/1227909437' entity='mgr.smithi045.nhtmwc' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.smithi169.cncbgh", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2023-12-16T21:35:08.773 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Dec 16 21:35:08 smithi045 ceph-mon[80763]: from='mgr.14214 172.21.15.45:0/1227909437' entity='mgr.smithi045.nhtmwc' cmd=[{"prefix": "mgr services"}]: dispatch 2023-12-16T21:35:08.773 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Dec 16 21:35:08 smithi045 ceph-mon[80763]: from='mgr.14214 172.21.15.45:0/1227909437' entity='mgr.smithi045.nhtmwc' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-12-16T21:35:08.832 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Dec 16 21:35:08 smithi169 ceph-mon[88350]: Reconfiguring crash.smithi169 (monmap changed)... 2023-12-16T21:35:08.832 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Dec 16 21:35:08 smithi169 ceph-mon[88350]: Reconfiguring daemon crash.smithi169 on smithi169 2023-12-16T21:35:08.832 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Dec 16 21:35:08 smithi169 ceph-mon[88350]: from='mgr.14214 172.21.15.45:0/1227909437' entity='mgr.smithi045.nhtmwc' 2023-12-16T21:35:08.833 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Dec 16 21:35:08 smithi169 ceph-mon[88350]: from='mgr.14214 172.21.15.45:0/1227909437' entity='mgr.smithi045.nhtmwc' 2023-12-16T21:35:08.833 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Dec 16 21:35:08 smithi169 ceph-mon[88350]: from='mgr.14214 172.21.15.45:0/1227909437' entity='mgr.smithi045.nhtmwc' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.smithi169.cncbgh", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2023-12-16T21:35:08.833 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Dec 16 21:35:08 smithi169 ceph-mon[88350]: from='mgr.14214 172.21.15.45:0/1227909437' entity='mgr.smithi045.nhtmwc' cmd=[{"prefix": "mgr services"}]: dispatch 2023-12-16T21:35:08.833 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Dec 16 21:35:08 smithi169 ceph-mon[88350]: from='mgr.14214 172.21.15.45:0/1227909437' entity='mgr.smithi045.nhtmwc' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-12-16T21:35:09.678 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Dec 16 21:35:09 smithi045 ceph-mon[80763]: Reconfiguring mgr.smithi169.cncbgh (monmap changed)... 2023-12-16T21:35:09.678 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Dec 16 21:35:09 smithi045 ceph-mon[80763]: Reconfiguring daemon mgr.smithi169.cncbgh on smithi169 2023-12-16T21:35:09.678 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Dec 16 21:35:09 smithi045 ceph-mon[80763]: pgmap v14: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-16T21:35:09.679 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Dec 16 21:35:09 smithi045 ceph-mon[80763]: from='mgr.14214 172.21.15.45:0/1227909437' entity='mgr.smithi045.nhtmwc' 2023-12-16T21:35:09.679 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Dec 16 21:35:09 smithi045 ceph-mon[80763]: from='mgr.14214 172.21.15.45:0/1227909437' entity='mgr.smithi045.nhtmwc' 2023-12-16T21:35:09.679 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Dec 16 21:35:09 smithi045 ceph-mon[80763]: from='mgr.14214 172.21.15.45:0/1227909437' entity='mgr.smithi045.nhtmwc' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2023-12-16T21:35:09.679 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Dec 16 21:35:09 smithi045 ceph-mon[80763]: from='mgr.14214 172.21.15.45:0/1227909437' entity='mgr.smithi045.nhtmwc' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2023-12-16T21:35:09.679 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Dec 16 21:35:09 smithi045 ceph-mon[80763]: from='mgr.14214 172.21.15.45:0/1227909437' entity='mgr.smithi045.nhtmwc' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-12-16T21:35:09.883 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Dec 16 21:35:09 smithi169 ceph-mon[88350]: Reconfiguring mgr.smithi169.cncbgh (monmap changed)... 2023-12-16T21:35:09.883 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Dec 16 21:35:09 smithi169 ceph-mon[88350]: Reconfiguring daemon mgr.smithi169.cncbgh on smithi169 2023-12-16T21:35:09.883 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Dec 16 21:35:09 smithi169 ceph-mon[88350]: pgmap v14: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-16T21:35:09.883 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Dec 16 21:35:09 smithi169 ceph-mon[88350]: from='mgr.14214 172.21.15.45:0/1227909437' entity='mgr.smithi045.nhtmwc' 2023-12-16T21:35:09.883 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Dec 16 21:35:09 smithi169 ceph-mon[88350]: from='mgr.14214 172.21.15.45:0/1227909437' entity='mgr.smithi045.nhtmwc' 2023-12-16T21:35:09.883 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Dec 16 21:35:09 smithi169 ceph-mon[88350]: from='mgr.14214 172.21.15.45:0/1227909437' entity='mgr.smithi045.nhtmwc' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2023-12-16T21:35:09.884 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Dec 16 21:35:09 smithi169 ceph-mon[88350]: from='mgr.14214 172.21.15.45:0/1227909437' entity='mgr.smithi045.nhtmwc' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2023-12-16T21:35:09.884 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Dec 16 21:35:09 smithi169 ceph-mon[88350]: from='mgr.14214 172.21.15.45:0/1227909437' entity='mgr.smithi045.nhtmwc' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-12-16T21:35:10.872 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Dec 16 21:35:10 smithi045 ceph-mon[80763]: Reconfiguring mon.smithi169 (monmap changed)... 2023-12-16T21:35:10.872 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Dec 16 21:35:10 smithi045 ceph-mon[80763]: Reconfiguring daemon mon.smithi169 on smithi169 2023-12-16T21:35:10.872 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Dec 16 21:35:10 smithi045 ceph-mon[80763]: from='mgr.14214 172.21.15.45:0/1227909437' entity='mgr.smithi045.nhtmwc' 2023-12-16T21:35:10.872 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Dec 16 21:35:10 smithi045 ceph-mon[80763]: from='mgr.14214 172.21.15.45:0/1227909437' entity='mgr.smithi045.nhtmwc' 2023-12-16T21:35:10.872 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Dec 16 21:35:10 smithi045 ceph-mon[80763]: from='mgr.14214 172.21.15.45:0/1227909437' entity='mgr.smithi045.nhtmwc' cmd=[{"prefix": "dashboard get-alertmanager-api-host"}]: dispatch 2023-12-16T21:35:10.872 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Dec 16 21:35:10 smithi045 ceph-mon[80763]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard get-alertmanager-api-host"}]: dispatch 2023-12-16T21:35:10.872 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Dec 16 21:35:10 smithi045 ceph-mon[80763]: from='mgr.14214 172.21.15.45:0/1227909437' entity='mgr.smithi045.nhtmwc' cmd=[{"prefix": "dashboard set-alertmanager-api-host", "value": "http://smithi045.front.sepia.ceph.com:9093"}]: dispatch 2023-12-16T21:35:10.872 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Dec 16 21:35:10 smithi045 ceph-mon[80763]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard set-alertmanager-api-host", "value": "http://smithi045.front.sepia.ceph.com:9093"}]: dispatch 2023-12-16T21:35:10.872 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Dec 16 21:35:10 smithi045 ceph-mon[80763]: from='mgr.14214 172.21.15.45:0/1227909437' entity='mgr.smithi045.nhtmwc' 2023-12-16T21:35:10.872 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Dec 16 21:35:10 smithi045 ceph-mon[80763]: from='mgr.14214 172.21.15.45:0/1227909437' entity='mgr.smithi045.nhtmwc' cmd=[{"prefix": "dashboard get-grafana-api-url"}]: dispatch 2023-12-16T21:35:10.872 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Dec 16 21:35:10 smithi045 ceph-mon[80763]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard get-grafana-api-url"}]: dispatch 2023-12-16T21:35:10.873 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Dec 16 21:35:10 smithi045 ceph-mon[80763]: from='mgr.14214 172.21.15.45:0/1227909437' entity='mgr.smithi045.nhtmwc' cmd=[{"prefix": "dashboard set-grafana-api-url", "value": "https://smithi045.front.sepia.ceph.com:3000"}]: dispatch 2023-12-16T21:35:10.873 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Dec 16 21:35:10 smithi045 ceph-mon[80763]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard set-grafana-api-url", "value": "https://smithi045.front.sepia.ceph.com:3000"}]: dispatch 2023-12-16T21:35:10.873 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Dec 16 21:35:10 smithi045 ceph-mon[80763]: from='mgr.14214 172.21.15.45:0/1227909437' entity='mgr.smithi045.nhtmwc' 2023-12-16T21:35:10.873 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Dec 16 21:35:10 smithi045 ceph-mon[80763]: from='mgr.14214 172.21.15.45:0/1227909437' entity='mgr.smithi045.nhtmwc' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2023-12-16T21:35:10.873 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Dec 16 21:35:10 smithi045 ceph-mon[80763]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2023-12-16T21:35:10.873 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Dec 16 21:35:10 smithi045 ceph-mon[80763]: from='mgr.14214 172.21.15.45:0/1227909437' entity='mgr.smithi045.nhtmwc' cmd=[{"prefix": "dashboard set-prometheus-api-host", "value": "http://smithi045.front.sepia.ceph.com:9095"}]: dispatch 2023-12-16T21:35:10.873 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Dec 16 21:35:10 smithi045 ceph-mon[80763]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard set-prometheus-api-host", "value": "http://smithi045.front.sepia.ceph.com:9095"}]: dispatch 2023-12-16T21:35:10.873 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Dec 16 21:35:10 smithi045 ceph-mon[80763]: from='mgr.14214 172.21.15.45:0/1227909437' entity='mgr.smithi045.nhtmwc' 2023-12-16T21:35:10.873 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Dec 16 21:35:10 smithi045 ceph-mon[80763]: from='mgr.14214 172.21.15.45:0/1227909437' entity='mgr.smithi045.nhtmwc' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-12-16T21:35:10.883 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Dec 16 21:35:10 smithi169 ceph-mon[88350]: Reconfiguring mon.smithi169 (monmap changed)... 2023-12-16T21:35:10.883 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Dec 16 21:35:10 smithi169 ceph-mon[88350]: Reconfiguring daemon mon.smithi169 on smithi169 2023-12-16T21:35:10.883 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Dec 16 21:35:10 smithi169 ceph-mon[88350]: from='mgr.14214 172.21.15.45:0/1227909437' entity='mgr.smithi045.nhtmwc' 2023-12-16T21:35:10.883 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Dec 16 21:35:10 smithi169 ceph-mon[88350]: from='mgr.14214 172.21.15.45:0/1227909437' entity='mgr.smithi045.nhtmwc' 2023-12-16T21:35:10.884 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Dec 16 21:35:10 smithi169 ceph-mon[88350]: from='mgr.14214 172.21.15.45:0/1227909437' entity='mgr.smithi045.nhtmwc' cmd=[{"prefix": "dashboard get-alertmanager-api-host"}]: dispatch 2023-12-16T21:35:10.884 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Dec 16 21:35:10 smithi169 ceph-mon[88350]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard get-alertmanager-api-host"}]: dispatch 2023-12-16T21:35:10.884 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Dec 16 21:35:10 smithi169 ceph-mon[88350]: from='mgr.14214 172.21.15.45:0/1227909437' entity='mgr.smithi045.nhtmwc' cmd=[{"prefix": "dashboard set-alertmanager-api-host", "value": "http://smithi045.front.sepia.ceph.com:9093"}]: dispatch 2023-12-16T21:35:10.884 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Dec 16 21:35:10 smithi169 ceph-mon[88350]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard set-alertmanager-api-host", "value": "http://smithi045.front.sepia.ceph.com:9093"}]: dispatch 2023-12-16T21:35:10.884 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Dec 16 21:35:10 smithi169 ceph-mon[88350]: from='mgr.14214 172.21.15.45:0/1227909437' entity='mgr.smithi045.nhtmwc' 2023-12-16T21:35:10.884 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Dec 16 21:35:10 smithi169 ceph-mon[88350]: from='mgr.14214 172.21.15.45:0/1227909437' entity='mgr.smithi045.nhtmwc' cmd=[{"prefix": "dashboard get-grafana-api-url"}]: dispatch 2023-12-16T21:35:10.884 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Dec 16 21:35:10 smithi169 ceph-mon[88350]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard get-grafana-api-url"}]: dispatch 2023-12-16T21:35:10.884 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Dec 16 21:35:10 smithi169 ceph-mon[88350]: from='mgr.14214 172.21.15.45:0/1227909437' entity='mgr.smithi045.nhtmwc' cmd=[{"prefix": "dashboard set-grafana-api-url", "value": "https://smithi045.front.sepia.ceph.com:3000"}]: dispatch 2023-12-16T21:35:10.884 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Dec 16 21:35:10 smithi169 ceph-mon[88350]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard set-grafana-api-url", "value": "https://smithi045.front.sepia.ceph.com:3000"}]: dispatch 2023-12-16T21:35:10.884 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Dec 16 21:35:10 smithi169 ceph-mon[88350]: from='mgr.14214 172.21.15.45:0/1227909437' entity='mgr.smithi045.nhtmwc' 2023-12-16T21:35:10.884 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Dec 16 21:35:10 smithi169 ceph-mon[88350]: from='mgr.14214 172.21.15.45:0/1227909437' entity='mgr.smithi045.nhtmwc' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2023-12-16T21:35:10.884 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Dec 16 21:35:10 smithi169 ceph-mon[88350]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2023-12-16T21:35:10.884 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Dec 16 21:35:10 smithi169 ceph-mon[88350]: from='mgr.14214 172.21.15.45:0/1227909437' entity='mgr.smithi045.nhtmwc' cmd=[{"prefix": "dashboard set-prometheus-api-host", "value": "http://smithi045.front.sepia.ceph.com:9095"}]: dispatch 2023-12-16T21:35:10.884 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Dec 16 21:35:10 smithi169 ceph-mon[88350]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard set-prometheus-api-host", "value": "http://smithi045.front.sepia.ceph.com:9095"}]: dispatch 2023-12-16T21:35:10.885 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Dec 16 21:35:10 smithi169 ceph-mon[88350]: from='mgr.14214 172.21.15.45:0/1227909437' entity='mgr.smithi045.nhtmwc' 2023-12-16T21:35:10.885 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Dec 16 21:35:10 smithi169 ceph-mon[88350]: from='mgr.14214 172.21.15.45:0/1227909437' entity='mgr.smithi045.nhtmwc' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-12-16T21:35:11.912 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Dec 16 21:35:11 smithi045 ceph-mon[80763]: pgmap v15: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-16T21:35:11.912 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Dec 16 21:35:11 smithi045 ceph-mon[80763]: from='mgr.14214 172.21.15.45:0/1227909437' entity='mgr.smithi045.nhtmwc' 2023-12-16T21:35:11.912 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Dec 16 21:35:11 smithi045 ceph-mon[80763]: from='mgr.14214 172.21.15.45:0/1227909437' entity='mgr.smithi045.nhtmwc' 2023-12-16T21:35:11.912 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Dec 16 21:35:11 smithi045 ceph-mon[80763]: from='mgr.14214 172.21.15.45:0/1227909437' entity='mgr.smithi045.nhtmwc' 2023-12-16T21:35:11.912 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Dec 16 21:35:11 smithi045 ceph-mon[80763]: from='mgr.14214 172.21.15.45:0/1227909437' entity='mgr.smithi045.nhtmwc' 2023-12-16T21:35:12.133 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Dec 16 21:35:11 smithi169 ceph-mon[88350]: pgmap v15: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-16T21:35:12.133 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Dec 16 21:35:11 smithi169 ceph-mon[88350]: from='mgr.14214 172.21.15.45:0/1227909437' entity='mgr.smithi045.nhtmwc' 2023-12-16T21:35:12.134 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Dec 16 21:35:11 smithi169 ceph-mon[88350]: from='mgr.14214 172.21.15.45:0/1227909437' entity='mgr.smithi045.nhtmwc' 2023-12-16T21:35:12.134 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Dec 16 21:35:11 smithi169 ceph-mon[88350]: from='mgr.14214 172.21.15.45:0/1227909437' entity='mgr.smithi045.nhtmwc' 2023-12-16T21:35:12.134 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Dec 16 21:35:11 smithi169 ceph-mon[88350]: from='mgr.14214 172.21.15.45:0/1227909437' entity='mgr.smithi045.nhtmwc' 2023-12-16T21:35:13.070 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:35:13.068+0000 7f62897fa700 1 -- 172.21.15.45:0/3186521198 <== mgr.14214 v2:172.21.15.45:6800/2012105536 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+37 (secure 0 0 0) 0x7f628c100a10 con 0x7f627c068fa0 2023-12-16T21:35:13.070 INFO:teuthology.orchestra.run.smithi045.stdout:Created osd(s) 0 on host 'smithi045' 2023-12-16T21:35:13.072 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:35:13.070+0000 7f6292f83700 1 -- 172.21.15.45:0/3186521198 >> [v2:172.21.15.45:6800/2012105536,v1:172.21.15.45:6801/2012105536] conn(0x7f627c068fa0 msgr2=0x7f627c06b450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:35:13.073 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:35:13.070+0000 7f6292f83700 1 --2- 172.21.15.45:0/3186521198 >> [v2:172.21.15.45:6800/2012105536,v1:172.21.15.45:6801/2012105536] conn(0x7f627c068fa0 0x7f627c06b450 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7f627800b3b0 tx=0x7f6278009e30 comp rx=0 tx=0).stop 2023-12-16T21:35:13.073 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:35:13.070+0000 7f6292f83700 1 -- 172.21.15.45:0/3186521198 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f628c066920 msgr2=0x7f628c10d580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:35:13.073 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:35:13.070+0000 7f6292f83700 1 --2- 172.21.15.45:0/3186521198 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f628c066920 0x7f628c10d580 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7f628000b790 tx=0x7f6280009250 comp rx=0 tx=0).stop 2023-12-16T21:35:13.073 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:35:13.070+0000 7f6292f83700 1 -- 172.21.15.45:0/3186521198 shutdown_connections 2023-12-16T21:35:13.073 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:35:13.070+0000 7f6292f83700 1 --2- 172.21.15.45:0/3186521198 >> [v2:172.21.15.45:6800/2012105536,v1:172.21.15.45:6801/2012105536] conn(0x7f627c068fa0 0x7f627c06b450 unknown :-1 s=CLOSED pgs=23 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:35:13.073 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:35:13.070+0000 7f6292f83700 1 --2- 172.21.15.45:0/3186521198 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f628c066920 0x7f628c10d580 unknown :-1 s=CLOSED pgs=152 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:35:13.073 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:35:13.070+0000 7f6292f83700 1 --2- 172.21.15.45:0/3186521198 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f628c0ffd70 0x7f628c10dac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:35:13.073 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:35:13.070+0000 7f6292f83700 1 -- 172.21.15.45:0/3186521198 >> 172.21.15.45:0/3186521198 conn(0x7f628c0f71b0 msgr2=0x7f628c0f95e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-16T21:35:13.073 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:35:13.070+0000 7f6292f83700 1 -- 172.21.15.45:0/3186521198 shutdown_connections 2023-12-16T21:35:13.073 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:35:13.070+0000 7f6292f83700 1 -- 172.21.15.45:0/3186521198 wait complete. 2023-12-16T21:35:13.380 DEBUG:teuthology.orchestra.run.smithi045:osd.0> sudo journalctl -f -n 0 -u ceph-88ff363c-9c5a-11ee-95a5-87774f69a715@osd.0.service 2023-12-16T21:35:13.382 INFO:tasks.cephadm:Deploying osd.1 on smithi045 with /dev/vg_nvme/lv_3... 2023-12-16T21:35:13.382 DEBUG:teuthology.orchestra.run.smithi045:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 88ff363c-9c5a-11ee-95a5-87774f69a715 -- ceph-volume lvm zap /dev/vg_nvme/lv_3 2023-12-16T21:35:13.883 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Dec 16 21:35:13 smithi169 ceph-mon[88350]: pgmap v16: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-16T21:35:13.883 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Dec 16 21:35:13 smithi169 ceph-mon[88350]: from='mgr.14214 172.21.15.45:0/1227909437' entity='mgr.smithi045.nhtmwc' 2023-12-16T21:35:13.883 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Dec 16 21:35:13 smithi169 ceph-mon[88350]: from='mgr.14214 172.21.15.45:0/1227909437' entity='mgr.smithi045.nhtmwc' 2023-12-16T21:35:13.884 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Dec 16 21:35:13 smithi169 ceph-mon[88350]: from='mgr.14214 172.21.15.45:0/1227909437' entity='mgr.smithi045.nhtmwc' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-12-16T21:35:13.884 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Dec 16 21:35:13 smithi169 ceph-mon[88350]: from='mgr.14214 172.21.15.45:0/1227909437' entity='mgr.smithi045.nhtmwc' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-12-16T21:35:13.884 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Dec 16 21:35:13 smithi169 ceph-mon[88350]: from='mgr.14214 172.21.15.45:0/1227909437' entity='mgr.smithi045.nhtmwc' 2023-12-16T21:35:13.884 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Dec 16 21:35:13 smithi169 ceph-mon[88350]: from='mgr.14214 172.21.15.45:0/1227909437' entity='mgr.smithi045.nhtmwc' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-12-16T21:35:13.884 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Dec 16 21:35:13 smithi169 ceph-mon[88350]: from='mgr.14214 172.21.15.45:0/1227909437' entity='mgr.smithi045.nhtmwc' 2023-12-16T21:35:13.884 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Dec 16 21:35:13 smithi169 ceph-mon[88350]: from='mgr.14214 172.21.15.45:0/1227909437' entity='mgr.smithi045.nhtmwc' 2023-12-16T21:35:13.884 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Dec 16 21:35:13 smithi169 ceph-mon[88350]: from='mgr.14214 172.21.15.45:0/1227909437' entity='mgr.smithi045.nhtmwc' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2023-12-16T21:35:13.977 INFO:teuthology.orchestra.run.smithi045.stderr:Inferring config /var/lib/ceph/88ff363c-9c5a-11ee-95a5-87774f69a715/mon.smithi045/config 2023-12-16T21:35:14.053 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Dec 16 21:35:13 smithi045 ceph-mon[80763]: pgmap v16: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-16T21:35:14.054 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Dec 16 21:35:13 smithi045 ceph-mon[80763]: from='mgr.14214 172.21.15.45:0/1227909437' entity='mgr.smithi045.nhtmwc' 2023-12-16T21:35:14.054 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Dec 16 21:35:13 smithi045 ceph-mon[80763]: from='mgr.14214 172.21.15.45:0/1227909437' entity='mgr.smithi045.nhtmwc' 2023-12-16T21:35:14.054 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Dec 16 21:35:13 smithi045 ceph-mon[80763]: from='mgr.14214 172.21.15.45:0/1227909437' entity='mgr.smithi045.nhtmwc' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-12-16T21:35:14.054 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Dec 16 21:35:13 smithi045 ceph-mon[80763]: from='mgr.14214 172.21.15.45:0/1227909437' entity='mgr.smithi045.nhtmwc' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-12-16T21:35:14.054 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Dec 16 21:35:13 smithi045 ceph-mon[80763]: from='mgr.14214 172.21.15.45:0/1227909437' entity='mgr.smithi045.nhtmwc' 2023-12-16T21:35:14.054 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Dec 16 21:35:13 smithi045 ceph-mon[80763]: from='mgr.14214 172.21.15.45:0/1227909437' entity='mgr.smithi045.nhtmwc' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-12-16T21:35:14.054 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Dec 16 21:35:13 smithi045 ceph-mon[80763]: from='mgr.14214 172.21.15.45:0/1227909437' entity='mgr.smithi045.nhtmwc' 2023-12-16T21:35:14.054 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Dec 16 21:35:13 smithi045 ceph-mon[80763]: from='mgr.14214 172.21.15.45:0/1227909437' entity='mgr.smithi045.nhtmwc' 2023-12-16T21:35:14.054 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Dec 16 21:35:13 smithi045 ceph-mon[80763]: from='mgr.14214 172.21.15.45:0/1227909437' entity='mgr.smithi045.nhtmwc' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2023-12-16T21:35:15.086 INFO:teuthology.orchestra.run.smithi045.stderr:--> Zapping: /dev/vg_nvme/lv_3 2023-12-16T21:35:15.087 INFO:teuthology.orchestra.run.smithi045.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_3 bs=1M count=10 conv=fsync 2023-12-16T21:35:15.087 INFO:teuthology.orchestra.run.smithi045.stderr: stderr: 10+0 records in 2023-12-16T21:35:15.087 INFO:teuthology.orchestra.run.smithi045.stderr:10+0 records out 2023-12-16T21:35:15.087 INFO:teuthology.orchestra.run.smithi045.stderr:10485760 bytes (10 MB, 10 MiB) copied, 0.0102898 s, 1.0 GB/s 2023-12-16T21:35:15.087 INFO:teuthology.orchestra.run.smithi045.stderr:--> Zapping successful for: 2023-12-16T21:35:15.246 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:35:14 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:35:14.978+0000 7f2bc9453540 -1 osd.0 0 log_to_monitors true 2023-12-16T21:35:15.540 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Dec 16 21:35:15 smithi169 ceph-mon[88350]: from='mgr.14214 172.21.15.45:0/1227909437' entity='mgr.smithi045.nhtmwc' 2023-12-16T21:35:15.540 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Dec 16 21:35:15 smithi169 ceph-mon[88350]: from='mgr.14214 172.21.15.45:0/1227909437' entity='mgr.smithi045.nhtmwc' 2023-12-16T21:35:15.540 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Dec 16 21:35:15 smithi169 ceph-mon[88350]: pgmap v17: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-16T21:35:15.540 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Dec 16 21:35:15 smithi169 ceph-mon[88350]: from='osd.0 [v2:172.21.15.45:6802/1443487903,v1:172.21.15.45:6803/1443487903]' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2023-12-16T21:35:15.566 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Dec 16 21:35:15 smithi045 ceph-mon[80763]: from='mgr.14214 172.21.15.45:0/1227909437' entity='mgr.smithi045.nhtmwc' 2023-12-16T21:35:15.566 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Dec 16 21:35:15 smithi045 ceph-mon[80763]: from='mgr.14214 172.21.15.45:0/1227909437' entity='mgr.smithi045.nhtmwc' 2023-12-16T21:35:15.566 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Dec 16 21:35:15 smithi045 ceph-mon[80763]: pgmap v17: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-16T21:35:15.566 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Dec 16 21:35:15 smithi045 ceph-mon[80763]: from='osd.0 [v2:172.21.15.45:6802/1443487903,v1:172.21.15.45:6803/1443487903]' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2023-12-16T21:35:15.639 DEBUG:teuthology.orchestra.run.smithi045:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 88ff363c-9c5a-11ee-95a5-87774f69a715 -- ceph orch daemon add osd smithi045:vg_nvme/lv_3 2023-12-16T21:35:15.915 INFO:teuthology.orchestra.run.smithi045.stderr:Inferring config /var/lib/ceph/88ff363c-9c5a-11ee-95a5-87774f69a715/mon.smithi045/config 2023-12-16T21:35:16.541 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Dec 16 21:35:16 smithi045 ceph-mon[80763]: from='osd.0 [v2:172.21.15.45:6802/1443487903,v1:172.21.15.45:6803/1443487903]' entity='osd.0' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]': finished 2023-12-16T21:35:16.541 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Dec 16 21:35:16 smithi045 ceph-mon[80763]: osdmap e7: 1 total, 0 up, 1 in 2023-12-16T21:35:16.541 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Dec 16 21:35:16 smithi045 ceph-mon[80763]: from='mgr.14214 172.21.15.45:0/1227909437' entity='mgr.smithi045.nhtmwc' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2023-12-16T21:35:16.541 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Dec 16 21:35:16 smithi045 ceph-mon[80763]: from='osd.0 [v2:172.21.15.45:6802/1443487903,v1:172.21.15.45:6803/1443487903]' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi045", "root=default"]}]: dispatch 2023-12-16T21:35:16.541 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:35:16 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:35:16.259+0000 7f2bbfc38700 -1 osd.0 0 waiting for initial osdmap 2023-12-16T21:35:16.542 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:35:16 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:35:16.266+0000 7f2bbaa33700 -1 osd.0 8 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2023-12-16T21:35:16.633 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Dec 16 21:35:16 smithi169 ceph-mon[88350]: from='osd.0 [v2:172.21.15.45:6802/1443487903,v1:172.21.15.45:6803/1443487903]' entity='osd.0' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]': finished 2023-12-16T21:35:16.633 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Dec 16 21:35:16 smithi169 ceph-mon[88350]: osdmap e7: 1 total, 0 up, 1 in 2023-12-16T21:35:16.633 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Dec 16 21:35:16 smithi169 ceph-mon[88350]: from='mgr.14214 172.21.15.45:0/1227909437' entity='mgr.smithi045.nhtmwc' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2023-12-16T21:35:16.633 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Dec 16 21:35:16 smithi169 ceph-mon[88350]: from='osd.0 [v2:172.21.15.45:6802/1443487903,v1:172.21.15.45:6803/1443487903]' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi045", "root=default"]}]: dispatch 2023-12-16T21:35:16.896 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:35:16.892+0000 7f98eba23700 1 -- 172.21.15.45:0/2589892726 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f98e40faed0 msgr2=0x7f98e40fb2e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:35:16.896 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:35:16.892+0000 7f98eba23700 1 --2- 172.21.15.45:0/2589892726 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f98e40faed0 0x7f98e40fb2e0 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7f98e0004940 tx=0x7f98e0009e30 comp rx=0 tx=0).stop 2023-12-16T21:35:16.896 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:35:16.892+0000 7f98eba23700 1 -- 172.21.15.45:0/2589892726 shutdown_connections 2023-12-16T21:35:16.897 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:35:16.892+0000 7f98eba23700 1 --2- 172.21.15.45:0/2589892726 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f98e40fdf60 0x7f98e40fe3b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:35:16.897 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:35:16.892+0000 7f98eba23700 1 --2- 172.21.15.45:0/2589892726 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f98e40faed0 0x7f98e40fb2e0 unknown :-1 s=CLOSED pgs=160 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:35:16.897 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:35:16.892+0000 7f98eba23700 1 -- 172.21.15.45:0/2589892726 >> 172.21.15.45:0/2589892726 conn(0x7f98e40f6950 msgr2=0x7f98e40f8d80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-16T21:35:16.897 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:35:16.892+0000 7f98eba23700 1 -- 172.21.15.45:0/2589892726 shutdown_connections 2023-12-16T21:35:16.897 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:35:16.892+0000 7f98eba23700 1 -- 172.21.15.45:0/2589892726 wait complete. 2023-12-16T21:35:16.897 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:35:16.892+0000 7f98eba23700 1 Processor -- start 2023-12-16T21:35:16.897 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:35:16.892+0000 7f98eba23700 1 -- start start 2023-12-16T21:35:16.897 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:35:16.893+0000 7f98eba23700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f98e40faed0 0x7f98e40fd9f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:35:16.897 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:35:16.893+0000 7f98eba23700 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f98e40fdf60 0x7f98e40fbe70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:35:16.897 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:35:16.893+0000 7f98eba23700 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f98e40fc500 con 0x7f98e40faed0 2023-12-16T21:35:16.897 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:35:16.893+0000 7f98eba23700 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f98e40fc670 con 0x7f98e40fdf60 2023-12-16T21:35:16.897 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:35:16.893+0000 7f98e97bf700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f98e40faed0 0x7f98e40fd9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:35:16.897 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:35:16.893+0000 7f98e97bf700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f98e40faed0 0x7f98e40fd9f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:47626/0 (socket says 172.21.15.45:47626) 2023-12-16T21:35:16.898 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:35:16.893+0000 7f98e97bf700 1 -- 172.21.15.45:0/742434661 learned_addr learned my addr 172.21.15.45:0/742434661 (peer_addr_for_me v2:172.21.15.45:0/0) 2023-12-16T21:35:16.898 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:35:16.893+0000 7f98e8fbe700 1 --2- 172.21.15.45:0/742434661 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f98e40fdf60 0x7f98e40fbe70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:35:16.898 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:35:16.893+0000 7f98e97bf700 1 -- 172.21.15.45:0/742434661 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f98e40fdf60 msgr2=0x7f98e40fbe70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:35:16.898 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:35:16.893+0000 7f98e97bf700 1 --2- 172.21.15.45:0/742434661 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f98e40fdf60 0x7f98e40fbe70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:35:16.898 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:35:16.893+0000 7f98e97bf700 1 -- 172.21.15.45:0/742434661 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f98e0005040 con 0x7f98e40faed0 2023-12-16T21:35:16.898 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:35:16.893+0000 7f98e97bf700 1 --2- 172.21.15.45:0/742434661 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f98e40faed0 0x7f98e40fd9f0 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7f98e0009fd0 tx=0x7f98e0009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:35:16.898 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:35:16.893+0000 7f98d9ffb700 1 -- 172.21.15.45:0/742434661 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f98e000e040 con 0x7f98e40faed0 2023-12-16T21:35:16.898 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:35:16.893+0000 7f98eba23700 1 -- 172.21.15.45:0/742434661 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f98e40fc8f0 con 0x7f98e40faed0 2023-12-16T21:35:16.898 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:35:16.893+0000 7f98d9ffb700 1 -- 172.21.15.45:0/742434661 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f98e000bc50 con 0x7f98e40faed0 2023-12-16T21:35:16.898 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:35:16.893+0000 7f98d9ffb700 1 -- 172.21.15.45:0/742434661 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f98e001c630 con 0x7f98e40faed0 2023-12-16T21:35:16.898 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:35:16.894+0000 7f98eba23700 1 -- 172.21.15.45:0/742434661 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f98e4115630 con 0x7f98e40faed0 2023-12-16T21:35:16.899 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:35:16.895+0000 7f98eba23700 1 -- 172.21.15.45:0/742434661 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f98e4107670 con 0x7f98e40faed0 2023-12-16T21:35:16.899 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:35:16.895+0000 7f98d9ffb700 1 -- 172.21.15.45:0/742434661 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 18) v1 ==== 86391+0+0 (secure 0 0 0) 0x7f98e0013420 con 0x7f98e40faed0 2023-12-16T21:35:16.899 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:35:16.895+0000 7f98d9ffb700 1 --2- 172.21.15.45:0/742434661 >> [v2:172.21.15.45:6800/2012105536,v1:172.21.15.45:6801/2012105536] conn(0x7f98d0068ed0 0x7f98d006b380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:35:16.899 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:35:16.896+0000 7f98d9ffb700 1 -- 172.21.15.45:0/742434661 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(8..8 src has 1..8) v4 ==== 1698+0+0 (secure 0 0 0) 0x7f98e00871d0 con 0x7f98e40faed0 2023-12-16T21:35:16.899 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:35:16.896+0000 7f98e8fbe700 1 --2- 172.21.15.45:0/742434661 >> [v2:172.21.15.45:6800/2012105536,v1:172.21.15.45:6801/2012105536] conn(0x7f98d0068ed0 0x7f98d006b380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:35:16.900 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:35:16.896+0000 7f98e8fbe700 1 --2- 172.21.15.45:0/742434661 >> [v2:172.21.15.45:6800/2012105536,v1:172.21.15.45:6801/2012105536] conn(0x7f98d0068ed0 0x7f98d006b380 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f98d4006fd0 tx=0x7f98d4008040 comp rx=0 tx=0).ready entity=mgr.14214 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:35:16.902 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:35:16.900+0000 7f98d9ffb700 1 -- 172.21.15.45:0/742434661 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+174570 (secure 0 0 0) 0x7f98e00580c0 con 0x7f98e40faed0 2023-12-16T21:35:17.110 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:35:17.108+0000 7f98eba23700 1 -- 172.21.15.45:0/742434661 --> [v2:172.21.15.45:6800/2012105536,v1:172.21.15.45:6801/2012105536] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi045:vg_nvme/lv_3", "target": ["mon-mgr", ""]}) v1 -- 0x7f98e402b1a0 con 0x7f98d0068ed0 2023-12-16T21:35:17.361 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Dec 16 21:35:17 smithi045 ceph-mon[80763]: from='osd.0 [v2:172.21.15.45:6802/1443487903,v1:172.21.15.45:6803/1443487903]' entity='osd.0' cmd='[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi045", "root=default"]}]': finished 2023-12-16T21:35:17.361 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Dec 16 21:35:17 smithi045 ceph-mon[80763]: osdmap e8: 1 total, 0 up, 1 in 2023-12-16T21:35:17.362 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Dec 16 21:35:17 smithi045 ceph-mon[80763]: from='mgr.14214 172.21.15.45:0/1227909437' entity='mgr.smithi045.nhtmwc' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2023-12-16T21:35:17.362 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Dec 16 21:35:17 smithi045 ceph-mon[80763]: from='mgr.14214 172.21.15.45:0/1227909437' entity='mgr.smithi045.nhtmwc' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2023-12-16T21:35:17.362 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Dec 16 21:35:17 smithi045 ceph-mon[80763]: pgmap v20: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-16T21:35:17.362 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Dec 16 21:35:17 smithi045 ceph-mon[80763]: from='mgr.14214 172.21.15.45:0/1227909437' entity='mgr.smithi045.nhtmwc' 2023-12-16T21:35:17.362 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Dec 16 21:35:17 smithi045 ceph-mon[80763]: from='mgr.14214 172.21.15.45:0/1227909437' entity='mgr.smithi045.nhtmwc' 2023-12-16T21:35:17.362 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Dec 16 21:35:17 smithi045 ceph-mon[80763]: from='mgr.14214 172.21.15.45:0/1227909437' entity='mgr.smithi045.nhtmwc' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2023-12-16T21:35:17.362 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Dec 16 21:35:17 smithi045 ceph-mon[80763]: from='mgr.14214 172.21.15.45:0/1227909437' entity='mgr.smithi045.nhtmwc' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2023-12-16T21:35:17.362 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Dec 16 21:35:17 smithi045 ceph-mon[80763]: from='mgr.14214 172.21.15.45:0/1227909437' entity='mgr.smithi045.nhtmwc' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-12-16T21:35:17.633 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Dec 16 21:35:17 smithi169 ceph-mon[88350]: from='osd.0 [v2:172.21.15.45:6802/1443487903,v1:172.21.15.45:6803/1443487903]' entity='osd.0' cmd='[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi045", "root=default"]}]': finished 2023-12-16T21:35:17.634 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Dec 16 21:35:17 smithi169 ceph-mon[88350]: osdmap e8: 1 total, 0 up, 1 in 2023-12-16T21:35:17.634 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Dec 16 21:35:17 smithi169 ceph-mon[88350]: from='mgr.14214 172.21.15.45:0/1227909437' entity='mgr.smithi045.nhtmwc' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2023-12-16T21:35:17.634 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Dec 16 21:35:17 smithi169 ceph-mon[88350]: from='mgr.14214 172.21.15.45:0/1227909437' entity='mgr.smithi045.nhtmwc' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2023-12-16T21:35:17.634 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Dec 16 21:35:17 smithi169 ceph-mon[88350]: pgmap v20: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-16T21:35:17.634 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Dec 16 21:35:17 smithi169 ceph-mon[88350]: from='mgr.14214 172.21.15.45:0/1227909437' entity='mgr.smithi045.nhtmwc' 2023-12-16T21:35:17.634 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Dec 16 21:35:17 smithi169 ceph-mon[88350]: from='mgr.14214 172.21.15.45:0/1227909437' entity='mgr.smithi045.nhtmwc' 2023-12-16T21:35:17.634 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Dec 16 21:35:17 smithi169 ceph-mon[88350]: from='mgr.14214 172.21.15.45:0/1227909437' entity='mgr.smithi045.nhtmwc' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2023-12-16T21:35:17.634 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Dec 16 21:35:17 smithi169 ceph-mon[88350]: from='mgr.14214 172.21.15.45:0/1227909437' entity='mgr.smithi045.nhtmwc' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2023-12-16T21:35:17.634 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Dec 16 21:35:17 smithi169 ceph-mon[88350]: from='mgr.14214 172.21.15.45:0/1227909437' entity='mgr.smithi045.nhtmwc' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-12-16T21:35:18.557 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Dec 16 21:35:18 smithi045 ceph-mon[80763]: purged_snaps scrub starts 2023-12-16T21:35:18.557 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Dec 16 21:35:18 smithi045 ceph-mon[80763]: purged_snaps scrub ok 2023-12-16T21:35:18.557 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Dec 16 21:35:18 smithi045 ceph-mon[80763]: from='client.14264 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi045:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2023-12-16T21:35:18.557 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Dec 16 21:35:18 smithi045 ceph-mon[80763]: from='mgr.14214 172.21.15.45:0/1227909437' entity='mgr.smithi045.nhtmwc' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2023-12-16T21:35:18.557 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Dec 16 21:35:18 smithi045 ceph-mon[80763]: osd.0 [v2:172.21.15.45:6802/1443487903,v1:172.21.15.45:6803/1443487903] boot 2023-12-16T21:35:18.557 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Dec 16 21:35:18 smithi045 ceph-mon[80763]: osdmap e9: 1 total, 1 up, 1 in 2023-12-16T21:35:18.557 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Dec 16 21:35:18 smithi045 ceph-mon[80763]: from='mgr.14214 172.21.15.45:0/1227909437' entity='mgr.smithi045.nhtmwc' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2023-12-16T21:35:18.633 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Dec 16 21:35:18 smithi169 ceph-mon[88350]: purged_snaps scrub starts 2023-12-16T21:35:18.633 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Dec 16 21:35:18 smithi169 ceph-mon[88350]: purged_snaps scrub ok 2023-12-16T21:35:18.633 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Dec 16 21:35:18 smithi169 ceph-mon[88350]: from='client.14264 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi045:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2023-12-16T21:35:18.634 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Dec 16 21:35:18 smithi169 ceph-mon[88350]: from='mgr.14214 172.21.15.45:0/1227909437' entity='mgr.smithi045.nhtmwc' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2023-12-16T21:35:18.634 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Dec 16 21:35:18 smithi169 ceph-mon[88350]: osd.0 [v2:172.21.15.45:6802/1443487903,v1:172.21.15.45:6803/1443487903] boot 2023-12-16T21:35:18.634 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Dec 16 21:35:18 smithi169 ceph-mon[88350]: osdmap e9: 1 total, 1 up, 1 in 2023-12-16T21:35:18.634 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Dec 16 21:35:18 smithi169 ceph-mon[88350]: from='mgr.14214 172.21.15.45:0/1227909437' entity='mgr.smithi045.nhtmwc' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2023-12-16T21:35:19.566 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Dec 16 21:35:19 smithi045 ceph-mon[80763]: osdmap e10: 1 total, 1 up, 1 in 2023-12-16T21:35:19.566 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Dec 16 21:35:19 smithi045 ceph-mon[80763]: pgmap v23: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2023-12-16T21:35:19.566 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Dec 16 21:35:19 smithi045 ceph-mon[80763]: from='mgr.14214 172.21.15.45:0/1227909437' entity='mgr.smithi045.nhtmwc' 2023-12-16T21:35:19.567 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Dec 16 21:35:19 smithi045 ceph-mon[80763]: from='mgr.14214 172.21.15.45:0/1227909437' entity='mgr.smithi045.nhtmwc' 2023-12-16T21:35:19.567 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Dec 16 21:35:19 smithi045 ceph-mon[80763]: from='mgr.14214 172.21.15.45:0/1227909437' entity='mgr.smithi045.nhtmwc' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2023-12-16T21:35:19.567 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Dec 16 21:35:19 smithi045 ceph-mon[80763]: from='mgr.14214 172.21.15.45:0/1227909437' entity='mgr.smithi045.nhtmwc' 2023-12-16T21:35:19.567 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Dec 16 21:35:19 smithi045 ceph-mon[80763]: from='mgr.14214 172.21.15.45:0/1227909437' entity='mgr.smithi045.nhtmwc' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-12-16T21:35:19.567 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Dec 16 21:35:19 smithi045 ceph-mon[80763]: from='mgr.14214 172.21.15.45:0/1227909437' entity='mgr.smithi045.nhtmwc' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-12-16T21:35:19.567 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Dec 16 21:35:19 smithi045 ceph-mon[80763]: from='mgr.14214 172.21.15.45:0/1227909437' entity='mgr.smithi045.nhtmwc' 2023-12-16T21:35:19.567 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Dec 16 21:35:19 smithi045 ceph-mon[80763]: from='mgr.14214 172.21.15.45:0/1227909437' entity='mgr.smithi045.nhtmwc' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-12-16T21:35:19.571 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Dec 16 21:35:19 smithi169 ceph-mon[88350]: osdmap e10: 1 total, 1 up, 1 in 2023-12-16T21:35:19.571 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Dec 16 21:35:19 smithi169 ceph-mon[88350]: pgmap v23: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2023-12-16T21:35:19.572 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Dec 16 21:35:19 smithi169 ceph-mon[88350]: from='mgr.14214 172.21.15.45:0/1227909437' entity='mgr.smithi045.nhtmwc' 2023-12-16T21:35:19.572 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Dec 16 21:35:19 smithi169 ceph-mon[88350]: from='mgr.14214 172.21.15.45:0/1227909437' entity='mgr.smithi045.nhtmwc' 2023-12-16T21:35:19.572 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Dec 16 21:35:19 smithi169 ceph-mon[88350]: from='mgr.14214 172.21.15.45:0/1227909437' entity='mgr.smithi045.nhtmwc' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2023-12-16T21:35:19.572 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Dec 16 21:35:19 smithi169 ceph-mon[88350]: from='mgr.14214 172.21.15.45:0/1227909437' entity='mgr.smithi045.nhtmwc' 2023-12-16T21:35:19.572 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Dec 16 21:35:19 smithi169 ceph-mon[88350]: from='mgr.14214 172.21.15.45:0/1227909437' entity='mgr.smithi045.nhtmwc' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-12-16T21:35:19.572 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Dec 16 21:35:19 smithi169 ceph-mon[88350]: from='mgr.14214 172.21.15.45:0/1227909437' entity='mgr.smithi045.nhtmwc' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-12-16T21:35:19.572 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Dec 16 21:35:19 smithi169 ceph-mon[88350]: from='mgr.14214 172.21.15.45:0/1227909437' entity='mgr.smithi045.nhtmwc' 2023-12-16T21:35:19.572 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Dec 16 21:35:19 smithi169 ceph-mon[88350]: from='mgr.14214 172.21.15.45:0/1227909437' entity='mgr.smithi045.nhtmwc' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-12-16T21:35:20.558 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Dec 16 21:35:20 smithi045 ceph-mon[80763]: Detected new or changed devices on smithi045 2023-12-16T21:35:20.559 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Dec 16 21:35:20 smithi045 ceph-mon[80763]: Adjusting osd_memory_target on smithi045 to 12869M 2023-12-16T21:35:20.559 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Dec 16 21:35:20 smithi045 ceph-mon[80763]: from='client.? 172.21.15.45:0/4022350394' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "66f2184c-26d0-492c-ae32-ece7b19ce7bd"}]: dispatch 2023-12-16T21:35:20.559 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Dec 16 21:35:20 smithi045 ceph-mon[80763]: from='client.? 172.21.15.45:0/4022350394' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "66f2184c-26d0-492c-ae32-ece7b19ce7bd"}]': finished 2023-12-16T21:35:20.559 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Dec 16 21:35:20 smithi045 ceph-mon[80763]: osdmap e11: 2 total, 1 up, 2 in 2023-12-16T21:35:20.559 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Dec 16 21:35:20 smithi045 ceph-mon[80763]: from='mgr.14214 172.21.15.45:0/1227909437' entity='mgr.smithi045.nhtmwc' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2023-12-16T21:35:20.559 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Dec 16 21:35:20 smithi045 ceph-mon[80763]: from='mgr.14214 172.21.15.45:0/1227909437' entity='mgr.smithi045.nhtmwc' 2023-12-16T21:35:20.559 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Dec 16 21:35:20 smithi045 ceph-mon[80763]: from='mgr.14214 172.21.15.45:0/1227909437' entity='mgr.smithi045.nhtmwc' 2023-12-16T21:35:20.633 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Dec 16 21:35:20 smithi169 ceph-mon[88350]: Detected new or changed devices on smithi045 2023-12-16T21:35:20.636 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Dec 16 21:35:20 smithi169 ceph-mon[88350]: Adjusting osd_memory_target on smithi045 to 12869M 2023-12-16T21:35:20.637 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Dec 16 21:35:20 smithi169 ceph-mon[88350]: from='client.? 172.21.15.45:0/4022350394' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "66f2184c-26d0-492c-ae32-ece7b19ce7bd"}]: dispatch 2023-12-16T21:35:20.637 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Dec 16 21:35:20 smithi169 ceph-mon[88350]: from='client.? 172.21.15.45:0/4022350394' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "66f2184c-26d0-492c-ae32-ece7b19ce7bd"}]': finished 2023-12-16T21:35:20.637 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Dec 16 21:35:20 smithi169 ceph-mon[88350]: osdmap e11: 2 total, 1 up, 2 in 2023-12-16T21:35:20.637 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Dec 16 21:35:20 smithi169 ceph-mon[88350]: from='mgr.14214 172.21.15.45:0/1227909437' entity='mgr.smithi045.nhtmwc' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2023-12-16T21:35:20.637 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Dec 16 21:35:20 smithi169 ceph-mon[88350]: from='mgr.14214 172.21.15.45:0/1227909437' entity='mgr.smithi045.nhtmwc' 2023-12-16T21:35:20.637 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Dec 16 21:35:20 smithi169 ceph-mon[88350]: from='mgr.14214 172.21.15.45:0/1227909437' entity='mgr.smithi045.nhtmwc' 2023-12-16T21:35:21.566 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Dec 16 21:35:21 smithi045 ceph-mon[80763]: from='client.? 172.21.15.45:0/1959949128' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2023-12-16T21:35:21.566 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Dec 16 21:35:21 smithi045 ceph-mon[80763]: pgmap v25: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2023-12-16T21:35:21.633 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Dec 16 21:35:21 smithi169 ceph-mon[88350]: from='client.? 172.21.15.45:0/1959949128' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2023-12-16T21:35:21.633 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Dec 16 21:35:21 smithi169 ceph-mon[88350]: pgmap v25: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2023-12-16T21:35:22.816 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Dec 16 21:35:22 smithi045 ceph-mon[80763]: from='mgr.14214 172.21.15.45:0/1227909437' entity='mgr.smithi045.nhtmwc' 2023-12-16T21:35:22.816 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Dec 16 21:35:22 smithi045 ceph-mon[80763]: from='mgr.14214 172.21.15.45:0/1227909437' entity='mgr.smithi045.nhtmwc' 2023-12-16T21:35:22.816 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Dec 16 21:35:22 smithi045 ceph-mon[80763]: from='mgr.14214 172.21.15.45:0/1227909437' entity='mgr.smithi045.nhtmwc' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-12-16T21:35:22.816 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Dec 16 21:35:22 smithi045 ceph-mon[80763]: from='mgr.14214 172.21.15.45:0/1227909437' entity='mgr.smithi045.nhtmwc' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-12-16T21:35:22.816 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Dec 16 21:35:22 smithi045 ceph-mon[80763]: from='mgr.14214 172.21.15.45:0/1227909437' entity='mgr.smithi045.nhtmwc' 2023-12-16T21:35:22.883 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Dec 16 21:35:22 smithi169 ceph-mon[88350]: from='mgr.14214 172.21.15.45:0/1227909437' entity='mgr.smithi045.nhtmwc' 2023-12-16T21:35:22.884 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Dec 16 21:35:22 smithi169 ceph-mon[88350]: from='mgr.14214 172.21.15.45:0/1227909437' entity='mgr.smithi045.nhtmwc' 2023-12-16T21:35:22.884 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Dec 16 21:35:22 smithi169 ceph-mon[88350]: from='mgr.14214 172.21.15.45:0/1227909437' entity='mgr.smithi045.nhtmwc' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-12-16T21:35:22.884 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Dec 16 21:35:22 smithi169 ceph-mon[88350]: from='mgr.14214 172.21.15.45:0/1227909437' entity='mgr.smithi045.nhtmwc' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-12-16T21:35:22.884 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Dec 16 21:35:22 smithi169 ceph-mon[88350]: from='mgr.14214 172.21.15.45:0/1227909437' entity='mgr.smithi045.nhtmwc' 2023-12-16T21:35:23.757 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Dec 16 21:35:23 smithi045 ceph-mon[80763]: pgmap v26: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2023-12-16T21:35:23.883 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Dec 16 21:35:23 smithi169 ceph-mon[88350]: pgmap v26: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2023-12-16T21:35:25.883 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Dec 16 21:35:25 smithi169 ceph-mon[88350]: pgmap v27: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2023-12-16T21:35:25.883 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Dec 16 21:35:25 smithi169 ceph-mon[88350]: from='mgr.14214 172.21.15.45:0/1227909437' entity='mgr.smithi045.nhtmwc' cmd=[{"prefix": "auth get", "entity": "osd.1"}]: dispatch 2023-12-16T21:35:25.883 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Dec 16 21:35:25 smithi169 ceph-mon[88350]: from='mgr.14214 172.21.15.45:0/1227909437' entity='mgr.smithi045.nhtmwc' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-12-16T21:35:25.960 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Dec 16 21:35:25 smithi045 ceph-mon[80763]: pgmap v27: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2023-12-16T21:35:25.960 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Dec 16 21:35:25 smithi045 ceph-mon[80763]: from='mgr.14214 172.21.15.45:0/1227909437' entity='mgr.smithi045.nhtmwc' cmd=[{"prefix": "auth get", "entity": "osd.1"}]: dispatch 2023-12-16T21:35:25.960 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Dec 16 21:35:25 smithi045 ceph-mon[80763]: from='mgr.14214 172.21.15.45:0/1227909437' entity='mgr.smithi045.nhtmwc' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-12-16T21:35:27.022 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Dec 16 21:35:26 smithi045 ceph-mon[80763]: Deploying daemon osd.1 on smithi045 2023-12-16T21:35:27.133 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Dec 16 21:35:26 smithi169 ceph-mon[88350]: Deploying daemon osd.1 on smithi045 2023-12-16T21:35:27.908 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Dec 16 21:35:27 smithi045 ceph-mon[80763]: pgmap v28: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2023-12-16T21:35:28.133 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Dec 16 21:35:27 smithi169 ceph-mon[88350]: pgmap v28: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2023-12-16T21:35:29.816 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Dec 16 21:35:29 smithi045 ceph-mon[80763]: from='mgr.14214 172.21.15.45:0/1227909437' entity='mgr.smithi045.nhtmwc' 2023-12-16T21:35:29.816 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Dec 16 21:35:29 smithi045 ceph-mon[80763]: from='mgr.14214 172.21.15.45:0/1227909437' entity='mgr.smithi045.nhtmwc' 2023-12-16T21:35:29.816 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Dec 16 21:35:29 smithi045 ceph-mon[80763]: from='mgr.14214 172.21.15.45:0/1227909437' entity='mgr.smithi045.nhtmwc' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-12-16T21:35:29.816 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Dec 16 21:35:29 smithi045 ceph-mon[80763]: pgmap v29: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2023-12-16T21:35:29.817 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Dec 16 21:35:29 smithi045 ceph-mon[80763]: from='mgr.14214 172.21.15.45:0/1227909437' entity='mgr.smithi045.nhtmwc' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2023-12-16T21:35:29.884 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Dec 16 21:35:29 smithi169 ceph-mon[88350]: from='mgr.14214 172.21.15.45:0/1227909437' entity='mgr.smithi045.nhtmwc' 2023-12-16T21:35:29.884 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Dec 16 21:35:29 smithi169 ceph-mon[88350]: from='mgr.14214 172.21.15.45:0/1227909437' entity='mgr.smithi045.nhtmwc' 2023-12-16T21:35:29.884 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Dec 16 21:35:29 smithi169 ceph-mon[88350]: from='mgr.14214 172.21.15.45:0/1227909437' entity='mgr.smithi045.nhtmwc' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-12-16T21:35:29.884 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Dec 16 21:35:29 smithi169 ceph-mon[88350]: pgmap v29: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2023-12-16T21:35:29.884 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Dec 16 21:35:29 smithi169 ceph-mon[88350]: from='mgr.14214 172.21.15.45:0/1227909437' entity='mgr.smithi045.nhtmwc' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2023-12-16T21:35:30.201 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:35:30.199+0000 7f98d9ffb700 1 -- 172.21.15.45:0/742434661 <== mgr.14214 v2:172.21.15.45:6800/2012105536 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+37 (secure 0 0 0) 0x7f98e402b1a0 con 0x7f98d0068ed0 2023-12-16T21:35:30.202 INFO:teuthology.orchestra.run.smithi045.stdout:Created osd(s) 1 on host 'smithi045' 2023-12-16T21:35:30.203 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:35:30.201+0000 7f98eba23700 1 -- 172.21.15.45:0/742434661 >> [v2:172.21.15.45:6800/2012105536,v1:172.21.15.45:6801/2012105536] conn(0x7f98d0068ed0 msgr2=0x7f98d006b380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:35:30.203 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:35:30.201+0000 7f98eba23700 1 --2- 172.21.15.45:0/742434661 >> [v2:172.21.15.45:6800/2012105536,v1:172.21.15.45:6801/2012105536] conn(0x7f98d0068ed0 0x7f98d006b380 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f98d4006fd0 tx=0x7f98d4008040 comp rx=0 tx=0).stop 2023-12-16T21:35:30.203 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:35:30.201+0000 7f98eba23700 1 -- 172.21.15.45:0/742434661 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f98e40faed0 msgr2=0x7f98e40fd9f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:35:30.204 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:35:30.201+0000 7f98eba23700 1 --2- 172.21.15.45:0/742434661 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f98e40faed0 0x7f98e40fd9f0 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7f98e0009fd0 tx=0x7f98e0009e30 comp rx=0 tx=0).stop 2023-12-16T21:35:30.204 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:35:30.201+0000 7f98eba23700 1 -- 172.21.15.45:0/742434661 shutdown_connections 2023-12-16T21:35:30.204 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:35:30.201+0000 7f98eba23700 1 --2- 172.21.15.45:0/742434661 >> [v2:172.21.15.45:6800/2012105536,v1:172.21.15.45:6801/2012105536] conn(0x7f98d0068ed0 0x7f98d006b380 unknown :-1 s=CLOSED pgs=27 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:35:30.204 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:35:30.201+0000 7f98eba23700 1 --2- 172.21.15.45:0/742434661 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f98e40faed0 0x7f98e40fd9f0 unknown :-1 s=CLOSED pgs=161 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:35:30.204 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:35:30.201+0000 7f98eba23700 1 --2- 172.21.15.45:0/742434661 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f98e40fdf60 0x7f98e40fbe70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:35:30.204 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:35:30.201+0000 7f98eba23700 1 -- 172.21.15.45:0/742434661 >> 172.21.15.45:0/742434661 conn(0x7f98e40f6950 msgr2=0x7f98e40f8c50 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-16T21:35:30.204 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:35:30.201+0000 7f98eba23700 1 -- 172.21.15.45:0/742434661 shutdown_connections 2023-12-16T21:35:30.204 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:35:30.201+0000 7f98eba23700 1 -- 172.21.15.45:0/742434661 wait complete. 2023-12-16T21:35:30.705 DEBUG:teuthology.orchestra.run.smithi045:osd.1> sudo journalctl -f -n 0 -u ceph-88ff363c-9c5a-11ee-95a5-87774f69a715@osd.1.service 2023-12-16T21:35:30.707 INFO:tasks.cephadm:Deploying osd.2 on smithi045 with /dev/vg_nvme/lv_2... 2023-12-16T21:35:30.707 DEBUG:teuthology.orchestra.run.smithi045:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 88ff363c-9c5a-11ee-95a5-87774f69a715 -- ceph-volume lvm zap /dev/vg_nvme/lv_2 2023-12-16T21:35:31.077 INFO:teuthology.orchestra.run.smithi045.stderr:Inferring config /var/lib/ceph/88ff363c-9c5a-11ee-95a5-87774f69a715/mon.smithi045/config 2023-12-16T21:35:31.427 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Dec 16 21:35:31 smithi045 ceph-mon[80763]: from='mgr.14214 172.21.15.45:0/1227909437' entity='mgr.smithi045.nhtmwc' 2023-12-16T21:35:31.427 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Dec 16 21:35:31 smithi045 ceph-mon[80763]: from='mgr.14214 172.21.15.45:0/1227909437' entity='mgr.smithi045.nhtmwc' 2023-12-16T21:35:31.427 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Dec 16 21:35:31 smithi045 ceph-mon[80763]: pgmap v30: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2023-12-16T21:35:31.633 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Dec 16 21:35:31 smithi169 ceph-mon[88350]: from='mgr.14214 172.21.15.45:0/1227909437' entity='mgr.smithi045.nhtmwc' 2023-12-16T21:35:31.633 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Dec 16 21:35:31 smithi169 ceph-mon[88350]: from='mgr.14214 172.21.15.45:0/1227909437' entity='mgr.smithi045.nhtmwc' 2023-12-16T21:35:31.633 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Dec 16 21:35:31 smithi169 ceph-mon[88350]: pgmap v30: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2023-12-16T21:35:32.471 INFO:teuthology.orchestra.run.smithi045.stderr:--> Zapping: /dev/vg_nvme/lv_2 2023-12-16T21:35:32.472 INFO:teuthology.orchestra.run.smithi045.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_2 bs=1M count=10 conv=fsync 2023-12-16T21:35:32.472 INFO:teuthology.orchestra.run.smithi045.stderr: stderr: 10+0 records in 2023-12-16T21:35:32.472 INFO:teuthology.orchestra.run.smithi045.stderr:10+0 records out 2023-12-16T21:35:32.472 INFO:teuthology.orchestra.run.smithi045.stderr:10485760 bytes (10 MB, 10 MiB) copied, 0.012177 s, 861 MB/s 2023-12-16T21:35:32.472 INFO:teuthology.orchestra.run.smithi045.stderr:--> Zapping successful for: 2023-12-16T21:35:32.734 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:35:32 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:35:32.493+0000 7fa1be792540 -1 osd.1 0 log_to_monitors true 2023-12-16T21:35:32.759 DEBUG:teuthology.orchestra.run.smithi045:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 88ff363c-9c5a-11ee-95a5-87774f69a715 -- ceph orch daemon add osd smithi045:vg_nvme/lv_2 2023-12-16T21:35:33.042 INFO:teuthology.orchestra.run.smithi045.stderr:Inferring config /var/lib/ceph/88ff363c-9c5a-11ee-95a5-87774f69a715/mon.smithi045/config 2023-12-16T21:35:33.633 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Dec 16 21:35:33 smithi169 ceph-mon[88350]: from='mgr.14214 172.21.15.45:0/1227909437' entity='mgr.smithi045.nhtmwc' 2023-12-16T21:35:33.633 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Dec 16 21:35:33 smithi169 ceph-mon[88350]: from='mgr.14214 172.21.15.45:0/1227909437' entity='mgr.smithi045.nhtmwc' 2023-12-16T21:35:33.633 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Dec 16 21:35:33 smithi169 ceph-mon[88350]: from='osd.1 [v2:172.21.15.45:6810/3869108635,v1:172.21.15.45:6811/3869108635]' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2023-12-16T21:35:33.633 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Dec 16 21:35:33 smithi169 ceph-mon[88350]: pgmap v31: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2023-12-16T21:35:33.650 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Dec 16 21:35:33 smithi045 ceph-mon[80763]: from='mgr.14214 172.21.15.45:0/1227909437' entity='mgr.smithi045.nhtmwc' 2023-12-16T21:35:33.650 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Dec 16 21:35:33 smithi045 ceph-mon[80763]: from='mgr.14214 172.21.15.45:0/1227909437' entity='mgr.smithi045.nhtmwc' 2023-12-16T21:35:33.650 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Dec 16 21:35:33 smithi045 ceph-mon[80763]: from='osd.1 [v2:172.21.15.45:6810/3869108635,v1:172.21.15.45:6811/3869108635]' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2023-12-16T21:35:33.650 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Dec 16 21:35:33 smithi045 ceph-mon[80763]: pgmap v31: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2023-12-16T21:35:33.907 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:35:33.905+0000 7f093cdd5700 1 -- 172.21.15.45:0/741046353 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f0938071bc0 msgr2=0x7f0938070030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:35:33.907 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:35:33.905+0000 7f093cdd5700 1 --2- 172.21.15.45:0/741046353 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f0938071bc0 0x7f0938070030 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7f092c004940 tx=0x7f092c009e30 comp rx=0 tx=0).stop 2023-12-16T21:35:33.907 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:35:33.905+0000 7f093cdd5700 1 -- 172.21.15.45:0/741046353 shutdown_connections 2023-12-16T21:35:33.908 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:35:33.905+0000 7f093cdd5700 1 --2- 172.21.15.45:0/741046353 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f0938070570 0x7f09380709c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:35:33.908 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:35:33.905+0000 7f093cdd5700 1 --2- 172.21.15.45:0/741046353 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f0938071bc0 0x7f0938070030 unknown :-1 s=CLOSED pgs=169 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:35:33.908 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:35:33.905+0000 7f093cdd5700 1 -- 172.21.15.45:0/741046353 >> 172.21.15.45:0/741046353 conn(0x7f09380f82d0 msgr2=0x7f09380fa6e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-16T21:35:33.908 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:35:33.905+0000 7f093cdd5700 1 -- 172.21.15.45:0/741046353 shutdown_connections 2023-12-16T21:35:33.908 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:35:33.905+0000 7f093cdd5700 1 -- 172.21.15.45:0/741046353 wait complete. 2023-12-16T21:35:33.910 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:35:33.905+0000 7f093cdd5700 1 Processor -- start 2023-12-16T21:35:33.910 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:35:33.905+0000 7f093cdd5700 1 -- start start 2023-12-16T21:35:33.910 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:35:33.906+0000 7f093cdd5700 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f0938070570 0x7f0938113b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:35:33.910 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:35:33.906+0000 7f093cdd5700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f0938071bc0 0x7f0938114050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:35:33.910 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:35:33.906+0000 7f093cdd5700 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0938114650 con 0x7f0938071bc0 2023-12-16T21:35:33.910 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:35:33.906+0000 7f093cdd5700 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0938114ef0 con 0x7f0938070570 2023-12-16T21:35:33.910 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:35:33.906+0000 7f093659c700 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f0938070570 0x7f0938113b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:35:33.910 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:35:33.906+0000 7f093659c700 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f0938070570 0x7f0938113b10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.169:3300/0 says I am v2:172.21.15.45:57868/0 (socket says 172.21.15.45:57868) 2023-12-16T21:35:33.910 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:35:33.906+0000 7f093659c700 1 -- 172.21.15.45:0/2533902676 learned_addr learned my addr 172.21.15.45:0/2533902676 (peer_addr_for_me v2:172.21.15.45:0/0) 2023-12-16T21:35:33.910 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:35:33.906+0000 7f0935d9b700 1 --2- 172.21.15.45:0/2533902676 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f0938071bc0 0x7f0938114050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:35:33.910 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:35:33.906+0000 7f093659c700 1 -- 172.21.15.45:0/2533902676 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f0938071bc0 msgr2=0x7f0938114050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:35:33.911 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:35:33.906+0000 7f093659c700 1 --2- 172.21.15.45:0/2533902676 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f0938071bc0 0x7f0938114050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:35:33.911 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:35:33.906+0000 7f093659c700 1 -- 172.21.15.45:0/2533902676 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f092c005040 con 0x7f0938070570 2023-12-16T21:35:33.911 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:35:33.907+0000 7f093659c700 1 --2- 172.21.15.45:0/2533902676 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f0938070570 0x7f0938113b10 secure :-1 s=READY pgs=3 cs=0 l=1 rev1=1 crypto rx=0x7f092c009360 tx=0x7f092c009e30 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:35:33.911 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:35:33.907+0000 7f0926ffd700 1 -- 172.21.15.45:0/2533902676 <== mon.1 v2:172.21.15.169:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f092c00e040 con 0x7f0938070570 2023-12-16T21:35:33.911 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:35:33.907+0000 7f0926ffd700 1 -- 172.21.15.45:0/2533902676 <== mon.1 v2:172.21.15.169:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f092c00bc50 con 0x7f0938070570 2023-12-16T21:35:33.911 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:35:33.907+0000 7f093cdd5700 1 -- 172.21.15.45:0/2533902676 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0938115170 con 0x7f0938070570 2023-12-16T21:35:33.911 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:35:33.907+0000 7f0926ffd700 1 -- 172.21.15.45:0/2533902676 <== mon.1 v2:172.21.15.169:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f092c01c3f0 con 0x7f0938070570 2023-12-16T21:35:33.911 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:35:33.907+0000 7f093cdd5700 1 -- 172.21.15.45:0/2533902676 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f09381156b0 con 0x7f0938070570 2023-12-16T21:35:33.911 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:35:33.908+0000 7f093cdd5700 1 -- 172.21.15.45:0/2533902676 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f093804bb80 con 0x7f0938070570 2023-12-16T21:35:33.912 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:35:33.908+0000 7f0926ffd700 1 -- 172.21.15.45:0/2533902676 <== mon.1 v2:172.21.15.169:3300/0 4 ==== mgrmap(e 18) v1 ==== 86391+0+0 (secure 0 0 0) 0x7f092c013420 con 0x7f0938070570 2023-12-16T21:35:33.912 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:35:33.909+0000 7f0926ffd700 1 --2- 172.21.15.45:0/2533902676 >> [v2:172.21.15.45:6800/2012105536,v1:172.21.15.45:6801/2012105536] conn(0x7f0920068ed0 0x7f092006b380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:35:33.912 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:35:33.909+0000 7f0926ffd700 1 -- 172.21.15.45:0/2533902676 <== mon.1 v2:172.21.15.169:3300/0 5 ==== osd_map(12..12 src has 1..12) v4 ==== 2114+0+0 (secure 0 0 0) 0x7f092c0872f0 con 0x7f0938070570 2023-12-16T21:35:33.912 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:35:33.909+0000 7f0935d9b700 1 --2- 172.21.15.45:0/2533902676 >> [v2:172.21.15.45:6800/2012105536,v1:172.21.15.45:6801/2012105536] conn(0x7f0920068ed0 0x7f092006b380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:35:33.912 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:35:33.909+0000 7f0935d9b700 1 --2- 172.21.15.45:0/2533902676 >> [v2:172.21.15.45:6800/2012105536,v1:172.21.15.45:6801/2012105536] conn(0x7f0920068ed0 0x7f092006b380 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7f0928004f40 tx=0x7f092800b040 comp rx=0 tx=0).ready entity=mgr.14214 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:35:33.916 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:35:33.914+0000 7f0926ffd700 1 -- 172.21.15.45:0/2533902676 <== mon.1 v2:172.21.15.169:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+174570 (secure 0 0 0) 0x7f092c057fc0 con 0x7f0938070570 2023-12-16T21:35:34.122 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:35:34.120+0000 7f093cdd5700 1 -- 172.21.15.45:0/2533902676 --> [v2:172.21.15.45:6800/2012105536,v1:172.21.15.45:6801/2012105536] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi045:vg_nvme/lv_2", "target": ["mon-mgr", ""]}) v1 -- 0x7f09380ffd50 con 0x7f0920068ed0 2023-12-16T21:35:34.561 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:35:34 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:35:34.391+0000 7fa1b4f77700 -1 osd.1 0 waiting for initial osdmap 2023-12-16T21:35:34.561 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:35:34 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:35:34.398+0000 7fa1b1d76700 -1 osd.1 13 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2023-12-16T21:35:34.561 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Dec 16 21:35:34 smithi045 ceph-mon[80763]: from='osd.1 [v2:172.21.15.45:6810/3869108635,v1:172.21.15.45:6811/3869108635]' entity='osd.1' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]': finished 2023-12-16T21:35:34.561 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Dec 16 21:35:34 smithi045 ceph-mon[80763]: osdmap e12: 2 total, 1 up, 2 in 2023-12-16T21:35:34.561 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Dec 16 21:35:34 smithi045 ceph-mon[80763]: from='mgr.14214 172.21.15.45:0/1227909437' entity='mgr.smithi045.nhtmwc' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2023-12-16T21:35:34.561 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Dec 16 21:35:34 smithi045 ceph-mon[80763]: from='osd.1 [v2:172.21.15.45:6810/3869108635,v1:172.21.15.45:6811/3869108635]' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi045", "root=default"]}]: dispatch 2023-12-16T21:35:34.561 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Dec 16 21:35:34 smithi045 ceph-mon[80763]: from='mgr.14214 172.21.15.45:0/1227909437' entity='mgr.smithi045.nhtmwc' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2023-12-16T21:35:34.561 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Dec 16 21:35:34 smithi045 ceph-mon[80763]: from='mgr.14214 172.21.15.45:0/1227909437' entity='mgr.smithi045.nhtmwc' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2023-12-16T21:35:34.562 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Dec 16 21:35:34 smithi045 ceph-mon[80763]: from='mgr.14214 172.21.15.45:0/1227909437' entity='mgr.smithi045.nhtmwc' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-12-16T21:35:34.883 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Dec 16 21:35:34 smithi169 ceph-mon[88350]: from='osd.1 [v2:172.21.15.45:6810/3869108635,v1:172.21.15.45:6811/3869108635]' entity='osd.1' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]': finished 2023-12-16T21:35:34.883 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Dec 16 21:35:34 smithi169 ceph-mon[88350]: osdmap e12: 2 total, 1 up, 2 in 2023-12-16T21:35:34.883 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Dec 16 21:35:34 smithi169 ceph-mon[88350]: from='mgr.14214 172.21.15.45:0/1227909437' entity='mgr.smithi045.nhtmwc' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2023-12-16T21:35:34.883 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Dec 16 21:35:34 smithi169 ceph-mon[88350]: from='osd.1 [v2:172.21.15.45:6810/3869108635,v1:172.21.15.45:6811/3869108635]' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi045", "root=default"]}]: dispatch 2023-12-16T21:35:34.884 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Dec 16 21:35:34 smithi169 ceph-mon[88350]: from='mgr.14214 172.21.15.45:0/1227909437' entity='mgr.smithi045.nhtmwc' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2023-12-16T21:35:34.884 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Dec 16 21:35:34 smithi169 ceph-mon[88350]: from='mgr.14214 172.21.15.45:0/1227909437' entity='mgr.smithi045.nhtmwc' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2023-12-16T21:35:34.884 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Dec 16 21:35:34 smithi169 ceph-mon[88350]: from='mgr.14214 172.21.15.45:0/1227909437' entity='mgr.smithi045.nhtmwc' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-12-16T21:35:35.500 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Dec 16 21:35:35 smithi045 ceph-mon[80763]: from='client.24127 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi045:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2023-12-16T21:35:35.500 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Dec 16 21:35:35 smithi045 ceph-mon[80763]: from='osd.1 [v2:172.21.15.45:6810/3869108635,v1:172.21.15.45:6811/3869108635]' entity='osd.1' cmd='[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi045", "root=default"]}]': finished 2023-12-16T21:35:35.500 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Dec 16 21:35:35 smithi045 ceph-mon[80763]: osdmap e13: 2 total, 1 up, 2 in 2023-12-16T21:35:35.501 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Dec 16 21:35:35 smithi045 ceph-mon[80763]: from='mgr.14214 172.21.15.45:0/1227909437' entity='mgr.smithi045.nhtmwc' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2023-12-16T21:35:35.501 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Dec 16 21:35:35 smithi045 ceph-mon[80763]: from='mgr.14214 172.21.15.45:0/1227909437' entity='mgr.smithi045.nhtmwc' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2023-12-16T21:35:35.501 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Dec 16 21:35:35 smithi045 ceph-mon[80763]: pgmap v34: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2023-12-16T21:35:35.501 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Dec 16 21:35:35 smithi045 ceph-mon[80763]: from='mgr.14214 172.21.15.45:0/1227909437' entity='mgr.smithi045.nhtmwc' 2023-12-16T21:35:35.501 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Dec 16 21:35:35 smithi045 ceph-mon[80763]: from='mgr.14214 172.21.15.45:0/1227909437' entity='mgr.smithi045.nhtmwc' 2023-12-16T21:35:35.501 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Dec 16 21:35:35 smithi045 ceph-mon[80763]: from='mgr.14214 172.21.15.45:0/1227909437' entity='mgr.smithi045.nhtmwc' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2023-12-16T21:35:35.501 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Dec 16 21:35:35 smithi045 ceph-mon[80763]: from='mgr.14214 172.21.15.45:0/1227909437' entity='mgr.smithi045.nhtmwc' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2023-12-16T21:35:35.501 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Dec 16 21:35:35 smithi045 ceph-mon[80763]: from='mgr.14214 172.21.15.45:0/1227909437' entity='mgr.smithi045.nhtmwc' 2023-12-16T21:35:35.501 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Dec 16 21:35:35 smithi045 ceph-mon[80763]: from='mgr.14214 172.21.15.45:0/1227909437' entity='mgr.smithi045.nhtmwc' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-12-16T21:35:35.501 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Dec 16 21:35:35 smithi045 ceph-mon[80763]: from='mgr.14214 172.21.15.45:0/1227909437' entity='mgr.smithi045.nhtmwc' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-12-16T21:35:35.502 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Dec 16 21:35:35 smithi045 ceph-mon[80763]: from='mgr.14214 172.21.15.45:0/1227909437' entity='mgr.smithi045.nhtmwc' 2023-12-16T21:35:35.502 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Dec 16 21:35:35 smithi045 ceph-mon[80763]: from='mgr.14214 172.21.15.45:0/1227909437' entity='mgr.smithi045.nhtmwc' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2023-12-16T21:35:35.883 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Dec 16 21:35:35 smithi169 ceph-mon[88350]: from='client.24127 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi045:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2023-12-16T21:35:35.883 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Dec 16 21:35:35 smithi169 ceph-mon[88350]: from='osd.1 [v2:172.21.15.45:6810/3869108635,v1:172.21.15.45:6811/3869108635]' entity='osd.1' cmd='[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi045", "root=default"]}]': finished 2023-12-16T21:35:35.884 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Dec 16 21:35:35 smithi169 ceph-mon[88350]: osdmap e13: 2 total, 1 up, 2 in 2023-12-16T21:35:35.884 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Dec 16 21:35:35 smithi169 ceph-mon[88350]: from='mgr.14214 172.21.15.45:0/1227909437' entity='mgr.smithi045.nhtmwc' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2023-12-16T21:35:35.884 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Dec 16 21:35:35 smithi169 ceph-mon[88350]: from='mgr.14214 172.21.15.45:0/1227909437' entity='mgr.smithi045.nhtmwc' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2023-12-16T21:35:35.884 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Dec 16 21:35:35 smithi169 ceph-mon[88350]: pgmap v34: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2023-12-16T21:35:35.884 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Dec 16 21:35:35 smithi169 ceph-mon[88350]: from='mgr.14214 172.21.15.45:0/1227909437' entity='mgr.smithi045.nhtmwc' 2023-12-16T21:35:35.884 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Dec 16 21:35:35 smithi169 ceph-mon[88350]: from='mgr.14214 172.21.15.45:0/1227909437' entity='mgr.smithi045.nhtmwc' 2023-12-16T21:35:35.884 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Dec 16 21:35:35 smithi169 ceph-mon[88350]: from='mgr.14214 172.21.15.45:0/1227909437' entity='mgr.smithi045.nhtmwc' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2023-12-16T21:35:35.884 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Dec 16 21:35:35 smithi169 ceph-mon[88350]: from='mgr.14214 172.21.15.45:0/1227909437' entity='mgr.smithi045.nhtmwc' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2023-12-16T21:35:35.884 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Dec 16 21:35:35 smithi169 ceph-mon[88350]: from='mgr.14214 172.21.15.45:0/1227909437' entity='mgr.smithi045.nhtmwc' 2023-12-16T21:35:35.884 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Dec 16 21:35:35 smithi169 ceph-mon[88350]: from='mgr.14214 172.21.15.45:0/1227909437' entity='mgr.smithi045.nhtmwc' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-12-16T21:35:35.884 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Dec 16 21:35:35 smithi169 ceph-mon[88350]: from='mgr.14214 172.21.15.45:0/1227909437' entity='mgr.smithi045.nhtmwc' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-12-16T21:35:35.884 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Dec 16 21:35:35 smithi169 ceph-mon[88350]: from='mgr.14214 172.21.15.45:0/1227909437' entity='mgr.smithi045.nhtmwc' 2023-12-16T21:35:35.884 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Dec 16 21:35:35 smithi169 ceph-mon[88350]: from='mgr.14214 172.21.15.45:0/1227909437' entity='mgr.smithi045.nhtmwc' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2023-12-16T21:35:36.660 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Dec 16 21:35:36 smithi045 ceph-mon[80763]: purged_snaps scrub starts 2023-12-16T21:35:36.660 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Dec 16 21:35:36 smithi045 ceph-mon[80763]: purged_snaps scrub ok 2023-12-16T21:35:36.660 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Dec 16 21:35:36 smithi045 ceph-mon[80763]: Detected new or changed devices on smithi045 2023-12-16T21:35:36.661 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Dec 16 21:35:36 smithi045 ceph-mon[80763]: Adjusting osd_memory_target on smithi045 to 6434M 2023-12-16T21:35:36.661 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Dec 16 21:35:36 smithi045 ceph-mon[80763]: osd.1 [v2:172.21.15.45:6810/3869108635,v1:172.21.15.45:6811/3869108635] boot 2023-12-16T21:35:36.661 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Dec 16 21:35:36 smithi045 ceph-mon[80763]: osdmap e14: 2 total, 2 up, 2 in 2023-12-16T21:35:36.661 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Dec 16 21:35:36 smithi045 ceph-mon[80763]: from='mgr.14214 172.21.15.45:0/1227909437' entity='mgr.smithi045.nhtmwc' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2023-12-16T21:35:36.661 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Dec 16 21:35:36 smithi045 ceph-mon[80763]: from='mgr.14214 172.21.15.45:0/1227909437' entity='mgr.smithi045.nhtmwc' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-12-16T21:35:36.661 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Dec 16 21:35:36 smithi045 ceph-mon[80763]: from='client.? 172.21.15.45:0/4200789613' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "e4c3ac33-08a6-439a-9a83-31488cc35202"}]: dispatch 2023-12-16T21:35:36.661 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Dec 16 21:35:36 smithi045 ceph-mon[80763]: from='client.? 172.21.15.45:0/4200789613' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "e4c3ac33-08a6-439a-9a83-31488cc35202"}]': finished 2023-12-16T21:35:36.661 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Dec 16 21:35:36 smithi045 ceph-mon[80763]: osdmap e15: 3 total, 2 up, 3 in 2023-12-16T21:35:36.661 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Dec 16 21:35:36 smithi045 ceph-mon[80763]: from='mgr.14214 172.21.15.45:0/1227909437' entity='mgr.smithi045.nhtmwc' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2023-12-16T21:35:36.883 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Dec 16 21:35:36 smithi169 ceph-mon[88350]: purged_snaps scrub starts 2023-12-16T21:35:36.884 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Dec 16 21:35:36 smithi169 ceph-mon[88350]: purged_snaps scrub ok 2023-12-16T21:35:36.884 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Dec 16 21:35:36 smithi169 ceph-mon[88350]: Detected new or changed devices on smithi045 2023-12-16T21:35:36.884 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Dec 16 21:35:36 smithi169 ceph-mon[88350]: Adjusting osd_memory_target on smithi045 to 6434M 2023-12-16T21:35:36.884 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Dec 16 21:35:36 smithi169 ceph-mon[88350]: osd.1 [v2:172.21.15.45:6810/3869108635,v1:172.21.15.45:6811/3869108635] boot 2023-12-16T21:35:36.884 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Dec 16 21:35:36 smithi169 ceph-mon[88350]: osdmap e14: 2 total, 2 up, 2 in 2023-12-16T21:35:36.884 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Dec 16 21:35:36 smithi169 ceph-mon[88350]: from='mgr.14214 172.21.15.45:0/1227909437' entity='mgr.smithi045.nhtmwc' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2023-12-16T21:35:36.884 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Dec 16 21:35:36 smithi169 ceph-mon[88350]: from='mgr.14214 172.21.15.45:0/1227909437' entity='mgr.smithi045.nhtmwc' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-12-16T21:35:36.884 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Dec 16 21:35:36 smithi169 ceph-mon[88350]: from='client.? 172.21.15.45:0/4200789613' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "e4c3ac33-08a6-439a-9a83-31488cc35202"}]: dispatch 2023-12-16T21:35:36.884 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Dec 16 21:35:36 smithi169 ceph-mon[88350]: from='client.? 172.21.15.45:0/4200789613' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "e4c3ac33-08a6-439a-9a83-31488cc35202"}]': finished 2023-12-16T21:35:36.884 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Dec 16 21:35:36 smithi169 ceph-mon[88350]: osdmap e15: 3 total, 2 up, 3 in 2023-12-16T21:35:36.884 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Dec 16 21:35:36 smithi169 ceph-mon[88350]: from='mgr.14214 172.21.15.45:0/1227909437' entity='mgr.smithi045.nhtmwc' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2023-12-16T21:35:37.816 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Dec 16 21:35:37 smithi045 ceph-mon[80763]: pgmap v37: 0 pgs: ; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2023-12-16T21:35:37.816 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Dec 16 21:35:37 smithi045 ceph-mon[80763]: from='client.? 172.21.15.45:0/2038880413' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2023-12-16T21:35:37.883 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Dec 16 21:35:37 smithi169 ceph-mon[88350]: pgmap v37: 0 pgs: ; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2023-12-16T21:35:37.883 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Dec 16 21:35:37 smithi169 ceph-mon[88350]: from='client.? 172.21.15.45:0/2038880413' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2023-12-16T21:35:39.566 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Dec 16 21:35:39 smithi045 ceph-mon[80763]: from='mgr.14214 172.21.15.45:0/1227909437' entity='mgr.smithi045.nhtmwc' 2023-12-16T21:35:39.566 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Dec 16 21:35:39 smithi045 ceph-mon[80763]: from='mgr.14214 172.21.15.45:0/1227909437' entity='mgr.smithi045.nhtmwc' 2023-12-16T21:35:39.566 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Dec 16 21:35:39 smithi045 ceph-mon[80763]: from='mgr.14214 172.21.15.45:0/1227909437' entity='mgr.smithi045.nhtmwc' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-12-16T21:35:39.566 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Dec 16 21:35:39 smithi045 ceph-mon[80763]: from='mgr.14214 172.21.15.45:0/1227909437' entity='mgr.smithi045.nhtmwc' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-12-16T21:35:39.566 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Dec 16 21:35:39 smithi045 ceph-mon[80763]: from='mgr.14214 172.21.15.45:0/1227909437' entity='mgr.smithi045.nhtmwc' 2023-12-16T21:35:39.566 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Dec 16 21:35:39 smithi045 ceph-mon[80763]: pgmap v38: 0 pgs: ; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2023-12-16T21:35:39.633 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Dec 16 21:35:39 smithi169 ceph-mon[88350]: from='mgr.14214 172.21.15.45:0/1227909437' entity='mgr.smithi045.nhtmwc' 2023-12-16T21:35:39.634 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Dec 16 21:35:39 smithi169 ceph-mon[88350]: from='mgr.14214 172.21.15.45:0/1227909437' entity='mgr.smithi045.nhtmwc' 2023-12-16T21:35:39.634 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Dec 16 21:35:39 smithi169 ceph-mon[88350]: from='mgr.14214 172.21.15.45:0/1227909437' entity='mgr.smithi045.nhtmwc' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-12-16T21:35:39.634 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Dec 16 21:35:39 smithi169 ceph-mon[88350]: from='mgr.14214 172.21.15.45:0/1227909437' entity='mgr.smithi045.nhtmwc' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-12-16T21:35:39.634 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Dec 16 21:35:39 smithi169 ceph-mon[88350]: from='mgr.14214 172.21.15.45:0/1227909437' entity='mgr.smithi045.nhtmwc' 2023-12-16T21:35:39.634 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Dec 16 21:35:39 smithi169 ceph-mon[88350]: pgmap v38: 0 pgs: ; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2023-12-16T21:35:42.066 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Dec 16 21:35:41 smithi045 ceph-mon[80763]: pgmap v39: 0 pgs: ; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2023-12-16T21:35:42.133 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Dec 16 21:35:41 smithi169 ceph-mon[88350]: pgmap v39: 0 pgs: ; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2023-12-16T21:35:42.899 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Dec 16 21:35:42 smithi045 ceph-mon[80763]: from='mgr.14214 172.21.15.45:0/1227909437' entity='mgr.smithi045.nhtmwc' cmd=[{"prefix": "auth get", "entity": "osd.2"}]: dispatch 2023-12-16T21:35:42.899 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Dec 16 21:35:42 smithi045 ceph-mon[80763]: from='mgr.14214 172.21.15.45:0/1227909437' entity='mgr.smithi045.nhtmwc' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-12-16T21:35:43.133 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Dec 16 21:35:42 smithi169 ceph-mon[88350]: from='mgr.14214 172.21.15.45:0/1227909437' entity='mgr.smithi045.nhtmwc' cmd=[{"prefix": "auth get", "entity": "osd.2"}]: dispatch 2023-12-16T21:35:43.133 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Dec 16 21:35:42 smithi169 ceph-mon[88350]: from='mgr.14214 172.21.15.45:0/1227909437' entity='mgr.smithi045.nhtmwc' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-12-16T21:35:43.816 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Dec 16 21:35:43 smithi045 ceph-mon[80763]: Deploying daemon osd.2 on smithi045 2023-12-16T21:35:43.816 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Dec 16 21:35:43 smithi045 ceph-mon[80763]: pgmap v40: 0 pgs: ; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2023-12-16T21:35:43.816 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Dec 16 21:35:43 smithi045 ceph-mon[80763]: from='mgr.14214 172.21.15.45:0/1227909437' entity='mgr.smithi045.nhtmwc' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2023-12-16T21:35:44.034 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Dec 16 21:35:43 smithi169 ceph-mon[88350]: Deploying daemon osd.2 on smithi045 2023-12-16T21:35:44.035 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Dec 16 21:35:43 smithi169 ceph-mon[88350]: pgmap v40: 0 pgs: ; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2023-12-16T21:35:44.035 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Dec 16 21:35:43 smithi169 ceph-mon[88350]: from='mgr.14214 172.21.15.45:0/1227909437' entity='mgr.smithi045.nhtmwc' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2023-12-16T21:35:45.849 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Dec 16 21:35:45 smithi045 ceph-mon[80763]: pgmap v41: 0 pgs: ; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2023-12-16T21:35:45.849 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Dec 16 21:35:45 smithi045 ceph-mon[80763]: from='mgr.14214 172.21.15.45:0/1227909437' entity='mgr.smithi045.nhtmwc' 2023-12-16T21:35:45.849 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Dec 16 21:35:45 smithi045 ceph-mon[80763]: from='mgr.14214 172.21.15.45:0/1227909437' entity='mgr.smithi045.nhtmwc' 2023-12-16T21:35:45.849 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Dec 16 21:35:45 smithi045 ceph-mon[80763]: from='mgr.14214 172.21.15.45:0/1227909437' entity='mgr.smithi045.nhtmwc' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-12-16T21:35:46.133 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Dec 16 21:35:45 smithi169 ceph-mon[88350]: pgmap v41: 0 pgs: ; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2023-12-16T21:35:46.134 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Dec 16 21:35:45 smithi169 ceph-mon[88350]: from='mgr.14214 172.21.15.45:0/1227909437' entity='mgr.smithi045.nhtmwc' 2023-12-16T21:35:46.134 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Dec 16 21:35:45 smithi169 ceph-mon[88350]: from='mgr.14214 172.21.15.45:0/1227909437' entity='mgr.smithi045.nhtmwc' 2023-12-16T21:35:46.134 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Dec 16 21:35:45 smithi169 ceph-mon[88350]: from='mgr.14214 172.21.15.45:0/1227909437' entity='mgr.smithi045.nhtmwc' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-12-16T21:35:47.217 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:35:47.214+0000 7f0926ffd700 1 -- 172.21.15.45:0/2533902676 <== mgr.14214 v2:172.21.15.45:6800/2012105536 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+37 (secure 0 0 0) 0x7f09380ffd50 con 0x7f0920068ed0 2023-12-16T21:35:47.217 INFO:teuthology.orchestra.run.smithi045.stdout:Created osd(s) 2 on host 'smithi045' 2023-12-16T21:35:47.218 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:35:47.216+0000 7f093cdd5700 1 -- 172.21.15.45:0/2533902676 >> [v2:172.21.15.45:6800/2012105536,v1:172.21.15.45:6801/2012105536] conn(0x7f0920068ed0 msgr2=0x7f092006b380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:35:47.219 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:35:47.216+0000 7f093cdd5700 1 --2- 172.21.15.45:0/2533902676 >> [v2:172.21.15.45:6800/2012105536,v1:172.21.15.45:6801/2012105536] conn(0x7f0920068ed0 0x7f092006b380 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7f0928004f40 tx=0x7f092800b040 comp rx=0 tx=0).stop 2023-12-16T21:35:47.219 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:35:47.216+0000 7f093cdd5700 1 -- 172.21.15.45:0/2533902676 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f0938070570 msgr2=0x7f0938113b10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:35:47.219 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:35:47.216+0000 7f093cdd5700 1 --2- 172.21.15.45:0/2533902676 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f0938070570 0x7f0938113b10 secure :-1 s=READY pgs=3 cs=0 l=1 rev1=1 crypto rx=0x7f092c009360 tx=0x7f092c009e30 comp rx=0 tx=0).stop 2023-12-16T21:35:47.219 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:35:47.216+0000 7f093cdd5700 1 -- 172.21.15.45:0/2533902676 shutdown_connections 2023-12-16T21:35:47.219 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:35:47.216+0000 7f093cdd5700 1 --2- 172.21.15.45:0/2533902676 >> [v2:172.21.15.45:6800/2012105536,v1:172.21.15.45:6801/2012105536] conn(0x7f0920068ed0 0x7f092006b380 unknown :-1 s=CLOSED pgs=32 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:35:47.219 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:35:47.216+0000 7f093cdd5700 1 --2- 172.21.15.45:0/2533902676 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f0938070570 0x7f0938113b10 unknown :-1 s=CLOSED pgs=3 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:35:47.219 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:35:47.216+0000 7f093cdd5700 1 --2- 172.21.15.45:0/2533902676 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f0938071bc0 0x7f0938114050 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:35:47.219 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:35:47.216+0000 7f093cdd5700 1 -- 172.21.15.45:0/2533902676 >> 172.21.15.45:0/2533902676 conn(0x7f09380f82d0 msgr2=0x7f09380fe630 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-16T21:35:47.219 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:35:47.216+0000 7f093cdd5700 1 -- 172.21.15.45:0/2533902676 shutdown_connections 2023-12-16T21:35:47.219 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:35:47.216+0000 7f093cdd5700 1 -- 172.21.15.45:0/2533902676 wait complete. 2023-12-16T21:35:47.394 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Dec 16 21:35:47 smithi045 ceph-mon[80763]: pgmap v42: 0 pgs: ; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2023-12-16T21:35:47.395 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Dec 16 21:35:47 smithi045 ceph-mon[80763]: from='mgr.14214 172.21.15.45:0/1227909437' entity='mgr.smithi045.nhtmwc' 2023-12-16T21:35:47.395 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Dec 16 21:35:47 smithi045 ceph-mon[80763]: from='mgr.14214 172.21.15.45:0/1227909437' entity='mgr.smithi045.nhtmwc' 2023-12-16T21:35:47.633 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Dec 16 21:35:47 smithi169 ceph-mon[88350]: pgmap v42: 0 pgs: ; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2023-12-16T21:35:47.633 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Dec 16 21:35:47 smithi169 ceph-mon[88350]: from='mgr.14214 172.21.15.45:0/1227909437' entity='mgr.smithi045.nhtmwc' 2023-12-16T21:35:47.634 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Dec 16 21:35:47 smithi169 ceph-mon[88350]: from='mgr.14214 172.21.15.45:0/1227909437' entity='mgr.smithi045.nhtmwc' 2023-12-16T21:35:47.706 DEBUG:teuthology.orchestra.run.smithi045:osd.2> sudo journalctl -f -n 0 -u ceph-88ff363c-9c5a-11ee-95a5-87774f69a715@osd.2.service 2023-12-16T21:35:47.708 INFO:tasks.cephadm:Deploying osd.3 on smithi169 with /dev/vg_nvme/lv_4... 2023-12-16T21:35:47.708 DEBUG:teuthology.orchestra.run.smithi169:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 88ff363c-9c5a-11ee-95a5-87774f69a715 -- ceph-volume lvm zap /dev/vg_nvme/lv_4 2023-12-16T21:35:47.969 INFO:teuthology.orchestra.run.smithi169.stderr:Inferring config /var/lib/ceph/88ff363c-9c5a-11ee-95a5-87774f69a715/mon.smithi169/config 2023-12-16T21:35:49.050 INFO:teuthology.orchestra.run.smithi169.stderr:--> Zapping: /dev/vg_nvme/lv_4 2023-12-16T21:35:49.050 INFO:teuthology.orchestra.run.smithi169.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_4 bs=1M count=10 conv=fsync 2023-12-16T21:35:49.050 INFO:teuthology.orchestra.run.smithi169.stderr: stderr: 10+0 records in 2023-12-16T21:35:49.050 INFO:teuthology.orchestra.run.smithi169.stderr:10+0 records out 2023-12-16T21:35:49.050 INFO:teuthology.orchestra.run.smithi169.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0127353 s, 823 MB/s 2023-12-16T21:35:49.050 INFO:teuthology.orchestra.run.smithi169.stderr:--> Zapping successful for: 2023-12-16T21:35:49.291 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:35:49 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:35:49.103+0000 7f9517ca3540 -1 osd.2 0 log_to_monitors true 2023-12-16T21:35:49.322 DEBUG:teuthology.orchestra.run.smithi169:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 88ff363c-9c5a-11ee-95a5-87774f69a715 -- ceph orch daemon add osd smithi169:vg_nvme/lv_4 2023-12-16T21:35:49.570 INFO:teuthology.orchestra.run.smithi169.stderr:Inferring config /var/lib/ceph/88ff363c-9c5a-11ee-95a5-87774f69a715/mon.smithi169/config 2023-12-16T21:35:49.816 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Dec 16 21:35:49 smithi045 ceph-mon[80763]: from='mgr.14214 172.21.15.45:0/1227909437' entity='mgr.smithi045.nhtmwc' 2023-12-16T21:35:49.816 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Dec 16 21:35:49 smithi045 ceph-mon[80763]: from='mgr.14214 172.21.15.45:0/1227909437' entity='mgr.smithi045.nhtmwc' 2023-12-16T21:35:49.816 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Dec 16 21:35:49 smithi045 ceph-mon[80763]: pgmap v43: 0 pgs: ; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2023-12-16T21:35:49.816 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Dec 16 21:35:49 smithi045 ceph-mon[80763]: from='osd.2 [v2:172.21.15.45:6818/844221874,v1:172.21.15.45:6819/844221874]' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2023-12-16T21:35:49.883 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Dec 16 21:35:49 smithi169 ceph-mon[88350]: from='mgr.14214 172.21.15.45:0/1227909437' entity='mgr.smithi045.nhtmwc' 2023-12-16T21:35:49.883 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Dec 16 21:35:49 smithi169 ceph-mon[88350]: from='mgr.14214 172.21.15.45:0/1227909437' entity='mgr.smithi045.nhtmwc' 2023-12-16T21:35:49.883 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Dec 16 21:35:49 smithi169 ceph-mon[88350]: pgmap v43: 0 pgs: ; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2023-12-16T21:35:49.883 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Dec 16 21:35:49 smithi169 ceph-mon[88350]: from='osd.2 [v2:172.21.15.45:6818/844221874,v1:172.21.15.45:6819/844221874]' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2023-12-16T21:35:50.295 INFO:teuthology.orchestra.run.smithi169.stderr:2023-12-16T21:35:50.290+0000 7ffaa568f700 1 -- 172.21.15.169:0/2768079657 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7ffaa00fbaf0 msgr2=0x7ffaa00fbf00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:35:50.295 INFO:teuthology.orchestra.run.smithi169.stderr:2023-12-16T21:35:50.290+0000 7ffaa568f700 1 --2- 172.21.15.169:0/2768079657 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7ffaa00fbaf0 0x7ffaa00fbf00 secure :-1 s=READY pgs=4 cs=0 l=1 rev1=1 crypto rx=0x7ffa90004940 tx=0x7ffa90009e30 comp rx=0 tx=0).stop 2023-12-16T21:35:50.295 INFO:teuthology.orchestra.run.smithi169.stderr:2023-12-16T21:35:50.290+0000 7ffaa568f700 1 -- 172.21.15.169:0/2768079657 shutdown_connections 2023-12-16T21:35:50.295 INFO:teuthology.orchestra.run.smithi169.stderr:2023-12-16T21:35:50.290+0000 7ffaa568f700 1 --2- 172.21.15.169:0/2768079657 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ffaa00fcd00 0x7ffaa00fd150 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:35:50.295 INFO:teuthology.orchestra.run.smithi169.stderr:2023-12-16T21:35:50.290+0000 7ffaa568f700 1 --2- 172.21.15.169:0/2768079657 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7ffaa00fbaf0 0x7ffaa00fbf00 unknown :-1 s=CLOSED pgs=4 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:35:50.295 INFO:teuthology.orchestra.run.smithi169.stderr:2023-12-16T21:35:50.290+0000 7ffaa568f700 1 -- 172.21.15.169:0/2768079657 >> 172.21.15.169:0/2768079657 conn(0x7ffaa00f71d0 msgr2=0x7ffaa00f95e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-16T21:35:50.295 INFO:teuthology.orchestra.run.smithi169.stderr:2023-12-16T21:35:50.290+0000 7ffaa568f700 1 -- 172.21.15.169:0/2768079657 shutdown_connections 2023-12-16T21:35:50.295 INFO:teuthology.orchestra.run.smithi169.stderr:2023-12-16T21:35:50.290+0000 7ffaa568f700 1 -- 172.21.15.169:0/2768079657 wait complete. 2023-12-16T21:35:50.295 INFO:teuthology.orchestra.run.smithi169.stderr:2023-12-16T21:35:50.291+0000 7ffaa568f700 1 Processor -- start 2023-12-16T21:35:50.295 INFO:teuthology.orchestra.run.smithi169.stderr:2023-12-16T21:35:50.291+0000 7ffaa568f700 1 -- start start 2023-12-16T21:35:50.295 INFO:teuthology.orchestra.run.smithi169.stderr:2023-12-16T21:35:50.291+0000 7ffaa568f700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ffaa00fbaf0 0x7ffaa010c470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:35:50.295 INFO:teuthology.orchestra.run.smithi169.stderr:2023-12-16T21:35:50.291+0000 7ffaa568f700 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7ffaa00fcd00 0x7ffaa010c9b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:35:50.296 INFO:teuthology.orchestra.run.smithi169.stderr:2023-12-16T21:35:50.291+0000 7ffaa568f700 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffaa010cfb0 con 0x7ffaa00fbaf0 2023-12-16T21:35:50.296 INFO:teuthology.orchestra.run.smithi169.stderr:2023-12-16T21:35:50.291+0000 7ffaa568f700 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffaa010d850 con 0x7ffaa00fcd00 2023-12-16T21:35:50.296 INFO:teuthology.orchestra.run.smithi169.stderr:2023-12-16T21:35:50.291+0000 7ffa9e7fc700 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7ffaa00fcd00 0x7ffaa010c9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:35:50.296 INFO:teuthology.orchestra.run.smithi169.stderr:2023-12-16T21:35:50.292+0000 7ffa9e7fc700 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7ffaa00fcd00 0x7ffaa010c9b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.169:3300/0 says I am v2:172.21.15.169:55314/0 (socket says 172.21.15.169:55314) 2023-12-16T21:35:50.296 INFO:teuthology.orchestra.run.smithi169.stderr:2023-12-16T21:35:50.292+0000 7ffa9e7fc700 1 -- 172.21.15.169:0/3340131933 learned_addr learned my addr 172.21.15.169:0/3340131933 (peer_addr_for_me v2:172.21.15.169:0/0) 2023-12-16T21:35:50.296 INFO:teuthology.orchestra.run.smithi169.stderr:2023-12-16T21:35:50.292+0000 7ffa9effd700 1 --2- 172.21.15.169:0/3340131933 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ffaa00fbaf0 0x7ffaa010c470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:35:50.296 INFO:teuthology.orchestra.run.smithi169.stderr:2023-12-16T21:35:50.292+0000 7ffa9e7fc700 1 -- 172.21.15.169:0/3340131933 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ffaa00fbaf0 msgr2=0x7ffaa010c470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:35:50.296 INFO:teuthology.orchestra.run.smithi169.stderr:2023-12-16T21:35:50.292+0000 7ffa9e7fc700 1 --2- 172.21.15.169:0/3340131933 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ffaa00fbaf0 0x7ffaa010c470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:35:50.296 INFO:teuthology.orchestra.run.smithi169.stderr:2023-12-16T21:35:50.292+0000 7ffa9e7fc700 1 -- 172.21.15.169:0/3340131933 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffa90005040 con 0x7ffaa00fcd00 2023-12-16T21:35:50.296 INFO:teuthology.orchestra.run.smithi169.stderr:2023-12-16T21:35:50.292+0000 7ffa9e7fc700 1 --2- 172.21.15.169:0/3340131933 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7ffaa00fcd00 0x7ffaa010c9b0 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 crypto rx=0x7ffa9400fce0 tx=0x7ffa940046b0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:35:50.296 INFO:teuthology.orchestra.run.smithi169.stderr:2023-12-16T21:35:50.292+0000 7ffa877fe700 1 -- 172.21.15.169:0/3340131933 <== mon.1 v2:172.21.15.169:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7ffa94007ea0 con 0x7ffaa00fcd00 2023-12-16T21:35:50.296 INFO:teuthology.orchestra.run.smithi169.stderr:2023-12-16T21:35:50.292+0000 7ffaa568f700 1 -- 172.21.15.169:0/3340131933 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ffaa010dad0 con 0x7ffaa00fcd00 2023-12-16T21:35:50.297 INFO:teuthology.orchestra.run.smithi169.stderr:2023-12-16T21:35:50.292+0000 7ffa877fe700 1 -- 172.21.15.169:0/3340131933 <== mon.1 v2:172.21.15.169:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7ffa9400dd80 con 0x7ffaa00fcd00 2023-12-16T21:35:50.297 INFO:teuthology.orchestra.run.smithi169.stderr:2023-12-16T21:35:50.292+0000 7ffa877fe700 1 -- 172.21.15.169:0/3340131933 <== mon.1 v2:172.21.15.169:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7ffa94012460 con 0x7ffaa00fcd00 2023-12-16T21:35:50.297 INFO:teuthology.orchestra.run.smithi169.stderr:2023-12-16T21:35:50.292+0000 7ffaa568f700 1 -- 172.21.15.169:0/3340131933 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ffaa010e010 con 0x7ffaa00fcd00 2023-12-16T21:35:50.298 INFO:teuthology.orchestra.run.smithi169.stderr:2023-12-16T21:35:50.293+0000 7ffaa568f700 1 -- 172.21.15.169:0/3340131933 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ffaa004bb80 con 0x7ffaa00fcd00 2023-12-16T21:35:50.298 INFO:teuthology.orchestra.run.smithi169.stderr:2023-12-16T21:35:50.294+0000 7ffa877fe700 1 -- 172.21.15.169:0/3340131933 <== mon.1 v2:172.21.15.169:3300/0 4 ==== mgrmap(e 18) v1 ==== 86391+0+0 (secure 0 0 0) 0x7ffa94009360 con 0x7ffaa00fcd00 2023-12-16T21:35:50.298 INFO:teuthology.orchestra.run.smithi169.stderr:2023-12-16T21:35:50.294+0000 7ffa877fe700 1 --2- 172.21.15.169:0/3340131933 >> [v2:172.21.15.45:6800/2012105536,v1:172.21.15.45:6801/2012105536] conn(0x7ffa88068ed0 0x7ffa8806b380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:35:50.298 INFO:teuthology.orchestra.run.smithi169.stderr:2023-12-16T21:35:50.294+0000 7ffa877fe700 1 -- 172.21.15.169:0/3340131933 <== mon.1 v2:172.21.15.169:3300/0 5 ==== osd_map(16..16 src has 1..16) v4 ==== 2546+0+0 (secure 0 0 0) 0x7ffa940887d0 con 0x7ffaa00fcd00 2023-12-16T21:35:50.298 INFO:teuthology.orchestra.run.smithi169.stderr:2023-12-16T21:35:50.294+0000 7ffa9effd700 1 --2- 172.21.15.169:0/3340131933 >> [v2:172.21.15.45:6800/2012105536,v1:172.21.15.45:6801/2012105536] conn(0x7ffa88068ed0 0x7ffa8806b380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:35:50.298 INFO:teuthology.orchestra.run.smithi169.stderr:2023-12-16T21:35:50.295+0000 7ffa9effd700 1 --2- 172.21.15.169:0/3340131933 >> [v2:172.21.15.45:6800/2012105536,v1:172.21.15.45:6801/2012105536] conn(0x7ffa88068ed0 0x7ffa8806b380 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7ffa90004890 tx=0x7ffa90009e30 comp rx=0 tx=0).ready entity=mgr.14214 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:35:50.301 INFO:teuthology.orchestra.run.smithi169.stderr:2023-12-16T21:35:50.299+0000 7ffa877fe700 1 -- 172.21.15.169:0/3340131933 <== mon.1 v2:172.21.15.169:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+174570 (secure 0 0 0) 0x7ffa940592f0 con 0x7ffaa00fcd00 2023-12-16T21:35:50.493 INFO:teuthology.orchestra.run.smithi169.stderr:2023-12-16T21:35:50.490+0000 7ffaa568f700 1 -- 172.21.15.169:0/3340131933 --> [v2:172.21.15.45:6800/2012105536,v1:172.21.15.45:6801/2012105536] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi169:vg_nvme/lv_4", "target": ["mon-mgr", ""]}) v1 -- 0x7ffaa010e480 con 0x7ffa88068ed0 2023-12-16T21:35:50.878 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Dec 16 21:35:50 smithi169 ceph-mon[88350]: from='osd.2 [v2:172.21.15.45:6818/844221874,v1:172.21.15.45:6819/844221874]' entity='osd.2' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]': finished 2023-12-16T21:35:50.878 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Dec 16 21:35:50 smithi169 ceph-mon[88350]: osdmap e16: 3 total, 2 up, 3 in 2023-12-16T21:35:50.878 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Dec 16 21:35:50 smithi169 ceph-mon[88350]: from='mgr.14214 172.21.15.45:0/1227909437' entity='mgr.smithi045.nhtmwc' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2023-12-16T21:35:50.878 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Dec 16 21:35:50 smithi169 ceph-mon[88350]: from='osd.2 [v2:172.21.15.45:6818/844221874,v1:172.21.15.45:6819/844221874]' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi045", "root=default"]}]: dispatch 2023-12-16T21:35:50.878 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Dec 16 21:35:50 smithi169 ceph-mon[88350]: from='mgr.14214 172.21.15.45:0/1227909437' entity='mgr.smithi045.nhtmwc' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2023-12-16T21:35:50.878 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Dec 16 21:35:50 smithi169 ceph-mon[88350]: from='mgr.14214 172.21.15.45:0/1227909437' entity='mgr.smithi045.nhtmwc' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2023-12-16T21:35:50.878 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Dec 16 21:35:50 smithi169 ceph-mon[88350]: from='mgr.14214 172.21.15.45:0/1227909437' entity='mgr.smithi045.nhtmwc' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-12-16T21:35:50.878 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Dec 16 21:35:50 smithi169 ceph-mon[88350]: from='mgr.14214 172.21.15.45:0/1227909437' entity='mgr.smithi045.nhtmwc' 2023-12-16T21:35:50.879 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Dec 16 21:35:50 smithi169 ceph-mon[88350]: from='mgr.14214 172.21.15.45:0/1227909437' entity='mgr.smithi045.nhtmwc' 2023-12-16T21:35:50.879 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Dec 16 21:35:50 smithi169 ceph-mon[88350]: from='mgr.14214 172.21.15.45:0/1227909437' entity='mgr.smithi045.nhtmwc' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2023-12-16T21:35:50.879 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Dec 16 21:35:50 smithi169 ceph-mon[88350]: from='mgr.14214 172.21.15.45:0/1227909437' entity='mgr.smithi045.nhtmwc' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2023-12-16T21:35:50.879 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Dec 16 21:35:50 smithi169 ceph-mon[88350]: from='mgr.14214 172.21.15.45:0/1227909437' entity='mgr.smithi045.nhtmwc' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2023-12-16T21:35:50.879 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Dec 16 21:35:50 smithi169 ceph-mon[88350]: from='mgr.14214 172.21.15.45:0/1227909437' entity='mgr.smithi045.nhtmwc' 2023-12-16T21:35:50.879 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Dec 16 21:35:50 smithi169 ceph-mon[88350]: from='mgr.14214 172.21.15.45:0/1227909437' entity='mgr.smithi045.nhtmwc' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-12-16T21:35:50.879 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Dec 16 21:35:50 smithi169 ceph-mon[88350]: from='mgr.14214 172.21.15.45:0/1227909437' entity='mgr.smithi045.nhtmwc' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-12-16T21:35:50.879 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Dec 16 21:35:50 smithi169 ceph-mon[88350]: from='mgr.14214 172.21.15.45:0/1227909437' entity='mgr.smithi045.nhtmwc' 2023-12-16T21:35:51.066 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:35:50 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:35:50.598+0000 7f950e488700 -1 osd.2 0 waiting for initial osdmap 2023-12-16T21:35:51.066 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:35:50 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:35:50.603+0000 7f9509a84700 -1 osd.2 17 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2023-12-16T21:35:51.066 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Dec 16 21:35:50 smithi045 ceph-mon[80763]: from='osd.2 [v2:172.21.15.45:6818/844221874,v1:172.21.15.45:6819/844221874]' entity='osd.2' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]': finished 2023-12-16T21:35:51.066 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Dec 16 21:35:50 smithi045 ceph-mon[80763]: osdmap e16: 3 total, 2 up, 3 in 2023-12-16T21:35:51.067 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Dec 16 21:35:50 smithi045 ceph-mon[80763]: from='mgr.14214 172.21.15.45:0/1227909437' entity='mgr.smithi045.nhtmwc' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2023-12-16T21:35:51.067 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Dec 16 21:35:50 smithi045 ceph-mon[80763]: from='osd.2 [v2:172.21.15.45:6818/844221874,v1:172.21.15.45:6819/844221874]' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi045", "root=default"]}]: dispatch 2023-12-16T21:35:51.067 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Dec 16 21:35:50 smithi045 ceph-mon[80763]: from='mgr.14214 172.21.15.45:0/1227909437' entity='mgr.smithi045.nhtmwc' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2023-12-16T21:35:51.067 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Dec 16 21:35:50 smithi045 ceph-mon[80763]: from='mgr.14214 172.21.15.45:0/1227909437' entity='mgr.smithi045.nhtmwc' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2023-12-16T21:35:51.067 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Dec 16 21:35:50 smithi045 ceph-mon[80763]: from='mgr.14214 172.21.15.45:0/1227909437' entity='mgr.smithi045.nhtmwc' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-12-16T21:35:51.067 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Dec 16 21:35:50 smithi045 ceph-mon[80763]: from='mgr.14214 172.21.15.45:0/1227909437' entity='mgr.smithi045.nhtmwc' 2023-12-16T21:35:51.067 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Dec 16 21:35:50 smithi045 ceph-mon[80763]: from='mgr.14214 172.21.15.45:0/1227909437' entity='mgr.smithi045.nhtmwc' 2023-12-16T21:35:51.067 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Dec 16 21:35:50 smithi045 ceph-mon[80763]: from='mgr.14214 172.21.15.45:0/1227909437' entity='mgr.smithi045.nhtmwc' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2023-12-16T21:35:51.067 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Dec 16 21:35:50 smithi045 ceph-mon[80763]: from='mgr.14214 172.21.15.45:0/1227909437' entity='mgr.smithi045.nhtmwc' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2023-12-16T21:35:51.067 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Dec 16 21:35:50 smithi045 ceph-mon[80763]: from='mgr.14214 172.21.15.45:0/1227909437' entity='mgr.smithi045.nhtmwc' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2023-12-16T21:35:51.067 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Dec 16 21:35:50 smithi045 ceph-mon[80763]: from='mgr.14214 172.21.15.45:0/1227909437' entity='mgr.smithi045.nhtmwc' 2023-12-16T21:35:51.067 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Dec 16 21:35:50 smithi045 ceph-mon[80763]: from='mgr.14214 172.21.15.45:0/1227909437' entity='mgr.smithi045.nhtmwc' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-12-16T21:35:51.067 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Dec 16 21:35:50 smithi045 ceph-mon[80763]: from='mgr.14214 172.21.15.45:0/1227909437' entity='mgr.smithi045.nhtmwc' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-12-16T21:35:51.067 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Dec 16 21:35:50 smithi045 ceph-mon[80763]: from='mgr.14214 172.21.15.45:0/1227909437' entity='mgr.smithi045.nhtmwc' 2023-12-16T21:35:51.883 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Dec 16 21:35:51 smithi169 ceph-mon[88350]: from='client.24141 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi169:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2023-12-16T21:35:51.884 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Dec 16 21:35:51 smithi169 ceph-mon[88350]: Detected new or changed devices on smithi045 2023-12-16T21:35:51.884 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Dec 16 21:35:51 smithi169 ceph-mon[88350]: Adjusting osd_memory_target on smithi045 to 4289M 2023-12-16T21:35:51.884 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Dec 16 21:35:51 smithi169 ceph-mon[88350]: from='osd.2 [v2:172.21.15.45:6818/844221874,v1:172.21.15.45:6819/844221874]' entity='osd.2' cmd='[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi045", "root=default"]}]': finished 2023-12-16T21:35:51.884 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Dec 16 21:35:51 smithi169 ceph-mon[88350]: osdmap e17: 3 total, 2 up, 3 in 2023-12-16T21:35:51.884 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Dec 16 21:35:51 smithi169 ceph-mon[88350]: from='mgr.14214 172.21.15.45:0/1227909437' entity='mgr.smithi045.nhtmwc' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2023-12-16T21:35:51.884 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Dec 16 21:35:51 smithi169 ceph-mon[88350]: pgmap v46: 0 pgs: ; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2023-12-16T21:35:51.884 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Dec 16 21:35:51 smithi169 ceph-mon[88350]: from='mgr.14214 172.21.15.45:0/1227909437' entity='mgr.smithi045.nhtmwc' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2023-12-16T21:35:51.884 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Dec 16 21:35:51 smithi169 ceph-mon[88350]: from='mgr.14214 172.21.15.45:0/1227909437' entity='mgr.smithi045.nhtmwc' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-12-16T21:35:51.884 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Dec 16 21:35:51 smithi169 ceph-mon[88350]: from='mgr.14214 172.21.15.45:0/1227909437' entity='mgr.smithi045.nhtmwc' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-12-16T21:35:51.884 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Dec 16 21:35:51 smithi169 ceph-mon[88350]: from='mgr.14214 172.21.15.45:0/1227909437' entity='mgr.smithi045.nhtmwc' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-12-16T21:35:51.884 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Dec 16 21:35:51 smithi169 ceph-mon[88350]: from='mgr.14214 172.21.15.45:0/1227909437' entity='mgr.smithi045.nhtmwc' 2023-12-16T21:35:52.066 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Dec 16 21:35:51 smithi045 ceph-mon[80763]: from='client.24141 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi169:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2023-12-16T21:35:52.066 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Dec 16 21:35:51 smithi045 ceph-mon[80763]: Detected new or changed devices on smithi045 2023-12-16T21:35:52.066 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Dec 16 21:35:51 smithi045 ceph-mon[80763]: Adjusting osd_memory_target on smithi045 to 4289M 2023-12-16T21:35:52.066 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Dec 16 21:35:51 smithi045 ceph-mon[80763]: from='osd.2 [v2:172.21.15.45:6818/844221874,v1:172.21.15.45:6819/844221874]' entity='osd.2' cmd='[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi045", "root=default"]}]': finished 2023-12-16T21:35:52.067 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Dec 16 21:35:51 smithi045 ceph-mon[80763]: osdmap e17: 3 total, 2 up, 3 in 2023-12-16T21:35:52.067 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Dec 16 21:35:51 smithi045 ceph-mon[80763]: from='mgr.14214 172.21.15.45:0/1227909437' entity='mgr.smithi045.nhtmwc' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2023-12-16T21:35:52.067 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Dec 16 21:35:51 smithi045 ceph-mon[80763]: pgmap v46: 0 pgs: ; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2023-12-16T21:35:52.067 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Dec 16 21:35:51 smithi045 ceph-mon[80763]: from='mgr.14214 172.21.15.45:0/1227909437' entity='mgr.smithi045.nhtmwc' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2023-12-16T21:35:52.067 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Dec 16 21:35:51 smithi045 ceph-mon[80763]: from='mgr.14214 172.21.15.45:0/1227909437' entity='mgr.smithi045.nhtmwc' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-12-16T21:35:52.067 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Dec 16 21:35:51 smithi045 ceph-mon[80763]: from='mgr.14214 172.21.15.45:0/1227909437' entity='mgr.smithi045.nhtmwc' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-12-16T21:35:52.067 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Dec 16 21:35:51 smithi045 ceph-mon[80763]: from='mgr.14214 172.21.15.45:0/1227909437' entity='mgr.smithi045.nhtmwc' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-12-16T21:35:52.067 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Dec 16 21:35:51 smithi045 ceph-mon[80763]: from='mgr.14214 172.21.15.45:0/1227909437' entity='mgr.smithi045.nhtmwc' 2023-12-16T21:35:52.883 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Dec 16 21:35:52 smithi169 ceph-mon[88350]: purged_snaps scrub starts 2023-12-16T21:35:52.883 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Dec 16 21:35:52 smithi169 ceph-mon[88350]: purged_snaps scrub ok 2023-12-16T21:35:52.883 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Dec 16 21:35:52 smithi169 ceph-mon[88350]: osd.2 [v2:172.21.15.45:6818/844221874,v1:172.21.15.45:6819/844221874] boot 2023-12-16T21:35:52.884 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Dec 16 21:35:52 smithi169 ceph-mon[88350]: osdmap e18: 3 total, 3 up, 3 in 2023-12-16T21:35:52.884 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Dec 16 21:35:52 smithi169 ceph-mon[88350]: from='mgr.14214 172.21.15.45:0/1227909437' entity='mgr.smithi045.nhtmwc' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2023-12-16T21:35:52.884 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Dec 16 21:35:52 smithi169 ceph-mon[88350]: from='client.? 172.21.15.169:0/1711541115' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "957064f1-6dda-475b-903e-130134845a90"}]: dispatch 2023-12-16T21:35:52.884 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Dec 16 21:35:52 smithi169 ceph-mon[88350]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "957064f1-6dda-475b-903e-130134845a90"}]: dispatch 2023-12-16T21:35:52.884 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Dec 16 21:35:52 smithi169 ceph-mon[88350]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "957064f1-6dda-475b-903e-130134845a90"}]': finished 2023-12-16T21:35:52.884 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Dec 16 21:35:52 smithi169 ceph-mon[88350]: osdmap e19: 4 total, 3 up, 4 in 2023-12-16T21:35:52.884 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Dec 16 21:35:52 smithi169 ceph-mon[88350]: from='mgr.14214 172.21.15.45:0/1227909437' entity='mgr.smithi045.nhtmwc' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2023-12-16T21:35:53.066 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Dec 16 21:35:52 smithi045 ceph-mon[80763]: purged_snaps scrub starts 2023-12-16T21:35:53.066 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Dec 16 21:35:52 smithi045 ceph-mon[80763]: purged_snaps scrub ok 2023-12-16T21:35:53.066 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Dec 16 21:35:52 smithi045 ceph-mon[80763]: osd.2 [v2:172.21.15.45:6818/844221874,v1:172.21.15.45:6819/844221874] boot 2023-12-16T21:35:53.066 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Dec 16 21:35:52 smithi045 ceph-mon[80763]: osdmap e18: 3 total, 3 up, 3 in 2023-12-16T21:35:53.066 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Dec 16 21:35:52 smithi045 ceph-mon[80763]: from='mgr.14214 172.21.15.45:0/1227909437' entity='mgr.smithi045.nhtmwc' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2023-12-16T21:35:53.066 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Dec 16 21:35:52 smithi045 ceph-mon[80763]: from='client.? 172.21.15.169:0/1711541115' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "957064f1-6dda-475b-903e-130134845a90"}]: dispatch 2023-12-16T21:35:53.066 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Dec 16 21:35:52 smithi045 ceph-mon[80763]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "957064f1-6dda-475b-903e-130134845a90"}]: dispatch 2023-12-16T21:35:53.066 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Dec 16 21:35:52 smithi045 ceph-mon[80763]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "957064f1-6dda-475b-903e-130134845a90"}]': finished 2023-12-16T21:35:53.066 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Dec 16 21:35:52 smithi045 ceph-mon[80763]: osdmap e19: 4 total, 3 up, 4 in 2023-12-16T21:35:53.067 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Dec 16 21:35:52 smithi045 ceph-mon[80763]: from='mgr.14214 172.21.15.45:0/1227909437' entity='mgr.smithi045.nhtmwc' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2023-12-16T21:35:53.883 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Dec 16 21:35:53 smithi169 ceph-mon[88350]: pgmap v49: 0 pgs: ; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2023-12-16T21:35:53.883 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Dec 16 21:35:53 smithi169 ceph-mon[88350]: from='mgr.14214 172.21.15.45:0/1227909437' entity='mgr.smithi045.nhtmwc' cmd=[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32}]: dispatch 2023-12-16T21:35:53.884 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Dec 16 21:35:53 smithi169 ceph-mon[88350]: from='client.? 172.21.15.169:0/323505460' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2023-12-16T21:35:53.884 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Dec 16 21:35:53 smithi169 ceph-mon[88350]: from='mgr.14214 172.21.15.45:0/1227909437' entity='mgr.smithi045.nhtmwc' cmd='[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32}]': finished 2023-12-16T21:35:53.884 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Dec 16 21:35:53 smithi169 ceph-mon[88350]: osdmap e20: 4 total, 3 up, 4 in 2023-12-16T21:35:53.884 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Dec 16 21:35:53 smithi169 ceph-mon[88350]: from='mgr.14214 172.21.15.45:0/1227909437' entity='mgr.smithi045.nhtmwc' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2023-12-16T21:35:53.884 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Dec 16 21:35:53 smithi169 ceph-mon[88350]: from='mgr.14214 172.21.15.45:0/1227909437' entity='mgr.smithi045.nhtmwc' cmd=[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]: dispatch 2023-12-16T21:35:54.066 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Dec 16 21:35:53 smithi045 ceph-mon[80763]: pgmap v49: 0 pgs: ; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2023-12-16T21:35:54.066 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Dec 16 21:35:53 smithi045 ceph-mon[80763]: from='mgr.14214 172.21.15.45:0/1227909437' entity='mgr.smithi045.nhtmwc' cmd=[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32}]: dispatch 2023-12-16T21:35:54.066 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Dec 16 21:35:53 smithi045 ceph-mon[80763]: from='client.? 172.21.15.169:0/323505460' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2023-12-16T21:35:54.066 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Dec 16 21:35:53 smithi045 ceph-mon[80763]: from='mgr.14214 172.21.15.45:0/1227909437' entity='mgr.smithi045.nhtmwc' cmd='[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32}]': finished 2023-12-16T21:35:54.066 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Dec 16 21:35:53 smithi045 ceph-mon[80763]: osdmap e20: 4 total, 3 up, 4 in 2023-12-16T21:35:54.066 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Dec 16 21:35:53 smithi045 ceph-mon[80763]: from='mgr.14214 172.21.15.45:0/1227909437' entity='mgr.smithi045.nhtmwc' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2023-12-16T21:35:54.066 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Dec 16 21:35:53 smithi045 ceph-mon[80763]: from='mgr.14214 172.21.15.45:0/1227909437' entity='mgr.smithi045.nhtmwc' cmd=[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]: dispatch 2023-12-16T21:35:55.119 INFO:teuthology.orchestra.run.smithi169.stderr:2023-12-16T21:35:55.116+0000 7ffa877fe700 1 -- 172.21.15.169:0/3340131933 <== mon.1 v2:172.21.15.169:3300/0 7 ==== mgrmap(e 19) v1 ==== 86447+0+0 (secure 0 0 0) 0x7ffa94055c30 con 0x7ffaa00fcd00 2023-12-16T21:35:55.383 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Dec 16 21:35:55 smithi169 ceph-mon[88350]: from='mgr.14214 172.21.15.45:0/1227909437' entity='mgr.smithi045.nhtmwc' cmd='[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]': finished 2023-12-16T21:35:55.384 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Dec 16 21:35:55 smithi169 ceph-mon[88350]: osdmap e21: 4 total, 3 up, 4 in 2023-12-16T21:35:55.384 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Dec 16 21:35:55 smithi169 ceph-mon[88350]: from='mgr.14214 172.21.15.45:0/1227909437' entity='mgr.smithi045.nhtmwc' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2023-12-16T21:35:55.384 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Dec 16 21:35:55 smithi169 ceph-mon[88350]: pgmap v52: 1 pgs: 1 unknown; 0 B data, 871 MiB used, 267 GiB / 268 GiB avail 2023-12-16T21:35:55.437 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Dec 16 21:35:55 smithi045 ceph-mon[80763]: from='mgr.14214 172.21.15.45:0/1227909437' entity='mgr.smithi045.nhtmwc' cmd='[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]': finished 2023-12-16T21:35:55.437 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Dec 16 21:35:55 smithi045 ceph-mon[80763]: osdmap e21: 4 total, 3 up, 4 in 2023-12-16T21:35:55.437 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Dec 16 21:35:55 smithi045 ceph-mon[80763]: from='mgr.14214 172.21.15.45:0/1227909437' entity='mgr.smithi045.nhtmwc' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2023-12-16T21:35:55.438 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Dec 16 21:35:55 smithi045 ceph-mon[80763]: pgmap v52: 1 pgs: 1 unknown; 0 B data, 871 MiB used, 267 GiB / 268 GiB avail 2023-12-16T21:35:55.816 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:35:55 smithi045 sudo[119334]: ceph : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/sbin/smartctl -x --json=o /dev/nvme0n1 2023-12-16T21:35:55.816 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:35:55 smithi045 sudo[119334]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2023-12-16T21:35:55.816 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:35:55 smithi045 sudo[119334]: pam_unix(sudo:session): session opened for user root by (uid=0) 2023-12-16T21:35:55.816 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:35:55 smithi045 sudo[119334]: pam_unix(sudo:session): session closed for user root 2023-12-16T21:35:56.316 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Dec 16 21:35:56 smithi045 ceph-mon[80763]: mgrmap e19: smithi045.nhtmwc(active, since 86s), standbys: smithi169.cncbgh 2023-12-16T21:35:56.316 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Dec 16 21:35:56 smithi045 ceph-mon[80763]: osdmap e22: 4 total, 3 up, 4 in 2023-12-16T21:35:56.316 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Dec 16 21:35:56 smithi045 ceph-mon[80763]: from='mgr.14214 172.21.15.45:0/1227909437' entity='mgr.smithi045.nhtmwc' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2023-12-16T21:35:56.316 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Dec 16 21:35:56 smithi045 ceph-mon[80763]: from='mgr.14214 172.21.15.45:0/1227909437' entity='mgr.smithi045.nhtmwc' 2023-12-16T21:35:56.316 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:35:55 smithi045 sudo[119337]: ceph : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2023-12-16T21:35:56.316 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:35:55 smithi045 sudo[119337]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2023-12-16T21:35:56.316 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:35:55 smithi045 sudo[119337]: pam_unix(sudo:session): session opened for user root by (uid=0) 2023-12-16T21:35:56.316 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:35:55 smithi045 sudo[119337]: pam_unix(sudo:session): session closed for user root 2023-12-16T21:35:56.383 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Dec 16 21:35:56 smithi169 ceph-mon[88350]: mgrmap e19: smithi045.nhtmwc(active, since 86s), standbys: smithi169.cncbgh 2023-12-16T21:35:56.384 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Dec 16 21:35:56 smithi169 ceph-mon[88350]: osdmap e22: 4 total, 3 up, 4 in 2023-12-16T21:35:56.384 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Dec 16 21:35:56 smithi169 ceph-mon[88350]: from='mgr.14214 172.21.15.45:0/1227909437' entity='mgr.smithi045.nhtmwc' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2023-12-16T21:35:56.384 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Dec 16 21:35:56 smithi169 ceph-mon[88350]: from='mgr.14214 172.21.15.45:0/1227909437' entity='mgr.smithi045.nhtmwc' 2023-12-16T21:35:56.816 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:35:56 smithi045 sudo[119340]: ceph : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/sbin/smartctl -x --json=o /dev/nvme0n1 2023-12-16T21:35:56.816 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:35:56 smithi045 sudo[119340]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2023-12-16T21:35:56.816 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:35:56 smithi045 sudo[119340]: pam_unix(sudo:session): session opened for user root by (uid=0) 2023-12-16T21:35:56.816 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:35:56 smithi045 sudo[119340]: pam_unix(sudo:session): session closed for user root 2023-12-16T21:35:57.271 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:35:56 smithi045 sudo[119343]: ceph : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2023-12-16T21:35:57.271 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:35:56 smithi045 sudo[119343]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2023-12-16T21:35:57.271 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:35:56 smithi045 sudo[119343]: pam_unix(sudo:session): session opened for user root by (uid=0) 2023-12-16T21:35:57.271 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:35:56 smithi045 sudo[119343]: pam_unix(sudo:session): session closed for user root 2023-12-16T21:35:57.566 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Dec 16 21:35:57 smithi045 ceph-mon[80763]: pgmap v54: 1 pgs: 1 active+clean; 577 KiB data, 872 MiB used, 267 GiB / 268 GiB avail 2023-12-16T21:35:57.566 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:35:57 smithi045 sudo[119346]: ceph : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/sbin/smartctl -x --json=o /dev/nvme0n1 2023-12-16T21:35:57.566 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:35:57 smithi045 sudo[119346]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2023-12-16T21:35:57.566 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:35:57 smithi045 sudo[119346]: pam_unix(sudo:session): session opened for user root by (uid=0) 2023-12-16T21:35:57.566 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:35:57 smithi045 sudo[119346]: pam_unix(sudo:session): session closed for user root 2023-12-16T21:35:57.621 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Dec 16 21:35:57 smithi169 ceph-mon[88350]: pgmap v54: 1 pgs: 1 active+clean; 577 KiB data, 872 MiB used, 267 GiB / 268 GiB avail 2023-12-16T21:35:58.066 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:35:57 smithi045 sudo[119349]: ceph : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2023-12-16T21:35:58.066 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:35:57 smithi045 sudo[119349]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2023-12-16T21:35:58.066 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:35:57 smithi045 sudo[119349]: pam_unix(sudo:session): session opened for user root by (uid=0) 2023-12-16T21:35:58.066 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:35:57 smithi045 sudo[119349]: pam_unix(sudo:session): session closed for user root 2023-12-16T21:35:58.566 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Dec 16 21:35:58 smithi045 sudo[119352]: ceph : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/sbin/smartctl -x --json=o /dev/nvme0n1 2023-12-16T21:35:58.566 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Dec 16 21:35:58 smithi045 sudo[119352]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2023-12-16T21:35:58.566 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Dec 16 21:35:58 smithi045 sudo[119352]: pam_unix(sudo:session): session opened for user root by (uid=0) 2023-12-16T21:35:58.566 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Dec 16 21:35:58 smithi045 sudo[119352]: pam_unix(sudo:session): session closed for user root 2023-12-16T21:35:58.883 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Dec 16 21:35:58 smithi169 ceph-mon[88350]: from='mgr.14214 172.21.15.45:0/1227909437' entity='mgr.smithi045.nhtmwc' cmd=[{"prefix": "auth get", "entity": "osd.3"}]: dispatch 2023-12-16T21:35:58.883 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Dec 16 21:35:58 smithi169 ceph-mon[88350]: from='mgr.14214 172.21.15.45:0/1227909437' entity='mgr.smithi045.nhtmwc' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-12-16T21:35:59.066 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Dec 16 21:35:58 smithi045 sudo[119355]: ceph : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2023-12-16T21:35:59.066 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Dec 16 21:35:58 smithi045 sudo[119355]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2023-12-16T21:35:59.066 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Dec 16 21:35:58 smithi045 sudo[119355]: pam_unix(sudo:session): session opened for user root by (uid=0) 2023-12-16T21:35:59.066 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Dec 16 21:35:58 smithi045 sudo[119355]: pam_unix(sudo:session): session closed for user root 2023-12-16T21:35:59.066 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Dec 16 21:35:58 smithi045 ceph-mon[80763]: from='mgr.14214 172.21.15.45:0/1227909437' entity='mgr.smithi045.nhtmwc' cmd=[{"prefix": "auth get", "entity": "osd.3"}]: dispatch 2023-12-16T21:35:59.066 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Dec 16 21:35:58 smithi045 ceph-mon[80763]: from='mgr.14214 172.21.15.45:0/1227909437' entity='mgr.smithi045.nhtmwc' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-12-16T21:35:59.140 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Dec 16 21:35:58 smithi169 sudo[93354]: ceph : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/sbin/smartctl -x --json=o /dev/nvme0n1 2023-12-16T21:35:59.140 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Dec 16 21:35:58 smithi169 sudo[93354]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2023-12-16T21:35:59.140 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Dec 16 21:35:58 smithi169 sudo[93354]: pam_unix(sudo:session): session opened for user root by (uid=0) 2023-12-16T21:35:59.140 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Dec 16 21:35:58 smithi169 sudo[93354]: pam_unix(sudo:session): session closed for user root 2023-12-16T21:35:59.473 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Dec 16 21:35:59 smithi169 sudo[93377]: ceph : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2023-12-16T21:35:59.474 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Dec 16 21:35:59 smithi169 sudo[93377]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2023-12-16T21:35:59.474 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Dec 16 21:35:59 smithi169 sudo[93377]: pam_unix(sudo:session): session opened for user root by (uid=0) 2023-12-16T21:35:59.474 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Dec 16 21:35:59 smithi169 sudo[93377]: pam_unix(sudo:session): session closed for user root 2023-12-16T21:35:59.739 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Dec 16 21:35:59 smithi169 ceph-mon[88350]: Deploying daemon osd.3 on smithi169 2023-12-16T21:35:59.739 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Dec 16 21:35:59 smithi169 ceph-mon[88350]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2023-12-16T21:35:59.739 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Dec 16 21:35:59 smithi169 ceph-mon[88350]: pgmap v55: 1 pgs: 1 active+clean; 577 KiB data, 874 MiB used, 267 GiB / 268 GiB avail 2023-12-16T21:35:59.739 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Dec 16 21:35:59 smithi169 ceph-mon[88350]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2023-12-16T21:35:59.739 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Dec 16 21:35:59 smithi169 ceph-mon[88350]: from='mgr.14214 172.21.15.45:0/1227909437' entity='mgr.smithi045.nhtmwc' cmd=[{"prefix": "mon metadata", "id": "smithi045"}]: dispatch 2023-12-16T21:35:59.739 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Dec 16 21:35:59 smithi169 ceph-mon[88350]: from='mgr.14214 172.21.15.45:0/1227909437' entity='mgr.smithi045.nhtmwc' cmd=[{"prefix": "mon metadata", "id": "smithi169"}]: dispatch 2023-12-16T21:35:59.740 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Dec 16 21:35:59 smithi169 ceph-mon[88350]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2023-12-16T21:35:59.740 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Dec 16 21:35:59 smithi169 ceph-mon[88350]: from='mgr.14214 172.21.15.45:0/1227909437' entity='mgr.smithi045.nhtmwc' cmd=[{"prefix": "mon metadata", "id": "smithi045"}]: dispatch 2023-12-16T21:35:59.740 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Dec 16 21:35:59 smithi169 ceph-mon[88350]: from='mgr.14214 172.21.15.45:0/1227909437' entity='mgr.smithi045.nhtmwc' cmd=[{"prefix": "mon metadata", "id": "smithi169"}]: dispatch 2023-12-16T21:35:59.740 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Dec 16 21:35:59 smithi169 ceph-mon[88350]: from='mgr.14214 172.21.15.45:0/1227909437' entity='mgr.smithi045.nhtmwc' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2023-12-16T21:35:59.740 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Dec 16 21:35:59 smithi169 ceph-mon[88350]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2023-12-16T21:35:59.740 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Dec 16 21:35:59 smithi169 ceph-mon[88350]: from='mgr.14214 172.21.15.45:0/1227909437' entity='mgr.smithi045.nhtmwc' 2023-12-16T21:36:00.066 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Dec 16 21:35:59 smithi045 ceph-mon[80763]: Deploying daemon osd.3 on smithi169 2023-12-16T21:36:00.066 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Dec 16 21:35:59 smithi045 ceph-mon[80763]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2023-12-16T21:36:00.066 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Dec 16 21:35:59 smithi045 ceph-mon[80763]: pgmap v55: 1 pgs: 1 active+clean; 577 KiB data, 874 MiB used, 267 GiB / 268 GiB avail 2023-12-16T21:36:00.067 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Dec 16 21:35:59 smithi045 ceph-mon[80763]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2023-12-16T21:36:00.067 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Dec 16 21:35:59 smithi045 ceph-mon[80763]: from='mgr.14214 172.21.15.45:0/1227909437' entity='mgr.smithi045.nhtmwc' cmd=[{"prefix": "mon metadata", "id": "smithi045"}]: dispatch 2023-12-16T21:36:00.067 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Dec 16 21:35:59 smithi045 ceph-mon[80763]: from='mgr.14214 172.21.15.45:0/1227909437' entity='mgr.smithi045.nhtmwc' cmd=[{"prefix": "mon metadata", "id": "smithi169"}]: dispatch 2023-12-16T21:36:00.067 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Dec 16 21:35:59 smithi045 ceph-mon[80763]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2023-12-16T21:36:00.067 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Dec 16 21:35:59 smithi045 ceph-mon[80763]: from='mgr.14214 172.21.15.45:0/1227909437' entity='mgr.smithi045.nhtmwc' cmd=[{"prefix": "mon metadata", "id": "smithi045"}]: dispatch 2023-12-16T21:36:00.067 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Dec 16 21:35:59 smithi045 ceph-mon[80763]: from='mgr.14214 172.21.15.45:0/1227909437' entity='mgr.smithi045.nhtmwc' cmd=[{"prefix": "mon metadata", "id": "smithi169"}]: dispatch 2023-12-16T21:36:00.067 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Dec 16 21:35:59 smithi045 ceph-mon[80763]: from='mgr.14214 172.21.15.45:0/1227909437' entity='mgr.smithi045.nhtmwc' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2023-12-16T21:36:00.067 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Dec 16 21:35:59 smithi045 ceph-mon[80763]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2023-12-16T21:36:00.068 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Dec 16 21:35:59 smithi045 ceph-mon[80763]: from='mgr.14214 172.21.15.45:0/1227909437' entity='mgr.smithi045.nhtmwc' 2023-12-16T21:36:02.013 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Dec 16 21:36:01 smithi169 ceph-mon[88350]: pgmap v56: 1 pgs: 1 active+clean; 577 KiB data, 874 MiB used, 267 GiB / 268 GiB avail 2023-12-16T21:36:02.013 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Dec 16 21:36:01 smithi169 ceph-mon[88350]: from='mgr.14214 172.21.15.45:0/1227909437' entity='mgr.smithi045.nhtmwc' 2023-12-16T21:36:02.013 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Dec 16 21:36:01 smithi169 ceph-mon[88350]: from='mgr.14214 172.21.15.45:0/1227909437' entity='mgr.smithi045.nhtmwc' 2023-12-16T21:36:02.013 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Dec 16 21:36:01 smithi169 ceph-mon[88350]: from='mgr.14214 172.21.15.45:0/1227909437' entity='mgr.smithi045.nhtmwc' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-12-16T21:36:02.066 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Dec 16 21:36:01 smithi045 ceph-mon[80763]: pgmap v56: 1 pgs: 1 active+clean; 577 KiB data, 874 MiB used, 267 GiB / 268 GiB avail 2023-12-16T21:36:02.066 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Dec 16 21:36:01 smithi045 ceph-mon[80763]: from='mgr.14214 172.21.15.45:0/1227909437' entity='mgr.smithi045.nhtmwc' 2023-12-16T21:36:02.066 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Dec 16 21:36:01 smithi045 ceph-mon[80763]: from='mgr.14214 172.21.15.45:0/1227909437' entity='mgr.smithi045.nhtmwc' 2023-12-16T21:36:02.066 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Dec 16 21:36:01 smithi045 ceph-mon[80763]: from='mgr.14214 172.21.15.45:0/1227909437' entity='mgr.smithi045.nhtmwc' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-12-16T21:36:02.536 INFO:teuthology.orchestra.run.smithi169.stderr:2023-12-16T21:36:02.533+0000 7ffa877fe700 1 -- 172.21.15.169:0/3340131933 <== mgr.14214 v2:172.21.15.45:6800/2012105536 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+37 (secure 0 0 0) 0x7ffaa010e480 con 0x7ffa88068ed0 2023-12-16T21:36:02.536 INFO:teuthology.orchestra.run.smithi169.stdout:Created osd(s) 3 on host 'smithi169' 2023-12-16T21:36:02.538 INFO:teuthology.orchestra.run.smithi169.stderr:2023-12-16T21:36:02.535+0000 7ffaa568f700 1 -- 172.21.15.169:0/3340131933 >> [v2:172.21.15.45:6800/2012105536,v1:172.21.15.45:6801/2012105536] conn(0x7ffa88068ed0 msgr2=0x7ffa8806b380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:36:02.538 INFO:teuthology.orchestra.run.smithi169.stderr:2023-12-16T21:36:02.535+0000 7ffaa568f700 1 --2- 172.21.15.169:0/3340131933 >> [v2:172.21.15.45:6800/2012105536,v1:172.21.15.45:6801/2012105536] conn(0x7ffa88068ed0 0x7ffa8806b380 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7ffa90004890 tx=0x7ffa90009e30 comp rx=0 tx=0).stop 2023-12-16T21:36:02.538 INFO:teuthology.orchestra.run.smithi169.stderr:2023-12-16T21:36:02.535+0000 7ffaa568f700 1 -- 172.21.15.169:0/3340131933 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7ffaa00fcd00 msgr2=0x7ffaa010c9b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:36:02.538 INFO:teuthology.orchestra.run.smithi169.stderr:2023-12-16T21:36:02.535+0000 7ffaa568f700 1 --2- 172.21.15.169:0/3340131933 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7ffaa00fcd00 0x7ffaa010c9b0 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 crypto rx=0x7ffa9400fce0 tx=0x7ffa940046b0 comp rx=0 tx=0).stop 2023-12-16T21:36:02.538 INFO:teuthology.orchestra.run.smithi169.stderr:2023-12-16T21:36:02.535+0000 7ffaa568f700 1 -- 172.21.15.169:0/3340131933 shutdown_connections 2023-12-16T21:36:02.538 INFO:teuthology.orchestra.run.smithi169.stderr:2023-12-16T21:36:02.535+0000 7ffaa568f700 1 --2- 172.21.15.169:0/3340131933 >> [v2:172.21.15.45:6800/2012105536,v1:172.21.15.45:6801/2012105536] conn(0x7ffa88068ed0 0x7ffa8806b380 unknown :-1 s=CLOSED pgs=38 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:36:02.538 INFO:teuthology.orchestra.run.smithi169.stderr:2023-12-16T21:36:02.535+0000 7ffaa568f700 1 --2- 172.21.15.169:0/3340131933 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ffaa00fbaf0 0x7ffaa010c470 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:36:02.538 INFO:teuthology.orchestra.run.smithi169.stderr:2023-12-16T21:36:02.535+0000 7ffaa568f700 1 --2- 172.21.15.169:0/3340131933 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7ffaa00fcd00 0x7ffaa010c9b0 unknown :-1 s=CLOSED pgs=5 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:36:02.538 INFO:teuthology.orchestra.run.smithi169.stderr:2023-12-16T21:36:02.535+0000 7ffaa568f700 1 -- 172.21.15.169:0/3340131933 >> 172.21.15.169:0/3340131933 conn(0x7ffaa00f71d0 msgr2=0x7ffaa00fff30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-16T21:36:02.538 INFO:teuthology.orchestra.run.smithi169.stderr:2023-12-16T21:36:02.535+0000 7ffaa568f700 1 -- 172.21.15.169:0/3340131933 shutdown_connections 2023-12-16T21:36:02.538 INFO:teuthology.orchestra.run.smithi169.stderr:2023-12-16T21:36:02.535+0000 7ffaa568f700 1 -- 172.21.15.169:0/3340131933 wait complete. 2023-12-16T21:36:02.843 DEBUG:teuthology.orchestra.run.smithi169:osd.3> sudo journalctl -f -n 0 -u ceph-88ff363c-9c5a-11ee-95a5-87774f69a715@osd.3.service 2023-12-16T21:36:02.846 INFO:tasks.cephadm:Deploying osd.4 on smithi169 with /dev/vg_nvme/lv_3... 2023-12-16T21:36:02.846 DEBUG:teuthology.orchestra.run.smithi169:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 88ff363c-9c5a-11ee-95a5-87774f69a715 -- ceph-volume lvm zap /dev/vg_nvme/lv_3 2023-12-16T21:36:03.444 INFO:teuthology.orchestra.run.smithi169.stderr:Inferring config /var/lib/ceph/88ff363c-9c5a-11ee-95a5-87774f69a715/mon.smithi169/config 2023-12-16T21:36:03.651 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Dec 16 21:36:03 smithi169 ceph-mon[88350]: from='mgr.14214 172.21.15.45:0/1227909437' entity='mgr.smithi045.nhtmwc' 2023-12-16T21:36:03.651 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Dec 16 21:36:03 smithi169 ceph-mon[88350]: from='mgr.14214 172.21.15.45:0/1227909437' entity='mgr.smithi045.nhtmwc' 2023-12-16T21:36:03.651 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Dec 16 21:36:03 smithi169 ceph-mon[88350]: from='mgr.14214 172.21.15.45:0/1227909437' entity='mgr.smithi045.nhtmwc' 2023-12-16T21:36:03.651 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Dec 16 21:36:03 smithi169 ceph-mon[88350]: from='mgr.14214 172.21.15.45:0/1227909437' entity='mgr.smithi045.nhtmwc' 2023-12-16T21:36:03.651 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Dec 16 21:36:03 smithi169 ceph-mon[88350]: pgmap v57: 1 pgs: 1 active+clean; 577 KiB data, 874 MiB used, 267 GiB / 268 GiB avail; 3.0 KiB/s rd, 131 KiB/s wr, 9 op/s 2023-12-16T21:36:03.816 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Dec 16 21:36:03 smithi045 ceph-mon[80763]: from='mgr.14214 172.21.15.45:0/1227909437' entity='mgr.smithi045.nhtmwc' 2023-12-16T21:36:03.816 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Dec 16 21:36:03 smithi045 ceph-mon[80763]: from='mgr.14214 172.21.15.45:0/1227909437' entity='mgr.smithi045.nhtmwc' 2023-12-16T21:36:03.816 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Dec 16 21:36:03 smithi045 ceph-mon[80763]: from='mgr.14214 172.21.15.45:0/1227909437' entity='mgr.smithi045.nhtmwc' 2023-12-16T21:36:03.816 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Dec 16 21:36:03 smithi045 ceph-mon[80763]: from='mgr.14214 172.21.15.45:0/1227909437' entity='mgr.smithi045.nhtmwc' 2023-12-16T21:36:03.816 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Dec 16 21:36:03 smithi045 ceph-mon[80763]: pgmap v57: 1 pgs: 1 active+clean; 577 KiB data, 874 MiB used, 267 GiB / 268 GiB avail; 3.0 KiB/s rd, 131 KiB/s wr, 9 op/s 2023-12-16T21:36:04.590 INFO:teuthology.orchestra.run.smithi169.stderr:--> Zapping: /dev/vg_nvme/lv_3 2023-12-16T21:36:04.590 INFO:teuthology.orchestra.run.smithi169.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_3 bs=1M count=10 conv=fsync 2023-12-16T21:36:04.590 INFO:teuthology.orchestra.run.smithi169.stderr: stderr: 10+0 records in 2023-12-16T21:36:04.590 INFO:teuthology.orchestra.run.smithi169.stderr:10+0 records out 2023-12-16T21:36:04.590 INFO:teuthology.orchestra.run.smithi169.stderr:10485760 bytes (10 MB, 10 MiB) copied, 0.0128252 s, 818 MB/s 2023-12-16T21:36:04.590 INFO:teuthology.orchestra.run.smithi169.stderr:--> Zapping successful for: 2023-12-16T21:36:04.883 INFO:journalctl@ceph.osd.3.smithi169.stdout:Dec 16 21:36:04 smithi169 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-3[93707]: 2023-12-16T21:36:04.671+0000 7f178a27f540 -1 osd.3 0 log_to_monitors true 2023-12-16T21:36:04.929 DEBUG:teuthology.orchestra.run.smithi169:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 88ff363c-9c5a-11ee-95a5-87774f69a715 -- ceph orch daemon add osd smithi169:vg_nvme/lv_3 2023-12-16T21:36:05.188 INFO:teuthology.orchestra.run.smithi169.stderr:Inferring config /var/lib/ceph/88ff363c-9c5a-11ee-95a5-87774f69a715/mon.smithi169/config 2023-12-16T21:36:05.928 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Dec 16 21:36:05 smithi169 ceph-mon[88350]: pgmap v58: 1 pgs: 1 active+clean; 577 KiB data, 874 MiB used, 267 GiB / 268 GiB avail; 2.2 KiB/s rd, 98 KiB/s wr, 6 op/s 2023-12-16T21:36:05.928 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Dec 16 21:36:05 smithi169 ceph-mon[88350]: from='osd.3 [v2:172.21.15.169:6800/201299104,v1:172.21.15.169:6801/201299104]' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2023-12-16T21:36:05.928 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Dec 16 21:36:05 smithi169 ceph-mon[88350]: from='osd.3 ' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2023-12-16T21:36:05.928 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Dec 16 21:36:05 smithi169 ceph-mon[88350]: from='mgr.14214 172.21.15.45:0/1227909437' entity='mgr.smithi045.nhtmwc' 2023-12-16T21:36:05.928 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Dec 16 21:36:05 smithi169 ceph-mon[88350]: from='mgr.14214 172.21.15.45:0/1227909437' entity='mgr.smithi045.nhtmwc' 2023-12-16T21:36:05.928 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Dec 16 21:36:05 smithi169 ceph-mon[88350]: from='mgr.14214 172.21.15.45:0/1227909437' entity='mgr.smithi045.nhtmwc' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2023-12-16T21:36:05.929 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Dec 16 21:36:05 smithi169 ceph-mon[88350]: from='mgr.14214 172.21.15.45:0/1227909437' entity='mgr.smithi045.nhtmwc' 2023-12-16T21:36:05.929 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Dec 16 21:36:05 smithi169 ceph-mon[88350]: from='mgr.14214 172.21.15.45:0/1227909437' entity='mgr.smithi045.nhtmwc' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-12-16T21:36:05.929 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Dec 16 21:36:05 smithi169 ceph-mon[88350]: from='mgr.14214 172.21.15.45:0/1227909437' entity='mgr.smithi045.nhtmwc' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-12-16T21:36:05.929 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Dec 16 21:36:05 smithi169 ceph-mon[88350]: from='mgr.14214 172.21.15.45:0/1227909437' entity='mgr.smithi045.nhtmwc' 2023-12-16T21:36:05.929 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Dec 16 21:36:05 smithi169 ceph-mon[88350]: from='mgr.14214 172.21.15.45:0/1227909437' entity='mgr.smithi045.nhtmwc' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-12-16T21:36:05.929 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Dec 16 21:36:05 smithi169 ceph-mon[88350]: from='mgr.14214 172.21.15.45:0/1227909437' entity='mgr.smithi045.nhtmwc' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-12-16T21:36:05.929 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Dec 16 21:36:05 smithi169 ceph-mon[88350]: from='mgr.14214 172.21.15.45:0/1227909437' entity='mgr.smithi045.nhtmwc' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-12-16T21:36:05.929 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Dec 16 21:36:05 smithi169 ceph-mon[88350]: from='mgr.14214 172.21.15.45:0/1227909437' entity='mgr.smithi045.nhtmwc' 2023-12-16T21:36:05.996 INFO:teuthology.orchestra.run.smithi169.stderr:2023-12-16T21:36:05.993+0000 7f6b0508c700 1 -- 172.21.15.169:0/1310817609 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f6b000fde00 msgr2=0x7f6b000fe250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:36:05.996 INFO:teuthology.orchestra.run.smithi169.stderr:2023-12-16T21:36:05.993+0000 7f6b0508c700 1 --2- 172.21.15.169:0/1310817609 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f6b000fde00 0x7f6b000fe250 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7f6af4004940 tx=0x7f6af4009e30 comp rx=0 tx=0).stop 2023-12-16T21:36:05.996 INFO:teuthology.orchestra.run.smithi169.stderr:2023-12-16T21:36:05.993+0000 7f6b0508c700 1 -- 172.21.15.169:0/1310817609 shutdown_connections 2023-12-16T21:36:05.996 INFO:teuthology.orchestra.run.smithi169.stderr:2023-12-16T21:36:05.993+0000 7f6b0508c700 1 --2- 172.21.15.169:0/1310817609 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f6b000fde00 0x7f6b000fe250 unknown :-1 s=CLOSED pgs=14 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:36:05.996 INFO:teuthology.orchestra.run.smithi169.stderr:2023-12-16T21:36:05.993+0000 7f6b0508c700 1 --2- 172.21.15.169:0/1310817609 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f6b000fcbf0 0x7f6b000fd000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:36:05.996 INFO:teuthology.orchestra.run.smithi169.stderr:2023-12-16T21:36:05.993+0000 7f6b0508c700 1 -- 172.21.15.169:0/1310817609 >> 172.21.15.169:0/1310817609 conn(0x7f6b000f82d0 msgr2=0x7f6b000fa6e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-16T21:36:05.996 INFO:teuthology.orchestra.run.smithi169.stderr:2023-12-16T21:36:05.993+0000 7f6b0508c700 1 -- 172.21.15.169:0/1310817609 shutdown_connections 2023-12-16T21:36:05.996 INFO:teuthology.orchestra.run.smithi169.stderr:2023-12-16T21:36:05.993+0000 7f6b0508c700 1 -- 172.21.15.169:0/1310817609 wait complete. 2023-12-16T21:36:05.997 INFO:teuthology.orchestra.run.smithi169.stderr:2023-12-16T21:36:05.994+0000 7f6b0508c700 1 Processor -- start 2023-12-16T21:36:05.997 INFO:teuthology.orchestra.run.smithi169.stderr:2023-12-16T21:36:05.994+0000 7f6b0508c700 1 -- start start 2023-12-16T21:36:06.000 INFO:teuthology.orchestra.run.smithi169.stderr:2023-12-16T21:36:05.994+0000 7f6b0508c700 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f6b000fcbf0 0x7f6b0010d5e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:36:06.000 INFO:teuthology.orchestra.run.smithi169.stderr:2023-12-16T21:36:05.994+0000 7f6b0508c700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f6b000fde00 0x7f6b0010db20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:36:06.000 INFO:teuthology.orchestra.run.smithi169.stderr:2023-12-16T21:36:05.994+0000 7f6b0508c700 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6b0010e120 con 0x7f6b000fde00 2023-12-16T21:36:06.000 INFO:teuthology.orchestra.run.smithi169.stderr:2023-12-16T21:36:05.994+0000 7f6b0508c700 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6b0010e9c0 con 0x7f6b000fcbf0 2023-12-16T21:36:06.000 INFO:teuthology.orchestra.run.smithi169.stderr:2023-12-16T21:36:05.994+0000 7f6afed9d700 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f6b000fcbf0 0x7f6b0010d5e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:36:06.000 INFO:teuthology.orchestra.run.smithi169.stderr:2023-12-16T21:36:05.994+0000 7f6afed9d700 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f6b000fcbf0 0x7f6b0010d5e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.169:3300/0 says I am v2:172.21.15.169:48608/0 (socket says 172.21.15.169:48608) 2023-12-16T21:36:06.000 INFO:teuthology.orchestra.run.smithi169.stderr:2023-12-16T21:36:05.994+0000 7f6afed9d700 1 -- 172.21.15.169:0/2215693675 learned_addr learned my addr 172.21.15.169:0/2215693675 (peer_addr_for_me v2:172.21.15.169:0/0) 2023-12-16T21:36:06.000 INFO:teuthology.orchestra.run.smithi169.stderr:2023-12-16T21:36:05.994+0000 7f6afed9d700 1 -- 172.21.15.169:0/2215693675 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f6b000fde00 msgr2=0x7f6b0010db20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:36:06.000 INFO:teuthology.orchestra.run.smithi169.stderr:2023-12-16T21:36:05.994+0000 7f6afe59c700 1 --2- 172.21.15.169:0/2215693675 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f6b000fde00 0x7f6b0010db20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:36:06.000 INFO:teuthology.orchestra.run.smithi169.stderr:2023-12-16T21:36:05.994+0000 7f6afed9d700 1 --2- 172.21.15.169:0/2215693675 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f6b000fde00 0x7f6b0010db20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:36:06.000 INFO:teuthology.orchestra.run.smithi169.stderr:2023-12-16T21:36:05.994+0000 7f6afed9d700 1 -- 172.21.15.169:0/2215693675 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6af4005040 con 0x7f6b000fcbf0 2023-12-16T21:36:06.000 INFO:teuthology.orchestra.run.smithi169.stderr:2023-12-16T21:36:05.995+0000 7f6afed9d700 1 --2- 172.21.15.169:0/2215693675 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f6b000fcbf0 0x7f6b0010d5e0 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0x7f6af0002a80 tx=0x7f6af0004600 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:36:06.000 INFO:teuthology.orchestra.run.smithi169.stderr:2023-12-16T21:36:05.995+0000 7f6ae77fe700 1 -- 172.21.15.169:0/2215693675 <== mon.1 v2:172.21.15.169:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f6af0007cc0 con 0x7f6b000fcbf0 2023-12-16T21:36:06.000 INFO:teuthology.orchestra.run.smithi169.stderr:2023-12-16T21:36:05.995+0000 7f6ae77fe700 1 -- 172.21.15.169:0/2215693675 <== mon.1 v2:172.21.15.169:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f6af000f040 con 0x7f6b000fcbf0 2023-12-16T21:36:06.001 INFO:teuthology.orchestra.run.smithi169.stderr:2023-12-16T21:36:05.995+0000 7f6b0508c700 1 -- 172.21.15.169:0/2215693675 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6b0010eca0 con 0x7f6b000fcbf0 2023-12-16T21:36:06.001 INFO:teuthology.orchestra.run.smithi169.stderr:2023-12-16T21:36:05.995+0000 7f6ae77fe700 1 -- 172.21.15.169:0/2215693675 <== mon.1 v2:172.21.15.169:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f6af000da20 con 0x7f6b000fcbf0 2023-12-16T21:36:06.001 INFO:teuthology.orchestra.run.smithi169.stderr:2023-12-16T21:36:05.995+0000 7f6b0508c700 1 -- 172.21.15.169:0/2215693675 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6b0010f1e0 con 0x7f6b000fcbf0 2023-12-16T21:36:06.001 INFO:teuthology.orchestra.run.smithi169.stderr:2023-12-16T21:36:05.996+0000 7f6b0508c700 1 -- 172.21.15.169:0/2215693675 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6b0004bb80 con 0x7f6b000fcbf0 2023-12-16T21:36:06.001 INFO:teuthology.orchestra.run.smithi169.stderr:2023-12-16T21:36:05.996+0000 7f6ae77fe700 1 -- 172.21.15.169:0/2215693675 <== mon.1 v2:172.21.15.169:3300/0 4 ==== mgrmap(e 19) v1 ==== 86447+0+0 (secure 0 0 0) 0x7f6af000db80 con 0x7f6b000fcbf0 2023-12-16T21:36:06.001 INFO:teuthology.orchestra.run.smithi169.stderr:2023-12-16T21:36:05.997+0000 7f6ae77fe700 1 --2- 172.21.15.169:0/2215693675 >> [v2:172.21.15.45:6800/2012105536,v1:172.21.15.45:6801/2012105536] conn(0x7f6ae8068f60 0x7f6ae806b410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:36:06.001 INFO:teuthology.orchestra.run.smithi169.stderr:2023-12-16T21:36:05.997+0000 7f6ae77fe700 1 -- 172.21.15.169:0/2215693675 <== mon.1 v2:172.21.15.169:3300/0 5 ==== osd_map(23..23 src has 1..23) v4 ==== 3364+0+0 (secure 0 0 0) 0x7f6af0086350 con 0x7f6b000fcbf0 2023-12-16T21:36:06.001 INFO:teuthology.orchestra.run.smithi169.stderr:2023-12-16T21:36:05.997+0000 7f6afe59c700 1 --2- 172.21.15.169:0/2215693675 >> [v2:172.21.15.45:6800/2012105536,v1:172.21.15.45:6801/2012105536] conn(0x7f6ae8068f60 0x7f6ae806b410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:36:06.001 INFO:teuthology.orchestra.run.smithi169.stderr:2023-12-16T21:36:05.997+0000 7f6afe59c700 1 --2- 172.21.15.169:0/2215693675 >> [v2:172.21.15.45:6800/2012105536,v1:172.21.15.45:6801/2012105536] conn(0x7f6ae8068f60 0x7f6ae806b410 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7f6af4004560 tx=0x7f6af4009e30 comp rx=0 tx=0).ready entity=mgr.14214 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:36:06.003 INFO:teuthology.orchestra.run.smithi169.stderr:2023-12-16T21:36:06.001+0000 7f6ae77fe700 1 -- 172.21.15.169:0/2215693675 <== mon.1 v2:172.21.15.169:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+174570 (secure 0 0 0) 0x7f6af0056a60 con 0x7f6b000fcbf0 2023-12-16T21:36:06.066 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Dec 16 21:36:05 smithi045 ceph-mon[80763]: pgmap v58: 1 pgs: 1 active+clean; 577 KiB data, 874 MiB used, 267 GiB / 268 GiB avail; 2.2 KiB/s rd, 98 KiB/s wr, 6 op/s 2023-12-16T21:36:06.066 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Dec 16 21:36:05 smithi045 ceph-mon[80763]: from='osd.3 [v2:172.21.15.169:6800/201299104,v1:172.21.15.169:6801/201299104]' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2023-12-16T21:36:06.067 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Dec 16 21:36:05 smithi045 ceph-mon[80763]: from='osd.3 ' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2023-12-16T21:36:06.067 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Dec 16 21:36:05 smithi045 ceph-mon[80763]: from='mgr.14214 172.21.15.45:0/1227909437' entity='mgr.smithi045.nhtmwc' 2023-12-16T21:36:06.067 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Dec 16 21:36:05 smithi045 ceph-mon[80763]: from='mgr.14214 172.21.15.45:0/1227909437' entity='mgr.smithi045.nhtmwc' 2023-12-16T21:36:06.067 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Dec 16 21:36:05 smithi045 ceph-mon[80763]: from='mgr.14214 172.21.15.45:0/1227909437' entity='mgr.smithi045.nhtmwc' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2023-12-16T21:36:06.067 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Dec 16 21:36:05 smithi045 ceph-mon[80763]: from='mgr.14214 172.21.15.45:0/1227909437' entity='mgr.smithi045.nhtmwc' 2023-12-16T21:36:06.067 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Dec 16 21:36:05 smithi045 ceph-mon[80763]: from='mgr.14214 172.21.15.45:0/1227909437' entity='mgr.smithi045.nhtmwc' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-12-16T21:36:06.067 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Dec 16 21:36:05 smithi045 ceph-mon[80763]: from='mgr.14214 172.21.15.45:0/1227909437' entity='mgr.smithi045.nhtmwc' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-12-16T21:36:06.067 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Dec 16 21:36:05 smithi045 ceph-mon[80763]: from='mgr.14214 172.21.15.45:0/1227909437' entity='mgr.smithi045.nhtmwc' 2023-12-16T21:36:06.067 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Dec 16 21:36:05 smithi045 ceph-mon[80763]: from='mgr.14214 172.21.15.45:0/1227909437' entity='mgr.smithi045.nhtmwc' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-12-16T21:36:06.067 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Dec 16 21:36:05 smithi045 ceph-mon[80763]: from='mgr.14214 172.21.15.45:0/1227909437' entity='mgr.smithi045.nhtmwc' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-12-16T21:36:06.067 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Dec 16 21:36:05 smithi045 ceph-mon[80763]: from='mgr.14214 172.21.15.45:0/1227909437' entity='mgr.smithi045.nhtmwc' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-12-16T21:36:06.067 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Dec 16 21:36:05 smithi045 ceph-mon[80763]: from='mgr.14214 172.21.15.45:0/1227909437' entity='mgr.smithi045.nhtmwc' 2023-12-16T21:36:06.196 INFO:teuthology.orchestra.run.smithi169.stderr:2023-12-16T21:36:06.194+0000 7f6b0508c700 1 -- 172.21.15.169:0/2215693675 --> [v2:172.21.15.45:6800/2012105536,v1:172.21.15.45:6801/2012105536] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi169:vg_nvme/lv_3", "target": ["mon-mgr", ""]}) v1 -- 0x7f6b0010f650 con 0x7f6ae8068f60 2023-12-16T21:36:06.861 INFO:journalctl@ceph.osd.3.smithi169.stdout:Dec 16 21:36:06 smithi169 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-3[93707]: 2023-12-16T21:36:06.667+0000 7f1780a64700 -1 osd.3 0 waiting for initial osdmap 2023-12-16T21:36:06.861 INFO:journalctl@ceph.osd.3.smithi169.stdout:Dec 16 21:36:06 smithi169 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-3[93707]: 2023-12-16T21:36:06.672+0000 7f177c060700 -1 osd.3 24 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2023-12-16T21:36:06.862 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Dec 16 21:36:06 smithi169 ceph-mon[88350]: Detected new or changed devices on smithi169 2023-12-16T21:36:06.862 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Dec 16 21:36:06 smithi169 ceph-mon[88350]: Adjusting osd_memory_target on smithi169 to 15937M 2023-12-16T21:36:06.862 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Dec 16 21:36:06 smithi169 ceph-mon[88350]: from='osd.3 ' entity='osd.3' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]': finished 2023-12-16T21:36:06.862 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Dec 16 21:36:06 smithi169 ceph-mon[88350]: osdmap e23: 4 total, 3 up, 4 in 2023-12-16T21:36:06.862 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Dec 16 21:36:06 smithi169 ceph-mon[88350]: from='mgr.14214 172.21.15.45:0/1227909437' entity='mgr.smithi045.nhtmwc' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2023-12-16T21:36:06.862 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Dec 16 21:36:06 smithi169 ceph-mon[88350]: from='osd.3 [v2:172.21.15.169:6800/201299104,v1:172.21.15.169:6801/201299104]' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi169", "root=default"]}]: dispatch 2023-12-16T21:36:06.862 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Dec 16 21:36:06 smithi169 ceph-mon[88350]: from='osd.3 ' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi169", "root=default"]}]: dispatch 2023-12-16T21:36:06.862 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Dec 16 21:36:06 smithi169 ceph-mon[88350]: from='mgr.14214 172.21.15.45:0/1227909437' entity='mgr.smithi045.nhtmwc' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2023-12-16T21:36:06.862 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Dec 16 21:36:06 smithi169 ceph-mon[88350]: from='mgr.14214 172.21.15.45:0/1227909437' entity='mgr.smithi045.nhtmwc' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2023-12-16T21:36:06.862 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Dec 16 21:36:06 smithi169 ceph-mon[88350]: from='mgr.14214 172.21.15.45:0/1227909437' entity='mgr.smithi045.nhtmwc' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-12-16T21:36:07.066 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Dec 16 21:36:06 smithi045 ceph-mon[80763]: Detected new or changed devices on smithi169 2023-12-16T21:36:07.066 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Dec 16 21:36:06 smithi045 ceph-mon[80763]: Adjusting osd_memory_target on smithi169 to 15937M 2023-12-16T21:36:07.066 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Dec 16 21:36:06 smithi045 ceph-mon[80763]: from='osd.3 ' entity='osd.3' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]': finished 2023-12-16T21:36:07.066 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Dec 16 21:36:06 smithi045 ceph-mon[80763]: osdmap e23: 4 total, 3 up, 4 in 2023-12-16T21:36:07.066 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Dec 16 21:36:06 smithi045 ceph-mon[80763]: from='mgr.14214 172.21.15.45:0/1227909437' entity='mgr.smithi045.nhtmwc' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2023-12-16T21:36:07.066 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Dec 16 21:36:06 smithi045 ceph-mon[80763]: from='osd.3 [v2:172.21.15.169:6800/201299104,v1:172.21.15.169:6801/201299104]' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi169", "root=default"]}]: dispatch 2023-12-16T21:36:07.066 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Dec 16 21:36:06 smithi045 ceph-mon[80763]: from='osd.3 ' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi169", "root=default"]}]: dispatch 2023-12-16T21:36:07.066 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Dec 16 21:36:06 smithi045 ceph-mon[80763]: from='mgr.14214 172.21.15.45:0/1227909437' entity='mgr.smithi045.nhtmwc' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2023-12-16T21:36:07.067 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Dec 16 21:36:06 smithi045 ceph-mon[80763]: from='mgr.14214 172.21.15.45:0/1227909437' entity='mgr.smithi045.nhtmwc' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2023-12-16T21:36:07.067 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Dec 16 21:36:06 smithi045 ceph-mon[80763]: from='mgr.14214 172.21.15.45:0/1227909437' entity='mgr.smithi045.nhtmwc' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-12-16T21:36:08.066 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Dec 16 21:36:07 smithi045 ceph-mon[80763]: from='client.24163 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi169:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2023-12-16T21:36:08.066 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Dec 16 21:36:07 smithi045 ceph-mon[80763]: pgmap v60: 1 pgs: 1 active+clean; 577 KiB data, 874 MiB used, 267 GiB / 268 GiB avail; 1.8 KiB/s rd, 78 KiB/s wr, 5 op/s 2023-12-16T21:36:08.066 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Dec 16 21:36:07 smithi045 ceph-mon[80763]: from='osd.3 ' entity='osd.3' cmd='[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi169", "root=default"]}]': finished 2023-12-16T21:36:08.066 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Dec 16 21:36:07 smithi045 ceph-mon[80763]: osdmap e24: 4 total, 3 up, 4 in 2023-12-16T21:36:08.066 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Dec 16 21:36:07 smithi045 ceph-mon[80763]: from='mgr.14214 172.21.15.45:0/1227909437' entity='mgr.smithi045.nhtmwc' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2023-12-16T21:36:08.066 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Dec 16 21:36:07 smithi045 ceph-mon[80763]: from='mgr.14214 172.21.15.45:0/1227909437' entity='mgr.smithi045.nhtmwc' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2023-12-16T21:36:08.132 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Dec 16 21:36:07 smithi169 ceph-mon[88350]: from='client.24163 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi169:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2023-12-16T21:36:08.132 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Dec 16 21:36:07 smithi169 ceph-mon[88350]: pgmap v60: 1 pgs: 1 active+clean; 577 KiB data, 874 MiB used, 267 GiB / 268 GiB avail; 1.8 KiB/s rd, 78 KiB/s wr, 5 op/s 2023-12-16T21:36:08.132 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Dec 16 21:36:07 smithi169 ceph-mon[88350]: from='osd.3 ' entity='osd.3' cmd='[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi169", "root=default"]}]': finished 2023-12-16T21:36:08.132 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Dec 16 21:36:07 smithi169 ceph-mon[88350]: osdmap e24: 4 total, 3 up, 4 in 2023-12-16T21:36:08.132 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Dec 16 21:36:07 smithi169 ceph-mon[88350]: from='mgr.14214 172.21.15.45:0/1227909437' entity='mgr.smithi045.nhtmwc' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2023-12-16T21:36:08.132 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Dec 16 21:36:07 smithi169 ceph-mon[88350]: from='mgr.14214 172.21.15.45:0/1227909437' entity='mgr.smithi045.nhtmwc' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2023-12-16T21:36:09.066 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Dec 16 21:36:08 smithi045 ceph-mon[80763]: purged_snaps scrub starts 2023-12-16T21:36:09.066 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Dec 16 21:36:08 smithi045 ceph-mon[80763]: purged_snaps scrub ok 2023-12-16T21:36:09.066 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Dec 16 21:36:08 smithi045 ceph-mon[80763]: osd.3 [v2:172.21.15.169:6800/201299104,v1:172.21.15.169:6801/201299104] boot 2023-12-16T21:36:09.066 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Dec 16 21:36:08 smithi045 ceph-mon[80763]: osdmap e25: 4 total, 4 up, 4 in 2023-12-16T21:36:09.066 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Dec 16 21:36:08 smithi045 ceph-mon[80763]: from='mgr.14214 172.21.15.45:0/1227909437' entity='mgr.smithi045.nhtmwc' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2023-12-16T21:36:09.066 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Dec 16 21:36:08 smithi045 ceph-mon[80763]: from='client.? 172.21.15.169:0/3969994042' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "0926f0d3-f077-4045-9d5a-e10a94421e9d"}]: dispatch 2023-12-16T21:36:09.066 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Dec 16 21:36:08 smithi045 ceph-mon[80763]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "0926f0d3-f077-4045-9d5a-e10a94421e9d"}]: dispatch 2023-12-16T21:36:09.066 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Dec 16 21:36:08 smithi045 ceph-mon[80763]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "0926f0d3-f077-4045-9d5a-e10a94421e9d"}]': finished 2023-12-16T21:36:09.066 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Dec 16 21:36:08 smithi045 ceph-mon[80763]: osdmap e26: 5 total, 4 up, 5 in 2023-12-16T21:36:09.067 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Dec 16 21:36:08 smithi045 ceph-mon[80763]: from='mgr.14214 172.21.15.45:0/1227909437' entity='mgr.smithi045.nhtmwc' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2023-12-16T21:36:09.067 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Dec 16 21:36:08 smithi045 ceph-mon[80763]: from='client.? 172.21.15.169:0/400383955' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2023-12-16T21:36:09.134 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Dec 16 21:36:08 smithi169 ceph-mon[88350]: purged_snaps scrub starts 2023-12-16T21:36:09.134 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Dec 16 21:36:08 smithi169 ceph-mon[88350]: purged_snaps scrub ok 2023-12-16T21:36:09.134 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Dec 16 21:36:08 smithi169 ceph-mon[88350]: osd.3 [v2:172.21.15.169:6800/201299104,v1:172.21.15.169:6801/201299104] boot 2023-12-16T21:36:09.134 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Dec 16 21:36:08 smithi169 ceph-mon[88350]: osdmap e25: 4 total, 4 up, 4 in 2023-12-16T21:36:09.134 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Dec 16 21:36:08 smithi169 ceph-mon[88350]: from='mgr.14214 172.21.15.45:0/1227909437' entity='mgr.smithi045.nhtmwc' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2023-12-16T21:36:09.134 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Dec 16 21:36:08 smithi169 ceph-mon[88350]: from='client.? 172.21.15.169:0/3969994042' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "0926f0d3-f077-4045-9d5a-e10a94421e9d"}]: dispatch 2023-12-16T21:36:09.134 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Dec 16 21:36:08 smithi169 ceph-mon[88350]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "0926f0d3-f077-4045-9d5a-e10a94421e9d"}]: dispatch 2023-12-16T21:36:09.134 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Dec 16 21:36:08 smithi169 ceph-mon[88350]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "0926f0d3-f077-4045-9d5a-e10a94421e9d"}]': finished 2023-12-16T21:36:09.134 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Dec 16 21:36:08 smithi169 ceph-mon[88350]: osdmap e26: 5 total, 4 up, 5 in 2023-12-16T21:36:09.134 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Dec 16 21:36:08 smithi169 ceph-mon[88350]: from='mgr.14214 172.21.15.45:0/1227909437' entity='mgr.smithi045.nhtmwc' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2023-12-16T21:36:09.135 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Dec 16 21:36:08 smithi169 ceph-mon[88350]: from='client.? 172.21.15.169:0/400383955' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2023-12-16T21:36:10.067 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Dec 16 21:36:09 smithi045 ceph-mon[80763]: pgmap v64: 1 pgs: 1 active+clean; 577 KiB data, 1.1 GiB used, 356 GiB / 358 GiB avail 2023-12-16T21:36:10.067 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Dec 16 21:36:09 smithi045 ceph-mon[80763]: osdmap e27: 5 total, 4 up, 5 in 2023-12-16T21:36:10.067 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Dec 16 21:36:09 smithi045 ceph-mon[80763]: from='mgr.14214 172.21.15.45:0/1227909437' entity='mgr.smithi045.nhtmwc' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2023-12-16T21:36:10.133 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Dec 16 21:36:09 smithi169 ceph-mon[88350]: pgmap v64: 1 pgs: 1 active+clean; 577 KiB data, 1.1 GiB used, 356 GiB / 358 GiB avail 2023-12-16T21:36:10.133 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Dec 16 21:36:09 smithi169 ceph-mon[88350]: osdmap e27: 5 total, 4 up, 5 in 2023-12-16T21:36:10.133 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Dec 16 21:36:09 smithi169 ceph-mon[88350]: from='mgr.14214 172.21.15.45:0/1227909437' entity='mgr.smithi045.nhtmwc' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2023-12-16T21:36:11.066 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Dec 16 21:36:10 smithi045 ceph-mon[80763]: osdmap e28: 5 total, 4 up, 5 in 2023-12-16T21:36:11.066 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Dec 16 21:36:10 smithi045 ceph-mon[80763]: from='mgr.14214 172.21.15.45:0/1227909437' entity='mgr.smithi045.nhtmwc' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2023-12-16T21:36:11.066 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Dec 16 21:36:10 smithi045 ceph-mon[80763]: osdmap e29: 5 total, 4 up, 5 in 2023-12-16T21:36:11.066 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Dec 16 21:36:10 smithi045 ceph-mon[80763]: from='mgr.14214 172.21.15.45:0/1227909437' entity='mgr.smithi045.nhtmwc' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2023-12-16T21:36:11.133 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Dec 16 21:36:10 smithi169 ceph-mon[88350]: osdmap e28: 5 total, 4 up, 5 in 2023-12-16T21:36:11.133 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Dec 16 21:36:10 smithi169 ceph-mon[88350]: from='mgr.14214 172.21.15.45:0/1227909437' entity='mgr.smithi045.nhtmwc' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2023-12-16T21:36:11.133 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Dec 16 21:36:10 smithi169 ceph-mon[88350]: osdmap e29: 5 total, 4 up, 5 in 2023-12-16T21:36:11.133 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Dec 16 21:36:10 smithi169 ceph-mon[88350]: from='mgr.14214 172.21.15.45:0/1227909437' entity='mgr.smithi045.nhtmwc' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2023-12-16T21:36:12.066 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Dec 16 21:36:11 smithi045 ceph-mon[80763]: pgmap v67: 1 pgs: 1 active+clean; 577 KiB data, 1.1 GiB used, 356 GiB / 358 GiB avail 2023-12-16T21:36:12.133 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Dec 16 21:36:11 smithi169 ceph-mon[88350]: pgmap v67: 1 pgs: 1 active+clean; 577 KiB data, 1.1 GiB used, 356 GiB / 358 GiB avail 2023-12-16T21:36:14.066 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Dec 16 21:36:13 smithi045 ceph-mon[80763]: pgmap v69: 1 pgs: 1 active+clean; 577 KiB data, 1.1 GiB used, 356 GiB / 358 GiB avail; 119 KiB/s, 0 objects/s recovering 2023-12-16T21:36:14.066 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Dec 16 21:36:13 smithi045 ceph-mon[80763]: from='mgr.14214 172.21.15.45:0/1227909437' entity='mgr.smithi045.nhtmwc' cmd=[{"prefix": "auth get", "entity": "osd.4"}]: dispatch 2023-12-16T21:36:14.066 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Dec 16 21:36:13 smithi045 ceph-mon[80763]: from='mgr.14214 172.21.15.45:0/1227909437' entity='mgr.smithi045.nhtmwc' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-12-16T21:36:14.066 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Dec 16 21:36:13 smithi045 ceph-mon[80763]: from='mgr.14214 172.21.15.45:0/1227909437' entity='mgr.smithi045.nhtmwc' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2023-12-16T21:36:14.102 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Dec 16 21:36:13 smithi169 ceph-mon[88350]: pgmap v69: 1 pgs: 1 active+clean; 577 KiB data, 1.1 GiB used, 356 GiB / 358 GiB avail; 119 KiB/s, 0 objects/s recovering 2023-12-16T21:36:14.102 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Dec 16 21:36:13 smithi169 ceph-mon[88350]: from='mgr.14214 172.21.15.45:0/1227909437' entity='mgr.smithi045.nhtmwc' cmd=[{"prefix": "auth get", "entity": "osd.4"}]: dispatch 2023-12-16T21:36:14.102 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Dec 16 21:36:13 smithi169 ceph-mon[88350]: from='mgr.14214 172.21.15.45:0/1227909437' entity='mgr.smithi045.nhtmwc' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-12-16T21:36:14.102 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Dec 16 21:36:13 smithi169 ceph-mon[88350]: from='mgr.14214 172.21.15.45:0/1227909437' entity='mgr.smithi045.nhtmwc' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2023-12-16T21:36:15.037 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Dec 16 21:36:14 smithi169 ceph-mon[88350]: Deploying daemon osd.4 on smithi169 2023-12-16T21:36:15.316 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Dec 16 21:36:14 smithi045 ceph-mon[80763]: Deploying daemon osd.4 on smithi169 2023-12-16T21:36:16.078 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Dec 16 21:36:15 smithi169 ceph-mon[88350]: pgmap v70: 1 pgs: 1 active+clean; 577 KiB data, 1.1 GiB used, 356 GiB / 358 GiB avail; 96 KiB/s, 0 objects/s recovering 2023-12-16T21:36:16.316 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Dec 16 21:36:15 smithi045 ceph-mon[80763]: pgmap v70: 1 pgs: 1 active+clean; 577 KiB data, 1.1 GiB used, 356 GiB / 358 GiB avail; 96 KiB/s, 0 objects/s recovering 2023-12-16T21:36:17.816 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Dec 16 21:36:17 smithi045 ceph-mon[80763]: from='mgr.14214 172.21.15.45:0/1227909437' entity='mgr.smithi045.nhtmwc' 2023-12-16T21:36:17.816 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Dec 16 21:36:17 smithi045 ceph-mon[80763]: from='mgr.14214 172.21.15.45:0/1227909437' entity='mgr.smithi045.nhtmwc' 2023-12-16T21:36:17.816 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Dec 16 21:36:17 smithi045 ceph-mon[80763]: from='mgr.14214 172.21.15.45:0/1227909437' entity='mgr.smithi045.nhtmwc' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-12-16T21:36:17.816 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Dec 16 21:36:17 smithi045 ceph-mon[80763]: pgmap v71: 1 pgs: 1 active+clean; 577 KiB data, 1.1 GiB used, 356 GiB / 358 GiB avail; 74 KiB/s, 0 objects/s recovering 2023-12-16T21:36:17.883 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Dec 16 21:36:17 smithi169 ceph-mon[88350]: from='mgr.14214 172.21.15.45:0/1227909437' entity='mgr.smithi045.nhtmwc' 2023-12-16T21:36:17.883 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Dec 16 21:36:17 smithi169 ceph-mon[88350]: from='mgr.14214 172.21.15.45:0/1227909437' entity='mgr.smithi045.nhtmwc' 2023-12-16T21:36:17.884 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Dec 16 21:36:17 smithi169 ceph-mon[88350]: from='mgr.14214 172.21.15.45:0/1227909437' entity='mgr.smithi045.nhtmwc' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-12-16T21:36:17.884 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Dec 16 21:36:17 smithi169 ceph-mon[88350]: pgmap v71: 1 pgs: 1 active+clean; 577 KiB data, 1.1 GiB used, 356 GiB / 358 GiB avail; 74 KiB/s, 0 objects/s recovering 2023-12-16T21:36:18.446 INFO:teuthology.orchestra.run.smithi169.stderr:2023-12-16T21:36:18.443+0000 7f6ae77fe700 1 -- 172.21.15.169:0/2215693675 <== mgr.14214 v2:172.21.15.45:6800/2012105536 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+37 (secure 0 0 0) 0x7f6b0010f650 con 0x7f6ae8068f60 2023-12-16T21:36:18.446 INFO:teuthology.orchestra.run.smithi169.stdout:Created osd(s) 4 on host 'smithi169' 2023-12-16T21:36:18.448 INFO:teuthology.orchestra.run.smithi169.stderr:2023-12-16T21:36:18.445+0000 7f6b0508c700 1 -- 172.21.15.169:0/2215693675 >> [v2:172.21.15.45:6800/2012105536,v1:172.21.15.45:6801/2012105536] conn(0x7f6ae8068f60 msgr2=0x7f6ae806b410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:36:18.448 INFO:teuthology.orchestra.run.smithi169.stderr:2023-12-16T21:36:18.445+0000 7f6b0508c700 1 --2- 172.21.15.169:0/2215693675 >> [v2:172.21.15.45:6800/2012105536,v1:172.21.15.45:6801/2012105536] conn(0x7f6ae8068f60 0x7f6ae806b410 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7f6af4004560 tx=0x7f6af4009e30 comp rx=0 tx=0).stop 2023-12-16T21:36:18.448 INFO:teuthology.orchestra.run.smithi169.stderr:2023-12-16T21:36:18.445+0000 7f6b0508c700 1 -- 172.21.15.169:0/2215693675 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f6b000fcbf0 msgr2=0x7f6b0010d5e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:36:18.448 INFO:teuthology.orchestra.run.smithi169.stderr:2023-12-16T21:36:18.445+0000 7f6b0508c700 1 --2- 172.21.15.169:0/2215693675 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f6b000fcbf0 0x7f6b0010d5e0 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0x7f6af0002a80 tx=0x7f6af0004600 comp rx=0 tx=0).stop 2023-12-16T21:36:18.448 INFO:teuthology.orchestra.run.smithi169.stderr:2023-12-16T21:36:18.445+0000 7f6b0508c700 1 -- 172.21.15.169:0/2215693675 shutdown_connections 2023-12-16T21:36:18.448 INFO:teuthology.orchestra.run.smithi169.stderr:2023-12-16T21:36:18.445+0000 7f6b0508c700 1 --2- 172.21.15.169:0/2215693675 >> [v2:172.21.15.45:6800/2012105536,v1:172.21.15.45:6801/2012105536] conn(0x7f6ae8068f60 0x7f6ae806b410 unknown :-1 s=CLOSED pgs=45 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:36:18.449 INFO:teuthology.orchestra.run.smithi169.stderr:2023-12-16T21:36:18.445+0000 7f6b0508c700 1 --2- 172.21.15.169:0/2215693675 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f6b000fcbf0 0x7f6b0010d5e0 unknown :-1 s=CLOSED pgs=15 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:36:18.449 INFO:teuthology.orchestra.run.smithi169.stderr:2023-12-16T21:36:18.445+0000 7f6b0508c700 1 --2- 172.21.15.169:0/2215693675 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f6b000fde00 0x7f6b0010db20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:36:18.449 INFO:teuthology.orchestra.run.smithi169.stderr:2023-12-16T21:36:18.445+0000 7f6b0508c700 1 -- 172.21.15.169:0/2215693675 >> 172.21.15.169:0/2215693675 conn(0x7f6b000f82d0 msgr2=0x7f6b00101030 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-16T21:36:18.449 INFO:teuthology.orchestra.run.smithi169.stderr:2023-12-16T21:36:18.445+0000 7f6b0508c700 1 -- 172.21.15.169:0/2215693675 shutdown_connections 2023-12-16T21:36:18.449 INFO:teuthology.orchestra.run.smithi169.stderr:2023-12-16T21:36:18.445+0000 7f6b0508c700 1 -- 172.21.15.169:0/2215693675 wait complete. 2023-12-16T21:36:18.701 DEBUG:teuthology.orchestra.run.smithi169:osd.4> sudo journalctl -f -n 0 -u ceph-88ff363c-9c5a-11ee-95a5-87774f69a715@osd.4.service 2023-12-16T21:36:18.703 INFO:tasks.cephadm:Deploying osd.5 on smithi169 with /dev/vg_nvme/lv_2... 2023-12-16T21:36:18.704 DEBUG:teuthology.orchestra.run.smithi169:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 88ff363c-9c5a-11ee-95a5-87774f69a715 -- ceph-volume lvm zap /dev/vg_nvme/lv_2 2023-12-16T21:36:19.018 INFO:teuthology.orchestra.run.smithi169.stderr:Inferring config /var/lib/ceph/88ff363c-9c5a-11ee-95a5-87774f69a715/mon.smithi169/config 2023-12-16T21:36:19.661 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Dec 16 21:36:19 smithi169 ceph-mon[88350]: from='mgr.14214 172.21.15.45:0/1227909437' entity='mgr.smithi045.nhtmwc' 2023-12-16T21:36:19.661 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Dec 16 21:36:19 smithi169 ceph-mon[88350]: from='mgr.14214 172.21.15.45:0/1227909437' entity='mgr.smithi045.nhtmwc' 2023-12-16T21:36:19.661 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Dec 16 21:36:19 smithi169 ceph-mon[88350]: pgmap v72: 1 pgs: 1 active+clean; 577 KiB data, 1.1 GiB used, 356 GiB / 358 GiB avail; 65 KiB/s, 0 objects/s recovering 2023-12-16T21:36:19.661 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Dec 16 21:36:19 smithi169 ceph-mon[88350]: from='mgr.14214 172.21.15.45:0/1227909437' entity='mgr.smithi045.nhtmwc' 2023-12-16T21:36:19.661 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Dec 16 21:36:19 smithi169 ceph-mon[88350]: from='mgr.14214 172.21.15.45:0/1227909437' entity='mgr.smithi045.nhtmwc' 2023-12-16T21:36:19.816 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Dec 16 21:36:19 smithi045 ceph-mon[80763]: from='mgr.14214 172.21.15.45:0/1227909437' entity='mgr.smithi045.nhtmwc' 2023-12-16T21:36:19.816 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Dec 16 21:36:19 smithi045 ceph-mon[80763]: from='mgr.14214 172.21.15.45:0/1227909437' entity='mgr.smithi045.nhtmwc' 2023-12-16T21:36:19.816 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Dec 16 21:36:19 smithi045 ceph-mon[80763]: pgmap v72: 1 pgs: 1 active+clean; 577 KiB data, 1.1 GiB used, 356 GiB / 358 GiB avail; 65 KiB/s, 0 objects/s recovering 2023-12-16T21:36:19.816 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Dec 16 21:36:19 smithi045 ceph-mon[80763]: from='mgr.14214 172.21.15.45:0/1227909437' entity='mgr.smithi045.nhtmwc' 2023-12-16T21:36:19.816 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Dec 16 21:36:19 smithi045 ceph-mon[80763]: from='mgr.14214 172.21.15.45:0/1227909437' entity='mgr.smithi045.nhtmwc' 2023-12-16T21:36:20.082 INFO:teuthology.orchestra.run.smithi169.stderr:--> Zapping: /dev/vg_nvme/lv_2 2023-12-16T21:36:20.082 INFO:teuthology.orchestra.run.smithi169.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_2 bs=1M count=10 conv=fsync 2023-12-16T21:36:20.083 INFO:teuthology.orchestra.run.smithi169.stderr: stderr: 10+0 records in 2023-12-16T21:36:20.083 INFO:teuthology.orchestra.run.smithi169.stderr:10+0 records out 2023-12-16T21:36:20.083 INFO:teuthology.orchestra.run.smithi169.stderr:10485760 bytes (10 MB, 10 MiB) copied, 0.0152482 s, 688 MB/s 2023-12-16T21:36:20.083 INFO:teuthology.orchestra.run.smithi169.stderr:--> Zapping successful for: 2023-12-16T21:36:20.351 DEBUG:teuthology.orchestra.run.smithi169:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 88ff363c-9c5a-11ee-95a5-87774f69a715 -- ceph orch daemon add osd smithi169:vg_nvme/lv_2 2023-12-16T21:36:20.566 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Dec 16 21:36:20 smithi045 ceph-mon[80763]: from='osd.4 [v2:172.21.15.169:6808/3840494264,v1:172.21.15.169:6809/3840494264]' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2023-12-16T21:36:20.566 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Dec 16 21:36:20 smithi045 ceph-mon[80763]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2023-12-16T21:36:20.633 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Dec 16 21:36:20 smithi169 ceph-mon[88350]: from='osd.4 [v2:172.21.15.169:6808/3840494264,v1:172.21.15.169:6809/3840494264]' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2023-12-16T21:36:20.633 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Dec 16 21:36:20 smithi169 ceph-mon[88350]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2023-12-16T21:36:20.634 INFO:journalctl@ceph.osd.4.smithi169.stdout:Dec 16 21:36:20 smithi169 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-4[100115]: 2023-12-16T21:36:20.483+0000 7fdb7f313540 -1 osd.4 0 log_to_monitors true 2023-12-16T21:36:20.637 INFO:teuthology.orchestra.run.smithi169.stderr:Inferring config /var/lib/ceph/88ff363c-9c5a-11ee-95a5-87774f69a715/mon.smithi169/config 2023-12-16T21:36:21.517 INFO:teuthology.orchestra.run.smithi169.stderr:2023-12-16T21:36:21.342+0000 7fae01cb4700 1 -- 172.21.15.169:0/1398321500 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fadfc0fdf60 msgr2=0x7fadfc0fe3b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:36:21.517 INFO:teuthology.orchestra.run.smithi169.stderr:2023-12-16T21:36:21.342+0000 7fae01cb4700 1 --2- 172.21.15.169:0/1398321500 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fadfc0fdf60 0x7fadfc0fe3b0 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7fadf0004940 tx=0x7fadf0009e30 comp rx=0 tx=0).stop 2023-12-16T21:36:21.517 INFO:teuthology.orchestra.run.smithi169.stderr:2023-12-16T21:36:21.342+0000 7fae01cb4700 1 -- 172.21.15.169:0/1398321500 shutdown_connections 2023-12-16T21:36:21.517 INFO:teuthology.orchestra.run.smithi169.stderr:2023-12-16T21:36:21.342+0000 7fae01cb4700 1 --2- 172.21.15.169:0/1398321500 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fadfc0fdf60 0x7fadfc0fe3b0 unknown :-1 s=CLOSED pgs=23 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:36:21.517 INFO:teuthology.orchestra.run.smithi169.stderr:2023-12-16T21:36:21.342+0000 7fae01cb4700 1 --2- 172.21.15.169:0/1398321500 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fadfc0faed0 0x7fadfc0fb2e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:36:21.517 INFO:teuthology.orchestra.run.smithi169.stderr:2023-12-16T21:36:21.342+0000 7fae01cb4700 1 -- 172.21.15.169:0/1398321500 >> 172.21.15.169:0/1398321500 conn(0x7fadfc0f6950 msgr2=0x7fadfc0f8d80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-16T21:36:21.517 INFO:teuthology.orchestra.run.smithi169.stderr:2023-12-16T21:36:21.342+0000 7fae01cb4700 1 -- 172.21.15.169:0/1398321500 shutdown_connections 2023-12-16T21:36:21.517 INFO:teuthology.orchestra.run.smithi169.stderr:2023-12-16T21:36:21.342+0000 7fae01cb4700 1 -- 172.21.15.169:0/1398321500 wait complete. 2023-12-16T21:36:21.518 INFO:teuthology.orchestra.run.smithi169.stderr:2023-12-16T21:36:21.343+0000 7fae01cb4700 1 Processor -- start 2023-12-16T21:36:21.518 INFO:teuthology.orchestra.run.smithi169.stderr:2023-12-16T21:36:21.343+0000 7fae01cb4700 1 -- start start 2023-12-16T21:36:21.518 INFO:teuthology.orchestra.run.smithi169.stderr:2023-12-16T21:36:21.343+0000 7fae01cb4700 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fadfc0faed0 0x7fadfc10d600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:36:21.518 INFO:teuthology.orchestra.run.smithi169.stderr:2023-12-16T21:36:21.343+0000 7fae01cb4700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fadfc0fdf60 0x7fadfc10db40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:36:21.518 INFO:teuthology.orchestra.run.smithi169.stderr:2023-12-16T21:36:21.343+0000 7fae01cb4700 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fadfc10e1d0 con 0x7fadfc0fdf60 2023-12-16T21:36:21.518 INFO:teuthology.orchestra.run.smithi169.stderr:2023-12-16T21:36:21.343+0000 7fae01cb4700 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fadfc10ea10 con 0x7fadfc0faed0 2023-12-16T21:36:21.518 INFO:teuthology.orchestra.run.smithi169.stderr:2023-12-16T21:36:21.343+0000 7fadfb7fe700 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fadfc0faed0 0x7fadfc10d600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:36:21.518 INFO:teuthology.orchestra.run.smithi169.stderr:2023-12-16T21:36:21.343+0000 7fadfb7fe700 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fadfc0faed0 0x7fadfc10d600 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.169:3300/0 says I am v2:172.21.15.169:36086/0 (socket says 172.21.15.169:36086) 2023-12-16T21:36:21.518 INFO:teuthology.orchestra.run.smithi169.stderr:2023-12-16T21:36:21.343+0000 7fadfb7fe700 1 -- 172.21.15.169:0/4211790209 learned_addr learned my addr 172.21.15.169:0/4211790209 (peer_addr_for_me v2:172.21.15.169:0/0) 2023-12-16T21:36:21.518 INFO:teuthology.orchestra.run.smithi169.stderr:2023-12-16T21:36:21.344+0000 7fadfb7fe700 1 -- 172.21.15.169:0/4211790209 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fadfc0fdf60 msgr2=0x7fadfc10db40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:36:21.518 INFO:teuthology.orchestra.run.smithi169.stderr:2023-12-16T21:36:21.344+0000 7fadfb7fe700 1 --2- 172.21.15.169:0/4211790209 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fadfc0fdf60 0x7fadfc10db40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:36:21.518 INFO:teuthology.orchestra.run.smithi169.stderr:2023-12-16T21:36:21.344+0000 7fadfb7fe700 1 -- 172.21.15.169:0/4211790209 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fadf0005040 con 0x7fadfc0faed0 2023-12-16T21:36:21.518 INFO:teuthology.orchestra.run.smithi169.stderr:2023-12-16T21:36:21.344+0000 7fadfb7fe700 1 --2- 172.21.15.169:0/4211790209 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fadfc0faed0 0x7fadfc10d600 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7fadec00b790 tx=0x7fadec009220 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:36:21.518 INFO:teuthology.orchestra.run.smithi169.stderr:2023-12-16T21:36:21.344+0000 7fade3fff700 1 -- 172.21.15.169:0/4211790209 <== mon.1 v2:172.21.15.169:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fadec00f6f0 con 0x7fadfc0faed0 2023-12-16T21:36:21.519 INFO:teuthology.orchestra.run.smithi169.stderr:2023-12-16T21:36:21.344+0000 7fae01cb4700 1 -- 172.21.15.169:0/4211790209 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fadfc10ecf0 con 0x7fadfc0faed0 2023-12-16T21:36:21.519 INFO:teuthology.orchestra.run.smithi169.stderr:2023-12-16T21:36:21.344+0000 7fade3fff700 1 -- 172.21.15.169:0/4211790209 <== mon.1 v2:172.21.15.169:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fadec00fd00 con 0x7fadfc0faed0 2023-12-16T21:36:21.519 INFO:teuthology.orchestra.run.smithi169.stderr:2023-12-16T21:36:21.344+0000 7fade3fff700 1 -- 172.21.15.169:0/4211790209 <== mon.1 v2:172.21.15.169:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fadec0103a0 con 0x7fadfc0faed0 2023-12-16T21:36:21.519 INFO:teuthology.orchestra.run.smithi169.stderr:2023-12-16T21:36:21.344+0000 7fae01cb4700 1 -- 172.21.15.169:0/4211790209 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fadfc10f230 con 0x7fadfc0faed0 2023-12-16T21:36:21.519 INFO:teuthology.orchestra.run.smithi169.stderr:2023-12-16T21:36:21.345+0000 7fade1ffb700 1 -- 172.21.15.169:0/4211790209 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7faddc0052b0 con 0x7fadfc0faed0 2023-12-16T21:36:21.519 INFO:teuthology.orchestra.run.smithi169.stderr:2023-12-16T21:36:21.345+0000 7fade3fff700 1 -- 172.21.15.169:0/4211790209 <== mon.1 v2:172.21.15.169:3300/0 4 ==== mgrmap(e 19) v1 ==== 86447+0+0 (secure 0 0 0) 0x7fadec00f850 con 0x7fadfc0faed0 2023-12-16T21:36:21.519 INFO:teuthology.orchestra.run.smithi169.stderr:2023-12-16T21:36:21.346+0000 7fade3fff700 1 --2- 172.21.15.169:0/4211790209 >> [v2:172.21.15.45:6800/2012105536,v1:172.21.15.45:6801/2012105536] conn(0x7fade4068f70 0x7fade406b420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:36:21.519 INFO:teuthology.orchestra.run.smithi169.stderr:2023-12-16T21:36:21.346+0000 7fade3fff700 1 -- 172.21.15.169:0/4211790209 <== mon.1 v2:172.21.15.169:3300/0 5 ==== osd_map(30..30 src has 1..30) v4 ==== 3906+0+0 (secure 0 0 0) 0x7fadec0865c0 con 0x7fadfc0faed0 2023-12-16T21:36:21.519 INFO:teuthology.orchestra.run.smithi169.stderr:2023-12-16T21:36:21.346+0000 7fadfaffd700 1 --2- 172.21.15.169:0/4211790209 >> [v2:172.21.15.45:6800/2012105536,v1:172.21.15.45:6801/2012105536] conn(0x7fade4068f70 0x7fade406b420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:36:21.519 INFO:teuthology.orchestra.run.smithi169.stderr:2023-12-16T21:36:21.346+0000 7fadfaffd700 1 --2- 172.21.15.169:0/4211790209 >> [v2:172.21.15.45:6800/2012105536,v1:172.21.15.45:6801/2012105536] conn(0x7fade4068f70 0x7fade406b420 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7fadf0004f90 tx=0x7fadf0009e30 comp rx=0 tx=0).ready entity=mgr.14214 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:36:21.519 INFO:teuthology.orchestra.run.smithi169.stderr:2023-12-16T21:36:21.350+0000 7fade3fff700 1 -- 172.21.15.169:0/4211790209 <== mon.1 v2:172.21.15.169:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+174570 (secure 0 0 0) 0x7fadec056ab0 con 0x7fadfc0faed0 2023-12-16T21:36:21.566 INFO:teuthology.orchestra.run.smithi169.stderr:2023-12-16T21:36:21.563+0000 7fade1ffb700 1 -- 172.21.15.169:0/4211790209 --> [v2:172.21.15.45:6800/2012105536,v1:172.21.15.45:6801/2012105536] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi169:vg_nvme/lv_2", "target": ["mon-mgr", ""]}) v1 -- 0x7faddc000bc0 con 0x7fade4068f70 2023-12-16T21:36:21.633 INFO:journalctl@ceph.osd.4.smithi169.stdout:Dec 16 21:36:21 smithi169 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-4[100115]: 2023-12-16T21:36:21.553+0000 7fdb75af8700 -1 osd.4 0 waiting for initial osdmap 2023-12-16T21:36:21.634 INFO:journalctl@ceph.osd.4.smithi169.stdout:Dec 16 21:36:21 smithi169 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-4[100115]: 2023-12-16T21:36:21.559+0000 7fdb718f5700 -1 osd.4 31 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2023-12-16T21:36:21.634 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Dec 16 21:36:21 smithi169 ceph-mon[88350]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]': finished 2023-12-16T21:36:21.634 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Dec 16 21:36:21 smithi169 ceph-mon[88350]: osdmap e30: 5 total, 4 up, 5 in 2023-12-16T21:36:21.634 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Dec 16 21:36:21 smithi169 ceph-mon[88350]: from='mgr.14214 172.21.15.45:0/1227909437' entity='mgr.smithi045.nhtmwc' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2023-12-16T21:36:21.634 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Dec 16 21:36:21 smithi169 ceph-mon[88350]: from='osd.4 [v2:172.21.15.169:6808/3840494264,v1:172.21.15.169:6809/3840494264]' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi169", "root=default"]}]: dispatch 2023-12-16T21:36:21.634 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Dec 16 21:36:21 smithi169 ceph-mon[88350]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi169", "root=default"]}]: dispatch 2023-12-16T21:36:21.634 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Dec 16 21:36:21 smithi169 ceph-mon[88350]: pgmap v74: 1 pgs: 1 active+clean; 577 KiB data, 1.1 GiB used, 356 GiB / 358 GiB avail; 59 KiB/s, 0 objects/s recovering 2023-12-16T21:36:21.635 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Dec 16 21:36:21 smithi169 ceph-mon[88350]: from='mgr.14214 172.21.15.45:0/1227909437' entity='mgr.smithi045.nhtmwc' 2023-12-16T21:36:21.635 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Dec 16 21:36:21 smithi169 ceph-mon[88350]: from='mgr.14214 172.21.15.45:0/1227909437' entity='mgr.smithi045.nhtmwc' 2023-12-16T21:36:21.635 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Dec 16 21:36:21 smithi169 ceph-mon[88350]: from='mgr.14214 172.21.15.45:0/1227909437' entity='mgr.smithi045.nhtmwc' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2023-12-16T21:36:21.635 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Dec 16 21:36:21 smithi169 ceph-mon[88350]: from='mgr.14214 172.21.15.45:0/1227909437' entity='mgr.smithi045.nhtmwc' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2023-12-16T21:36:21.635 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Dec 16 21:36:21 smithi169 ceph-mon[88350]: from='mgr.14214 172.21.15.45:0/1227909437' entity='mgr.smithi045.nhtmwc' 2023-12-16T21:36:21.635 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Dec 16 21:36:21 smithi169 ceph-mon[88350]: from='mgr.14214 172.21.15.45:0/1227909437' entity='mgr.smithi045.nhtmwc' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-12-16T21:36:21.635 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Dec 16 21:36:21 smithi169 ceph-mon[88350]: from='mgr.14214 172.21.15.45:0/1227909437' entity='mgr.smithi045.nhtmwc' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-12-16T21:36:21.816 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Dec 16 21:36:21 smithi045 ceph-mon[80763]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]': finished 2023-12-16T21:36:21.816 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Dec 16 21:36:21 smithi045 ceph-mon[80763]: osdmap e30: 5 total, 4 up, 5 in 2023-12-16T21:36:21.816 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Dec 16 21:36:21 smithi045 ceph-mon[80763]: from='mgr.14214 172.21.15.45:0/1227909437' entity='mgr.smithi045.nhtmwc' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2023-12-16T21:36:21.816 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Dec 16 21:36:21 smithi045 ceph-mon[80763]: from='osd.4 [v2:172.21.15.169:6808/3840494264,v1:172.21.15.169:6809/3840494264]' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi169", "root=default"]}]: dispatch 2023-12-16T21:36:21.816 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Dec 16 21:36:21 smithi045 ceph-mon[80763]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi169", "root=default"]}]: dispatch 2023-12-16T21:36:21.816 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Dec 16 21:36:21 smithi045 ceph-mon[80763]: pgmap v74: 1 pgs: 1 active+clean; 577 KiB data, 1.1 GiB used, 356 GiB / 358 GiB avail; 59 KiB/s, 0 objects/s recovering 2023-12-16T21:36:21.816 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Dec 16 21:36:21 smithi045 ceph-mon[80763]: from='mgr.14214 172.21.15.45:0/1227909437' entity='mgr.smithi045.nhtmwc' 2023-12-16T21:36:21.816 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Dec 16 21:36:21 smithi045 ceph-mon[80763]: from='mgr.14214 172.21.15.45:0/1227909437' entity='mgr.smithi045.nhtmwc' 2023-12-16T21:36:21.816 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Dec 16 21:36:21 smithi045 ceph-mon[80763]: from='mgr.14214 172.21.15.45:0/1227909437' entity='mgr.smithi045.nhtmwc' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2023-12-16T21:36:21.817 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Dec 16 21:36:21 smithi045 ceph-mon[80763]: from='mgr.14214 172.21.15.45:0/1227909437' entity='mgr.smithi045.nhtmwc' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2023-12-16T21:36:21.817 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Dec 16 21:36:21 smithi045 ceph-mon[80763]: from='mgr.14214 172.21.15.45:0/1227909437' entity='mgr.smithi045.nhtmwc' 2023-12-16T21:36:21.817 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Dec 16 21:36:21 smithi045 ceph-mon[80763]: from='mgr.14214 172.21.15.45:0/1227909437' entity='mgr.smithi045.nhtmwc' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-12-16T21:36:21.817 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Dec 16 21:36:21 smithi045 ceph-mon[80763]: from='mgr.14214 172.21.15.45:0/1227909437' entity='mgr.smithi045.nhtmwc' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-12-16T21:36:22.816 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Dec 16 21:36:22 smithi045 ceph-mon[80763]: Detected new or changed devices on smithi169 2023-12-16T21:36:22.816 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Dec 16 21:36:22 smithi045 ceph-mon[80763]: Adjusting osd_memory_target on smithi169 to 7968M 2023-12-16T21:36:22.816 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Dec 16 21:36:22 smithi045 ceph-mon[80763]: from='mgr.14214 172.21.15.45:0/1227909437' entity='mgr.smithi045.nhtmwc' 2023-12-16T21:36:22.816 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Dec 16 21:36:22 smithi045 ceph-mon[80763]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi169", "root=default"]}]': finished 2023-12-16T21:36:22.817 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Dec 16 21:36:22 smithi045 ceph-mon[80763]: osdmap e31: 5 total, 4 up, 5 in 2023-12-16T21:36:22.817 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Dec 16 21:36:22 smithi045 ceph-mon[80763]: from='mgr.14214 172.21.15.45:0/1227909437' entity='mgr.smithi045.nhtmwc' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2023-12-16T21:36:22.817 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Dec 16 21:36:22 smithi045 ceph-mon[80763]: from='mgr.14214 172.21.15.45:0/1227909437' entity='mgr.smithi045.nhtmwc' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2023-12-16T21:36:22.817 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Dec 16 21:36:22 smithi045 ceph-mon[80763]: from='client.24181 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi169:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2023-12-16T21:36:22.817 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Dec 16 21:36:22 smithi045 ceph-mon[80763]: from='mgr.14214 172.21.15.45:0/1227909437' entity='mgr.smithi045.nhtmwc' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2023-12-16T21:36:22.817 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Dec 16 21:36:22 smithi045 ceph-mon[80763]: from='mgr.14214 172.21.15.45:0/1227909437' entity='mgr.smithi045.nhtmwc' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2023-12-16T21:36:22.817 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Dec 16 21:36:22 smithi045 ceph-mon[80763]: from='mgr.14214 172.21.15.45:0/1227909437' entity='mgr.smithi045.nhtmwc' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-12-16T21:36:22.817 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Dec 16 21:36:22 smithi045 ceph-mon[80763]: from='mgr.14214 172.21.15.45:0/1227909437' entity='mgr.smithi045.nhtmwc' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-12-16T21:36:22.817 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Dec 16 21:36:22 smithi045 ceph-mon[80763]: from='mgr.14214 172.21.15.45:0/1227909437' entity='mgr.smithi045.nhtmwc' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-12-16T21:36:22.817 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Dec 16 21:36:22 smithi045 ceph-mon[80763]: from='mgr.14214 172.21.15.45:0/1227909437' entity='mgr.smithi045.nhtmwc' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-12-16T21:36:22.817 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Dec 16 21:36:22 smithi045 ceph-mon[80763]: from='mgr.14214 172.21.15.45:0/1227909437' entity='mgr.smithi045.nhtmwc' 2023-12-16T21:36:22.883 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Dec 16 21:36:22 smithi169 ceph-mon[88350]: Detected new or changed devices on smithi169 2023-12-16T21:36:22.884 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Dec 16 21:36:22 smithi169 ceph-mon[88350]: Adjusting osd_memory_target on smithi169 to 7968M 2023-12-16T21:36:22.884 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Dec 16 21:36:22 smithi169 ceph-mon[88350]: from='mgr.14214 172.21.15.45:0/1227909437' entity='mgr.smithi045.nhtmwc' 2023-12-16T21:36:22.884 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Dec 16 21:36:22 smithi169 ceph-mon[88350]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi169", "root=default"]}]': finished 2023-12-16T21:36:22.884 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Dec 16 21:36:22 smithi169 ceph-mon[88350]: osdmap e31: 5 total, 4 up, 5 in 2023-12-16T21:36:22.884 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Dec 16 21:36:22 smithi169 ceph-mon[88350]: from='mgr.14214 172.21.15.45:0/1227909437' entity='mgr.smithi045.nhtmwc' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2023-12-16T21:36:22.884 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Dec 16 21:36:22 smithi169 ceph-mon[88350]: from='mgr.14214 172.21.15.45:0/1227909437' entity='mgr.smithi045.nhtmwc' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2023-12-16T21:36:22.884 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Dec 16 21:36:22 smithi169 ceph-mon[88350]: from='client.24181 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi169:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2023-12-16T21:36:22.884 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Dec 16 21:36:22 smithi169 ceph-mon[88350]: from='mgr.14214 172.21.15.45:0/1227909437' entity='mgr.smithi045.nhtmwc' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2023-12-16T21:36:22.884 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Dec 16 21:36:22 smithi169 ceph-mon[88350]: from='mgr.14214 172.21.15.45:0/1227909437' entity='mgr.smithi045.nhtmwc' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2023-12-16T21:36:22.884 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Dec 16 21:36:22 smithi169 ceph-mon[88350]: from='mgr.14214 172.21.15.45:0/1227909437' entity='mgr.smithi045.nhtmwc' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-12-16T21:36:22.884 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Dec 16 21:36:22 smithi169 ceph-mon[88350]: from='mgr.14214 172.21.15.45:0/1227909437' entity='mgr.smithi045.nhtmwc' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-12-16T21:36:22.884 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Dec 16 21:36:22 smithi169 ceph-mon[88350]: from='mgr.14214 172.21.15.45:0/1227909437' entity='mgr.smithi045.nhtmwc' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-12-16T21:36:22.884 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Dec 16 21:36:22 smithi169 ceph-mon[88350]: from='mgr.14214 172.21.15.45:0/1227909437' entity='mgr.smithi045.nhtmwc' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-12-16T21:36:22.885 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Dec 16 21:36:22 smithi169 ceph-mon[88350]: from='mgr.14214 172.21.15.45:0/1227909437' entity='mgr.smithi045.nhtmwc' 2023-12-16T21:36:23.816 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Dec 16 21:36:23 smithi045 ceph-mon[80763]: purged_snaps scrub starts 2023-12-16T21:36:23.816 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Dec 16 21:36:23 smithi045 ceph-mon[80763]: purged_snaps scrub ok 2023-12-16T21:36:23.816 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Dec 16 21:36:23 smithi045 ceph-mon[80763]: osd.4 [v2:172.21.15.169:6808/3840494264,v1:172.21.15.169:6809/3840494264] boot 2023-12-16T21:36:23.816 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Dec 16 21:36:23 smithi045 ceph-mon[80763]: osdmap e32: 5 total, 5 up, 5 in 2023-12-16T21:36:23.816 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Dec 16 21:36:23 smithi045 ceph-mon[80763]: from='mgr.14214 172.21.15.45:0/1227909437' entity='mgr.smithi045.nhtmwc' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2023-12-16T21:36:23.816 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Dec 16 21:36:23 smithi045 ceph-mon[80763]: pgmap v77: 1 pgs: 1 active+clean; 577 KiB data, 1.4 GiB used, 446 GiB / 447 GiB avail 2023-12-16T21:36:23.816 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Dec 16 21:36:23 smithi045 ceph-mon[80763]: from='client.? 172.21.15.169:0/2346710584' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "363ba5fe-e249-4935-a370-6bc071a2f405"}]: dispatch 2023-12-16T21:36:23.817 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Dec 16 21:36:23 smithi045 ceph-mon[80763]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "363ba5fe-e249-4935-a370-6bc071a2f405"}]: dispatch 2023-12-16T21:36:23.817 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Dec 16 21:36:23 smithi045 ceph-mon[80763]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "363ba5fe-e249-4935-a370-6bc071a2f405"}]': finished 2023-12-16T21:36:23.817 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Dec 16 21:36:23 smithi045 ceph-mon[80763]: osdmap e33: 6 total, 5 up, 6 in 2023-12-16T21:36:23.817 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Dec 16 21:36:23 smithi045 ceph-mon[80763]: from='mgr.14214 172.21.15.45:0/1227909437' entity='mgr.smithi045.nhtmwc' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2023-12-16T21:36:23.883 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Dec 16 21:36:23 smithi169 ceph-mon[88350]: purged_snaps scrub starts 2023-12-16T21:36:23.884 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Dec 16 21:36:23 smithi169 ceph-mon[88350]: purged_snaps scrub ok 2023-12-16T21:36:23.884 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Dec 16 21:36:23 smithi169 ceph-mon[88350]: osd.4 [v2:172.21.15.169:6808/3840494264,v1:172.21.15.169:6809/3840494264] boot 2023-12-16T21:36:23.884 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Dec 16 21:36:23 smithi169 ceph-mon[88350]: osdmap e32: 5 total, 5 up, 5 in 2023-12-16T21:36:23.884 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Dec 16 21:36:23 smithi169 ceph-mon[88350]: from='mgr.14214 172.21.15.45:0/1227909437' entity='mgr.smithi045.nhtmwc' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2023-12-16T21:36:23.884 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Dec 16 21:36:23 smithi169 ceph-mon[88350]: pgmap v77: 1 pgs: 1 active+clean; 577 KiB data, 1.4 GiB used, 446 GiB / 447 GiB avail 2023-12-16T21:36:23.884 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Dec 16 21:36:23 smithi169 ceph-mon[88350]: from='client.? 172.21.15.169:0/2346710584' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "363ba5fe-e249-4935-a370-6bc071a2f405"}]: dispatch 2023-12-16T21:36:23.884 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Dec 16 21:36:23 smithi169 ceph-mon[88350]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "363ba5fe-e249-4935-a370-6bc071a2f405"}]: dispatch 2023-12-16T21:36:23.884 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Dec 16 21:36:23 smithi169 ceph-mon[88350]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "363ba5fe-e249-4935-a370-6bc071a2f405"}]': finished 2023-12-16T21:36:23.884 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Dec 16 21:36:23 smithi169 ceph-mon[88350]: osdmap e33: 6 total, 5 up, 6 in 2023-12-16T21:36:23.884 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Dec 16 21:36:23 smithi169 ceph-mon[88350]: from='mgr.14214 172.21.15.45:0/1227909437' entity='mgr.smithi045.nhtmwc' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2023-12-16T21:36:24.816 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Dec 16 21:36:24 smithi045 ceph-mon[80763]: from='client.? 172.21.15.169:0/2017646473' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2023-12-16T21:36:24.884 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Dec 16 21:36:24 smithi169 ceph-mon[88350]: from='client.? 172.21.15.169:0/2017646473' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2023-12-16T21:36:25.816 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Dec 16 21:36:25 smithi045 ceph-mon[80763]: pgmap v79: 1 pgs: 1 active+clean; 577 KiB data, 1.4 GiB used, 446 GiB / 447 GiB avail 2023-12-16T21:36:25.883 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Dec 16 21:36:25 smithi169 ceph-mon[88350]: pgmap v79: 1 pgs: 1 active+clean; 577 KiB data, 1.4 GiB used, 446 GiB / 447 GiB avail 2023-12-16T21:36:27.566 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Dec 16 21:36:27 smithi169 ceph-mon[88350]: pgmap v80: 1 pgs: 1 active+clean; 577 KiB data, 1.4 GiB used, 446 GiB / 447 GiB avail 2023-12-16T21:36:27.566 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Dec 16 21:36:27 smithi045 ceph-mon[80763]: pgmap v80: 1 pgs: 1 active+clean; 577 KiB data, 1.4 GiB used, 446 GiB / 447 GiB avail 2023-12-16T21:36:28.999 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Dec 16 21:36:28 smithi169 ceph-mon[88350]: from='mgr.14214 172.21.15.45:0/1227909437' entity='mgr.smithi045.nhtmwc' cmd=[{"prefix": "auth get", "entity": "osd.5"}]: dispatch 2023-12-16T21:36:28.999 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Dec 16 21:36:28 smithi169 ceph-mon[88350]: from='mgr.14214 172.21.15.45:0/1227909437' entity='mgr.smithi045.nhtmwc' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-12-16T21:36:28.999 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Dec 16 21:36:28 smithi169 ceph-mon[88350]: from='mgr.14214 172.21.15.45:0/1227909437' entity='mgr.smithi045.nhtmwc' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2023-12-16T21:36:29.066 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Dec 16 21:36:28 smithi045 ceph-mon[80763]: from='mgr.14214 172.21.15.45:0/1227909437' entity='mgr.smithi045.nhtmwc' cmd=[{"prefix": "auth get", "entity": "osd.5"}]: dispatch 2023-12-16T21:36:29.066 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Dec 16 21:36:28 smithi045 ceph-mon[80763]: from='mgr.14214 172.21.15.45:0/1227909437' entity='mgr.smithi045.nhtmwc' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-12-16T21:36:29.066 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Dec 16 21:36:28 smithi045 ceph-mon[80763]: from='mgr.14214 172.21.15.45:0/1227909437' entity='mgr.smithi045.nhtmwc' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2023-12-16T21:36:29.880 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Dec 16 21:36:29 smithi169 ceph-mon[88350]: pgmap v81: 1 pgs: 1 active+clean; 577 KiB data, 1.4 GiB used, 446 GiB / 447 GiB avail 2023-12-16T21:36:29.880 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Dec 16 21:36:29 smithi169 ceph-mon[88350]: Deploying daemon osd.5 on smithi169 2023-12-16T21:36:30.066 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Dec 16 21:36:29 smithi045 ceph-mon[80763]: pgmap v81: 1 pgs: 1 active+clean; 577 KiB data, 1.4 GiB used, 446 GiB / 447 GiB avail 2023-12-16T21:36:30.066 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Dec 16 21:36:29 smithi045 ceph-mon[80763]: Deploying daemon osd.5 on smithi169 2023-12-16T21:36:31.828 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Dec 16 21:36:31 smithi169 ceph-mon[88350]: pgmap v82: 1 pgs: 1 active+clean; 577 KiB data, 1.4 GiB used, 446 GiB / 447 GiB avail 2023-12-16T21:36:32.066 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Dec 16 21:36:31 smithi045 ceph-mon[80763]: pgmap v82: 1 pgs: 1 active+clean; 577 KiB data, 1.4 GiB used, 446 GiB / 447 GiB avail 2023-12-16T21:36:33.134 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Dec 16 21:36:32 smithi169 ceph-mon[88350]: from='mgr.14214 172.21.15.45:0/1227909437' entity='mgr.smithi045.nhtmwc' 2023-12-16T21:36:33.134 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Dec 16 21:36:32 smithi169 ceph-mon[88350]: from='mgr.14214 172.21.15.45:0/1227909437' entity='mgr.smithi045.nhtmwc' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-12-16T21:36:33.134 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Dec 16 21:36:32 smithi169 ceph-mon[88350]: from='mgr.14214 172.21.15.45:0/1227909437' entity='mgr.smithi045.nhtmwc' 2023-12-16T21:36:33.134 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Dec 16 21:36:32 smithi169 ceph-mon[88350]: pgmap v83: 1 pgs: 1 active+clean; 577 KiB data, 1.4 GiB used, 446 GiB / 447 GiB avail 2023-12-16T21:36:33.316 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Dec 16 21:36:32 smithi045 ceph-mon[80763]: from='mgr.14214 172.21.15.45:0/1227909437' entity='mgr.smithi045.nhtmwc' 2023-12-16T21:36:33.316 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Dec 16 21:36:32 smithi045 ceph-mon[80763]: from='mgr.14214 172.21.15.45:0/1227909437' entity='mgr.smithi045.nhtmwc' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-12-16T21:36:33.316 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Dec 16 21:36:32 smithi045 ceph-mon[80763]: from='mgr.14214 172.21.15.45:0/1227909437' entity='mgr.smithi045.nhtmwc' 2023-12-16T21:36:33.316 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Dec 16 21:36:32 smithi045 ceph-mon[80763]: pgmap v83: 1 pgs: 1 active+clean; 577 KiB data, 1.4 GiB used, 446 GiB / 447 GiB avail 2023-12-16T21:36:33.915 INFO:teuthology.orchestra.run.smithi169.stderr:2023-12-16T21:36:33.912+0000 7fade3fff700 1 -- 172.21.15.169:0/4211790209 <== mgr.14214 v2:172.21.15.45:6800/2012105536 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+37 (secure 0 0 0) 0x7faddc000bc0 con 0x7fade4068f70 2023-12-16T21:36:33.915 INFO:teuthology.orchestra.run.smithi169.stdout:Created osd(s) 5 on host 'smithi169' 2023-12-16T21:36:33.917 INFO:teuthology.orchestra.run.smithi169.stderr:2023-12-16T21:36:33.915+0000 7fade1ffb700 1 -- 172.21.15.169:0/4211790209 >> [v2:172.21.15.45:6800/2012105536,v1:172.21.15.45:6801/2012105536] conn(0x7fade4068f70 msgr2=0x7fade406b420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:36:33.917 INFO:teuthology.orchestra.run.smithi169.stderr:2023-12-16T21:36:33.915+0000 7fade1ffb700 1 --2- 172.21.15.169:0/4211790209 >> [v2:172.21.15.45:6800/2012105536,v1:172.21.15.45:6801/2012105536] conn(0x7fade4068f70 0x7fade406b420 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7fadf0004f90 tx=0x7fadf0009e30 comp rx=0 tx=0).stop 2023-12-16T21:36:33.917 INFO:teuthology.orchestra.run.smithi169.stderr:2023-12-16T21:36:33.915+0000 7fade1ffb700 1 -- 172.21.15.169:0/4211790209 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fadfc0faed0 msgr2=0x7fadfc10d600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:36:33.917 INFO:teuthology.orchestra.run.smithi169.stderr:2023-12-16T21:36:33.915+0000 7fade1ffb700 1 --2- 172.21.15.169:0/4211790209 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fadfc0faed0 0x7fadfc10d600 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7fadec00b790 tx=0x7fadec009220 comp rx=0 tx=0).stop 2023-12-16T21:36:33.917 INFO:teuthology.orchestra.run.smithi169.stderr:2023-12-16T21:36:33.915+0000 7fade1ffb700 1 -- 172.21.15.169:0/4211790209 shutdown_connections 2023-12-16T21:36:33.917 INFO:teuthology.orchestra.run.smithi169.stderr:2023-12-16T21:36:33.915+0000 7fade1ffb700 1 --2- 172.21.15.169:0/4211790209 >> [v2:172.21.15.45:6800/2012105536,v1:172.21.15.45:6801/2012105536] conn(0x7fade4068f70 0x7fade406b420 unknown :-1 s=CLOSED pgs=50 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:36:33.917 INFO:teuthology.orchestra.run.smithi169.stderr:2023-12-16T21:36:33.915+0000 7fade1ffb700 1 --2- 172.21.15.169:0/4211790209 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fadfc0faed0 0x7fadfc10d600 unknown :-1 s=CLOSED pgs=24 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:36:33.917 INFO:teuthology.orchestra.run.smithi169.stderr:2023-12-16T21:36:33.915+0000 7fade1ffb700 1 --2- 172.21.15.169:0/4211790209 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fadfc0fdf60 0x7fadfc10db40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:36:33.917 INFO:teuthology.orchestra.run.smithi169.stderr:2023-12-16T21:36:33.915+0000 7fade1ffb700 1 -- 172.21.15.169:0/4211790209 >> 172.21.15.169:0/4211790209 conn(0x7fadfc0f6950 msgr2=0x7fadfc0f8c20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-16T21:36:33.918 INFO:teuthology.orchestra.run.smithi169.stderr:2023-12-16T21:36:33.915+0000 7fade1ffb700 1 -- 172.21.15.169:0/4211790209 shutdown_connections 2023-12-16T21:36:33.918 INFO:teuthology.orchestra.run.smithi169.stderr:2023-12-16T21:36:33.915+0000 7fade1ffb700 1 -- 172.21.15.169:0/4211790209 wait complete. 2023-12-16T21:36:34.147 DEBUG:teuthology.orchestra.run.smithi169:osd.5> sudo journalctl -f -n 0 -u ceph-88ff363c-9c5a-11ee-95a5-87774f69a715@osd.5.service 2023-12-16T21:36:34.149 INFO:tasks.cephadm:Waiting for 6 OSDs to come up... 2023-12-16T21:36:34.150 DEBUG:teuthology.orchestra.run.smithi045:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 88ff363c-9c5a-11ee-95a5-87774f69a715 -- ceph osd stat -f json 2023-12-16T21:36:34.434 INFO:teuthology.orchestra.run.smithi045.stderr:Inferring config /var/lib/ceph/88ff363c-9c5a-11ee-95a5-87774f69a715/mon.smithi045/config 2023-12-16T21:36:35.159 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Dec 16 21:36:34 smithi045 ceph-mon[80763]: from='mgr.14214 172.21.15.45:0/1227909437' entity='mgr.smithi045.nhtmwc' 2023-12-16T21:36:35.160 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Dec 16 21:36:34 smithi045 ceph-mon[80763]: from='mgr.14214 172.21.15.45:0/1227909437' entity='mgr.smithi045.nhtmwc' 2023-12-16T21:36:35.160 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Dec 16 21:36:34 smithi045 ceph-mon[80763]: from='mgr.14214 172.21.15.45:0/1227909437' entity='mgr.smithi045.nhtmwc' 2023-12-16T21:36:35.160 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Dec 16 21:36:34 smithi045 ceph-mon[80763]: from='mgr.14214 172.21.15.45:0/1227909437' entity='mgr.smithi045.nhtmwc' 2023-12-16T21:36:35.160 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Dec 16 21:36:34 smithi045 ceph-mon[80763]: pgmap v84: 1 pgs: 1 active+clean; 577 KiB data, 1.4 GiB used, 446 GiB / 447 GiB avail 2023-12-16T21:36:35.310 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:36:35.307+0000 7fcb9a84a700 1 -- 172.21.15.45:0/2485262361 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fcb940fbaf0 msgr2=0x7fcb940fbf00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:36:35.310 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:36:35.307+0000 7fcb9a84a700 1 --2- 172.21.15.45:0/2485262361 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fcb940fbaf0 0x7fcb940fbf00 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7fcb84004940 tx=0x7fcb84009e30 comp rx=0 tx=0).stop 2023-12-16T21:36:35.310 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:36:35.307+0000 7fcb9a84a700 1 -- 172.21.15.45:0/2485262361 shutdown_connections 2023-12-16T21:36:35.310 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:36:35.307+0000 7fcb9a84a700 1 --2- 172.21.15.45:0/2485262361 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fcb940fcd00 0x7fcb940fd150 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:36:35.310 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:36:35.307+0000 7fcb9a84a700 1 --2- 172.21.15.45:0/2485262361 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fcb940fbaf0 0x7fcb940fbf00 unknown :-1 s=CLOSED pgs=180 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:36:35.310 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:36:35.307+0000 7fcb9a84a700 1 -- 172.21.15.45:0/2485262361 >> 172.21.15.45:0/2485262361 conn(0x7fcb940f71d0 msgr2=0x7fcb940f95e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-16T21:36:35.310 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:36:35.307+0000 7fcb9a84a700 1 -- 172.21.15.45:0/2485262361 shutdown_connections 2023-12-16T21:36:35.310 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:36:35.307+0000 7fcb9a84a700 1 -- 172.21.15.45:0/2485262361 wait complete. 2023-12-16T21:36:35.310 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:36:35.307+0000 7fcb9a84a700 1 Processor -- start 2023-12-16T21:36:35.312 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:36:35.308+0000 7fcb9a84a700 1 -- start start 2023-12-16T21:36:35.312 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:36:35.308+0000 7fcb9a84a700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fcb940fbaf0 0x7fcb9410d4d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:36:35.312 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:36:35.308+0000 7fcb9a84a700 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fcb940fcd00 0x7fcb9410da10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:36:35.312 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:36:35.308+0000 7fcb9a84a700 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcb9410e010 con 0x7fcb940fbaf0 2023-12-16T21:36:35.312 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:36:35.308+0000 7fcb9a84a700 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcb9410e8b0 con 0x7fcb940fcd00 2023-12-16T21:36:35.312 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:36:35.308+0000 7fcb93fff700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fcb940fbaf0 0x7fcb9410d4d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:36:35.312 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:36:35.308+0000 7fcb93fff700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fcb940fbaf0 0x7fcb9410d4d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:43810/0 (socket says 172.21.15.45:43810) 2023-12-16T21:36:35.312 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:36:35.308+0000 7fcb93fff700 1 -- 172.21.15.45:0/3626056535 learned_addr learned my addr 172.21.15.45:0/3626056535 (peer_addr_for_me v2:172.21.15.45:0/0) 2023-12-16T21:36:35.312 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:36:35.308+0000 7fcb937fe700 1 --2- 172.21.15.45:0/3626056535 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fcb940fcd00 0x7fcb9410da10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:36:35.312 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:36:35.308+0000 7fcb93fff700 1 -- 172.21.15.45:0/3626056535 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fcb940fcd00 msgr2=0x7fcb9410da10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:36:35.312 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:36:35.308+0000 7fcb93fff700 1 --2- 172.21.15.45:0/3626056535 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fcb940fcd00 0x7fcb9410da10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:36:35.313 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:36:35.308+0000 7fcb93fff700 1 -- 172.21.15.45:0/3626056535 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcb84005040 con 0x7fcb940fbaf0 2023-12-16T21:36:35.313 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:36:35.308+0000 7fcb93fff700 1 --2- 172.21.15.45:0/3626056535 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fcb940fbaf0 0x7fcb9410d4d0 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7fcb84000c00 tx=0x7fcb84009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:36:35.313 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:36:35.309+0000 7fcb90ff9700 1 -- 172.21.15.45:0/3626056535 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fcb84021070 con 0x7fcb940fbaf0 2023-12-16T21:36:35.313 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:36:35.309+0000 7fcb90ff9700 1 -- 172.21.15.45:0/3626056535 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fcb8401c400 con 0x7fcb940fbaf0 2023-12-16T21:36:35.313 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:36:35.309+0000 7fcb9a84a700 1 -- 172.21.15.45:0/3626056535 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcb9410eb30 con 0x7fcb940fbaf0 2023-12-16T21:36:35.313 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:36:35.309+0000 7fcb90ff9700 1 -- 172.21.15.45:0/3626056535 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fcb8400e070 con 0x7fcb940fbaf0 2023-12-16T21:36:35.313 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:36:35.309+0000 7fcb9a84a700 1 -- 172.21.15.45:0/3626056535 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcb9410f010 con 0x7fcb940fbaf0 2023-12-16T21:36:35.313 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:36:35.310+0000 7fcb9a84a700 1 -- 172.21.15.45:0/3626056535 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcb9404bb80 con 0x7fcb940fbaf0 2023-12-16T21:36:35.314 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:36:35.310+0000 7fcb90ff9700 1 -- 172.21.15.45:0/3626056535 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 19) v1 ==== 86447+0+0 (secure 0 0 0) 0x7fcb8401ca00 con 0x7fcb940fbaf0 2023-12-16T21:36:35.314 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:36:35.311+0000 7fcb90ff9700 1 --2- 172.21.15.45:0/3626056535 >> [v2:172.21.15.45:6800/2012105536,v1:172.21.15.45:6801/2012105536] conn(0x7fcb7c068f60 0x7fcb7c06b410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:36:35.314 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:36:35.311+0000 7fcb90ff9700 1 -- 172.21.15.45:0/3626056535 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 4317+0+0 (secure 0 0 0) 0x7fcb84087ad0 con 0x7fcb940fbaf0 2023-12-16T21:36:35.314 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:36:35.311+0000 7fcb937fe700 1 --2- 172.21.15.45:0/3626056535 >> [v2:172.21.15.45:6800/2012105536,v1:172.21.15.45:6801/2012105536] conn(0x7fcb7c068f60 0x7fcb7c06b410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:36:35.314 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:36:35.311+0000 7fcb937fe700 1 --2- 172.21.15.45:0/3626056535 >> [v2:172.21.15.45:6800/2012105536,v1:172.21.15.45:6801/2012105536] conn(0x7fcb7c068f60 0x7fcb7c06b410 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7fcb88009b80 tx=0x7fcb8800b040 comp rx=0 tx=0).ready entity=mgr.14214 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:36:35.318 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:36:35.315+0000 7fcb90ff9700 1 -- 172.21.15.45:0/3626056535 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+174570 (secure 0 0 0) 0x7fcb84057eb0 con 0x7fcb940fbaf0 2023-12-16T21:36:35.384 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Dec 16 21:36:34 smithi169 ceph-mon[88350]: from='mgr.14214 172.21.15.45:0/1227909437' entity='mgr.smithi045.nhtmwc' 2023-12-16T21:36:35.384 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Dec 16 21:36:34 smithi169 ceph-mon[88350]: from='mgr.14214 172.21.15.45:0/1227909437' entity='mgr.smithi045.nhtmwc' 2023-12-16T21:36:35.384 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Dec 16 21:36:34 smithi169 ceph-mon[88350]: from='mgr.14214 172.21.15.45:0/1227909437' entity='mgr.smithi045.nhtmwc' 2023-12-16T21:36:35.384 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Dec 16 21:36:34 smithi169 ceph-mon[88350]: from='mgr.14214 172.21.15.45:0/1227909437' entity='mgr.smithi045.nhtmwc' 2023-12-16T21:36:35.384 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Dec 16 21:36:34 smithi169 ceph-mon[88350]: pgmap v84: 1 pgs: 1 active+clean; 577 KiB data, 1.4 GiB used, 446 GiB / 447 GiB avail 2023-12-16T21:36:35.515 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:36:35.513+0000 7fcb9a84a700 1 -- 172.21.15.45:0/3626056535 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "osd stat", "format": "json"} v 0) v1 -- 0x7fcb9410f390 con 0x7fcb940fbaf0 2023-12-16T21:36:35.516 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:36:35.513+0000 7fcb90ff9700 1 -- 172.21.15.45:0/3626056535 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mon_command_ack([{"prefix": "osd stat", "format": "json"}]=0 v33) v1 ==== 74+0+130 (secure 0 0 0) 0x7fcb84057a40 con 0x7fcb940fbaf0 2023-12-16T21:36:35.516 INFO:teuthology.orchestra.run.smithi045.stdout: 2023-12-16T21:36:35.518 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:36:35.515+0000 7fcb9a84a700 1 -- 172.21.15.45:0/3626056535 >> [v2:172.21.15.45:6800/2012105536,v1:172.21.15.45:6801/2012105536] conn(0x7fcb7c068f60 msgr2=0x7fcb7c06b410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:36:35.518 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:36:35.515+0000 7fcb9a84a700 1 --2- 172.21.15.45:0/3626056535 >> [v2:172.21.15.45:6800/2012105536,v1:172.21.15.45:6801/2012105536] conn(0x7fcb7c068f60 0x7fcb7c06b410 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7fcb88009b80 tx=0x7fcb8800b040 comp rx=0 tx=0).stop 2023-12-16T21:36:35.518 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:36:35.515+0000 7fcb9a84a700 1 -- 172.21.15.45:0/3626056535 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fcb940fbaf0 msgr2=0x7fcb9410d4d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:36:35.518 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:36:35.515+0000 7fcb9a84a700 1 --2- 172.21.15.45:0/3626056535 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fcb940fbaf0 0x7fcb9410d4d0 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7fcb84000c00 tx=0x7fcb84009e30 comp rx=0 tx=0).stop 2023-12-16T21:36:35.518 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:36:35.515+0000 7fcb9a84a700 1 -- 172.21.15.45:0/3626056535 shutdown_connections 2023-12-16T21:36:35.518 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:36:35.515+0000 7fcb9a84a700 1 --2- 172.21.15.45:0/3626056535 >> [v2:172.21.15.45:6800/2012105536,v1:172.21.15.45:6801/2012105536] conn(0x7fcb7c068f60 0x7fcb7c06b410 unknown :-1 s=CLOSED pgs=55 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:36:35.519 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:36:35.515+0000 7fcb9a84a700 1 --2- 172.21.15.45:0/3626056535 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fcb940fbaf0 0x7fcb9410d4d0 unknown :-1 s=CLOSED pgs=181 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:36:35.519 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:36:35.515+0000 7fcb9a84a700 1 --2- 172.21.15.45:0/3626056535 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fcb940fcd00 0x7fcb9410da10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:36:35.519 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:36:35.515+0000 7fcb9a84a700 1 -- 172.21.15.45:0/3626056535 >> 172.21.15.45:0/3626056535 conn(0x7fcb940f71d0 msgr2=0x7fcb94065c00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-16T21:36:35.519 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:36:35.516+0000 7fcb9a84a700 1 -- 172.21.15.45:0/3626056535 shutdown_connections 2023-12-16T21:36:35.519 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:36:35.516+0000 7fcb9a84a700 1 -- 172.21.15.45:0/3626056535 wait complete. 2023-12-16T21:36:35.791 INFO:teuthology.orchestra.run.smithi045.stdout:{"epoch":33,"num_osds":6,"num_up_osds":5,"osd_up_since":1702762582,"num_in_osds":6,"osd_in_since":1702762583,"num_remapped_pgs":0} 2023-12-16T21:36:36.133 INFO:journalctl@ceph.osd.5.smithi169.stdout:Dec 16 21:36:35 smithi169 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-5[107009]: 2023-12-16T21:36:35.882+0000 7fe662cd4540 -1 osd.5 0 log_to_monitors true 2023-12-16T21:36:36.334 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Dec 16 21:36:36 smithi045 ceph-mon[80763]: from='client.? 172.21.15.45:0/3626056535' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2023-12-16T21:36:36.335 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Dec 16 21:36:36 smithi045 ceph-mon[80763]: from='osd.5 [v2:172.21.15.169:6816/3826842549,v1:172.21.15.169:6817/3826842549]' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2023-12-16T21:36:36.335 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Dec 16 21:36:36 smithi045 ceph-mon[80763]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2023-12-16T21:36:36.335 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Dec 16 21:36:36 smithi045 ceph-mon[80763]: from='mgr.14214 172.21.15.45:0/1227909437' entity='mgr.smithi045.nhtmwc' 2023-12-16T21:36:36.335 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Dec 16 21:36:36 smithi045 ceph-mon[80763]: from='mgr.14214 172.21.15.45:0/1227909437' entity='mgr.smithi045.nhtmwc' 2023-12-16T21:36:36.335 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Dec 16 21:36:36 smithi045 ceph-mon[80763]: from='mgr.14214 172.21.15.45:0/1227909437' entity='mgr.smithi045.nhtmwc' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2023-12-16T21:36:36.335 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Dec 16 21:36:36 smithi045 ceph-mon[80763]: from='mgr.14214 172.21.15.45:0/1227909437' entity='mgr.smithi045.nhtmwc' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2023-12-16T21:36:36.335 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Dec 16 21:36:36 smithi045 ceph-mon[80763]: from='mgr.14214 172.21.15.45:0/1227909437' entity='mgr.smithi045.nhtmwc' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2023-12-16T21:36:36.335 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Dec 16 21:36:36 smithi045 ceph-mon[80763]: from='mgr.14214 172.21.15.45:0/1227909437' entity='mgr.smithi045.nhtmwc' 2023-12-16T21:36:36.335 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Dec 16 21:36:36 smithi045 ceph-mon[80763]: from='mgr.14214 172.21.15.45:0/1227909437' entity='mgr.smithi045.nhtmwc' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-12-16T21:36:36.335 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Dec 16 21:36:36 smithi045 ceph-mon[80763]: from='mgr.14214 172.21.15.45:0/1227909437' entity='mgr.smithi045.nhtmwc' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-12-16T21:36:36.335 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Dec 16 21:36:36 smithi045 ceph-mon[80763]: from='mgr.14214 172.21.15.45:0/1227909437' entity='mgr.smithi045.nhtmwc' 2023-12-16T21:36:36.335 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Dec 16 21:36:36 smithi045 ceph-mon[80763]: from='mgr.14214 172.21.15.45:0/1227909437' entity='mgr.smithi045.nhtmwc' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-12-16T21:36:36.633 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Dec 16 21:36:36 smithi169 ceph-mon[88350]: from='client.? 172.21.15.45:0/3626056535' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2023-12-16T21:36:36.634 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Dec 16 21:36:36 smithi169 ceph-mon[88350]: from='osd.5 [v2:172.21.15.169:6816/3826842549,v1:172.21.15.169:6817/3826842549]' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2023-12-16T21:36:36.634 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Dec 16 21:36:36 smithi169 ceph-mon[88350]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2023-12-16T21:36:36.634 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Dec 16 21:36:36 smithi169 ceph-mon[88350]: from='mgr.14214 172.21.15.45:0/1227909437' entity='mgr.smithi045.nhtmwc' 2023-12-16T21:36:36.634 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Dec 16 21:36:36 smithi169 ceph-mon[88350]: from='mgr.14214 172.21.15.45:0/1227909437' entity='mgr.smithi045.nhtmwc' 2023-12-16T21:36:36.634 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Dec 16 21:36:36 smithi169 ceph-mon[88350]: from='mgr.14214 172.21.15.45:0/1227909437' entity='mgr.smithi045.nhtmwc' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2023-12-16T21:36:36.634 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Dec 16 21:36:36 smithi169 ceph-mon[88350]: from='mgr.14214 172.21.15.45:0/1227909437' entity='mgr.smithi045.nhtmwc' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2023-12-16T21:36:36.634 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Dec 16 21:36:36 smithi169 ceph-mon[88350]: from='mgr.14214 172.21.15.45:0/1227909437' entity='mgr.smithi045.nhtmwc' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2023-12-16T21:36:36.634 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Dec 16 21:36:36 smithi169 ceph-mon[88350]: from='mgr.14214 172.21.15.45:0/1227909437' entity='mgr.smithi045.nhtmwc' 2023-12-16T21:36:36.634 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Dec 16 21:36:36 smithi169 ceph-mon[88350]: from='mgr.14214 172.21.15.45:0/1227909437' entity='mgr.smithi045.nhtmwc' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-12-16T21:36:36.634 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Dec 16 21:36:36 smithi169 ceph-mon[88350]: from='mgr.14214 172.21.15.45:0/1227909437' entity='mgr.smithi045.nhtmwc' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-12-16T21:36:36.634 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Dec 16 21:36:36 smithi169 ceph-mon[88350]: from='mgr.14214 172.21.15.45:0/1227909437' entity='mgr.smithi045.nhtmwc' 2023-12-16T21:36:36.634 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Dec 16 21:36:36 smithi169 ceph-mon[88350]: from='mgr.14214 172.21.15.45:0/1227909437' entity='mgr.smithi045.nhtmwc' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-12-16T21:36:36.793 DEBUG:teuthology.orchestra.run.smithi045:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 88ff363c-9c5a-11ee-95a5-87774f69a715 -- ceph osd stat -f json 2023-12-16T21:36:37.065 INFO:teuthology.orchestra.run.smithi045.stderr:Inferring config /var/lib/ceph/88ff363c-9c5a-11ee-95a5-87774f69a715/mon.smithi045/config 2023-12-16T21:36:37.238 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Dec 16 21:36:37 smithi045 ceph-mon[80763]: Detected new or changed devices on smithi169 2023-12-16T21:36:37.238 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Dec 16 21:36:37 smithi045 ceph-mon[80763]: Adjusting osd_memory_target on smithi169 to 5312M 2023-12-16T21:36:37.238 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Dec 16 21:36:37 smithi045 ceph-mon[80763]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]': finished 2023-12-16T21:36:37.238 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Dec 16 21:36:37 smithi045 ceph-mon[80763]: osdmap e34: 6 total, 5 up, 6 in 2023-12-16T21:36:37.238 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Dec 16 21:36:37 smithi045 ceph-mon[80763]: from='mgr.14214 172.21.15.45:0/1227909437' entity='mgr.smithi045.nhtmwc' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2023-12-16T21:36:37.238 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Dec 16 21:36:37 smithi045 ceph-mon[80763]: from='osd.5 [v2:172.21.15.169:6816/3826842549,v1:172.21.15.169:6817/3826842549]' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi169", "root=default"]}]: dispatch 2023-12-16T21:36:37.238 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Dec 16 21:36:37 smithi045 ceph-mon[80763]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi169", "root=default"]}]: dispatch 2023-12-16T21:36:37.238 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Dec 16 21:36:37 smithi045 ceph-mon[80763]: pgmap v86: 1 pgs: 1 active+clean; 577 KiB data, 1.4 GiB used, 446 GiB / 447 GiB avail 2023-12-16T21:36:37.238 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Dec 16 21:36:37 smithi045 ceph-mon[80763]: from='mgr.14214 172.21.15.45:0/1227909437' entity='mgr.smithi045.nhtmwc' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-12-16T21:36:37.238 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Dec 16 21:36:37 smithi045 ceph-mon[80763]: from='mgr.14214 172.21.15.45:0/1227909437' entity='mgr.smithi045.nhtmwc' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-12-16T21:36:37.238 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Dec 16 21:36:37 smithi045 ceph-mon[80763]: from='mgr.14214 172.21.15.45:0/1227909437' entity='mgr.smithi045.nhtmwc' 2023-12-16T21:36:37.633 INFO:journalctl@ceph.osd.5.smithi169.stdout:Dec 16 21:36:37 smithi169 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-5[107009]: 2023-12-16T21:36:37.236+0000 7fe6594b9700 -1 osd.5 0 waiting for initial osdmap 2023-12-16T21:36:37.633 INFO:journalctl@ceph.osd.5.smithi169.stdout:Dec 16 21:36:37 smithi169 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-5[107009]: 2023-12-16T21:36:37.245+0000 7fe6542b4700 -1 osd.5 35 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2023-12-16T21:36:37.634 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Dec 16 21:36:37 smithi169 ceph-mon[88350]: Detected new or changed devices on smithi169 2023-12-16T21:36:37.634 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Dec 16 21:36:37 smithi169 ceph-mon[88350]: Adjusting osd_memory_target on smithi169 to 5312M 2023-12-16T21:36:37.634 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Dec 16 21:36:37 smithi169 ceph-mon[88350]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]': finished 2023-12-16T21:36:37.634 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Dec 16 21:36:37 smithi169 ceph-mon[88350]: osdmap e34: 6 total, 5 up, 6 in 2023-12-16T21:36:37.634 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Dec 16 21:36:37 smithi169 ceph-mon[88350]: from='mgr.14214 172.21.15.45:0/1227909437' entity='mgr.smithi045.nhtmwc' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2023-12-16T21:36:37.634 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Dec 16 21:36:37 smithi169 ceph-mon[88350]: from='osd.5 [v2:172.21.15.169:6816/3826842549,v1:172.21.15.169:6817/3826842549]' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi169", "root=default"]}]: dispatch 2023-12-16T21:36:37.634 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Dec 16 21:36:37 smithi169 ceph-mon[88350]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi169", "root=default"]}]: dispatch 2023-12-16T21:36:37.634 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Dec 16 21:36:37 smithi169 ceph-mon[88350]: pgmap v86: 1 pgs: 1 active+clean; 577 KiB data, 1.4 GiB used, 446 GiB / 447 GiB avail 2023-12-16T21:36:37.634 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Dec 16 21:36:37 smithi169 ceph-mon[88350]: from='mgr.14214 172.21.15.45:0/1227909437' entity='mgr.smithi045.nhtmwc' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-12-16T21:36:37.634 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Dec 16 21:36:37 smithi169 ceph-mon[88350]: from='mgr.14214 172.21.15.45:0/1227909437' entity='mgr.smithi045.nhtmwc' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-12-16T21:36:37.634 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Dec 16 21:36:37 smithi169 ceph-mon[88350]: from='mgr.14214 172.21.15.45:0/1227909437' entity='mgr.smithi045.nhtmwc' 2023-12-16T21:36:37.934 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:36:37.930+0000 7fab909c8700 1 -- 172.21.15.45:0/1221972058 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fab8c0fbaf0 msgr2=0x7fab8c0fbf00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:36:37.934 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:36:37.930+0000 7fab909c8700 1 --2- 172.21.15.45:0/1221972058 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fab8c0fbaf0 0x7fab8c0fbf00 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7fab7c004940 tx=0x7fab7c009e30 comp rx=0 tx=0).stop 2023-12-16T21:36:37.934 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:36:37.930+0000 7fab909c8700 1 -- 172.21.15.45:0/1221972058 shutdown_connections 2023-12-16T21:36:37.934 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:36:37.930+0000 7fab909c8700 1 --2- 172.21.15.45:0/1221972058 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fab8c0fcd00 0x7fab8c0fd150 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:36:37.935 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:36:37.930+0000 7fab909c8700 1 --2- 172.21.15.45:0/1221972058 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fab8c0fbaf0 0x7fab8c0fbf00 unknown :-1 s=CLOSED pgs=182 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:36:37.935 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:36:37.930+0000 7fab909c8700 1 -- 172.21.15.45:0/1221972058 >> 172.21.15.45:0/1221972058 conn(0x7fab8c0f71d0 msgr2=0x7fab8c0f95e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-16T21:36:37.935 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:36:37.930+0000 7fab909c8700 1 -- 172.21.15.45:0/1221972058 shutdown_connections 2023-12-16T21:36:37.935 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:36:37.930+0000 7fab909c8700 1 -- 172.21.15.45:0/1221972058 wait complete. 2023-12-16T21:36:37.935 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:36:37.930+0000 7fab909c8700 1 Processor -- start 2023-12-16T21:36:37.935 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:36:37.930+0000 7fab909c8700 1 -- start start 2023-12-16T21:36:37.935 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:36:37.931+0000 7fab909c8700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fab8c0fbaf0 0x7fab8c10c470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:36:37.935 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:36:37.931+0000 7fab909c8700 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fab8c0fcd00 0x7fab8c10c9b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:36:37.935 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:36:37.931+0000 7fab909c8700 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fab8c10cfb0 con 0x7fab8c0fbaf0 2023-12-16T21:36:37.935 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:36:37.931+0000 7fab909c8700 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fab8c10d850 con 0x7fab8c0fcd00 2023-12-16T21:36:37.935 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:36:37.931+0000 7fab8a59c700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fab8c0fbaf0 0x7fab8c10c470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:36:37.935 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:36:37.931+0000 7fab8a59c700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fab8c0fbaf0 0x7fab8c10c470 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:43832/0 (socket says 172.21.15.45:43832) 2023-12-16T21:36:37.935 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:36:37.931+0000 7fab8a59c700 1 -- 172.21.15.45:0/42827965 learned_addr learned my addr 172.21.15.45:0/42827965 (peer_addr_for_me v2:172.21.15.45:0/0) 2023-12-16T21:36:37.935 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:36:37.931+0000 7fab89d9b700 1 --2- 172.21.15.45:0/42827965 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fab8c0fcd00 0x7fab8c10c9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:36:37.936 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:36:37.931+0000 7fab8a59c700 1 -- 172.21.15.45:0/42827965 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fab8c0fcd00 msgr2=0x7fab8c10c9b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:36:37.936 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:36:37.931+0000 7fab8a59c700 1 --2- 172.21.15.45:0/42827965 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fab8c0fcd00 0x7fab8c10c9b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:36:37.936 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:36:37.931+0000 7fab8a59c700 1 -- 172.21.15.45:0/42827965 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fab7c005040 con 0x7fab8c0fbaf0 2023-12-16T21:36:37.936 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:36:37.931+0000 7fab8a59c700 1 --2- 172.21.15.45:0/42827965 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fab8c0fbaf0 0x7fab8c10c470 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7fab7c000c00 tx=0x7fab7c009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:36:37.936 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:36:37.931+0000 7fab7affd700 1 -- 172.21.15.45:0/42827965 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fab7c00e070 con 0x7fab8c0fbaf0 2023-12-16T21:36:37.936 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:36:37.931+0000 7fab7affd700 1 -- 172.21.15.45:0/42827965 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fab7c013c40 con 0x7fab8c0fbaf0 2023-12-16T21:36:37.936 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:36:37.931+0000 7fab909c8700 1 -- 172.21.15.45:0/42827965 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fab8c10dad0 con 0x7fab8c0fbaf0 2023-12-16T21:36:37.936 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:36:37.931+0000 7fab7affd700 1 -- 172.21.15.45:0/42827965 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fab7c023630 con 0x7fab8c0fbaf0 2023-12-16T21:36:37.936 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:36:37.931+0000 7fab909c8700 1 -- 172.21.15.45:0/42827965 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fab8c10dfb0 con 0x7fab8c0fbaf0 2023-12-16T21:36:37.937 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:36:37.933+0000 7fab909c8700 1 -- 172.21.15.45:0/42827965 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fab8c04bb80 con 0x7fab8c0fbaf0 2023-12-16T21:36:37.937 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:36:37.933+0000 7fab7affd700 1 -- 172.21.15.45:0/42827965 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 19) v1 ==== 86447+0+0 (secure 0 0 0) 0x7fab7c01a480 con 0x7fab8c0fbaf0 2023-12-16T21:36:37.937 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:36:37.933+0000 7fab7affd700 1 --2- 172.21.15.45:0/42827965 >> [v2:172.21.15.45:6800/2012105536,v1:172.21.15.45:6801/2012105536] conn(0x7fab74068f60 0x7fab7406b410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:36:37.937 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:36:37.933+0000 7fab7affd700 1 -- 172.21.15.45:0/42827965 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(35..35 src has 1..35) v4 ==== 4354+0+0 (secure 0 0 0) 0x7fab7c08a1b0 con 0x7fab8c0fbaf0 2023-12-16T21:36:37.937 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:36:37.933+0000 7fab89d9b700 1 --2- 172.21.15.45:0/42827965 >> [v2:172.21.15.45:6800/2012105536,v1:172.21.15.45:6801/2012105536] conn(0x7fab74068f60 0x7fab7406b410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:36:37.937 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:36:37.934+0000 7fab89d9b700 1 --2- 172.21.15.45:0/42827965 >> [v2:172.21.15.45:6800/2012105536,v1:172.21.15.45:6801/2012105536] conn(0x7fab74068f60 0x7fab7406b410 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7fab80009b80 tx=0x7fab8000b040 comp rx=0 tx=0).ready entity=mgr.14214 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:36:37.940 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:36:37.938+0000 7fab7affd700 1 -- 172.21.15.45:0/42827965 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+174570 (secure 0 0 0) 0x7fab7c05a560 con 0x7fab8c0fbaf0 2023-12-16T21:36:38.137 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:36:38.134+0000 7fab909c8700 1 -- 172.21.15.45:0/42827965 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "osd stat", "format": "json"} v 0) v1 -- 0x7fab8c10e2e0 con 0x7fab8c0fbaf0 2023-12-16T21:36:38.137 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:36:38.135+0000 7fab7affd700 1 -- 172.21.15.45:0/42827965 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mon_command_ack([{"prefix": "osd stat", "format": "json"}]=0 v35) v1 ==== 74+0+130 (secure 0 0 0) 0x7fab7c05a0f0 con 0x7fab8c0fbaf0 2023-12-16T21:36:38.137 INFO:teuthology.orchestra.run.smithi045.stdout: 2023-12-16T21:36:38.139 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:36:38.137+0000 7fab909c8700 1 -- 172.21.15.45:0/42827965 >> [v2:172.21.15.45:6800/2012105536,v1:172.21.15.45:6801/2012105536] conn(0x7fab74068f60 msgr2=0x7fab7406b410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:36:38.139 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:36:38.137+0000 7fab909c8700 1 --2- 172.21.15.45:0/42827965 >> [v2:172.21.15.45:6800/2012105536,v1:172.21.15.45:6801/2012105536] conn(0x7fab74068f60 0x7fab7406b410 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7fab80009b80 tx=0x7fab8000b040 comp rx=0 tx=0).stop 2023-12-16T21:36:38.139 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:36:38.137+0000 7fab909c8700 1 -- 172.21.15.45:0/42827965 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fab8c0fbaf0 msgr2=0x7fab8c10c470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:36:38.139 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:36:38.137+0000 7fab909c8700 1 --2- 172.21.15.45:0/42827965 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fab8c0fbaf0 0x7fab8c10c470 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7fab7c000c00 tx=0x7fab7c009e30 comp rx=0 tx=0).stop 2023-12-16T21:36:38.139 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:36:38.137+0000 7fab909c8700 1 -- 172.21.15.45:0/42827965 shutdown_connections 2023-12-16T21:36:38.140 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:36:38.137+0000 7fab909c8700 1 --2- 172.21.15.45:0/42827965 >> [v2:172.21.15.45:6800/2012105536,v1:172.21.15.45:6801/2012105536] conn(0x7fab74068f60 0x7fab7406b410 unknown :-1 s=CLOSED pgs=57 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:36:38.140 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:36:38.137+0000 7fab909c8700 1 --2- 172.21.15.45:0/42827965 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fab8c0fbaf0 0x7fab8c10c470 unknown :-1 s=CLOSED pgs=183 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:36:38.140 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:36:38.137+0000 7fab909c8700 1 --2- 172.21.15.45:0/42827965 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fab8c0fcd00 0x7fab8c10c9b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:36:38.140 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:36:38.137+0000 7fab909c8700 1 -- 172.21.15.45:0/42827965 >> 172.21.15.45:0/42827965 conn(0x7fab8c0f71d0 msgr2=0x7fab8c0fff30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-16T21:36:38.140 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:36:38.137+0000 7fab909c8700 1 -- 172.21.15.45:0/42827965 shutdown_connections 2023-12-16T21:36:38.140 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:36:38.137+0000 7fab909c8700 1 -- 172.21.15.45:0/42827965 wait complete. 2023-12-16T21:36:38.418 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Dec 16 21:36:38 smithi045 ceph-mon[80763]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi169", "root=default"]}]': finished 2023-12-16T21:36:38.418 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Dec 16 21:36:38 smithi045 ceph-mon[80763]: osdmap e35: 6 total, 5 up, 6 in 2023-12-16T21:36:38.418 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Dec 16 21:36:38 smithi045 ceph-mon[80763]: from='mgr.14214 172.21.15.45:0/1227909437' entity='mgr.smithi045.nhtmwc' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2023-12-16T21:36:38.418 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Dec 16 21:36:38 smithi045 ceph-mon[80763]: from='mgr.14214 172.21.15.45:0/1227909437' entity='mgr.smithi045.nhtmwc' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2023-12-16T21:36:38.418 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Dec 16 21:36:38 smithi045 ceph-mon[80763]: from='client.? 172.21.15.45:0/42827965' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2023-12-16T21:36:38.558 INFO:teuthology.orchestra.run.smithi045.stdout:{"epoch":35,"num_osds":6,"num_up_osds":5,"osd_up_since":1702762582,"num_in_osds":6,"osd_in_since":1702762583,"num_remapped_pgs":0} 2023-12-16T21:36:38.633 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Dec 16 21:36:38 smithi169 ceph-mon[88350]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi169", "root=default"]}]': finished 2023-12-16T21:36:38.634 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Dec 16 21:36:38 smithi169 ceph-mon[88350]: osdmap e35: 6 total, 5 up, 6 in 2023-12-16T21:36:38.634 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Dec 16 21:36:38 smithi169 ceph-mon[88350]: from='mgr.14214 172.21.15.45:0/1227909437' entity='mgr.smithi045.nhtmwc' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2023-12-16T21:36:38.634 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Dec 16 21:36:38 smithi169 ceph-mon[88350]: from='mgr.14214 172.21.15.45:0/1227909437' entity='mgr.smithi045.nhtmwc' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2023-12-16T21:36:38.634 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Dec 16 21:36:38 smithi169 ceph-mon[88350]: from='client.? 172.21.15.45:0/42827965' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2023-12-16T21:36:39.559 DEBUG:teuthology.orchestra.run.smithi045:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 88ff363c-9c5a-11ee-95a5-87774f69a715 -- ceph osd stat -f json 2023-12-16T21:36:39.566 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Dec 16 21:36:39 smithi045 ceph-mon[80763]: purged_snaps scrub starts 2023-12-16T21:36:39.566 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Dec 16 21:36:39 smithi045 ceph-mon[80763]: purged_snaps scrub ok 2023-12-16T21:36:39.566 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Dec 16 21:36:39 smithi045 ceph-mon[80763]: osd.5 [v2:172.21.15.169:6816/3826842549,v1:172.21.15.169:6817/3826842549] boot 2023-12-16T21:36:39.566 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Dec 16 21:36:39 smithi045 ceph-mon[80763]: osdmap e36: 6 total, 6 up, 6 in 2023-12-16T21:36:39.566 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Dec 16 21:36:39 smithi045 ceph-mon[80763]: from='mgr.14214 172.21.15.45:0/1227909437' entity='mgr.smithi045.nhtmwc' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2023-12-16T21:36:39.566 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Dec 16 21:36:39 smithi045 ceph-mon[80763]: pgmap v89: 1 pgs: 1 active+clean; 577 KiB data, 1.4 GiB used, 446 GiB / 447 GiB avail 2023-12-16T21:36:39.634 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Dec 16 21:36:39 smithi169 ceph-mon[88350]: purged_snaps scrub starts 2023-12-16T21:36:39.634 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Dec 16 21:36:39 smithi169 ceph-mon[88350]: purged_snaps scrub ok 2023-12-16T21:36:39.634 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Dec 16 21:36:39 smithi169 ceph-mon[88350]: osd.5 [v2:172.21.15.169:6816/3826842549,v1:172.21.15.169:6817/3826842549] boot 2023-12-16T21:36:39.634 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Dec 16 21:36:39 smithi169 ceph-mon[88350]: osdmap e36: 6 total, 6 up, 6 in 2023-12-16T21:36:39.634 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Dec 16 21:36:39 smithi169 ceph-mon[88350]: from='mgr.14214 172.21.15.45:0/1227909437' entity='mgr.smithi045.nhtmwc' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2023-12-16T21:36:39.635 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Dec 16 21:36:39 smithi169 ceph-mon[88350]: pgmap v89: 1 pgs: 1 active+clean; 577 KiB data, 1.4 GiB used, 446 GiB / 447 GiB avail 2023-12-16T21:36:39.838 INFO:teuthology.orchestra.run.smithi045.stderr:Inferring config /var/lib/ceph/88ff363c-9c5a-11ee-95a5-87774f69a715/mon.smithi045/config 2023-12-16T21:36:40.507 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Dec 16 21:36:40 smithi045 ceph-mon[80763]: osdmap e37: 6 total, 6 up, 6 in 2023-12-16T21:36:40.633 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Dec 16 21:36:40 smithi169 ceph-mon[88350]: osdmap e37: 6 total, 6 up, 6 in 2023-12-16T21:36:40.708 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:36:40.705+0000 7f9ae1a5a700 1 -- 172.21.15.45:0/3974810672 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f9adc0fec60 msgr2=0x7f9adc1010e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:36:40.708 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:36:40.705+0000 7f9ae1a5a700 1 --2- 172.21.15.45:0/3974810672 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f9adc0fec60 0x7f9adc1010e0 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7f9ad0004940 tx=0x7f9ad0009e30 comp rx=0 tx=0).stop 2023-12-16T21:36:40.708 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:36:40.705+0000 7f9ae1a5a700 1 -- 172.21.15.45:0/3974810672 shutdown_connections 2023-12-16T21:36:40.708 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:36:40.705+0000 7f9ae1a5a700 1 --2- 172.21.15.45:0/3974810672 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f9adc0fec60 0x7f9adc1010e0 unknown :-1 s=CLOSED pgs=184 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:36:40.708 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:36:40.705+0000 7f9ae1a5a700 1 --2- 172.21.15.45:0/3974810672 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f9adc0fc2b0 0x7f9adc0fe690 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:36:40.708 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:36:40.705+0000 7f9ae1a5a700 1 -- 172.21.15.45:0/3974810672 >> 172.21.15.45:0/3974810672 conn(0x7f9adc0f64f0 msgr2=0x7f9adc0f8920 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-16T21:36:40.708 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:36:40.705+0000 7f9ae1a5a700 1 -- 172.21.15.45:0/3974810672 shutdown_connections 2023-12-16T21:36:40.708 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:36:40.705+0000 7f9ae1a5a700 1 -- 172.21.15.45:0/3974810672 wait complete. 2023-12-16T21:36:40.710 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:36:40.706+0000 7f9ae1a5a700 1 Processor -- start 2023-12-16T21:36:40.710 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:36:40.706+0000 7f9ae1a5a700 1 -- start start 2023-12-16T21:36:40.710 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:36:40.706+0000 7f9ae1a5a700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f9adc0fc2b0 0x7f9adc110210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:36:40.710 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:36:40.706+0000 7f9ae1a5a700 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f9adc0fec60 0x7f9adc112760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:36:40.710 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:36:40.706+0000 7f9ae1a5a700 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9adc0fbed0 con 0x7f9adc0fc2b0 2023-12-16T21:36:40.710 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:36:40.706+0000 7f9ae1a5a700 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9adc0fc010 con 0x7f9adc0fec60 2023-12-16T21:36:40.710 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:36:40.706+0000 7f9adaffd700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f9adc0fc2b0 0x7f9adc110210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:36:40.711 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:36:40.706+0000 7f9adaffd700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f9adc0fc2b0 0x7f9adc110210 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:43852/0 (socket says 172.21.15.45:43852) 2023-12-16T21:36:40.711 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:36:40.706+0000 7f9adaffd700 1 -- 172.21.15.45:0/2558897036 learned_addr learned my addr 172.21.15.45:0/2558897036 (peer_addr_for_me v2:172.21.15.45:0/0) 2023-12-16T21:36:40.711 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:36:40.706+0000 7f9ada7fc700 1 --2- 172.21.15.45:0/2558897036 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f9adc0fec60 0x7f9adc112760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:36:40.711 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:36:40.707+0000 7f9adaffd700 1 -- 172.21.15.45:0/2558897036 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f9adc0fec60 msgr2=0x7f9adc112760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:36:40.711 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:36:40.707+0000 7f9adaffd700 1 --2- 172.21.15.45:0/2558897036 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f9adc0fec60 0x7f9adc112760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:36:40.711 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:36:40.707+0000 7f9adaffd700 1 -- 172.21.15.45:0/2558897036 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9ad0005040 con 0x7f9adc0fc2b0 2023-12-16T21:36:40.711 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:36:40.707+0000 7f9adaffd700 1 --2- 172.21.15.45:0/2558897036 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f9adc0fc2b0 0x7f9adc110210 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7f9acc00c8c0 tx=0x7f9acc0044c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:36:40.711 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:36:40.707+0000 7f9ac37fe700 1 -- 172.21.15.45:0/2558897036 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f9acc00a760 con 0x7f9adc0fc2b0 2023-12-16T21:36:40.711 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:36:40.707+0000 7f9ae1a5a700 1 -- 172.21.15.45:0/2558897036 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9adc112dc0 con 0x7f9adc0fc2b0 2023-12-16T21:36:40.711 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:36:40.707+0000 7f9ac37fe700 1 -- 172.21.15.45:0/2558897036 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f9acc00ad70 con 0x7f9adc0fc2b0 2023-12-16T21:36:40.711 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:36:40.707+0000 7f9ac37fe700 1 -- 172.21.15.45:0/2558897036 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f9acc0113a0 con 0x7f9adc0fc2b0 2023-12-16T21:36:40.711 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:36:40.707+0000 7f9ae1a5a700 1 -- 172.21.15.45:0/2558897036 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9adc113250 con 0x7f9adc0fc2b0 2023-12-16T21:36:40.712 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:36:40.708+0000 7f9ae1a5a700 1 -- 172.21.15.45:0/2558897036 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9adc1064d0 con 0x7f9adc0fc2b0 2023-12-16T21:36:40.712 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:36:40.708+0000 7f9ac37fe700 1 -- 172.21.15.45:0/2558897036 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 19) v1 ==== 86447+0+0 (secure 0 0 0) 0x7f9acc012400 con 0x7f9adc0fc2b0 2023-12-16T21:36:40.712 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:36:40.709+0000 7f9ac37fe700 1 --2- 172.21.15.45:0/2558897036 >> [v2:172.21.15.45:6800/2012105536,v1:172.21.15.45:6801/2012105536] conn(0x7f9ac4068f70 0x7f9ac406b420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:36:40.713 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:36:40.709+0000 7f9ac37fe700 1 -- 172.21.15.45:0/2558897036 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(37..37 src has 1..37) v4 ==== 4634+0+0 (secure 0 0 0) 0x7f9acc086ac0 con 0x7f9adc0fc2b0 2023-12-16T21:36:40.713 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:36:40.709+0000 7f9ada7fc700 1 --2- 172.21.15.45:0/2558897036 >> [v2:172.21.15.45:6800/2012105536,v1:172.21.15.45:6801/2012105536] conn(0x7f9ac4068f70 0x7f9ac406b420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:36:40.713 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:36:40.709+0000 7f9ada7fc700 1 --2- 172.21.15.45:0/2558897036 >> [v2:172.21.15.45:6800/2012105536,v1:172.21.15.45:6801/2012105536] conn(0x7f9ac4068f70 0x7f9ac406b420 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f9ad00043d0 tx=0x7f9ad0009e30 comp rx=0 tx=0).ready entity=mgr.14214 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:36:40.715 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:36:40.713+0000 7f9ac37fe700 1 -- 172.21.15.45:0/2558897036 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+174570 (secure 0 0 0) 0x7f9acc0532d0 con 0x7f9adc0fc2b0 2023-12-16T21:36:40.910 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:36:40.908+0000 7f9ae1a5a700 1 -- 172.21.15.45:0/2558897036 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "osd stat", "format": "json"} v 0) v1 -- 0x7f9adc04bb80 con 0x7f9adc0fc2b0 2023-12-16T21:36:40.910 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:36:40.908+0000 7f9ac37fe700 1 -- 172.21.15.45:0/2558897036 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mon_command_ack([{"prefix": "osd stat", "format": "json"}]=0 v37) v1 ==== 74+0+130 (secure 0 0 0) 0x7f9acc0568f0 con 0x7f9adc0fc2b0 2023-12-16T21:36:40.911 INFO:teuthology.orchestra.run.smithi045.stdout: 2023-12-16T21:36:40.912 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:36:40.910+0000 7f9ae1a5a700 1 -- 172.21.15.45:0/2558897036 >> [v2:172.21.15.45:6800/2012105536,v1:172.21.15.45:6801/2012105536] conn(0x7f9ac4068f70 msgr2=0x7f9ac406b420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:36:40.913 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:36:40.910+0000 7f9ae1a5a700 1 --2- 172.21.15.45:0/2558897036 >> [v2:172.21.15.45:6800/2012105536,v1:172.21.15.45:6801/2012105536] conn(0x7f9ac4068f70 0x7f9ac406b420 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f9ad00043d0 tx=0x7f9ad0009e30 comp rx=0 tx=0).stop 2023-12-16T21:36:40.913 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:36:40.910+0000 7f9ae1a5a700 1 -- 172.21.15.45:0/2558897036 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f9adc0fc2b0 msgr2=0x7f9adc110210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:36:40.913 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:36:40.910+0000 7f9ae1a5a700 1 --2- 172.21.15.45:0/2558897036 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f9adc0fc2b0 0x7f9adc110210 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7f9acc00c8c0 tx=0x7f9acc0044c0 comp rx=0 tx=0).stop 2023-12-16T21:36:40.913 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:36:40.910+0000 7f9ae1a5a700 1 -- 172.21.15.45:0/2558897036 shutdown_connections 2023-12-16T21:36:40.913 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:36:40.910+0000 7f9ae1a5a700 1 --2- 172.21.15.45:0/2558897036 >> [v2:172.21.15.45:6800/2012105536,v1:172.21.15.45:6801/2012105536] conn(0x7f9ac4068f70 0x7f9ac406b420 unknown :-1 s=CLOSED pgs=60 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:36:40.913 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:36:40.910+0000 7f9ae1a5a700 1 --2- 172.21.15.45:0/2558897036 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f9adc0fc2b0 0x7f9adc110210 unknown :-1 s=CLOSED pgs=185 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:36:40.913 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:36:40.910+0000 7f9ae1a5a700 1 --2- 172.21.15.45:0/2558897036 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f9adc0fec60 0x7f9adc112760 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:36:40.913 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:36:40.910+0000 7f9ae1a5a700 1 -- 172.21.15.45:0/2558897036 >> 172.21.15.45:0/2558897036 conn(0x7f9adc0f64f0 msgr2=0x7f9adc0f8920 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-16T21:36:40.913 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:36:40.910+0000 7f9ae1a5a700 1 -- 172.21.15.45:0/2558897036 shutdown_connections 2023-12-16T21:36:40.913 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:36:40.910+0000 7f9ae1a5a700 1 -- 172.21.15.45:0/2558897036 wait complete. 2023-12-16T21:36:41.181 INFO:teuthology.orchestra.run.smithi045.stdout:{"epoch":37,"num_osds":6,"num_up_osds":6,"osd_up_since":1702762598,"num_in_osds":6,"osd_in_since":1702762583,"num_remapped_pgs":0} 2023-12-16T21:36:41.182 DEBUG:teuthology.orchestra.run.smithi045:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 88ff363c-9c5a-11ee-95a5-87774f69a715 -- ceph osd dump --format=json 2023-12-16T21:36:41.487 INFO:teuthology.orchestra.run.smithi045.stderr:Inferring config /var/lib/ceph/88ff363c-9c5a-11ee-95a5-87774f69a715/mon.smithi045/config 2023-12-16T21:36:41.548 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Dec 16 21:36:41 smithi045 ceph-mon[80763]: pgmap v91: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2023-12-16T21:36:41.548 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Dec 16 21:36:41 smithi045 ceph-mon[80763]: from='client.? 172.21.15.45:0/2558897036' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2023-12-16T21:36:41.633 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Dec 16 21:36:41 smithi169 ceph-mon[88350]: pgmap v91: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2023-12-16T21:36:41.633 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Dec 16 21:36:41 smithi169 ceph-mon[88350]: from='client.? 172.21.15.45:0/2558897036' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2023-12-16T21:36:42.211 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:36:42.206+0000 7f68349e3700 1 -- 172.21.15.45:0/2066759005 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f6830102c00 msgr2=0x7f6830102fd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:36:42.211 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:36:42.206+0000 7f68349e3700 1 --2- 172.21.15.45:0/2066759005 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f6830102c00 0x7f6830102fd0 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7f6820004940 tx=0x7f6820009e30 comp rx=0 tx=0).stop 2023-12-16T21:36:42.211 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:36:42.207+0000 7f68349e3700 1 -- 172.21.15.45:0/2066759005 shutdown_connections 2023-12-16T21:36:42.211 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:36:42.207+0000 7f68349e3700 1 --2- 172.21.15.45:0/2066759005 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f68301035a0 0x7f68300fcc10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:36:42.211 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:36:42.207+0000 7f68349e3700 1 --2- 172.21.15.45:0/2066759005 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f6830102c00 0x7f6830102fd0 unknown :-1 s=CLOSED pgs=186 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:36:42.211 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:36:42.207+0000 7f68349e3700 1 -- 172.21.15.45:0/2066759005 >> 172.21.15.45:0/2066759005 conn(0x7f68300f8790 msgr2=0x7f68300faba0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-16T21:36:42.211 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:36:42.207+0000 7f68349e3700 1 -- 172.21.15.45:0/2066759005 shutdown_connections 2023-12-16T21:36:42.211 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:36:42.207+0000 7f68349e3700 1 -- 172.21.15.45:0/2066759005 wait complete. 2023-12-16T21:36:42.211 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:36:42.207+0000 7f68349e3700 1 Processor -- start 2023-12-16T21:36:42.211 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:36:42.207+0000 7f68349e3700 1 -- start start 2023-12-16T21:36:42.211 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:36:42.207+0000 7f68349e3700 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f6830102c00 0x7f6830113650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:36:42.212 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:36:42.207+0000 7f68349e3700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f68301035a0 0x7f6830113bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:36:42.212 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:36:42.207+0000 7f68349e3700 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f683010d510 con 0x7f68301035a0 2023-12-16T21:36:42.212 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:36:42.208+0000 7f68349e3700 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f683010d680 con 0x7f6830102c00 2023-12-16T21:36:42.212 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:36:42.208+0000 7f682dd9b700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f68301035a0 0x7f6830113bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:36:42.212 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:36:42.208+0000 7f682dd9b700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f68301035a0 0x7f6830113bd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:43880/0 (socket says 172.21.15.45:43880) 2023-12-16T21:36:42.212 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:36:42.208+0000 7f682dd9b700 1 -- 172.21.15.45:0/1204422108 learned_addr learned my addr 172.21.15.45:0/1204422108 (peer_addr_for_me v2:172.21.15.45:0/0) 2023-12-16T21:36:42.212 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:36:42.208+0000 7f682e59c700 1 --2- 172.21.15.45:0/1204422108 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f6830102c00 0x7f6830113650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:36:42.212 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:36:42.208+0000 7f682dd9b700 1 -- 172.21.15.45:0/1204422108 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f6830102c00 msgr2=0x7f6830113650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:36:42.212 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:36:42.208+0000 7f682dd9b700 1 --2- 172.21.15.45:0/1204422108 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f6830102c00 0x7f6830113650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:36:42.212 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:36:42.208+0000 7f682dd9b700 1 -- 172.21.15.45:0/1204422108 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6820005040 con 0x7f68301035a0 2023-12-16T21:36:42.212 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:36:42.208+0000 7f682dd9b700 1 --2- 172.21.15.45:0/1204422108 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f68301035a0 0x7f6830113bd0 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7f682400b790 tx=0x7f6824009220 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:36:42.212 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:36:42.208+0000 7f681effd700 1 -- 172.21.15.45:0/1204422108 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f682400f6f0 con 0x7f68301035a0 2023-12-16T21:36:42.212 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:36:42.208+0000 7f68349e3700 1 -- 172.21.15.45:0/1204422108 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f683010d960 con 0x7f68301035a0 2023-12-16T21:36:42.212 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:36:42.208+0000 7f681effd700 1 -- 172.21.15.45:0/1204422108 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f682400fd00 con 0x7f68301035a0 2023-12-16T21:36:42.213 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:36:42.208+0000 7f681effd700 1 -- 172.21.15.45:0/1204422108 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f68240103a0 con 0x7f68301035a0 2023-12-16T21:36:42.213 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:36:42.208+0000 7f68349e3700 1 -- 172.21.15.45:0/1204422108 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f683010dea0 con 0x7f68301035a0 2023-12-16T21:36:42.213 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:36:42.210+0000 7f68349e3700 1 -- 172.21.15.45:0/1204422108 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f68300fc390 con 0x7f68301035a0 2023-12-16T21:36:42.214 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:36:42.210+0000 7f681effd700 1 -- 172.21.15.45:0/1204422108 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 19) v1 ==== 86447+0+0 (secure 0 0 0) 0x7f682400f850 con 0x7f68301035a0 2023-12-16T21:36:42.214 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:36:42.210+0000 7f681effd700 1 --2- 172.21.15.45:0/1204422108 >> [v2:172.21.15.45:6800/2012105536,v1:172.21.15.45:6801/2012105536] conn(0x7f681806d370 0x7f681806f820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:36:42.214 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:36:42.210+0000 7f681effd700 1 -- 172.21.15.45:0/1204422108 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(37..37 src has 1..37) v4 ==== 4634+0+0 (secure 0 0 0) 0x7f6824086810 con 0x7f68301035a0 2023-12-16T21:36:42.214 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:36:42.210+0000 7f682e59c700 1 --2- 172.21.15.45:0/1204422108 >> [v2:172.21.15.45:6800/2012105536,v1:172.21.15.45:6801/2012105536] conn(0x7f681806d370 0x7f681806f820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:36:42.214 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:36:42.211+0000 7f682e59c700 1 --2- 172.21.15.45:0/1204422108 >> [v2:172.21.15.45:6800/2012105536,v1:172.21.15.45:6801/2012105536] conn(0x7f681806d370 0x7f681806f820 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f682000b3b0 tx=0x7f6820009e30 comp rx=0 tx=0).ready entity=mgr.14214 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:36:42.218 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:36:42.215+0000 7f681effd700 1 -- 172.21.15.45:0/1204422108 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+174570 (secure 0 0 0) 0x7f6824056ab0 con 0x7f68301035a0 2023-12-16T21:36:42.412 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:36:42.410+0000 7f68349e3700 1 -- 172.21.15.45:0/1204422108 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f683010ec30 con 0x7f68301035a0 2023-12-16T21:36:42.413 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:36:42.410+0000 7f681effd700 1 -- 172.21.15.45:0/1204422108 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v37) v1 ==== 74+0+11070 (secure 0 0 0) 0x7f6824056640 con 0x7f68301035a0 2023-12-16T21:36:42.413 INFO:teuthology.orchestra.run.smithi045.stdout: 2023-12-16T21:36:42.413 INFO:teuthology.orchestra.run.smithi045.stdout:{"epoch":37,"fsid":"88ff363c-9c5a-11ee-95a5-87774f69a715","created":"2023-12-16T21:32:48.502711+0000","modified":"2023-12-16T21:36:39.239674+0000","last_up_change":"2023-12-16T21:36:38.234094+0000","last_in_change":"2023-12-16T21:36:23.065916+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":14,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":6,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-12-16T21:35:53.102992+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}}],"osds":[{"osd":0,"uuid":"bc814a3e-c753-4948-b9eb-0d9d2d30904c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6802","nonce":1443487903},{"type":"v1","addr":"172.21.15.45:6803","nonce":1443487903}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6804","nonce":1443487903},{"type":"v1","addr":"172.21.15.45:6805","nonce":1443487903}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6808","nonce":1443487903},{"type":"v1","addr":"172.21.15.45:6809","nonce":1443487903}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6806","nonce":1443487903},{"type":"v1","addr":"172.21.15.45:6807","nonce":1443487903}]},"public_addr":"172.21.15.45:6803/1443487903","cluster_addr":"172.21.15.45:6805/1443487903","heartbeat_back_addr":"172.21.15.45:6809/1443487903","heartbeat_front_addr":"172.21.15.45:6807/1443487903","state":["exists","up"]},{"osd":1,"uuid":"66f2184c-26d0-492c-ae32-ece7b19ce7bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":28,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6810","nonce":3869108635},{"type":"v1","addr":"172.21.15.45:6811","nonce":3869108635}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6812","nonce":3869108635},{"type":"v1","addr":"172.21.15.45:6813","nonce":3869108635}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6816","nonce":3869108635},{"type":"v1","addr":"172.21.15.45:6817","nonce":3869108635}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6814","nonce":3869108635},{"type":"v1","addr":"172.21.15.45:6815","nonce":3869108635}]},"public_addr":"172.21.15.45:6811/3869108635","cluster_addr":"172.21.15.45:6813/3869108635","heartbeat_back_addr":"172.21.15.45:6817/3869108635","heartbeat_front_addr":"172.21.15.45:6815/3869108635","state":["exists","up"]},{"osd":2,"uuid":"e4c3ac33-08a6-439a-9a83-31488cc35202","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6818","nonce":844221874},{"type":"v1","addr":"172.21.15.45:6819","nonce":844221874}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6820","nonce":844221874},{"type":"v1","addr":"172.21.15.45:6821","nonce":844221874}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6824","nonce":844221874},{"type":"v1","addr":"172.21.15.45:6825","nonce":844221874}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6822","nonce":844221874},{"type":"v1","addr":"172.21.15.45:6823","nonce":844221874}]},"public_addr":"172.21.15.45:6819/844221874","cluster_addr":"172.21.15.45:6821/844221874","heartbeat_back_addr":"172.21.15.45:6825/844221874","heartbeat_front_addr":"172.21.15.45:6823/844221874","state":["exists","up"]},{"osd":3,"uuid":"957064f1-6dda-475b-903e-130134845a90","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":31,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6800","nonce":201299104},{"type":"v1","addr":"172.21.15.169:6801","nonce":201299104}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6802","nonce":201299104},{"type":"v1","addr":"172.21.15.169:6803","nonce":201299104}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6806","nonce":201299104},{"type":"v1","addr":"172.21.15.169:6807","nonce":201299104}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6804","nonce":201299104},{"type":"v1","addr":"172.21.15.169:6805","nonce":201299104}]},"public_addr":"172.21.15.169:6801/201299104","cluster_addr":"172.21.15.169:6803/201299104","heartbeat_back_addr":"172.21.15.169:6807/201299104","heartbeat_front_addr":"172.21.15.169:6805/201299104","state":["exists","up"]},{"osd":4,"uuid":"0926f0d3-f077-4045-9d5a-e10a94421e9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":32,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6808","nonce":3840494264},{"type":"v1","addr":"172.21.15.169:6809","nonce":3840494264}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6810","nonce":3840494264},{"type":"v1","addr":"172.21.15.169:6811","nonce":3840494264}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6814","nonce":3840494264},{"type":"v1","addr":"172.21.15.169:6815","nonce":3840494264}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6812","nonce":3840494264},{"type":"v1","addr":"172.21.15.169:6813","nonce":3840494264}]},"public_addr":"172.21.15.169:6809/3840494264","cluster_addr":"172.21.15.169:6811/3840494264","heartbeat_back_addr":"172.21.15.169:6815/3840494264","heartbeat_front_addr":"172.21.15.169:6813/3840494264","state":["exists","up"]},{"osd":5,"uuid":"363ba5fe-e249-4935-a370-6bc071a2f405","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":36,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6816","nonce":3826842549},{"type":"v1","addr":"172.21.15.169:6817","nonce":3826842549}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6818","nonce":3826842549},{"type":"v1","addr":"172.21.15.169:6819","nonce":3826842549}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6822","nonce":3826842549},{"type":"v1","addr":"172.21.15.169:6823","nonce":3826842549}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6820","nonce":3826842549},{"type":"v1","addr":"172.21.15.169:6821","nonce":3826842549}]},"public_addr":"172.21.15.169:6817/3826842549","cluster_addr":"172.21.15.169:6819/3826842549","heartbeat_back_addr":"172.21.15.169:6823/3826842549","heartbeat_front_addr":"172.21.15.169:6821/3826842549","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-16T21:35:15.986469+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-16T21:35:33.469306+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-16T21:35:50.142969+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-16T21:36:05.641289+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-16T21:36:21.464972+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-16T21:36:36.899654+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.45:0/103340221":"2023-12-17T21:34:28.582912+0000","172.21.15.45:6801/2215417738":"2023-12-17T21:34:28.582912+0000","172.21.15.45:0/3536329610":"2023-12-17T21:33:38.101959+0000","172.21.15.45:6800/2215417738":"2023-12-17T21:34:28.582912+0000","172.21.15.45:0/566984549":"2023-12-17T21:33:38.101959+0000","172.21.15.45:0/2329702517":"2023-12-17T21:33:16.105866+0000","172.21.15.45:0/1828324795":"2023-12-17T21:34:28.582912+0000","172.21.15.45:0/1866561551":"2023-12-17T21:33:16.105866+0000","172.21.15.45:0/3248165816":"2023-12-17T21:34:28.582912+0000","172.21.15.45:6801/1656764601":"2023-12-17T21:33:16.105866+0000","172.21.15.45:0/989600782":"2023-12-17T21:33:38.101959+0000","172.21.15.45:6800/2502560408":"2023-12-17T21:33:38.101959+0000","172.21.15.45:6800/1656764601":"2023-12-17T21:33:16.105866+0000","172.21.15.45:6801/2502560408":"2023-12-17T21:33:38.101959+0000","172.21.15.45:0/3764934134":"2023-12-17T21:33:16.105866+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-12-16T21:36:42.415 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:36:42.412+0000 7f68349e3700 1 -- 172.21.15.45:0/1204422108 >> [v2:172.21.15.45:6800/2012105536,v1:172.21.15.45:6801/2012105536] conn(0x7f681806d370 msgr2=0x7f681806f820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:36:42.415 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:36:42.412+0000 7f68349e3700 1 --2- 172.21.15.45:0/1204422108 >> [v2:172.21.15.45:6800/2012105536,v1:172.21.15.45:6801/2012105536] conn(0x7f681806d370 0x7f681806f820 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f682000b3b0 tx=0x7f6820009e30 comp rx=0 tx=0).stop 2023-12-16T21:36:42.415 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:36:42.412+0000 7f68349e3700 1 -- 172.21.15.45:0/1204422108 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f68301035a0 msgr2=0x7f6830113bd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:36:42.415 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:36:42.412+0000 7f68349e3700 1 --2- 172.21.15.45:0/1204422108 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f68301035a0 0x7f6830113bd0 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7f682400b790 tx=0x7f6824009220 comp rx=0 tx=0).stop 2023-12-16T21:36:42.415 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:36:42.413+0000 7f68349e3700 1 -- 172.21.15.45:0/1204422108 shutdown_connections 2023-12-16T21:36:42.415 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:36:42.413+0000 7f68349e3700 1 --2- 172.21.15.45:0/1204422108 >> [v2:172.21.15.45:6800/2012105536,v1:172.21.15.45:6801/2012105536] conn(0x7f681806d370 0x7f681806f820 unknown :-1 s=CLOSED pgs=61 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:36:42.416 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:36:42.413+0000 7f68349e3700 1 --2- 172.21.15.45:0/1204422108 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f6830102c00 0x7f6830113650 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:36:42.416 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:36:42.413+0000 7f68349e3700 1 --2- 172.21.15.45:0/1204422108 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f68301035a0 0x7f6830113bd0 unknown :-1 s=CLOSED pgs=187 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:36:42.416 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:36:42.413+0000 7f68349e3700 1 -- 172.21.15.45:0/1204422108 >> 172.21.15.45:0/1204422108 conn(0x7f68300f8790 msgr2=0x7f68300fa150 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-16T21:36:42.416 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:36:42.413+0000 7f68349e3700 1 -- 172.21.15.45:0/1204422108 shutdown_connections 2023-12-16T21:36:42.416 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:36:42.413+0000 7f68349e3700 1 -- 172.21.15.45:0/1204422108 wait complete. 2023-12-16T21:36:42.673 INFO:tasks.cephadm.ceph_manager.ceph:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2023-12-16T21:35:53.102992+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '22', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}}] 2023-12-16T21:36:42.674 DEBUG:teuthology.orchestra.run.smithi045:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 88ff363c-9c5a-11ee-95a5-87774f69a715 -- ceph osd pool get .mgr pg_num 2023-12-16T21:36:42.816 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Dec 16 21:36:42 smithi045 ceph-mon[80763]: from='client.? 172.21.15.45:0/1204422108' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2023-12-16T21:36:42.883 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Dec 16 21:36:42 smithi169 ceph-mon[88350]: from='client.? 172.21.15.45:0/1204422108' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2023-12-16T21:36:42.936 INFO:teuthology.orchestra.run.smithi045.stderr:Inferring config /var/lib/ceph/88ff363c-9c5a-11ee-95a5-87774f69a715/mon.smithi045/config 2023-12-16T21:36:43.606 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Dec 16 21:36:43 smithi045 ceph-mon[80763]: pgmap v92: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2023-12-16T21:36:43.726 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:36:43.723+0000 7f1d7a438700 1 -- 172.21.15.45:0/1855911524 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f1d740fbaf0 msgr2=0x7f1d740fbf60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:36:43.726 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:36:43.723+0000 7f1d7a438700 1 --2- 172.21.15.45:0/1855911524 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f1d740fbaf0 0x7f1d740fbf60 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7f1d68004940 tx=0x7f1d68009e30 comp rx=0 tx=0).stop 2023-12-16T21:36:43.726 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:36:43.723+0000 7f1d7a438700 1 -- 172.21.15.45:0/1855911524 shutdown_connections 2023-12-16T21:36:43.726 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:36:43.723+0000 7f1d7a438700 1 --2- 172.21.15.45:0/1855911524 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f1d740fbaf0 0x7f1d740fbf60 unknown :-1 s=CLOSED pgs=188 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:36:43.726 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:36:43.723+0000 7f1d7a438700 1 --2- 172.21.15.45:0/1855911524 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f1d74101b50 0x7f1d74101f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:36:43.727 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:36:43.723+0000 7f1d7a438700 1 -- 172.21.15.45:0/1855911524 >> 172.21.15.45:0/1855911524 conn(0x7f1d740f7670 msgr2=0x7f1d740f9a80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-16T21:36:43.727 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:36:43.723+0000 7f1d7a438700 1 -- 172.21.15.45:0/1855911524 shutdown_connections 2023-12-16T21:36:43.727 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:36:43.724+0000 7f1d7a438700 1 -- 172.21.15.45:0/1855911524 wait complete. 2023-12-16T21:36:43.730 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:36:43.724+0000 7f1d7a438700 1 Processor -- start 2023-12-16T21:36:43.730 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:36:43.724+0000 7f1d7a438700 1 -- start start 2023-12-16T21:36:43.731 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:36:43.725+0000 7f1d7a438700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f1d740fbaf0 0x7f1d74112580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:36:43.731 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:36:43.725+0000 7f1d7a438700 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f1d74101b50 0x7f1d74112ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:36:43.731 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:36:43.725+0000 7f1d7a438700 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1d74113110 con 0x7f1d740fbaf0 2023-12-16T21:36:43.731 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:36:43.725+0000 7f1d7a438700 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1d7410c440 con 0x7f1d74101b50 2023-12-16T21:36:43.731 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:36:43.725+0000 7f1d73fff700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f1d740fbaf0 0x7f1d74112580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:36:43.731 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:36:43.725+0000 7f1d73fff700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f1d740fbaf0 0x7f1d74112580 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:52548/0 (socket says 172.21.15.45:52548) 2023-12-16T21:36:43.731 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:36:43.725+0000 7f1d73fff700 1 -- 172.21.15.45:0/3224321973 learned_addr learned my addr 172.21.15.45:0/3224321973 (peer_addr_for_me v2:172.21.15.45:0/0) 2023-12-16T21:36:43.731 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:36:43.725+0000 7f1d737fe700 1 --2- 172.21.15.45:0/3224321973 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f1d74101b50 0x7f1d74112ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:36:43.731 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:36:43.725+0000 7f1d73fff700 1 -- 172.21.15.45:0/3224321973 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f1d74101b50 msgr2=0x7f1d74112ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:36:43.732 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:36:43.725+0000 7f1d73fff700 1 --2- 172.21.15.45:0/3224321973 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f1d74101b50 0x7f1d74112ac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:36:43.732 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:36:43.725+0000 7f1d73fff700 1 -- 172.21.15.45:0/3224321973 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1d68005040 con 0x7f1d740fbaf0 2023-12-16T21:36:43.732 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:36:43.725+0000 7f1d737fe700 1 --2- 172.21.15.45:0/3224321973 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f1d74101b50 0x7f1d74112ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-12-16T21:36:43.732 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:36:43.725+0000 7f1d73fff700 1 --2- 172.21.15.45:0/3224321973 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f1d740fbaf0 0x7f1d74112580 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7f1d6400c810 tx=0x7f1d64004470 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:36:43.732 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:36:43.725+0000 7f1d70ff9700 1 -- 172.21.15.45:0/3224321973 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f1d64012810 con 0x7f1d740fbaf0 2023-12-16T21:36:43.732 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:36:43.726+0000 7f1d70ff9700 1 -- 172.21.15.45:0/3224321973 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f1d64012e20 con 0x7f1d740fbaf0 2023-12-16T21:36:43.732 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:36:43.726+0000 7f1d7a438700 1 -- 172.21.15.45:0/3224321973 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1d7410c720 con 0x7f1d740fbaf0 2023-12-16T21:36:43.732 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:36:43.726+0000 7f1d70ff9700 1 -- 172.21.15.45:0/3224321973 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f1d6400a370 con 0x7f1d740fbaf0 2023-12-16T21:36:43.732 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:36:43.726+0000 7f1d7a438700 1 -- 172.21.15.45:0/3224321973 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1d7410cc60 con 0x7f1d740fbaf0 2023-12-16T21:36:43.733 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:36:43.727+0000 7f1d7a438700 1 -- 172.21.15.45:0/3224321973 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1d74103f30 con 0x7f1d740fbaf0 2023-12-16T21:36:43.733 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:36:43.727+0000 7f1d70ff9700 1 -- 172.21.15.45:0/3224321973 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 19) v1 ==== 86447+0+0 (secure 0 0 0) 0x7f1d64011780 con 0x7f1d740fbaf0 2023-12-16T21:36:43.733 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:36:43.727+0000 7f1d70ff9700 1 --2- 172.21.15.45:0/3224321973 >> [v2:172.21.15.45:6800/2012105536,v1:172.21.15.45:6801/2012105536] conn(0x7f1d5c068f60 0x7f1d5c06b410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:36:43.733 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:36:43.728+0000 7f1d70ff9700 1 -- 172.21.15.45:0/3224321973 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(37..37 src has 1..37) v4 ==== 4634+0+0 (secure 0 0 0) 0x7f1d64086e50 con 0x7f1d740fbaf0 2023-12-16T21:36:43.733 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:36:43.728+0000 7f1d737fe700 1 --2- 172.21.15.45:0/3224321973 >> [v2:172.21.15.45:6800/2012105536,v1:172.21.15.45:6801/2012105536] conn(0x7f1d5c068f60 0x7f1d5c06b410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:36:43.733 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:36:43.728+0000 7f1d737fe700 1 --2- 172.21.15.45:0/3224321973 >> [v2:172.21.15.45:6800/2012105536,v1:172.21.15.45:6801/2012105536] conn(0x7f1d5c068f60 0x7f1d5c06b410 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f1d68009fd0 tx=0x7f1d68009e30 comp rx=0 tx=0).ready entity=mgr.14214 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:36:43.734 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:36:43.732+0000 7f1d70ff9700 1 -- 172.21.15.45:0/3224321973 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+174570 (secure 0 0 0) 0x7f1d640570f0 con 0x7f1d740fbaf0 2023-12-16T21:36:43.883 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Dec 16 21:36:43 smithi169 ceph-mon[88350]: pgmap v92: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2023-12-16T21:36:43.930 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:36:43.927+0000 7f1d7a438700 1 -- 172.21.15.45:0/3224321973 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f1d7404bb80 con 0x7f1d740fbaf0 2023-12-16T21:36:43.930 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:36:43.928+0000 7f1d70ff9700 1 -- 172.21.15.45:0/3224321973 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v37) v1 ==== 93+0+10 (secure 0 0 0) 0x7f1d64016020 con 0x7f1d740fbaf0 2023-12-16T21:36:43.930 INFO:teuthology.orchestra.run.smithi045.stdout:pg_num: 1 2023-12-16T21:36:43.933 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:36:43.931+0000 7f1d7a438700 1 -- 172.21.15.45:0/3224321973 >> [v2:172.21.15.45:6800/2012105536,v1:172.21.15.45:6801/2012105536] conn(0x7f1d5c068f60 msgr2=0x7f1d5c06b410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:36:43.933 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:36:43.931+0000 7f1d7a438700 1 --2- 172.21.15.45:0/3224321973 >> [v2:172.21.15.45:6800/2012105536,v1:172.21.15.45:6801/2012105536] conn(0x7f1d5c068f60 0x7f1d5c06b410 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f1d68009fd0 tx=0x7f1d68009e30 comp rx=0 tx=0).stop 2023-12-16T21:36:43.934 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:36:43.931+0000 7f1d7a438700 1 -- 172.21.15.45:0/3224321973 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f1d740fbaf0 msgr2=0x7f1d74112580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:36:43.934 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:36:43.931+0000 7f1d7a438700 1 --2- 172.21.15.45:0/3224321973 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f1d740fbaf0 0x7f1d74112580 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7f1d6400c810 tx=0x7f1d64004470 comp rx=0 tx=0).stop 2023-12-16T21:36:43.934 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:36:43.931+0000 7f1d7a438700 1 -- 172.21.15.45:0/3224321973 shutdown_connections 2023-12-16T21:36:43.934 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:36:43.931+0000 7f1d7a438700 1 --2- 172.21.15.45:0/3224321973 >> [v2:172.21.15.45:6800/2012105536,v1:172.21.15.45:6801/2012105536] conn(0x7f1d5c068f60 0x7f1d5c06b410 unknown :-1 s=CLOSED pgs=62 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:36:43.934 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:36:43.931+0000 7f1d7a438700 1 --2- 172.21.15.45:0/3224321973 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f1d740fbaf0 0x7f1d74112580 unknown :-1 s=CLOSED pgs=189 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:36:43.934 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:36:43.931+0000 7f1d7a438700 1 --2- 172.21.15.45:0/3224321973 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f1d74101b50 0x7f1d74112ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:36:43.934 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:36:43.931+0000 7f1d7a438700 1 -- 172.21.15.45:0/3224321973 >> 172.21.15.45:0/3224321973 conn(0x7f1d740f7670 msgr2=0x7f1d740f82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-16T21:36:43.934 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:36:43.931+0000 7f1d7a438700 1 -- 172.21.15.45:0/3224321973 shutdown_connections 2023-12-16T21:36:43.934 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:36:43.931+0000 7f1d7a438700 1 -- 172.21.15.45:0/3224321973 wait complete. 2023-12-16T21:36:44.181 INFO:tasks.cephadm:Setting up client nodes... 2023-12-16T21:36:44.182 DEBUG:teuthology.orchestra.run.smithi045:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 88ff363c-9c5a-11ee-95a5-87774f69a715 -- ceph auth get-or-create client.0 mon 'allow *' osd 'allow *' mds 'allow *' mgr 'allow *' 2023-12-16T21:36:44.445 INFO:teuthology.orchestra.run.smithi045.stderr:Inferring config /var/lib/ceph/88ff363c-9c5a-11ee-95a5-87774f69a715/mon.smithi045/config 2023-12-16T21:36:44.781 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Dec 16 21:36:44 smithi045 ceph-mon[80763]: from='mgr.14214 172.21.15.45:0/1227909437' entity='mgr.smithi045.nhtmwc' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2023-12-16T21:36:44.782 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Dec 16 21:36:44 smithi045 ceph-mon[80763]: from='client.? 172.21.15.45:0/3224321973' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2023-12-16T21:36:44.784 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Dec 16 21:36:44 smithi169 ceph-mon[88350]: from='mgr.14214 172.21.15.45:0/1227909437' entity='mgr.smithi045.nhtmwc' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2023-12-16T21:36:44.785 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Dec 16 21:36:44 smithi169 ceph-mon[88350]: from='client.? 172.21.15.45:0/3224321973' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2023-12-16T21:36:45.068 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:36:45.063+0000 7fab7869f700 1 -- 172.21.15.45:0/153309332 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fab70100050 msgr2=0x7fab70102430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:36:45.068 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:36:45.063+0000 7fab7869f700 1 --2- 172.21.15.45:0/153309332 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fab70100050 0x7fab70102430 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7fab60004940 tx=0x7fab60009e30 comp rx=0 tx=0).stop 2023-12-16T21:36:45.068 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:36:45.064+0000 7fab7869f700 1 -- 172.21.15.45:0/153309332 shutdown_connections 2023-12-16T21:36:45.068 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:36:45.064+0000 7fab7869f700 1 --2- 172.21.15.45:0/153309332 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fab70100050 0x7fab70102430 unknown :-1 s=CLOSED pgs=190 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:36:45.068 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:36:45.064+0000 7fab7869f700 1 --2- 172.21.15.45:0/153309332 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fab700fd730 0x7fab700ffb10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:36:45.068 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:36:45.064+0000 7fab7869f700 1 -- 172.21.15.45:0/153309332 >> 172.21.15.45:0/153309332 conn(0x7fab700f71d0 msgr2=0x7fab700f95e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-16T21:36:45.068 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:36:45.064+0000 7fab7869f700 1 -- 172.21.15.45:0/153309332 shutdown_connections 2023-12-16T21:36:45.068 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:36:45.064+0000 7fab7869f700 1 -- 172.21.15.45:0/153309332 wait complete. 2023-12-16T21:36:45.068 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:36:45.064+0000 7fab7869f700 1 Processor -- start 2023-12-16T21:36:45.068 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:36:45.064+0000 7fab7869f700 1 -- start start 2023-12-16T21:36:45.068 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:36:45.064+0000 7fab7869f700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fab700fd730 0x7fab7010d520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:36:45.069 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:36:45.064+0000 7fab7869f700 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fab70100050 0x7fab7010da60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:36:45.069 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:36:45.064+0000 7fab7869f700 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fab7010e060 con 0x7fab700fd730 2023-12-16T21:36:45.069 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:36:45.064+0000 7fab7869f700 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fab7010e900 con 0x7fab70100050 2023-12-16T21:36:45.069 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:36:45.065+0000 7fab7643b700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fab700fd730 0x7fab7010d520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:36:45.069 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:36:45.065+0000 7fab7643b700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fab700fd730 0x7fab7010d520 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:52562/0 (socket says 172.21.15.45:52562) 2023-12-16T21:36:45.069 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:36:45.065+0000 7fab7643b700 1 -- 172.21.15.45:0/957262311 learned_addr learned my addr 172.21.15.45:0/957262311 (peer_addr_for_me v2:172.21.15.45:0/0) 2023-12-16T21:36:45.069 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:36:45.065+0000 7fab75c3a700 1 --2- 172.21.15.45:0/957262311 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fab70100050 0x7fab7010da60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:36:45.069 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:36:45.065+0000 7fab7643b700 1 -- 172.21.15.45:0/957262311 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fab70100050 msgr2=0x7fab7010da60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:36:45.069 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:36:45.065+0000 7fab7643b700 1 --2- 172.21.15.45:0/957262311 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fab70100050 0x7fab7010da60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:36:45.069 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:36:45.065+0000 7fab7643b700 1 -- 172.21.15.45:0/957262311 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fab60005040 con 0x7fab700fd730 2023-12-16T21:36:45.069 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:36:45.065+0000 7fab7643b700 1 --2- 172.21.15.45:0/957262311 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fab700fd730 0x7fab7010d520 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7fab6c002a50 tx=0x7fab6c0046b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:36:45.069 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:36:45.065+0000 7fab66ffd700 1 -- 172.21.15.45:0/957262311 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fab6c007cc0 con 0x7fab700fd730 2023-12-16T21:36:45.069 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:36:45.065+0000 7fab66ffd700 1 -- 172.21.15.45:0/957262311 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fab6c00f040 con 0x7fab700fd730 2023-12-16T21:36:45.069 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:36:45.065+0000 7fab7869f700 1 -- 172.21.15.45:0/957262311 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fab7010ebe0 con 0x7fab700fd730 2023-12-16T21:36:45.070 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:36:45.065+0000 7fab66ffd700 1 -- 172.21.15.45:0/957262311 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fab6c00d9c0 con 0x7fab700fd730 2023-12-16T21:36:45.070 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:36:45.065+0000 7fab7869f700 1 -- 172.21.15.45:0/957262311 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fab7010f120 con 0x7fab700fd730 2023-12-16T21:36:45.070 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:36:45.066+0000 7fab7869f700 1 -- 172.21.15.45:0/957262311 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fab701075c0 con 0x7fab700fd730 2023-12-16T21:36:45.070 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:36:45.067+0000 7fab66ffd700 1 -- 172.21.15.45:0/957262311 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 19) v1 ==== 86447+0+0 (secure 0 0 0) 0x7fab6c00db20 con 0x7fab700fd730 2023-12-16T21:36:45.071 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:36:45.067+0000 7fab66ffd700 1 --2- 172.21.15.45:0/957262311 >> [v2:172.21.15.45:6800/2012105536,v1:172.21.15.45:6801/2012105536] conn(0x7fab5c068f60 0x7fab5c06b410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:36:45.071 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:36:45.067+0000 7fab66ffd700 1 -- 172.21.15.45:0/957262311 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(37..37 src has 1..37) v4 ==== 4634+0+0 (secure 0 0 0) 0x7fab6c086790 con 0x7fab700fd730 2023-12-16T21:36:45.071 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:36:45.067+0000 7fab75c3a700 1 --2- 172.21.15.45:0/957262311 >> [v2:172.21.15.45:6800/2012105536,v1:172.21.15.45:6801/2012105536] conn(0x7fab5c068f60 0x7fab5c06b410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:36:45.071 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:36:45.068+0000 7fab75c3a700 1 --2- 172.21.15.45:0/957262311 >> [v2:172.21.15.45:6800/2012105536,v1:172.21.15.45:6801/2012105536] conn(0x7fab5c068f60 0x7fab5c06b410 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7fab6000b3b0 tx=0x7fab60009e30 comp rx=0 tx=0).ready entity=mgr.14214 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:36:45.074 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:36:45.072+0000 7fab66ffd700 1 -- 172.21.15.45:0/957262311 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+174570 (secure 0 0 0) 0x7fab6c0569b0 con 0x7fab700fd730 2023-12-16T21:36:45.349 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:36:45.347+0000 7fab7869f700 1 -- 172.21.15.45:0/957262311 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]} v 0) v1 -- 0x7fab7004bb80 con 0x7fab700fd730 2023-12-16T21:36:45.353 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:36:45.351+0000 7fab66ffd700 1 -- 172.21.15.45:0/957262311 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mon_command_ack([{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]=0 v14) v1 ==== 170+0+59 (secure 0 0 0) 0x7fab6c056540 con 0x7fab700fd730 2023-12-16T21:36:45.353 INFO:teuthology.orchestra.run.smithi045.stdout:[client.0] 2023-12-16T21:36:45.353 INFO:teuthology.orchestra.run.smithi045.stdout: key = AQBtGH5luGDJFBAAODuSGu3vy+bJrziEPNJ1/A== 2023-12-16T21:36:45.355 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:36:45.353+0000 7fab7869f700 1 -- 172.21.15.45:0/957262311 >> [v2:172.21.15.45:6800/2012105536,v1:172.21.15.45:6801/2012105536] conn(0x7fab5c068f60 msgr2=0x7fab5c06b410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:36:45.355 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:36:45.353+0000 7fab7869f700 1 --2- 172.21.15.45:0/957262311 >> [v2:172.21.15.45:6800/2012105536,v1:172.21.15.45:6801/2012105536] conn(0x7fab5c068f60 0x7fab5c06b410 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7fab6000b3b0 tx=0x7fab60009e30 comp rx=0 tx=0).stop 2023-12-16T21:36:45.356 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:36:45.353+0000 7fab7869f700 1 -- 172.21.15.45:0/957262311 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fab700fd730 msgr2=0x7fab7010d520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:36:45.356 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:36:45.353+0000 7fab7869f700 1 --2- 172.21.15.45:0/957262311 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fab700fd730 0x7fab7010d520 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7fab6c002a50 tx=0x7fab6c0046b0 comp rx=0 tx=0).stop 2023-12-16T21:36:45.356 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:36:45.353+0000 7fab7869f700 1 -- 172.21.15.45:0/957262311 shutdown_connections 2023-12-16T21:36:45.356 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:36:45.353+0000 7fab7869f700 1 --2- 172.21.15.45:0/957262311 >> [v2:172.21.15.45:6800/2012105536,v1:172.21.15.45:6801/2012105536] conn(0x7fab5c068f60 0x7fab5c06b410 unknown :-1 s=CLOSED pgs=63 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:36:45.356 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:36:45.353+0000 7fab7869f700 1 --2- 172.21.15.45:0/957262311 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fab700fd730 0x7fab7010d520 unknown :-1 s=CLOSED pgs=191 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:36:45.356 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:36:45.353+0000 7fab7869f700 1 --2- 172.21.15.45:0/957262311 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fab70100050 0x7fab7010da60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:36:45.356 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:36:45.353+0000 7fab7869f700 1 -- 172.21.15.45:0/957262311 >> 172.21.15.45:0/957262311 conn(0x7fab700f71d0 msgr2=0x7fab700f95b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-16T21:36:45.356 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:36:45.353+0000 7fab7869f700 1 -- 172.21.15.45:0/957262311 shutdown_connections 2023-12-16T21:36:45.356 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:36:45.353+0000 7fab7869f700 1 -- 172.21.15.45:0/957262311 wait complete. 2023-12-16T21:36:45.604 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Dec 16 21:36:45 smithi045 ceph-mon[80763]: pgmap v93: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2023-12-16T21:36:45.604 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Dec 16 21:36:45 smithi045 ceph-mon[80763]: from='client.? 172.21.15.45:0/957262311' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2023-12-16T21:36:45.604 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Dec 16 21:36:45 smithi045 ceph-mon[80763]: from='client.? 172.21.15.45:0/957262311' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2023-12-16T21:36:45.606 DEBUG:teuthology.orchestra.run.smithi045:> set -ex 2023-12-16T21:36:45.606 DEBUG:teuthology.orchestra.run.smithi045:> sudo dd of=/etc/ceph/ceph.client.0.keyring 2023-12-16T21:36:45.606 DEBUG:teuthology.orchestra.run.smithi045:> sudo chmod 0644 /etc/ceph/ceph.client.0.keyring 2023-12-16T21:36:45.654 DEBUG:teuthology.orchestra.run.smithi169:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 88ff363c-9c5a-11ee-95a5-87774f69a715 -- ceph auth get-or-create client.1 mon 'allow *' osd 'allow *' mds 'allow *' mgr 'allow *' 2023-12-16T21:36:45.883 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Dec 16 21:36:45 smithi169 ceph-mon[88350]: pgmap v93: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2023-12-16T21:36:45.883 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Dec 16 21:36:45 smithi169 ceph-mon[88350]: from='client.? 172.21.15.45:0/957262311' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2023-12-16T21:36:45.883 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Dec 16 21:36:45 smithi169 ceph-mon[88350]: from='client.? 172.21.15.45:0/957262311' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2023-12-16T21:36:45.924 INFO:teuthology.orchestra.run.smithi169.stderr:Inferring config /var/lib/ceph/88ff363c-9c5a-11ee-95a5-87774f69a715/mon.smithi169/config 2023-12-16T21:36:46.629 INFO:teuthology.orchestra.run.smithi169.stderr:2023-12-16T21:36:46.625+0000 7f2e5e51f700 1 -- 172.21.15.169:0/1826868787 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f2e580fd730 msgr2=0x7f2e580ffb10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:36:46.629 INFO:teuthology.orchestra.run.smithi169.stderr:2023-12-16T21:36:46.625+0000 7f2e5e51f700 1 --2- 172.21.15.169:0/1826868787 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f2e580fd730 0x7f2e580ffb10 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7f2e48004940 tx=0x7f2e48009e30 comp rx=0 tx=0).stop 2023-12-16T21:36:46.630 INFO:teuthology.orchestra.run.smithi169.stderr:2023-12-16T21:36:46.625+0000 7f2e5e51f700 1 -- 172.21.15.169:0/1826868787 shutdown_connections 2023-12-16T21:36:46.630 INFO:teuthology.orchestra.run.smithi169.stderr:2023-12-16T21:36:46.625+0000 7f2e5e51f700 1 --2- 172.21.15.169:0/1826868787 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f2e58100050 0x7f2e58102430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:36:46.630 INFO:teuthology.orchestra.run.smithi169.stderr:2023-12-16T21:36:46.625+0000 7f2e5e51f700 1 --2- 172.21.15.169:0/1826868787 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f2e580fd730 0x7f2e580ffb10 unknown :-1 s=CLOSED pgs=32 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:36:46.630 INFO:teuthology.orchestra.run.smithi169.stderr:2023-12-16T21:36:46.625+0000 7f2e5e51f700 1 -- 172.21.15.169:0/1826868787 >> 172.21.15.169:0/1826868787 conn(0x7f2e580f7190 msgr2=0x7f2e580f95e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-16T21:36:46.630 INFO:teuthology.orchestra.run.smithi169.stderr:2023-12-16T21:36:46.625+0000 7f2e5e51f700 1 -- 172.21.15.169:0/1826868787 shutdown_connections 2023-12-16T21:36:46.630 INFO:teuthology.orchestra.run.smithi169.stderr:2023-12-16T21:36:46.625+0000 7f2e5e51f700 1 -- 172.21.15.169:0/1826868787 wait complete. 2023-12-16T21:36:46.630 INFO:teuthology.orchestra.run.smithi169.stderr:2023-12-16T21:36:46.625+0000 7f2e5e51f700 1 Processor -- start 2023-12-16T21:36:46.630 INFO:teuthology.orchestra.run.smithi169.stderr:2023-12-16T21:36:46.626+0000 7f2e5e51f700 1 -- start start 2023-12-16T21:36:46.630 INFO:teuthology.orchestra.run.smithi169.stderr:2023-12-16T21:36:46.626+0000 7f2e5e51f700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f2e580fd730 0x7f2e580fd1c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:36:46.630 INFO:teuthology.orchestra.run.smithi169.stderr:2023-12-16T21:36:46.626+0000 7f2e5e51f700 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f2e58100050 0x7f2e580fb640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:36:46.630 INFO:teuthology.orchestra.run.smithi169.stderr:2023-12-16T21:36:46.626+0000 7f2e5e51f700 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2e580fbc40 con 0x7f2e580fd730 2023-12-16T21:36:46.631 INFO:teuthology.orchestra.run.smithi169.stderr:2023-12-16T21:36:46.626+0000 7f2e5e51f700 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2e580fbdb0 con 0x7f2e58100050 2023-12-16T21:36:46.631 INFO:teuthology.orchestra.run.smithi169.stderr:2023-12-16T21:36:46.626+0000 7f2e577fe700 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f2e58100050 0x7f2e580fb640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:36:46.631 INFO:teuthology.orchestra.run.smithi169.stderr:2023-12-16T21:36:46.626+0000 7f2e577fe700 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f2e58100050 0x7f2e580fb640 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.169:3300/0 says I am v2:172.21.15.169:36090/0 (socket says 172.21.15.169:36090) 2023-12-16T21:36:46.631 INFO:teuthology.orchestra.run.smithi169.stderr:2023-12-16T21:36:46.626+0000 7f2e577fe700 1 -- 172.21.15.169:0/3931419834 learned_addr learned my addr 172.21.15.169:0/3931419834 (peer_addr_for_me v2:172.21.15.169:0/0) 2023-12-16T21:36:46.631 INFO:teuthology.orchestra.run.smithi169.stderr:2023-12-16T21:36:46.626+0000 7f2e57fff700 1 --2- 172.21.15.169:0/3931419834 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f2e580fd730 0x7f2e580fd1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:36:46.631 INFO:teuthology.orchestra.run.smithi169.stderr:2023-12-16T21:36:46.626+0000 7f2e577fe700 1 -- 172.21.15.169:0/3931419834 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f2e580fd730 msgr2=0x7f2e580fd1c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:36:46.631 INFO:teuthology.orchestra.run.smithi169.stderr:2023-12-16T21:36:46.626+0000 7f2e577fe700 1 --2- 172.21.15.169:0/3931419834 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f2e580fd730 0x7f2e580fd1c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:36:46.631 INFO:teuthology.orchestra.run.smithi169.stderr:2023-12-16T21:36:46.626+0000 7f2e577fe700 1 -- 172.21.15.169:0/3931419834 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2e48005040 con 0x7f2e58100050 2023-12-16T21:36:46.631 INFO:teuthology.orchestra.run.smithi169.stderr:2023-12-16T21:36:46.627+0000 7f2e577fe700 1 --2- 172.21.15.169:0/3931419834 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f2e58100050 0x7f2e580fb640 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7f2e4c002a50 tx=0x7f2e4c0046b0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:36:46.631 INFO:teuthology.orchestra.run.smithi169.stderr:2023-12-16T21:36:46.627+0000 7f2e54ff9700 1 -- 172.21.15.169:0/3931419834 <== mon.1 v2:172.21.15.169:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f2e4c007cc0 con 0x7f2e58100050 2023-12-16T21:36:46.631 INFO:teuthology.orchestra.run.smithi169.stderr:2023-12-16T21:36:46.627+0000 7f2e54ff9700 1 -- 172.21.15.169:0/3931419834 <== mon.1 v2:172.21.15.169:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f2e4c00f040 con 0x7f2e58100050 2023-12-16T21:36:46.631 INFO:teuthology.orchestra.run.smithi169.stderr:2023-12-16T21:36:46.627+0000 7f2e5e51f700 1 -- 172.21.15.169:0/3931419834 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2e580fc090 con 0x7f2e58100050 2023-12-16T21:36:46.632 INFO:teuthology.orchestra.run.smithi169.stderr:2023-12-16T21:36:46.627+0000 7f2e54ff9700 1 -- 172.21.15.169:0/3931419834 <== mon.1 v2:172.21.15.169:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f2e4c00d9c0 con 0x7f2e58100050 2023-12-16T21:36:46.632 INFO:teuthology.orchestra.run.smithi169.stderr:2023-12-16T21:36:46.627+0000 7f2e5e51f700 1 -- 172.21.15.169:0/3931419834 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2e58115530 con 0x7f2e58100050 2023-12-16T21:36:46.633 INFO:teuthology.orchestra.run.smithi169.stderr:2023-12-16T21:36:46.628+0000 7f2e5e51f700 1 -- 172.21.15.169:0/3931419834 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2e5804bb80 con 0x7f2e58100050 2023-12-16T21:36:46.633 INFO:teuthology.orchestra.run.smithi169.stderr:2023-12-16T21:36:46.628+0000 7f2e54ff9700 1 -- 172.21.15.169:0/3931419834 <== mon.1 v2:172.21.15.169:3300/0 4 ==== mgrmap(e 19) v1 ==== 86447+0+0 (secure 0 0 0) 0x7f2e4c00db20 con 0x7f2e58100050 2023-12-16T21:36:46.633 INFO:teuthology.orchestra.run.smithi169.stderr:2023-12-16T21:36:46.629+0000 7f2e54ff9700 1 --2- 172.21.15.169:0/3931419834 >> [v2:172.21.15.45:6800/2012105536,v1:172.21.15.45:6801/2012105536] conn(0x7f2e40068f60 0x7f2e4006b410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:36:46.633 INFO:teuthology.orchestra.run.smithi169.stderr:2023-12-16T21:36:46.629+0000 7f2e54ff9700 1 -- 172.21.15.169:0/3931419834 <== mon.1 v2:172.21.15.169:3300/0 5 ==== osd_map(37..37 src has 1..37) v4 ==== 4634+0+0 (secure 0 0 0) 0x7f2e4c0867d0 con 0x7f2e58100050 2023-12-16T21:36:46.633 INFO:teuthology.orchestra.run.smithi169.stderr:2023-12-16T21:36:46.629+0000 7f2e57fff700 1 --2- 172.21.15.169:0/3931419834 >> [v2:172.21.15.45:6800/2012105536,v1:172.21.15.45:6801/2012105536] conn(0x7f2e40068f60 0x7f2e4006b410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:36:46.633 INFO:teuthology.orchestra.run.smithi169.stderr:2023-12-16T21:36:46.629+0000 7f2e57fff700 1 --2- 172.21.15.169:0/3931419834 >> [v2:172.21.15.45:6800/2012105536,v1:172.21.15.45:6801/2012105536] conn(0x7f2e40068f60 0x7f2e4006b410 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7f2e48004890 tx=0x7f2e48009e30 comp rx=0 tx=0).ready entity=mgr.14214 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:36:46.635 INFO:teuthology.orchestra.run.smithi169.stderr:2023-12-16T21:36:46.633+0000 7f2e54ff9700 1 -- 172.21.15.169:0/3931419834 <== mon.1 v2:172.21.15.169:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+174570 (secure 0 0 0) 0x7f2e4c0569f0 con 0x7f2e58100050 2023-12-16T21:36:46.890 INFO:teuthology.orchestra.run.smithi169.stderr:2023-12-16T21:36:46.888+0000 7f2e5e51f700 1 -- 172.21.15.169:0/3931419834 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_command({"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]} v 0) v1 -- 0x7f2e58105580 con 0x7f2e58100050 2023-12-16T21:36:46.895 INFO:teuthology.orchestra.run.smithi169.stderr:2023-12-16T21:36:46.893+0000 7f2e54ff9700 1 -- 172.21.15.169:0/3931419834 <== mon.1 v2:172.21.15.169:3300/0 7 ==== mon_command_ack([{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]=0 v15) v1 ==== 170+0+59 (secure 0 0 0) 0x7f2e4c056580 con 0x7f2e58100050 2023-12-16T21:36:46.896 INFO:teuthology.orchestra.run.smithi169.stdout:[client.1] 2023-12-16T21:36:46.896 INFO:teuthology.orchestra.run.smithi169.stdout: key = AQBuGH5lULMVNRAACeIdPZueYqjLS/P+vHZyig== 2023-12-16T21:36:46.898 INFO:teuthology.orchestra.run.smithi169.stderr:2023-12-16T21:36:46.896+0000 7f2e5e51f700 1 -- 172.21.15.169:0/3931419834 >> [v2:172.21.15.45:6800/2012105536,v1:172.21.15.45:6801/2012105536] conn(0x7f2e40068f60 msgr2=0x7f2e4006b410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:36:46.898 INFO:teuthology.orchestra.run.smithi169.stderr:2023-12-16T21:36:46.896+0000 7f2e5e51f700 1 --2- 172.21.15.169:0/3931419834 >> [v2:172.21.15.45:6800/2012105536,v1:172.21.15.45:6801/2012105536] conn(0x7f2e40068f60 0x7f2e4006b410 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7f2e48004890 tx=0x7f2e48009e30 comp rx=0 tx=0).stop 2023-12-16T21:36:46.898 INFO:teuthology.orchestra.run.smithi169.stderr:2023-12-16T21:36:46.896+0000 7f2e5e51f700 1 -- 172.21.15.169:0/3931419834 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f2e58100050 msgr2=0x7f2e580fb640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:36:46.898 INFO:teuthology.orchestra.run.smithi169.stderr:2023-12-16T21:36:46.896+0000 7f2e5e51f700 1 --2- 172.21.15.169:0/3931419834 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f2e58100050 0x7f2e580fb640 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7f2e4c002a50 tx=0x7f2e4c0046b0 comp rx=0 tx=0).stop 2023-12-16T21:36:46.898 INFO:teuthology.orchestra.run.smithi169.stderr:2023-12-16T21:36:46.896+0000 7f2e5e51f700 1 -- 172.21.15.169:0/3931419834 shutdown_connections 2023-12-16T21:36:46.898 INFO:teuthology.orchestra.run.smithi169.stderr:2023-12-16T21:36:46.896+0000 7f2e5e51f700 1 --2- 172.21.15.169:0/3931419834 >> [v2:172.21.15.45:6800/2012105536,v1:172.21.15.45:6801/2012105536] conn(0x7f2e40068f60 0x7f2e4006b410 unknown :-1 s=CLOSED pgs=64 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:36:46.898 INFO:teuthology.orchestra.run.smithi169.stderr:2023-12-16T21:36:46.896+0000 7f2e5e51f700 1 --2- 172.21.15.169:0/3931419834 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f2e580fd730 0x7f2e580fd1c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:36:46.898 INFO:teuthology.orchestra.run.smithi169.stderr:2023-12-16T21:36:46.896+0000 7f2e5e51f700 1 --2- 172.21.15.169:0/3931419834 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f2e58100050 0x7f2e580fb640 unknown :-1 s=CLOSED pgs=33 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:36:46.899 INFO:teuthology.orchestra.run.smithi169.stderr:2023-12-16T21:36:46.896+0000 7f2e5e51f700 1 -- 172.21.15.169:0/3931419834 >> 172.21.15.169:0/3931419834 conn(0x7f2e580f7190 msgr2=0x7f2e580f95e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-16T21:36:46.899 INFO:teuthology.orchestra.run.smithi169.stderr:2023-12-16T21:36:46.896+0000 7f2e5e51f700 1 -- 172.21.15.169:0/3931419834 shutdown_connections 2023-12-16T21:36:46.899 INFO:teuthology.orchestra.run.smithi169.stderr:2023-12-16T21:36:46.896+0000 7f2e5e51f700 1 -- 172.21.15.169:0/3931419834 wait complete. 2023-12-16T21:36:47.188 DEBUG:teuthology.orchestra.run.smithi169:> set -ex 2023-12-16T21:36:47.188 DEBUG:teuthology.orchestra.run.smithi169:> sudo dd of=/etc/ceph/ceph.client.1.keyring 2023-12-16T21:36:47.188 DEBUG:teuthology.orchestra.run.smithi169:> sudo chmod 0644 /etc/ceph/ceph.client.1.keyring 2023-12-16T21:36:47.230 INFO:tasks.ceph:Waiting until ceph daemons up and pgs clean... 2023-12-16T21:36:47.230 INFO:tasks.cephadm.ceph_manager.ceph:waiting for mgr available 2023-12-16T21:36:47.230 DEBUG:teuthology.orchestra.run.smithi045:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 88ff363c-9c5a-11ee-95a5-87774f69a715 -- ceph mgr dump --format=json 2023-12-16T21:36:47.528 INFO:teuthology.orchestra.run.smithi045.stderr:Inferring config /var/lib/ceph/88ff363c-9c5a-11ee-95a5-87774f69a715/mon.smithi045/config 2023-12-16T21:36:47.566 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Dec 16 21:36:47 smithi045 ceph-mon[80763]: pgmap v94: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2023-12-16T21:36:47.566 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Dec 16 21:36:47 smithi045 ceph-mon[80763]: from='client.? 172.21.15.169:0/3931419834' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2023-12-16T21:36:47.566 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Dec 16 21:36:47 smithi045 ceph-mon[80763]: from='client.? ' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2023-12-16T21:36:47.566 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Dec 16 21:36:47 smithi045 ceph-mon[80763]: from='client.? ' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2023-12-16T21:36:47.633 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Dec 16 21:36:47 smithi169 ceph-mon[88350]: pgmap v94: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2023-12-16T21:36:47.634 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Dec 16 21:36:47 smithi169 ceph-mon[88350]: from='client.? 172.21.15.169:0/3931419834' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2023-12-16T21:36:47.634 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Dec 16 21:36:47 smithi169 ceph-mon[88350]: from='client.? ' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2023-12-16T21:36:47.634 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Dec 16 21:36:47 smithi169 ceph-mon[88350]: from='client.? ' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2023-12-16T21:36:48.235 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:36:48.232+0000 7fc273845700 1 -- 172.21.15.45:0/2601718843 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fc26c102c00 msgr2=0x7fc26c102fd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:36:48.235 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:36:48.232+0000 7fc273845700 1 --2- 172.21.15.45:0/2601718843 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fc26c102c00 0x7fc26c102fd0 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7fc264009ce0 tx=0x7fc264004d10 comp rx=0 tx=0).stop 2023-12-16T21:36:48.235 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:36:48.232+0000 7fc273845700 1 -- 172.21.15.45:0/2601718843 shutdown_connections 2023-12-16T21:36:48.235 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:36:48.232+0000 7fc273845700 1 --2- 172.21.15.45:0/2601718843 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fc26c1035a0 0x7fc26c0fcc10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:36:48.235 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:36:48.232+0000 7fc273845700 1 --2- 172.21.15.45:0/2601718843 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fc26c102c00 0x7fc26c102fd0 unknown :-1 s=CLOSED pgs=192 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:36:48.235 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:36:48.232+0000 7fc273845700 1 -- 172.21.15.45:0/2601718843 >> 172.21.15.45:0/2601718843 conn(0x7fc26c0f8790 msgr2=0x7fc26c0faba0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-16T21:36:48.235 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:36:48.232+0000 7fc273845700 1 -- 172.21.15.45:0/2601718843 shutdown_connections 2023-12-16T21:36:48.235 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:36:48.232+0000 7fc273845700 1 -- 172.21.15.45:0/2601718843 wait complete. 2023-12-16T21:36:48.237 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:36:48.233+0000 7fc273845700 1 Processor -- start 2023-12-16T21:36:48.237 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:36:48.233+0000 7fc273845700 1 -- start start 2023-12-16T21:36:48.237 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:36:48.233+0000 7fc273845700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fc26c102c00 0x7fc26c113770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:36:48.237 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:36:48.233+0000 7fc273845700 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fc26c1035a0 0x7fc26c113cb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:36:48.237 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:36:48.233+0000 7fc273845700 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc26c114300 con 0x7fc26c102c00 2023-12-16T21:36:48.237 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:36:48.233+0000 7fc273845700 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc26c10d630 con 0x7fc26c1035a0 2023-12-16T21:36:48.237 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:36:48.233+0000 7fc2715e1700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fc26c102c00 0x7fc26c113770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:36:48.237 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:36:48.233+0000 7fc2715e1700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fc26c102c00 0x7fc26c113770 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:52576/0 (socket says 172.21.15.45:52576) 2023-12-16T21:36:48.237 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:36:48.233+0000 7fc2715e1700 1 -- 172.21.15.45:0/150003563 learned_addr learned my addr 172.21.15.45:0/150003563 (peer_addr_for_me v2:172.21.15.45:0/0) 2023-12-16T21:36:48.238 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:36:48.233+0000 7fc270de0700 1 --2- 172.21.15.45:0/150003563 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fc26c1035a0 0x7fc26c113cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:36:48.238 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:36:48.234+0000 7fc2715e1700 1 -- 172.21.15.45:0/150003563 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fc26c1035a0 msgr2=0x7fc26c113cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:36:48.238 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:36:48.234+0000 7fc2715e1700 1 --2- 172.21.15.45:0/150003563 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fc26c1035a0 0x7fc26c113cb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:36:48.238 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:36:48.234+0000 7fc2715e1700 1 -- 172.21.15.45:0/150003563 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc264009940 con 0x7fc26c102c00 2023-12-16T21:36:48.238 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:36:48.234+0000 7fc2715e1700 1 --2- 172.21.15.45:0/150003563 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fc26c102c00 0x7fc26c113770 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7fc264003400 tx=0x7fc264004d10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:36:48.238 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:36:48.234+0000 7fc269ffb700 1 -- 172.21.15.45:0/150003563 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fc26400e040 con 0x7fc26c102c00 2023-12-16T21:36:48.238 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:36:48.234+0000 7fc269ffb700 1 -- 172.21.15.45:0/150003563 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fc264008e20 con 0x7fc26c102c00 2023-12-16T21:36:48.238 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:36:48.234+0000 7fc273845700 1 -- 172.21.15.45:0/150003563 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc26c10d8b0 con 0x7fc26c102c00 2023-12-16T21:36:48.238 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:36:48.234+0000 7fc269ffb700 1 -- 172.21.15.45:0/150003563 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fc26401c640 con 0x7fc26c102c00 2023-12-16T21:36:48.238 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:36:48.234+0000 7fc273845700 1 -- 172.21.15.45:0/150003563 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc26c10ddf0 con 0x7fc26c102c00 2023-12-16T21:36:48.239 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:36:48.235+0000 7fc273845700 1 -- 172.21.15.45:0/150003563 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc26c0fff10 con 0x7fc26c102c00 2023-12-16T21:36:48.239 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:36:48.236+0000 7fc269ffb700 1 -- 172.21.15.45:0/150003563 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 19) v1 ==== 86447+0+0 (secure 0 0 0) 0x7fc264012410 con 0x7fc26c102c00 2023-12-16T21:36:48.239 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:36:48.236+0000 7fc269ffb700 1 --2- 172.21.15.45:0/150003563 >> [v2:172.21.15.45:6800/2012105536,v1:172.21.15.45:6801/2012105536] conn(0x7fc254068f60 0x7fc25406b410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:36:48.239 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:36:48.236+0000 7fc269ffb700 1 -- 172.21.15.45:0/150003563 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(37..37 src has 1..37) v4 ==== 4634+0+0 (secure 0 0 0) 0x7fc2640884b0 con 0x7fc26c102c00 2023-12-16T21:36:48.239 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:36:48.236+0000 7fc270de0700 1 --2- 172.21.15.45:0/150003563 >> [v2:172.21.15.45:6800/2012105536,v1:172.21.15.45:6801/2012105536] conn(0x7fc254068f60 0x7fc25406b410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:36:48.239 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:36:48.237+0000 7fc270de0700 1 --2- 172.21.15.45:0/150003563 >> [v2:172.21.15.45:6800/2012105536,v1:172.21.15.45:6801/2012105536] conn(0x7fc254068f60 0x7fc25406b410 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7fc26c10e840 tx=0x7fc26000b040 comp rx=0 tx=0).ready entity=mgr.14214 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:36:48.243 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:36:48.241+0000 7fc269ffb700 1 -- 172.21.15.45:0/150003563 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+174570 (secure 0 0 0) 0x7fc264058750 con 0x7fc26c102c00 2023-12-16T21:36:48.493 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:36:48.490+0000 7fc273845700 1 -- 172.21.15.45:0/150003563 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "mgr dump", "format": "json"} v 0) v1 -- 0x7fc26c04bb80 con 0x7fc26c102c00 2023-12-16T21:36:48.495 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:36:48.493+0000 7fc269ffb700 1 -- 172.21.15.45:0/150003563 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mon_command_ack([{"prefix": "mgr dump", "format": "json"}]=0 v19) v1 ==== 74+0+166775 (secure 0 0 0) 0x7fc2640582e0 con 0x7fc26c102c00 2023-12-16T21:36:48.498 INFO:teuthology.orchestra.run.smithi045.stdout: 2023-12-16T21:36:48.500 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:36:48.496+0000 7fc273845700 1 -- 172.21.15.45:0/150003563 >> [v2:172.21.15.45:6800/2012105536,v1:172.21.15.45:6801/2012105536] conn(0x7fc254068f60 msgr2=0x7fc25406b410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:36:48.500 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:36:48.496+0000 7fc273845700 1 --2- 172.21.15.45:0/150003563 >> [v2:172.21.15.45:6800/2012105536,v1:172.21.15.45:6801/2012105536] conn(0x7fc254068f60 0x7fc25406b410 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7fc26c10e840 tx=0x7fc26000b040 comp rx=0 tx=0).stop 2023-12-16T21:36:48.500 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:36:48.496+0000 7fc273845700 1 -- 172.21.15.45:0/150003563 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fc26c102c00 msgr2=0x7fc26c113770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:36:48.500 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:36:48.496+0000 7fc273845700 1 --2- 172.21.15.45:0/150003563 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fc26c102c00 0x7fc26c113770 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7fc264003400 tx=0x7fc264004d10 comp rx=0 tx=0).stop 2023-12-16T21:36:48.500 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:36:48.496+0000 7fc273845700 1 -- 172.21.15.45:0/150003563 shutdown_connections 2023-12-16T21:36:48.500 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:36:48.496+0000 7fc273845700 1 --2- 172.21.15.45:0/150003563 >> [v2:172.21.15.45:6800/2012105536,v1:172.21.15.45:6801/2012105536] conn(0x7fc254068f60 0x7fc25406b410 unknown :-1 s=CLOSED pgs=65 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:36:48.500 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:36:48.496+0000 7fc273845700 1 --2- 172.21.15.45:0/150003563 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fc26c102c00 0x7fc26c113770 unknown :-1 s=CLOSED pgs=193 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:36:48.501 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:36:48.496+0000 7fc273845700 1 --2- 172.21.15.45:0/150003563 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fc26c1035a0 0x7fc26c113cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:36:48.501 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:36:48.496+0000 7fc273845700 1 -- 172.21.15.45:0/150003563 >> 172.21.15.45:0/150003563 conn(0x7fc26c0f8790 msgr2=0x7fc26c0fa2d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-16T21:36:48.501 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:36:48.496+0000 7fc273845700 1 -- 172.21.15.45:0/150003563 shutdown_connections 2023-12-16T21:36:48.501 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:36:48.496+0000 7fc273845700 1 -- 172.21.15.45:0/150003563 wait complete. 2023-12-16T21:36:48.633 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Dec 16 21:36:48 smithi169 ceph-mon[88350]: from='client.? 172.21.15.45:0/150003563' entity='client.admin' cmd=[{"prefix": "mgr dump", "format": "json"}]: dispatch 2023-12-16T21:36:48.774 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Dec 16 21:36:48 smithi045 ceph-mon[80763]: from='client.? 172.21.15.45:0/150003563' entity='client.admin' cmd=[{"prefix": "mgr dump", "format": "json"}]: dispatch 2023-12-16T21:36:48.931 INFO:teuthology.orchestra.run.smithi045.stdout:{"epoch":19,"active_gid":14214,"active_name":"smithi045.nhtmwc","active_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6800","nonce":2012105536},{"type":"v1","addr":"172.21.15.45:6801","nonce":2012105536}]},"active_addr":"172.21.15.45:6801/2012105536","active_change":"2023-12-16T21:34:28.583032+0000","active_mgr_features":4540138320759226367,"available":true,"standbys":[{"gid":14236,"name":"smithi169.cncbgh","mgr_features":4540138320759226367,"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"7","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 or 7 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2400","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"7","min":"0","max":"7","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 or 7 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","upmap"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.25.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/ceph-grafana:9.4.7","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.1.5","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.5.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.43.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"secs","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"inventory_list_all":{"name":"inventory_list_all","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"k8sevents","can_run":true,"error_string":"","module_options":{"ceph_event_retention_days":{"name":"ceph_event_retention_days","type":"int","level":"advanced","flags":0,"default_value":"7","min":"","max":"","enum_allowed":[],"desc":"Days to hold ceph event information within local cache","long_desc":"","tags":[],"see_also":[]},"config_check_secs":{"name":"config_check_secs","type":"int","level":"advanced","flags":0,"default_value":"10","min":"10","max":"","enum_allowed":[],"desc":"interval (secs) to check for cluster configuration changes","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rgw","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"drive_group_interval":{"name":"drive_group_interval","type":"float","level":"advanced","flags":0,"default_value":"300.0","min":"","max":"","enum_allowed":[],"desc":"interval in seconds between re-application of applied drive_groups","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"storage_class":{"name":"storage_class","type":"str","level":"advanced","flags":0,"default_value":"local","min":"","max":"","enum_allowed":[],"desc":"storage class name for LSO-discovered PVs","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}]}],"modules":["cephadm","dashboard","iostat","nfs","prometheus","restful"],"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"7","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 or 7 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2400","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"7","min":"0","max":"7","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 or 7 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","upmap"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.25.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/ceph-grafana:9.4.7","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.1.5","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.5.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.43.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"secs","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"inventory_list_all":{"name":"inventory_list_all","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"k8sevents","can_run":true,"error_string":"","module_options":{"ceph_event_retention_days":{"name":"ceph_event_retention_days","type":"int","level":"advanced","flags":0,"default_value":"7","min":"","max":"","enum_allowed":[],"desc":"Days to hold ceph event information within local cache","long_desc":"","tags":[],"see_also":[]},"config_check_secs":{"name":"config_check_secs","type":"int","level":"advanced","flags":0,"default_value":"10","min":"10","max":"","enum_allowed":[],"desc":"interval (secs) to check for cluster configuration changes","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rgw","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"drive_group_interval":{"name":"drive_group_interval","type":"float","level":"advanced","flags":0,"default_value":"300.0","min":"","max":"","enum_allowed":[],"desc":"interval in seconds between re-application of applied drive_groups","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"storage_class":{"name":"storage_class","type":"str","level":"advanced","flags":0,"default_value":"local","min":"","max":"","enum_allowed":[],"desc":"storage class name for LSO-discovered PVs","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}],"services":{"dashboard":"https://172.21.15.45:8443/","prometheus":"http://172.21.15.45:9283/"},"always_on_modules":{"octopus":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"pacific":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"quincy":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"]},"last_failure_osd_epoch":5,"active_clients":[{"name":"devicehealth","addrvec":[{"type":"v2","addr":"172.21.15.45:0","nonce":2230584933}]},{"name":"libcephsqlite","addrvec":[{"type":"v2","addr":"172.21.15.45:0","nonce":3539241412}]},{"name":"rbd_support","addrvec":[{"type":"v2","addr":"172.21.15.45:0","nonce":4111067193}]},{"name":"volumes","addrvec":[{"type":"v2","addr":"172.21.15.45:0","nonce":34444321}]}]} 2023-12-16T21:36:48.935 INFO:tasks.cephadm.ceph_manager.ceph:mgr available! 2023-12-16T21:36:48.935 INFO:tasks.cephadm.ceph_manager.ceph:waiting for all up 2023-12-16T21:36:48.936 DEBUG:teuthology.orchestra.run.smithi045:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 88ff363c-9c5a-11ee-95a5-87774f69a715 -- ceph osd dump --format=json 2023-12-16T21:36:49.201 INFO:teuthology.orchestra.run.smithi045.stderr:Inferring config /var/lib/ceph/88ff363c-9c5a-11ee-95a5-87774f69a715/mon.smithi045/config 2023-12-16T21:36:49.595 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Dec 16 21:36:49 smithi045 ceph-mon[80763]: pgmap v95: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2023-12-16T21:36:49.883 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Dec 16 21:36:49 smithi169 ceph-mon[88350]: pgmap v95: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2023-12-16T21:36:49.964 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:36:49.960+0000 7f56b33cd700 1 -- 172.21.15.45:0/1370628930 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f56ac1035a0 msgr2=0x7f56ac0fcc10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:36:49.965 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:36:49.960+0000 7f56b33cd700 1 --2- 172.21.15.45:0/1370628930 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f56ac1035a0 0x7f56ac0fcc10 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7f56a0004940 tx=0x7f56a0009e30 comp rx=0 tx=0).stop 2023-12-16T21:36:49.965 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:36:49.960+0000 7f56b33cd700 1 -- 172.21.15.45:0/1370628930 shutdown_connections 2023-12-16T21:36:49.965 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:36:49.960+0000 7f56b33cd700 1 --2- 172.21.15.45:0/1370628930 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f56ac1035a0 0x7f56ac0fcc10 unknown :-1 s=CLOSED pgs=194 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:36:49.965 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:36:49.960+0000 7f56b33cd700 1 --2- 172.21.15.45:0/1370628930 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f56ac102c00 0x7f56ac102fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:36:49.965 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:36:49.960+0000 7f56b33cd700 1 -- 172.21.15.45:0/1370628930 >> 172.21.15.45:0/1370628930 conn(0x7f56ac0f8790 msgr2=0x7f56ac0faba0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-16T21:36:49.965 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:36:49.960+0000 7f56b33cd700 1 -- 172.21.15.45:0/1370628930 shutdown_connections 2023-12-16T21:36:49.965 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:36:49.960+0000 7f56b33cd700 1 -- 172.21.15.45:0/1370628930 wait complete. 2023-12-16T21:36:49.965 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:36:49.961+0000 7f56b33cd700 1 Processor -- start 2023-12-16T21:36:49.965 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:36:49.961+0000 7f56b33cd700 1 -- start start 2023-12-16T21:36:49.966 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:36:49.961+0000 7f56b33cd700 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f56ac102c00 0x7f56ac113660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:36:49.966 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:36:49.961+0000 7f56b33cd700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f56ac1035a0 0x7f56ac113be0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:36:49.966 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:36:49.961+0000 7f56b33cd700 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f56ac10d520 con 0x7f56ac1035a0 2023-12-16T21:36:49.966 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:36:49.961+0000 7f56b33cd700 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f56ac10d690 con 0x7f56ac102c00 2023-12-16T21:36:49.966 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:36:49.961+0000 7f56b0968700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f56ac1035a0 0x7f56ac113be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:36:49.966 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:36:49.961+0000 7f56b0968700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f56ac1035a0 0x7f56ac113be0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:52594/0 (socket says 172.21.15.45:52594) 2023-12-16T21:36:49.966 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:36:49.961+0000 7f56b0968700 1 -- 172.21.15.45:0/3760830143 learned_addr learned my addr 172.21.15.45:0/3760830143 (peer_addr_for_me v2:172.21.15.45:0/0) 2023-12-16T21:36:49.966 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:36:49.961+0000 7f56b1169700 1 --2- 172.21.15.45:0/3760830143 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f56ac102c00 0x7f56ac113660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:36:49.967 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:36:49.961+0000 7f56b0968700 1 -- 172.21.15.45:0/3760830143 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f56ac102c00 msgr2=0x7f56ac113660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:36:49.967 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:36:49.961+0000 7f56b0968700 1 --2- 172.21.15.45:0/3760830143 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f56ac102c00 0x7f56ac113660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:36:49.967 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:36:49.961+0000 7f56b0968700 1 -- 172.21.15.45:0/3760830143 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f56a0005040 con 0x7f56ac1035a0 2023-12-16T21:36:49.967 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:36:49.962+0000 7f56b0968700 1 --2- 172.21.15.45:0/3760830143 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f56ac1035a0 0x7f56ac113be0 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7f56a0009fd0 tx=0x7f56a0009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:36:49.967 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:36:49.962+0000 7f56a9ffb700 1 -- 172.21.15.45:0/3760830143 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f56a000e040 con 0x7f56ac1035a0 2023-12-16T21:36:49.967 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:36:49.962+0000 7f56b33cd700 1 -- 172.21.15.45:0/3760830143 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f56ac10d910 con 0x7f56ac1035a0 2023-12-16T21:36:49.967 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:36:49.962+0000 7f56a9ffb700 1 -- 172.21.15.45:0/3760830143 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f56a000bc50 con 0x7f56ac1035a0 2023-12-16T21:36:49.968 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:36:49.962+0000 7f56a9ffb700 1 -- 172.21.15.45:0/3760830143 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f56a001c3f0 con 0x7f56ac1035a0 2023-12-16T21:36:49.968 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:36:49.962+0000 7f56b33cd700 1 -- 172.21.15.45:0/3760830143 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f56ac10de50 con 0x7f56ac1035a0 2023-12-16T21:36:49.970 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:36:49.963+0000 7f56b33cd700 1 -- 172.21.15.45:0/3760830143 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f56ac0fff10 con 0x7f56ac1035a0 2023-12-16T21:36:49.970 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:36:49.963+0000 7f56a9ffb700 1 -- 172.21.15.45:0/3760830143 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 19) v1 ==== 86447+0+0 (secure 0 0 0) 0x7f56a0013420 con 0x7f56ac1035a0 2023-12-16T21:36:49.970 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:36:49.964+0000 7f56a9ffb700 1 --2- 172.21.15.45:0/3760830143 >> [v2:172.21.15.45:6800/2012105536,v1:172.21.15.45:6801/2012105536] conn(0x7f5694068f60 0x7f569406b410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:36:49.970 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:36:49.964+0000 7f56a9ffb700 1 -- 172.21.15.45:0/3760830143 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(37..37 src has 1..37) v4 ==== 4634+0+0 (secure 0 0 0) 0x7f56a0087de0 con 0x7f56ac1035a0 2023-12-16T21:36:49.970 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:36:49.964+0000 7f56b1169700 1 --2- 172.21.15.45:0/3760830143 >> [v2:172.21.15.45:6800/2012105536,v1:172.21.15.45:6801/2012105536] conn(0x7f5694068f60 0x7f569406b410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:36:49.970 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:36:49.964+0000 7f56b1169700 1 --2- 172.21.15.45:0/3760830143 >> [v2:172.21.15.45:6800/2012105536,v1:172.21.15.45:6801/2012105536] conn(0x7f5694068f60 0x7f569406b410 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7f56ac0f9dd0 tx=0x7f569c009620 comp rx=0 tx=0).ready entity=mgr.14214 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:36:49.971 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:36:49.968+0000 7f56a9ffb700 1 -- 172.21.15.45:0/3760830143 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+174570 (secure 0 0 0) 0x7f56a0058080 con 0x7f56ac1035a0 2023-12-16T21:36:50.168 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:36:50.166+0000 7f56b33cd700 1 -- 172.21.15.45:0/3760830143 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f56ac04bb80 con 0x7f56ac1035a0 2023-12-16T21:36:50.169 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:36:50.166+0000 7f56a9ffb700 1 -- 172.21.15.45:0/3760830143 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v37) v1 ==== 74+0+11070 (secure 0 0 0) 0x7f56a0057c10 con 0x7f56ac1035a0 2023-12-16T21:36:50.169 INFO:teuthology.orchestra.run.smithi045.stdout: 2023-12-16T21:36:50.169 INFO:teuthology.orchestra.run.smithi045.stdout:{"epoch":37,"fsid":"88ff363c-9c5a-11ee-95a5-87774f69a715","created":"2023-12-16T21:32:48.502711+0000","modified":"2023-12-16T21:36:39.239674+0000","last_up_change":"2023-12-16T21:36:38.234094+0000","last_in_change":"2023-12-16T21:36:23.065916+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":14,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":6,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-12-16T21:35:53.102992+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}}],"osds":[{"osd":0,"uuid":"bc814a3e-c753-4948-b9eb-0d9d2d30904c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6802","nonce":1443487903},{"type":"v1","addr":"172.21.15.45:6803","nonce":1443487903}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6804","nonce":1443487903},{"type":"v1","addr":"172.21.15.45:6805","nonce":1443487903}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6808","nonce":1443487903},{"type":"v1","addr":"172.21.15.45:6809","nonce":1443487903}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6806","nonce":1443487903},{"type":"v1","addr":"172.21.15.45:6807","nonce":1443487903}]},"public_addr":"172.21.15.45:6803/1443487903","cluster_addr":"172.21.15.45:6805/1443487903","heartbeat_back_addr":"172.21.15.45:6809/1443487903","heartbeat_front_addr":"172.21.15.45:6807/1443487903","state":["exists","up"]},{"osd":1,"uuid":"66f2184c-26d0-492c-ae32-ece7b19ce7bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":28,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6810","nonce":3869108635},{"type":"v1","addr":"172.21.15.45:6811","nonce":3869108635}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6812","nonce":3869108635},{"type":"v1","addr":"172.21.15.45:6813","nonce":3869108635}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6816","nonce":3869108635},{"type":"v1","addr":"172.21.15.45:6817","nonce":3869108635}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6814","nonce":3869108635},{"type":"v1","addr":"172.21.15.45:6815","nonce":3869108635}]},"public_addr":"172.21.15.45:6811/3869108635","cluster_addr":"172.21.15.45:6813/3869108635","heartbeat_back_addr":"172.21.15.45:6817/3869108635","heartbeat_front_addr":"172.21.15.45:6815/3869108635","state":["exists","up"]},{"osd":2,"uuid":"e4c3ac33-08a6-439a-9a83-31488cc35202","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6818","nonce":844221874},{"type":"v1","addr":"172.21.15.45:6819","nonce":844221874}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6820","nonce":844221874},{"type":"v1","addr":"172.21.15.45:6821","nonce":844221874}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6824","nonce":844221874},{"type":"v1","addr":"172.21.15.45:6825","nonce":844221874}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6822","nonce":844221874},{"type":"v1","addr":"172.21.15.45:6823","nonce":844221874}]},"public_addr":"172.21.15.45:6819/844221874","cluster_addr":"172.21.15.45:6821/844221874","heartbeat_back_addr":"172.21.15.45:6825/844221874","heartbeat_front_addr":"172.21.15.45:6823/844221874","state":["exists","up"]},{"osd":3,"uuid":"957064f1-6dda-475b-903e-130134845a90","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":31,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6800","nonce":201299104},{"type":"v1","addr":"172.21.15.169:6801","nonce":201299104}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6802","nonce":201299104},{"type":"v1","addr":"172.21.15.169:6803","nonce":201299104}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6806","nonce":201299104},{"type":"v1","addr":"172.21.15.169:6807","nonce":201299104}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6804","nonce":201299104},{"type":"v1","addr":"172.21.15.169:6805","nonce":201299104}]},"public_addr":"172.21.15.169:6801/201299104","cluster_addr":"172.21.15.169:6803/201299104","heartbeat_back_addr":"172.21.15.169:6807/201299104","heartbeat_front_addr":"172.21.15.169:6805/201299104","state":["exists","up"]},{"osd":4,"uuid":"0926f0d3-f077-4045-9d5a-e10a94421e9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":32,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6808","nonce":3840494264},{"type":"v1","addr":"172.21.15.169:6809","nonce":3840494264}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6810","nonce":3840494264},{"type":"v1","addr":"172.21.15.169:6811","nonce":3840494264}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6814","nonce":3840494264},{"type":"v1","addr":"172.21.15.169:6815","nonce":3840494264}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6812","nonce":3840494264},{"type":"v1","addr":"172.21.15.169:6813","nonce":3840494264}]},"public_addr":"172.21.15.169:6809/3840494264","cluster_addr":"172.21.15.169:6811/3840494264","heartbeat_back_addr":"172.21.15.169:6815/3840494264","heartbeat_front_addr":"172.21.15.169:6813/3840494264","state":["exists","up"]},{"osd":5,"uuid":"363ba5fe-e249-4935-a370-6bc071a2f405","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":36,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6816","nonce":3826842549},{"type":"v1","addr":"172.21.15.169:6817","nonce":3826842549}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6818","nonce":3826842549},{"type":"v1","addr":"172.21.15.169:6819","nonce":3826842549}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6822","nonce":3826842549},{"type":"v1","addr":"172.21.15.169:6823","nonce":3826842549}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6820","nonce":3826842549},{"type":"v1","addr":"172.21.15.169:6821","nonce":3826842549}]},"public_addr":"172.21.15.169:6817/3826842549","cluster_addr":"172.21.15.169:6819/3826842549","heartbeat_back_addr":"172.21.15.169:6823/3826842549","heartbeat_front_addr":"172.21.15.169:6821/3826842549","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-16T21:35:15.986469+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-16T21:35:33.469306+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-16T21:35:50.142969+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-16T21:36:05.641289+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-16T21:36:21.464972+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-16T21:36:36.899654+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.45:0/103340221":"2023-12-17T21:34:28.582912+0000","172.21.15.45:6801/2215417738":"2023-12-17T21:34:28.582912+0000","172.21.15.45:0/3536329610":"2023-12-17T21:33:38.101959+0000","172.21.15.45:6800/2215417738":"2023-12-17T21:34:28.582912+0000","172.21.15.45:0/566984549":"2023-12-17T21:33:38.101959+0000","172.21.15.45:0/2329702517":"2023-12-17T21:33:16.105866+0000","172.21.15.45:0/1828324795":"2023-12-17T21:34:28.582912+0000","172.21.15.45:0/1866561551":"2023-12-17T21:33:16.105866+0000","172.21.15.45:0/3248165816":"2023-12-17T21:34:28.582912+0000","172.21.15.45:6801/1656764601":"2023-12-17T21:33:16.105866+0000","172.21.15.45:0/989600782":"2023-12-17T21:33:38.101959+0000","172.21.15.45:6800/2502560408":"2023-12-17T21:33:38.101959+0000","172.21.15.45:6800/1656764601":"2023-12-17T21:33:16.105866+0000","172.21.15.45:6801/2502560408":"2023-12-17T21:33:38.101959+0000","172.21.15.45:0/3764934134":"2023-12-17T21:33:16.105866+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-12-16T21:36:50.172 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:36:50.169+0000 7f56b33cd700 1 -- 172.21.15.45:0/3760830143 >> [v2:172.21.15.45:6800/2012105536,v1:172.21.15.45:6801/2012105536] conn(0x7f5694068f60 msgr2=0x7f569406b410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:36:50.172 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:36:50.169+0000 7f56b33cd700 1 --2- 172.21.15.45:0/3760830143 >> [v2:172.21.15.45:6800/2012105536,v1:172.21.15.45:6801/2012105536] conn(0x7f5694068f60 0x7f569406b410 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7f56ac0f9dd0 tx=0x7f569c009620 comp rx=0 tx=0).stop 2023-12-16T21:36:50.172 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:36:50.169+0000 7f56b33cd700 1 -- 172.21.15.45:0/3760830143 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f56ac1035a0 msgr2=0x7f56ac113be0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:36:50.172 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:36:50.169+0000 7f56b33cd700 1 --2- 172.21.15.45:0/3760830143 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f56ac1035a0 0x7f56ac113be0 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7f56a0009fd0 tx=0x7f56a0009e30 comp rx=0 tx=0).stop 2023-12-16T21:36:50.172 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:36:50.169+0000 7f56b33cd700 1 -- 172.21.15.45:0/3760830143 shutdown_connections 2023-12-16T21:36:50.172 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:36:50.169+0000 7f56b33cd700 1 --2- 172.21.15.45:0/3760830143 >> [v2:172.21.15.45:6800/2012105536,v1:172.21.15.45:6801/2012105536] conn(0x7f5694068f60 0x7f569406b410 unknown :-1 s=CLOSED pgs=66 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:36:50.172 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:36:50.169+0000 7f56b33cd700 1 --2- 172.21.15.45:0/3760830143 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f56ac102c00 0x7f56ac113660 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:36:50.172 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:36:50.169+0000 7f56b33cd700 1 --2- 172.21.15.45:0/3760830143 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f56ac1035a0 0x7f56ac113be0 unknown :-1 s=CLOSED pgs=195 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:36:50.172 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:36:50.169+0000 7f56b33cd700 1 -- 172.21.15.45:0/3760830143 >> 172.21.15.45:0/3760830143 conn(0x7f56ac0f8790 msgr2=0x7f56ac0fa260 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-16T21:36:50.172 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:36:50.169+0000 7f56b33cd700 1 -- 172.21.15.45:0/3760830143 shutdown_connections 2023-12-16T21:36:50.173 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:36:50.169+0000 7f56b33cd700 1 -- 172.21.15.45:0/3760830143 wait complete. 2023-12-16T21:36:50.424 INFO:tasks.cephadm.ceph_manager.ceph:all up! 2023-12-16T21:36:50.425 DEBUG:teuthology.orchestra.run.smithi045:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 88ff363c-9c5a-11ee-95a5-87774f69a715 -- ceph osd dump --format=json 2023-12-16T21:36:50.702 INFO:teuthology.orchestra.run.smithi045.stderr:Inferring config /var/lib/ceph/88ff363c-9c5a-11ee-95a5-87774f69a715/mon.smithi045/config 2023-12-16T21:36:50.764 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Dec 16 21:36:50 smithi045 ceph-mon[80763]: from='client.? 172.21.15.45:0/3760830143' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2023-12-16T21:36:50.883 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Dec 16 21:36:50 smithi169 ceph-mon[88350]: from='client.? 172.21.15.45:0/3760830143' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2023-12-16T21:36:51.402 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:36:51.398+0000 7ff92566d700 1 -- 172.21.15.45:0/2798441353 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff9200fdc10 msgr2=0x7ff9200fdfe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:36:51.402 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:36:51.398+0000 7ff92566d700 1 --2- 172.21.15.45:0/2798441353 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff9200fdc10 0x7ff9200fdfe0 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7ff908004940 tx=0x7ff908009e30 comp rx=0 tx=0).stop 2023-12-16T21:36:51.402 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:36:51.398+0000 7ff92566d700 1 -- 172.21.15.45:0/2798441353 shutdown_connections 2023-12-16T21:36:51.402 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:36:51.398+0000 7ff92566d700 1 --2- 172.21.15.45:0/2798441353 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7ff9200fe5b0 0x7ff920106bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:36:51.402 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:36:51.398+0000 7ff92566d700 1 --2- 172.21.15.45:0/2798441353 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff9200fdc10 0x7ff9200fdfe0 unknown :-1 s=CLOSED pgs=196 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:36:51.402 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:36:51.398+0000 7ff92566d700 1 -- 172.21.15.45:0/2798441353 >> 172.21.15.45:0/2798441353 conn(0x7ff9200f76c0 msgr2=0x7ff9200f9ad0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-16T21:36:51.402 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:36:51.398+0000 7ff92566d700 1 -- 172.21.15.45:0/2798441353 shutdown_connections 2023-12-16T21:36:51.402 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:36:51.398+0000 7ff92566d700 1 -- 172.21.15.45:0/2798441353 wait complete. 2023-12-16T21:36:51.402 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:36:51.399+0000 7ff92566d700 1 Processor -- start 2023-12-16T21:36:51.402 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:36:51.399+0000 7ff92566d700 1 -- start start 2023-12-16T21:36:51.403 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:36:51.399+0000 7ff92566d700 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7ff9200fdc10 0x7ff920113650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:36:51.403 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:36:51.399+0000 7ff92566d700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff9200fe5b0 0x7ff920113bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:36:51.403 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:36:51.399+0000 7ff92566d700 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff92010d510 con 0x7ff9200fe5b0 2023-12-16T21:36:51.403 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:36:51.399+0000 7ff92566d700 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff92010d680 con 0x7ff9200fdc10 2023-12-16T21:36:51.403 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:36:51.399+0000 7ff91e7fc700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff9200fe5b0 0x7ff920113bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:36:51.403 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:36:51.399+0000 7ff91e7fc700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff9200fe5b0 0x7ff920113bd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:52610/0 (socket says 172.21.15.45:52610) 2023-12-16T21:36:51.403 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:36:51.399+0000 7ff91e7fc700 1 -- 172.21.15.45:0/4151389805 learned_addr learned my addr 172.21.15.45:0/4151389805 (peer_addr_for_me v2:172.21.15.45:0/0) 2023-12-16T21:36:51.403 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:36:51.399+0000 7ff91effd700 1 --2- 172.21.15.45:0/4151389805 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7ff9200fdc10 0x7ff920113650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:36:51.403 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:36:51.399+0000 7ff91e7fc700 1 -- 172.21.15.45:0/4151389805 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7ff9200fdc10 msgr2=0x7ff920113650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:36:51.403 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:36:51.399+0000 7ff91e7fc700 1 --2- 172.21.15.45:0/4151389805 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7ff9200fdc10 0x7ff920113650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:36:51.403 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:36:51.399+0000 7ff91e7fc700 1 -- 172.21.15.45:0/4151389805 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff908005040 con 0x7ff9200fe5b0 2023-12-16T21:36:51.403 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:36:51.400+0000 7ff91e7fc700 1 --2- 172.21.15.45:0/4151389805 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff9200fe5b0 0x7ff920113bd0 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7ff91000b790 tx=0x7ff910009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:36:51.404 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:36:51.400+0000 7ff9177fe700 1 -- 172.21.15.45:0/4151389805 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7ff91000f6f0 con 0x7ff9200fe5b0 2023-12-16T21:36:51.405 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:36:51.400+0000 7ff92566d700 1 -- 172.21.15.45:0/4151389805 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff92010d960 con 0x7ff9200fe5b0 2023-12-16T21:36:51.405 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:36:51.400+0000 7ff92566d700 1 -- 172.21.15.45:0/4151389805 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff92010dea0 con 0x7ff9200fe5b0 2023-12-16T21:36:51.405 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:36:51.400+0000 7ff9177fe700 1 -- 172.21.15.45:0/4151389805 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7ff91000fd00 con 0x7ff9200fe5b0 2023-12-16T21:36:51.405 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:36:51.400+0000 7ff9177fe700 1 -- 172.21.15.45:0/4151389805 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7ff9100103a0 con 0x7ff9200fe5b0 2023-12-16T21:36:51.405 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:36:51.401+0000 7ff92566d700 1 -- 172.21.15.45:0/4151389805 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff9000052e0 con 0x7ff9200fe5b0 2023-12-16T21:36:51.405 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:36:51.401+0000 7ff9177fe700 1 -- 172.21.15.45:0/4151389805 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 19) v1 ==== 86447+0+0 (secure 0 0 0) 0x7ff910010500 con 0x7ff9200fe5b0 2023-12-16T21:36:51.405 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:36:51.402+0000 7ff9177fe700 1 --2- 172.21.15.45:0/4151389805 >> [v2:172.21.15.45:6800/2012105536,v1:172.21.15.45:6801/2012105536] conn(0x7ff90c068f60 0x7ff90c06b410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:36:51.405 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:36:51.402+0000 7ff9177fe700 1 -- 172.21.15.45:0/4151389805 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(37..37 src has 1..37) v4 ==== 4634+0+0 (secure 0 0 0) 0x7ff910086700 con 0x7ff9200fe5b0 2023-12-16T21:36:51.405 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:36:51.402+0000 7ff91effd700 1 --2- 172.21.15.45:0/4151389805 >> [v2:172.21.15.45:6800/2012105536,v1:172.21.15.45:6801/2012105536] conn(0x7ff90c068f60 0x7ff90c06b410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:36:51.405 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:36:51.402+0000 7ff91effd700 1 --2- 172.21.15.45:0/4151389805 >> [v2:172.21.15.45:6800/2012105536,v1:172.21.15.45:6801/2012105536] conn(0x7ff90c068f60 0x7ff90c06b410 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7ff908009fd0 tx=0x7ff908009e30 comp rx=0 tx=0).ready entity=mgr.14214 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:36:51.408 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:36:51.406+0000 7ff9177fe700 1 -- 172.21.15.45:0/4151389805 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+174570 (secure 0 0 0) 0x7ff910056ab0 con 0x7ff9200fe5b0 2023-12-16T21:36:51.595 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Dec 16 21:36:51 smithi045 ceph-mon[80763]: pgmap v96: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2023-12-16T21:36:51.609 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:36:51.606+0000 7ff92566d700 1 -- 172.21.15.45:0/4151389805 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ff900005590 con 0x7ff9200fe5b0 2023-12-16T21:36:51.609 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:36:51.607+0000 7ff9177fe700 1 -- 172.21.15.45:0/4151389805 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v37) v1 ==== 74+0+11070 (secure 0 0 0) 0x7ff910056640 con 0x7ff9200fe5b0 2023-12-16T21:36:51.610 INFO:teuthology.orchestra.run.smithi045.stdout: 2023-12-16T21:36:51.610 INFO:teuthology.orchestra.run.smithi045.stdout:{"epoch":37,"fsid":"88ff363c-9c5a-11ee-95a5-87774f69a715","created":"2023-12-16T21:32:48.502711+0000","modified":"2023-12-16T21:36:39.239674+0000","last_up_change":"2023-12-16T21:36:38.234094+0000","last_in_change":"2023-12-16T21:36:23.065916+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":14,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":6,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-12-16T21:35:53.102992+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}}],"osds":[{"osd":0,"uuid":"bc814a3e-c753-4948-b9eb-0d9d2d30904c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6802","nonce":1443487903},{"type":"v1","addr":"172.21.15.45:6803","nonce":1443487903}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6804","nonce":1443487903},{"type":"v1","addr":"172.21.15.45:6805","nonce":1443487903}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6808","nonce":1443487903},{"type":"v1","addr":"172.21.15.45:6809","nonce":1443487903}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6806","nonce":1443487903},{"type":"v1","addr":"172.21.15.45:6807","nonce":1443487903}]},"public_addr":"172.21.15.45:6803/1443487903","cluster_addr":"172.21.15.45:6805/1443487903","heartbeat_back_addr":"172.21.15.45:6809/1443487903","heartbeat_front_addr":"172.21.15.45:6807/1443487903","state":["exists","up"]},{"osd":1,"uuid":"66f2184c-26d0-492c-ae32-ece7b19ce7bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":28,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6810","nonce":3869108635},{"type":"v1","addr":"172.21.15.45:6811","nonce":3869108635}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6812","nonce":3869108635},{"type":"v1","addr":"172.21.15.45:6813","nonce":3869108635}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6816","nonce":3869108635},{"type":"v1","addr":"172.21.15.45:6817","nonce":3869108635}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6814","nonce":3869108635},{"type":"v1","addr":"172.21.15.45:6815","nonce":3869108635}]},"public_addr":"172.21.15.45:6811/3869108635","cluster_addr":"172.21.15.45:6813/3869108635","heartbeat_back_addr":"172.21.15.45:6817/3869108635","heartbeat_front_addr":"172.21.15.45:6815/3869108635","state":["exists","up"]},{"osd":2,"uuid":"e4c3ac33-08a6-439a-9a83-31488cc35202","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6818","nonce":844221874},{"type":"v1","addr":"172.21.15.45:6819","nonce":844221874}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6820","nonce":844221874},{"type":"v1","addr":"172.21.15.45:6821","nonce":844221874}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6824","nonce":844221874},{"type":"v1","addr":"172.21.15.45:6825","nonce":844221874}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6822","nonce":844221874},{"type":"v1","addr":"172.21.15.45:6823","nonce":844221874}]},"public_addr":"172.21.15.45:6819/844221874","cluster_addr":"172.21.15.45:6821/844221874","heartbeat_back_addr":"172.21.15.45:6825/844221874","heartbeat_front_addr":"172.21.15.45:6823/844221874","state":["exists","up"]},{"osd":3,"uuid":"957064f1-6dda-475b-903e-130134845a90","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":31,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6800","nonce":201299104},{"type":"v1","addr":"172.21.15.169:6801","nonce":201299104}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6802","nonce":201299104},{"type":"v1","addr":"172.21.15.169:6803","nonce":201299104}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6806","nonce":201299104},{"type":"v1","addr":"172.21.15.169:6807","nonce":201299104}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6804","nonce":201299104},{"type":"v1","addr":"172.21.15.169:6805","nonce":201299104}]},"public_addr":"172.21.15.169:6801/201299104","cluster_addr":"172.21.15.169:6803/201299104","heartbeat_back_addr":"172.21.15.169:6807/201299104","heartbeat_front_addr":"172.21.15.169:6805/201299104","state":["exists","up"]},{"osd":4,"uuid":"0926f0d3-f077-4045-9d5a-e10a94421e9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":32,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6808","nonce":3840494264},{"type":"v1","addr":"172.21.15.169:6809","nonce":3840494264}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6810","nonce":3840494264},{"type":"v1","addr":"172.21.15.169:6811","nonce":3840494264}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6814","nonce":3840494264},{"type":"v1","addr":"172.21.15.169:6815","nonce":3840494264}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6812","nonce":3840494264},{"type":"v1","addr":"172.21.15.169:6813","nonce":3840494264}]},"public_addr":"172.21.15.169:6809/3840494264","cluster_addr":"172.21.15.169:6811/3840494264","heartbeat_back_addr":"172.21.15.169:6815/3840494264","heartbeat_front_addr":"172.21.15.169:6813/3840494264","state":["exists","up"]},{"osd":5,"uuid":"363ba5fe-e249-4935-a370-6bc071a2f405","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":36,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6816","nonce":3826842549},{"type":"v1","addr":"172.21.15.169:6817","nonce":3826842549}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6818","nonce":3826842549},{"type":"v1","addr":"172.21.15.169:6819","nonce":3826842549}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6822","nonce":3826842549},{"type":"v1","addr":"172.21.15.169:6823","nonce":3826842549}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6820","nonce":3826842549},{"type":"v1","addr":"172.21.15.169:6821","nonce":3826842549}]},"public_addr":"172.21.15.169:6817/3826842549","cluster_addr":"172.21.15.169:6819/3826842549","heartbeat_back_addr":"172.21.15.169:6823/3826842549","heartbeat_front_addr":"172.21.15.169:6821/3826842549","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-16T21:35:15.986469+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-16T21:35:33.469306+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-16T21:35:50.142969+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-16T21:36:05.641289+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-16T21:36:21.464972+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-16T21:36:36.899654+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.45:0/103340221":"2023-12-17T21:34:28.582912+0000","172.21.15.45:6801/2215417738":"2023-12-17T21:34:28.582912+0000","172.21.15.45:0/3536329610":"2023-12-17T21:33:38.101959+0000","172.21.15.45:6800/2215417738":"2023-12-17T21:34:28.582912+0000","172.21.15.45:0/566984549":"2023-12-17T21:33:38.101959+0000","172.21.15.45:0/2329702517":"2023-12-17T21:33:16.105866+0000","172.21.15.45:0/1828324795":"2023-12-17T21:34:28.582912+0000","172.21.15.45:0/1866561551":"2023-12-17T21:33:16.105866+0000","172.21.15.45:0/3248165816":"2023-12-17T21:34:28.582912+0000","172.21.15.45:6801/1656764601":"2023-12-17T21:33:16.105866+0000","172.21.15.45:0/989600782":"2023-12-17T21:33:38.101959+0000","172.21.15.45:6800/2502560408":"2023-12-17T21:33:38.101959+0000","172.21.15.45:6800/1656764601":"2023-12-17T21:33:16.105866+0000","172.21.15.45:6801/2502560408":"2023-12-17T21:33:38.101959+0000","172.21.15.45:0/3764934134":"2023-12-17T21:33:16.105866+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-12-16T21:36:51.611 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:36:51.609+0000 7ff92566d700 1 -- 172.21.15.45:0/4151389805 >> [v2:172.21.15.45:6800/2012105536,v1:172.21.15.45:6801/2012105536] conn(0x7ff90c068f60 msgr2=0x7ff90c06b410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:36:51.611 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:36:51.609+0000 7ff92566d700 1 --2- 172.21.15.45:0/4151389805 >> [v2:172.21.15.45:6800/2012105536,v1:172.21.15.45:6801/2012105536] conn(0x7ff90c068f60 0x7ff90c06b410 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7ff908009fd0 tx=0x7ff908009e30 comp rx=0 tx=0).stop 2023-12-16T21:36:51.611 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:36:51.609+0000 7ff92566d700 1 -- 172.21.15.45:0/4151389805 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff9200fe5b0 msgr2=0x7ff920113bd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:36:51.612 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:36:51.609+0000 7ff92566d700 1 --2- 172.21.15.45:0/4151389805 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff9200fe5b0 0x7ff920113bd0 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7ff91000b790 tx=0x7ff910009250 comp rx=0 tx=0).stop 2023-12-16T21:36:51.612 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:36:51.609+0000 7ff92566d700 1 -- 172.21.15.45:0/4151389805 shutdown_connections 2023-12-16T21:36:51.612 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:36:51.609+0000 7ff92566d700 1 --2- 172.21.15.45:0/4151389805 >> [v2:172.21.15.45:6800/2012105536,v1:172.21.15.45:6801/2012105536] conn(0x7ff90c068f60 0x7ff90c06b410 unknown :-1 s=CLOSED pgs=67 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:36:51.612 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:36:51.609+0000 7ff92566d700 1 --2- 172.21.15.45:0/4151389805 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7ff9200fdc10 0x7ff920113650 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:36:51.612 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:36:51.609+0000 7ff92566d700 1 --2- 172.21.15.45:0/4151389805 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff9200fe5b0 0x7ff920113bd0 unknown :-1 s=CLOSED pgs=197 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:36:51.612 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:36:51.609+0000 7ff92566d700 1 -- 172.21.15.45:0/4151389805 >> 172.21.15.45:0/4151389805 conn(0x7ff9200f76c0 msgr2=0x7ff920101380 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-16T21:36:51.612 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:36:51.609+0000 7ff92566d700 1 -- 172.21.15.45:0/4151389805 shutdown_connections 2023-12-16T21:36:51.612 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:36:51.609+0000 7ff92566d700 1 -- 172.21.15.45:0/4151389805 wait complete. 2023-12-16T21:36:51.856 DEBUG:teuthology.orchestra.run.smithi045:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 88ff363c-9c5a-11ee-95a5-87774f69a715 -- ceph tell osd.0 flush_pg_stats 2023-12-16T21:36:51.857 DEBUG:teuthology.orchestra.run.smithi045:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 88ff363c-9c5a-11ee-95a5-87774f69a715 -- ceph tell osd.1 flush_pg_stats 2023-12-16T21:36:51.857 DEBUG:teuthology.orchestra.run.smithi045:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 88ff363c-9c5a-11ee-95a5-87774f69a715 -- ceph tell osd.2 flush_pg_stats 2023-12-16T21:36:51.857 DEBUG:teuthology.orchestra.run.smithi045:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 88ff363c-9c5a-11ee-95a5-87774f69a715 -- ceph tell osd.3 flush_pg_stats 2023-12-16T21:36:51.858 DEBUG:teuthology.orchestra.run.smithi045:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 88ff363c-9c5a-11ee-95a5-87774f69a715 -- ceph tell osd.4 flush_pg_stats 2023-12-16T21:36:51.858 DEBUG:teuthology.orchestra.run.smithi045:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 88ff363c-9c5a-11ee-95a5-87774f69a715 -- ceph tell osd.5 flush_pg_stats 2023-12-16T21:36:51.883 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Dec 16 21:36:51 smithi169 ceph-mon[88350]: pgmap v96: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2023-12-16T21:36:52.157 INFO:teuthology.orchestra.run.smithi045.stderr:Inferring config /var/lib/ceph/88ff363c-9c5a-11ee-95a5-87774f69a715/mon.smithi045/config 2023-12-16T21:36:52.430 INFO:teuthology.orchestra.run.smithi045.stderr:Inferring config /var/lib/ceph/88ff363c-9c5a-11ee-95a5-87774f69a715/mon.smithi045/config 2023-12-16T21:36:52.816 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Dec 16 21:36:52 smithi045 ceph-mon[80763]: from='client.? 172.21.15.45:0/4151389805' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2023-12-16T21:36:52.883 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Dec 16 21:36:52 smithi169 ceph-mon[88350]: from='client.? 172.21.15.45:0/4151389805' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2023-12-16T21:36:53.107 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:36:53.104+0000 7f2517167700 1 -- 172.21.15.45:0/1500642522 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f25101035a0 msgr2=0x7f25100fcc10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:36:53.107 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:36:53.104+0000 7f2517167700 1 --2- 172.21.15.45:0/1500642522 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f25101035a0 0x7f25100fcc10 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7f24fc004940 tx=0x7f24fc009e30 comp rx=0 tx=0).stop 2023-12-16T21:36:53.108 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:36:53.104+0000 7f2517167700 1 -- 172.21.15.45:0/1500642522 shutdown_connections 2023-12-16T21:36:53.108 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:36:53.104+0000 7f2517167700 1 --2- 172.21.15.45:0/1500642522 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f25101035a0 0x7f25100fcc10 unknown :-1 s=CLOSED pgs=198 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:36:53.108 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:36:53.104+0000 7f2517167700 1 --2- 172.21.15.45:0/1500642522 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f2510102c00 0x7f2510102fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:36:53.108 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:36:53.104+0000 7f2517167700 1 -- 172.21.15.45:0/1500642522 >> 172.21.15.45:0/1500642522 conn(0x7f25100f8790 msgr2=0x7f25100faba0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-16T21:36:53.108 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:36:53.104+0000 7f2517167700 1 -- 172.21.15.45:0/1500642522 shutdown_connections 2023-12-16T21:36:53.108 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:36:53.104+0000 7f2517167700 1 -- 172.21.15.45:0/1500642522 wait complete. 2023-12-16T21:36:53.108 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:36:53.105+0000 7f2517167700 1 Processor -- start 2023-12-16T21:36:53.108 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:36:53.105+0000 7f2517167700 1 -- start start 2023-12-16T21:36:53.108 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:36:53.105+0000 7f2517167700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f2510102c00 0x7f251006ff40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:36:53.108 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:36:53.105+0000 7f2517167700 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f25101035a0 0x7f2510070480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:36:53.108 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:36:53.105+0000 7f2517167700 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2510070b10 con 0x7f2510102c00 2023-12-16T21:36:53.108 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:36:53.105+0000 7f2517167700 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2510070c80 con 0x7f25101035a0 2023-12-16T21:36:53.110 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:36:53.105+0000 7f2514f03700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f2510102c00 0x7f251006ff40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:36:53.110 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:36:53.105+0000 7f2514f03700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f2510102c00 0x7f251006ff40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:46708/0 (socket says 172.21.15.45:46708) 2023-12-16T21:36:53.110 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:36:53.105+0000 7f2514f03700 1 -- 172.21.15.45:0/3344385015 learned_addr learned my addr 172.21.15.45:0/3344385015 (peer_addr_for_me v2:172.21.15.45:0/0) 2023-12-16T21:36:53.110 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:36:53.106+0000 7f250ffff700 1 --2- 172.21.15.45:0/3344385015 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f25101035a0 0x7f2510070480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:36:53.110 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:36:53.106+0000 7f2514f03700 1 -- 172.21.15.45:0/3344385015 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f25101035a0 msgr2=0x7f2510070480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:36:53.110 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:36:53.106+0000 7f2514f03700 1 --2- 172.21.15.45:0/3344385015 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f25101035a0 0x7f2510070480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:36:53.110 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:36:53.106+0000 7f2514f03700 1 -- 172.21.15.45:0/3344385015 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f24fc005040 con 0x7f2510102c00 2023-12-16T21:36:53.110 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:36:53.106+0000 7f2514f03700 1 --2- 172.21.15.45:0/3344385015 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f2510102c00 0x7f251006ff40 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7f250400b790 tx=0x7f2504009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:36:53.110 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:36:53.106+0000 7f250d7fa700 1 -- 172.21.15.45:0/3344385015 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f250400f770 con 0x7f2510102c00 2023-12-16T21:36:53.110 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:36:53.106+0000 7f250d7fa700 1 -- 172.21.15.45:0/3344385015 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f250400fd80 con 0x7f2510102c00 2023-12-16T21:36:53.110 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:36:53.106+0000 7f2517167700 1 -- 172.21.15.45:0/3344385015 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2510119910 con 0x7f2510102c00 2023-12-16T21:36:53.110 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:36:53.106+0000 7f250d7fa700 1 -- 172.21.15.45:0/3344385015 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f250400d310 con 0x7f2510102c00 2023-12-16T21:36:53.110 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:36:53.106+0000 7f2517167700 1 -- 172.21.15.45:0/3344385015 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2510119e00 con 0x7f2510102c00 2023-12-16T21:36:53.111 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:36:53.107+0000 7f2517167700 1 -- 172.21.15.45:0/3344385015 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f251005e380 con 0x7f2510102c00 2023-12-16T21:36:53.112 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:36:53.108+0000 7f250d7fa700 1 -- 172.21.15.45:0/3344385015 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 19) v1 ==== 86447+0+0 (secure 0 0 0) 0x7f2504010420 con 0x7f2510102c00 2023-12-16T21:36:53.112 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:36:53.108+0000 7f250d7fa700 1 --2- 172.21.15.45:0/3344385015 >> [v2:172.21.15.45:6800/2012105536,v1:172.21.15.45:6801/2012105536] conn(0x7f2500068f60 0x7f250006b410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:36:53.112 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:36:53.108+0000 7f250d7fa700 1 -- 172.21.15.45:0/3344385015 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(37..37 src has 1..37) v4 ==== 4634+0+0 (secure 0 0 0) 0x7f2504086a40 con 0x7f2510102c00 2023-12-16T21:36:53.112 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:36:53.108+0000 7f250ffff700 1 --2- 172.21.15.45:0/3344385015 >> [v2:172.21.15.45:6800/2012105536,v1:172.21.15.45:6801/2012105536] conn(0x7f2500068f60 0x7f250006b410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:36:53.112 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:36:53.108+0000 7f250d7fa700 1 --2- 172.21.15.45:0/3344385015 >> [v2:172.21.15.169:6800/201299104,v1:172.21.15.169:6801/201299104] conn(0x7f250006ec30 0x7f2500071040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:36:53.112 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:36:53.108+0000 7f250d7fa700 1 -- 172.21.15.45:0/3344385015 --> [v2:172.21.15.169:6800/201299104,v1:172.21.15.169:6801/201299104] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f25000716f0 con 0x7f250006ec30 2023-12-16T21:36:53.112 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:36:53.108+0000 7f250d7fa700 1 -- 172.21.15.45:0/3344385015 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_get_version_reply(handle=1 version=37) v2 ==== 24+0+0 (secure 0 0 0) 0x7f250408b220 con 0x7f2510102c00 2023-12-16T21:36:53.112 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:36:53.109+0000 7f2515704700 1 --2- 172.21.15.45:0/3344385015 >> [v2:172.21.15.169:6800/201299104,v1:172.21.15.169:6801/201299104] conn(0x7f250006ec30 0x7f2500071040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:36:53.112 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:36:53.109+0000 7f250ffff700 1 --2- 172.21.15.45:0/3344385015 >> [v2:172.21.15.45:6800/2012105536,v1:172.21.15.45:6801/2012105536] conn(0x7f2500068f60 0x7f250006b410 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7f2510071940 tx=0x7f24fc009e30 comp rx=0 tx=0).ready entity=mgr.14214 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:36:53.112 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:36:53.109+0000 7f2515704700 1 --2- 172.21.15.45:0/3344385015 >> [v2:172.21.15.169:6800/201299104,v1:172.21.15.169:6801/201299104] conn(0x7f250006ec30 0x7f2500071040 crc :-1 s=READY pgs=5 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:36:53.113 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:36:53.110+0000 7f250d7fa700 1 -- 172.21.15.45:0/3344385015 <== osd.3 v2:172.21.15.169:6800/201299104 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+23999 (crc 0 0 0) 0x7f25000716f0 con 0x7f250006ec30 2023-12-16T21:36:53.127 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:36:53.125+0000 7f2517167700 1 -- 172.21.15.45:0/3344385015 --> [v2:172.21.15.169:6800/201299104,v1:172.21.15.169:6801/201299104] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7f251005e800 con 0x7f250006ec30 2023-12-16T21:36:53.128 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:36:53.125+0000 7f250d7fa700 1 -- 172.21.15.45:0/3344385015 <== osd.3 v2:172.21.15.169:6800/201299104 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+12 (crc 0 0 0) 0x7f251005e800 con 0x7f250006ec30 2023-12-16T21:36:53.129 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:36:53.126+0000 7f2517167700 1 -- 172.21.15.45:0/3344385015 >> [v2:172.21.15.169:6800/201299104,v1:172.21.15.169:6801/201299104] conn(0x7f250006ec30 msgr2=0x7f2500071040 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:36:53.129 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:36:53.126+0000 7f2517167700 1 --2- 172.21.15.45:0/3344385015 >> [v2:172.21.15.169:6800/201299104,v1:172.21.15.169:6801/201299104] conn(0x7f250006ec30 0x7f2500071040 crc :-1 s=READY pgs=5 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:36:53.129 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:36:53.126+0000 7f2517167700 1 -- 172.21.15.45:0/3344385015 >> [v2:172.21.15.45:6800/2012105536,v1:172.21.15.45:6801/2012105536] conn(0x7f2500068f60 msgr2=0x7f250006b410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:36:53.129 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:36:53.126+0000 7f2517167700 1 --2- 172.21.15.45:0/3344385015 >> [v2:172.21.15.45:6800/2012105536,v1:172.21.15.45:6801/2012105536] conn(0x7f2500068f60 0x7f250006b410 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7f2510071940 tx=0x7f24fc009e30 comp rx=0 tx=0).stop 2023-12-16T21:36:53.129 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:36:53.126+0000 7f2517167700 1 -- 172.21.15.45:0/3344385015 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f2510102c00 msgr2=0x7f251006ff40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:36:53.129 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:36:53.126+0000 7f2517167700 1 --2- 172.21.15.45:0/3344385015 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f2510102c00 0x7f251006ff40 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7f250400b790 tx=0x7f2504009250 comp rx=0 tx=0).stop 2023-12-16T21:36:53.129 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:36:53.126+0000 7f2517167700 1 -- 172.21.15.45:0/3344385015 shutdown_connections 2023-12-16T21:36:53.129 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:36:53.126+0000 7f2517167700 1 --2- 172.21.15.45:0/3344385015 >> [v2:172.21.15.169:6800/201299104,v1:172.21.15.169:6801/201299104] conn(0x7f250006ec30 0x7f2500071040 unknown :-1 s=CLOSED pgs=5 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:36:53.129 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:36:53.126+0000 7f2517167700 1 --2- 172.21.15.45:0/3344385015 >> [v2:172.21.15.45:6800/2012105536,v1:172.21.15.45:6801/2012105536] conn(0x7f2500068f60 0x7f250006b410 unknown :-1 s=CLOSED pgs=68 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:36:53.130 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:36:53.126+0000 7f2517167700 1 --2- 172.21.15.45:0/3344385015 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f2510102c00 0x7f251006ff40 unknown :-1 s=CLOSED pgs=199 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:36:53.130 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:36:53.126+0000 7f2517167700 1 --2- 172.21.15.45:0/3344385015 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f25101035a0 0x7f2510070480 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:36:53.130 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:36:53.126+0000 7f2517167700 1 -- 172.21.15.45:0/3344385015 >> 172.21.15.45:0/3344385015 conn(0x7f25100f8790 msgr2=0x7f25100fa1b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-16T21:36:53.130 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:36:53.126+0000 7f2517167700 1 -- 172.21.15.45:0/3344385015 shutdown_connections 2023-12-16T21:36:53.130 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:36:53.126+0000 7f2517167700 1 -- 172.21.15.45:0/3344385015 wait complete. 2023-12-16T21:36:53.504 INFO:teuthology.orchestra.run.smithi045.stderr:Inferring config /var/lib/ceph/88ff363c-9c5a-11ee-95a5-87774f69a715/mon.smithi045/config 2023-12-16T21:36:53.526 INFO:teuthology.orchestra.run.smithi045.stdout:107374182412 2023-12-16T21:36:53.526 DEBUG:teuthology.orchestra.run.smithi045:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 88ff363c-9c5a-11ee-95a5-87774f69a715 -- ceph osd last-stat-seq osd.3 2023-12-16T21:36:53.761 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Dec 16 21:36:53 smithi045 ceph-mon[80763]: pgmap v97: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2023-12-16T21:36:53.830 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:36:53.798+0000 7fcd22fb3700 1 -- 172.21.15.45:0/2575095402 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fcd1c103590 msgr2=0x7fcd1c0fcc00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:36:53.830 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:36:53.798+0000 7fcd22fb3700 1 --2- 172.21.15.45:0/2575095402 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fcd1c103590 0x7fcd1c0fcc00 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7fcd08004940 tx=0x7fcd08009e30 comp rx=0 tx=0).stop 2023-12-16T21:36:53.830 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:36:53.798+0000 7fcd22fb3700 1 -- 172.21.15.45:0/2575095402 shutdown_connections 2023-12-16T21:36:53.830 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:36:53.798+0000 7fcd22fb3700 1 --2- 172.21.15.45:0/2575095402 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fcd1c103590 0x7fcd1c0fcc00 unknown :-1 s=CLOSED pgs=200 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:36:53.830 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:36:53.798+0000 7fcd22fb3700 1 --2- 172.21.15.45:0/2575095402 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fcd1c102bf0 0x7fcd1c102fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:36:53.830 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:36:53.798+0000 7fcd22fb3700 1 -- 172.21.15.45:0/2575095402 >> 172.21.15.45:0/2575095402 conn(0x7fcd1c0f87a0 msgr2=0x7fcd1c0fabb0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-16T21:36:53.830 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:36:53.798+0000 7fcd22fb3700 1 -- 172.21.15.45:0/2575095402 shutdown_connections 2023-12-16T21:36:53.830 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:36:53.799+0000 7fcd22fb3700 1 -- 172.21.15.45:0/2575095402 wait complete. 2023-12-16T21:36:53.830 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:36:53.799+0000 7fcd22fb3700 1 Processor -- start 2023-12-16T21:36:53.830 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:36:53.799+0000 7fcd22fb3700 1 -- start start 2023-12-16T21:36:53.830 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:36:53.800+0000 7fcd22fb3700 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fcd1c102bf0 0x7fcd1c1135c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:36:53.830 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:36:53.800+0000 7fcd22fb3700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fcd1c103590 0x7fcd1c113b00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:36:53.831 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:36:53.800+0000 7fcd22fb3700 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcd1c114150 con 0x7fcd1c103590 2023-12-16T21:36:53.831 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:36:53.800+0000 7fcd22fb3700 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcd1c10d480 con 0x7fcd1c102bf0 2023-12-16T21:36:53.831 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:36:53.800+0000 7fcd1bfff700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fcd1c103590 0x7fcd1c113b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:36:53.831 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:36:53.800+0000 7fcd1bfff700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fcd1c103590 0x7fcd1c113b00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:46732/0 (socket says 172.21.15.45:46732) 2023-12-16T21:36:53.831 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:36:53.800+0000 7fcd1bfff700 1 -- 172.21.15.45:0/714433677 learned_addr learned my addr 172.21.15.45:0/714433677 (peer_addr_for_me v2:172.21.15.45:0/0) 2023-12-16T21:36:53.831 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:36:53.800+0000 7fcd20d4f700 1 --2- 172.21.15.45:0/714433677 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fcd1c102bf0 0x7fcd1c1135c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:36:53.831 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:36:53.800+0000 7fcd1bfff700 1 -- 172.21.15.45:0/714433677 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fcd1c102bf0 msgr2=0x7fcd1c1135c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:36:53.832 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:36:53.800+0000 7fcd1bfff700 1 --2- 172.21.15.45:0/714433677 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fcd1c102bf0 0x7fcd1c1135c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:36:53.832 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:36:53.800+0000 7fcd1bfff700 1 -- 172.21.15.45:0/714433677 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcd08005040 con 0x7fcd1c103590 2023-12-16T21:36:53.832 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:36:53.801+0000 7fcd1bfff700 1 --2- 172.21.15.45:0/714433677 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fcd1c103590 0x7fcd1c113b00 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7fcd08008d80 tx=0x7fcd08009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:36:53.832 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:36:53.801+0000 7fcd197fa700 1 -- 172.21.15.45:0/714433677 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fcd08021070 con 0x7fcd1c103590 2023-12-16T21:36:53.832 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:36:53.801+0000 7fcd197fa700 1 -- 172.21.15.45:0/714433677 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fcd080093b0 con 0x7fcd1c103590 2023-12-16T21:36:53.832 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:36:53.801+0000 7fcd22fb3700 1 -- 172.21.15.45:0/714433677 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcd1c10d700 con 0x7fcd1c103590 2023-12-16T21:36:53.832 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:36:53.801+0000 7fcd197fa700 1 -- 172.21.15.45:0/714433677 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fcd0800e070 con 0x7fcd1c103590 2023-12-16T21:36:53.832 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:36:53.801+0000 7fcd22fb3700 1 -- 172.21.15.45:0/714433677 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcd1c10dbe0 con 0x7fcd1c103590 2023-12-16T21:36:53.832 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:36:53.802+0000 7fcd22fb3700 1 -- 172.21.15.45:0/714433677 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7fcd1c10e100 con 0x7fcd1c103590 2023-12-16T21:36:53.832 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:36:53.802+0000 7fcd197fa700 1 -- 172.21.15.45:0/714433677 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 19) v1 ==== 86447+0+0 (secure 0 0 0) 0x7fcd08013a30 con 0x7fcd1c103590 2023-12-16T21:36:53.832 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:36:53.803+0000 7fcd197fa700 1 --2- 172.21.15.45:0/714433677 >> [v2:172.21.15.45:6800/2012105536,v1:172.21.15.45:6801/2012105536] conn(0x7fcd0c068f70 0x7fcd0c06b420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:36:53.832 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:36:53.803+0000 7fcd197fa700 1 -- 172.21.15.45:0/714433677 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(37..37 src has 1..37) v4 ==== 4634+0+0 (secure 0 0 0) 0x7fcd08087b30 con 0x7fcd1c103590 2023-12-16T21:36:53.832 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:36:53.803+0000 7fcd20d4f700 1 --2- 172.21.15.45:0/714433677 >> [v2:172.21.15.45:6800/2012105536,v1:172.21.15.45:6801/2012105536] conn(0x7fcd0c068f70 0x7fcd0c06b420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:36:53.832 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:36:53.803+0000 7fcd197fa700 1 --2- 172.21.15.45:0/714433677 >> [v2:172.21.15.45:6818/844221874,v1:172.21.15.45:6819/844221874] conn(0x7fcd0c06ec40 0x7fcd0c071050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:36:53.833 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:36:53.803+0000 7fcd197fa700 1 -- 172.21.15.45:0/714433677 --> [v2:172.21.15.45:6818/844221874,v1:172.21.15.45:6819/844221874] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7fcd0c071700 con 0x7fcd0c06ec40 2023-12-16T21:36:53.833 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:36:53.803+0000 7fcd197fa700 1 -- 172.21.15.45:0/714433677 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_get_version_reply(handle=1 version=37) v2 ==== 24+0+0 (secure 0 0 0) 0x7fcd0808c1c0 con 0x7fcd1c103590 2023-12-16T21:36:53.833 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:36:53.803+0000 7fcd21550700 1 --2- 172.21.15.45:0/714433677 >> [v2:172.21.15.45:6818/844221874,v1:172.21.15.45:6819/844221874] conn(0x7fcd0c06ec40 0x7fcd0c071050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:36:53.833 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:36:53.803+0000 7fcd20d4f700 1 --2- 172.21.15.45:0/714433677 >> [v2:172.21.15.45:6800/2012105536,v1:172.21.15.45:6801/2012105536] conn(0x7fcd0c068f70 0x7fcd0c06b420 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7fcd10009b80 tx=0x7fcd1000b040 comp rx=0 tx=0).ready entity=mgr.14214 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:36:53.833 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:36:53.804+0000 7fcd21550700 1 --2- 172.21.15.45:0/714433677 >> [v2:172.21.15.45:6818/844221874,v1:172.21.15.45:6819/844221874] conn(0x7fcd0c06ec40 0x7fcd0c071050 crc :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:36:53.833 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:36:53.804+0000 7fcd197fa700 1 -- 172.21.15.45:0/714433677 <== osd.2 v2:172.21.15.45:6818/844221874 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+23999 (crc 0 0 0) 0x7fcd0c071700 con 0x7fcd0c06ec40 2023-12-16T21:36:53.833 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:36:53.821+0000 7fcd22fb3700 1 -- 172.21.15.45:0/714433677 --> [v2:172.21.15.45:6818/844221874,v1:172.21.15.45:6819/844221874] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7fcd1c05e340 con 0x7fcd0c06ec40 2023-12-16T21:36:53.833 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:36:53.821+0000 7fcd197fa700 1 -- 172.21.15.45:0/714433677 <== osd.2 v2:172.21.15.45:6818/844221874 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+11 (crc 0 0 0) 0x7fcd1c05e340 con 0x7fcd0c06ec40 2023-12-16T21:36:53.833 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:36:53.822+0000 7fcd22fb3700 1 -- 172.21.15.45:0/714433677 >> [v2:172.21.15.45:6818/844221874,v1:172.21.15.45:6819/844221874] conn(0x7fcd0c06ec40 msgr2=0x7fcd0c071050 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:36:53.833 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:36:53.822+0000 7fcd22fb3700 1 --2- 172.21.15.45:0/714433677 >> [v2:172.21.15.45:6818/844221874,v1:172.21.15.45:6819/844221874] conn(0x7fcd0c06ec40 0x7fcd0c071050 crc :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:36:53.833 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:36:53.822+0000 7fcd22fb3700 1 -- 172.21.15.45:0/714433677 >> [v2:172.21.15.45:6800/2012105536,v1:172.21.15.45:6801/2012105536] conn(0x7fcd0c068f70 msgr2=0x7fcd0c06b420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:36:53.833 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:36:53.822+0000 7fcd22fb3700 1 --2- 172.21.15.45:0/714433677 >> [v2:172.21.15.45:6800/2012105536,v1:172.21.15.45:6801/2012105536] conn(0x7fcd0c068f70 0x7fcd0c06b420 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7fcd10009b80 tx=0x7fcd1000b040 comp rx=0 tx=0).stop 2023-12-16T21:36:53.833 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:36:53.822+0000 7fcd22fb3700 1 -- 172.21.15.45:0/714433677 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fcd1c103590 msgr2=0x7fcd1c113b00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:36:53.833 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:36:53.822+0000 7fcd22fb3700 1 --2- 172.21.15.45:0/714433677 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fcd1c103590 0x7fcd1c113b00 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7fcd08008d80 tx=0x7fcd08009e30 comp rx=0 tx=0).stop 2023-12-16T21:36:53.834 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:36:53.822+0000 7fcd22fb3700 1 -- 172.21.15.45:0/714433677 shutdown_connections 2023-12-16T21:36:53.834 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:36:53.822+0000 7fcd22fb3700 1 --2- 172.21.15.45:0/714433677 >> [v2:172.21.15.45:6818/844221874,v1:172.21.15.45:6819/844221874] conn(0x7fcd0c06ec40 0x7fcd0c071050 unknown :-1 s=CLOSED pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:36:53.834 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:36:53.822+0000 7fcd22fb3700 1 --2- 172.21.15.45:0/714433677 >> [v2:172.21.15.45:6800/2012105536,v1:172.21.15.45:6801/2012105536] conn(0x7fcd0c068f70 0x7fcd0c06b420 unknown :-1 s=CLOSED pgs=69 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:36:53.834 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:36:53.822+0000 7fcd22fb3700 1 --2- 172.21.15.45:0/714433677 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fcd1c102bf0 0x7fcd1c1135c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:36:53.834 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:36:53.822+0000 7fcd22fb3700 1 --2- 172.21.15.45:0/714433677 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fcd1c103590 0x7fcd1c113b00 unknown :-1 s=CLOSED pgs=201 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:36:53.834 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:36:53.822+0000 7fcd22fb3700 1 -- 172.21.15.45:0/714433677 >> 172.21.15.45:0/714433677 conn(0x7fcd1c0f87a0 msgr2=0x7fcd1c0fa290 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-16T21:36:53.834 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:36:53.822+0000 7fcd22fb3700 1 -- 172.21.15.45:0/714433677 shutdown_connections 2023-12-16T21:36:53.834 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:36:53.822+0000 7fcd22fb3700 1 -- 172.21.15.45:0/714433677 wait complete. 2023-12-16T21:36:53.883 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Dec 16 21:36:53 smithi169 ceph-mon[88350]: pgmap v97: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2023-12-16T21:36:54.505 INFO:teuthology.orchestra.run.smithi045.stdout:77309411342 2023-12-16T21:36:54.505 DEBUG:teuthology.orchestra.run.smithi045:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 88ff363c-9c5a-11ee-95a5-87774f69a715 -- ceph osd last-stat-seq osd.2 2023-12-16T21:36:54.607 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:36:54.603+0000 7f25b0226700 1 -- 172.21.15.45:0/3229017566 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f25a80fbb40 msgr2=0x7f25a80fbfb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:36:54.607 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:36:54.603+0000 7f25b0226700 1 --2- 172.21.15.45:0/3229017566 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f25a80fbb40 0x7f25a80fbfb0 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7f2598004940 tx=0x7f2598009e30 comp rx=0 tx=0).stop 2023-12-16T21:36:54.607 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:36:54.603+0000 7f25b0226700 1 -- 172.21.15.45:0/3229017566 shutdown_connections 2023-12-16T21:36:54.607 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:36:54.603+0000 7f25b0226700 1 --2- 172.21.15.45:0/3229017566 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f25a80fbb40 0x7f25a80fbfb0 unknown :-1 s=CLOSED pgs=202 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:36:54.607 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:36:54.603+0000 7f25b0226700 1 --2- 172.21.15.45:0/3229017566 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f25a8101b00 0x7f25a8101ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:36:54.607 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:36:54.603+0000 7f25b0226700 1 -- 172.21.15.45:0/3229017566 >> 172.21.15.45:0/3229017566 conn(0x7f25a80719b0 msgr2=0x7f25a8071db0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-16T21:36:54.607 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:36:54.604+0000 7f25b0226700 1 -- 172.21.15.45:0/3229017566 shutdown_connections 2023-12-16T21:36:54.607 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:36:54.604+0000 7f25b0226700 1 -- 172.21.15.45:0/3229017566 wait complete. 2023-12-16T21:36:54.607 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:36:54.604+0000 7f25b0226700 1 Processor -- start 2023-12-16T21:36:54.607 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:36:54.604+0000 7f25b0226700 1 -- start start 2023-12-16T21:36:54.609 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:36:54.605+0000 7f25b0226700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f25a80fbb40 0x7f25a8113660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:36:54.609 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:36:54.605+0000 7f25b0226700 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f25a8101b00 0x7f25a8113be0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:36:54.609 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:36:54.605+0000 7f25b0226700 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f25a810d520 con 0x7f25a80fbb40 2023-12-16T21:36:54.609 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:36:54.605+0000 7f25b0226700 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f25a810d690 con 0x7f25a8101b00 2023-12-16T21:36:54.609 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:36:54.605+0000 7f25adfc2700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f25a80fbb40 0x7f25a8113660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:36:54.609 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:36:54.605+0000 7f25adfc2700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f25a80fbb40 0x7f25a8113660 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:46748/0 (socket says 172.21.15.45:46748) 2023-12-16T21:36:54.609 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:36:54.605+0000 7f25adfc2700 1 -- 172.21.15.45:0/3763201941 learned_addr learned my addr 172.21.15.45:0/3763201941 (peer_addr_for_me v2:172.21.15.45:0/0) 2023-12-16T21:36:54.609 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:36:54.605+0000 7f25ad7c1700 1 --2- 172.21.15.45:0/3763201941 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f25a8101b00 0x7f25a8113be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:36:54.609 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:36:54.605+0000 7f25adfc2700 1 -- 172.21.15.45:0/3763201941 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f25a8101b00 msgr2=0x7f25a8113be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:36:54.609 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:36:54.605+0000 7f25adfc2700 1 --2- 172.21.15.45:0/3763201941 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f25a8101b00 0x7f25a8113be0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:36:54.609 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:36:54.605+0000 7f25adfc2700 1 -- 172.21.15.45:0/3763201941 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2598005040 con 0x7f25a80fbb40 2023-12-16T21:36:54.610 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:36:54.605+0000 7f25adfc2700 1 --2- 172.21.15.45:0/3763201941 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f25a80fbb40 0x7f25a8113660 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7f25a400bbb0 tx=0x7f25a4009840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:36:54.610 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:36:54.605+0000 7f259e7fc700 1 -- 172.21.15.45:0/3763201941 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f25a40092e0 con 0x7f25a80fbb40 2023-12-16T21:36:54.610 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:36:54.605+0000 7f25b0226700 1 -- 172.21.15.45:0/3763201941 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f25a810d970 con 0x7f25a80fbb40 2023-12-16T21:36:54.610 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:36:54.605+0000 7f259e7fc700 1 -- 172.21.15.45:0/3763201941 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f25a400cb00 con 0x7f25a80fbb40 2023-12-16T21:36:54.610 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:36:54.606+0000 7f259e7fc700 1 -- 172.21.15.45:0/3763201941 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f25a4015d10 con 0x7f25a80fbb40 2023-12-16T21:36:54.610 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:36:54.606+0000 7f25b0226700 1 -- 172.21.15.45:0/3763201941 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f25a810deb0 con 0x7f25a80fbb40 2023-12-16T21:36:54.611 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:36:54.607+0000 7f25b0226700 1 -- 172.21.15.45:0/3763201941 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f25a810e350 con 0x7f25a80fbb40 2023-12-16T21:36:54.611 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:36:54.607+0000 7f259e7fc700 1 -- 172.21.15.45:0/3763201941 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 19) v1 ==== 86447+0+0 (secure 0 0 0) 0x7f25a4012070 con 0x7f25a80fbb40 2023-12-16T21:36:54.611 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:36:54.608+0000 7f259e7fc700 1 --2- 172.21.15.45:0/3763201941 >> [v2:172.21.15.45:6800/2012105536,v1:172.21.15.45:6801/2012105536] conn(0x7f2594068f60 0x7f259406b410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:36:54.611 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:36:54.608+0000 7f259e7fc700 1 -- 172.21.15.45:0/3763201941 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(37..37 src has 1..37) v4 ==== 4634+0+0 (secure 0 0 0) 0x7f25a4086690 con 0x7f25a80fbb40 2023-12-16T21:36:54.611 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:36:54.608+0000 7f25ad7c1700 1 --2- 172.21.15.45:0/3763201941 >> [v2:172.21.15.45:6800/2012105536,v1:172.21.15.45:6801/2012105536] conn(0x7f2594068f60 0x7f259406b410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:36:54.611 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:36:54.608+0000 7f259e7fc700 1 --2- 172.21.15.45:0/3763201941 >> [v2:172.21.15.169:6816/3826842549,v1:172.21.15.169:6817/3826842549] conn(0x7f259406ec30 0x7f2594071040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:36:54.611 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:36:54.608+0000 7f259e7fc700 1 -- 172.21.15.45:0/3763201941 --> [v2:172.21.15.169:6816/3826842549,v1:172.21.15.169:6817/3826842549] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f25940716f0 con 0x7f259406ec30 2023-12-16T21:36:54.611 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:36:54.608+0000 7f259e7fc700 1 -- 172.21.15.45:0/3763201941 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_get_version_reply(handle=1 version=37) v2 ==== 24+0+0 (secure 0 0 0) 0x7f25a4088ce0 con 0x7f25a80fbb40 2023-12-16T21:36:54.612 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:36:54.608+0000 7f25ad7c1700 1 --2- 172.21.15.45:0/3763201941 >> [v2:172.21.15.45:6800/2012105536,v1:172.21.15.45:6801/2012105536] conn(0x7f2594068f60 0x7f259406b410 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f25a810ef00 tx=0x7f2598009e30 comp rx=0 tx=0).ready entity=mgr.14214 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:36:54.612 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:36:54.608+0000 7f25ae7c3700 1 --2- 172.21.15.45:0/3763201941 >> [v2:172.21.15.169:6816/3826842549,v1:172.21.15.169:6817/3826842549] conn(0x7f259406ec30 0x7f2594071040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:36:54.612 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:36:54.609+0000 7f25ae7c3700 1 --2- 172.21.15.45:0/3763201941 >> [v2:172.21.15.169:6816/3826842549,v1:172.21.15.169:6817/3826842549] conn(0x7f259406ec30 0x7f2594071040 crc :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:36:54.612 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:36:54.609+0000 7f259e7fc700 1 -- 172.21.15.45:0/3763201941 <== osd.5 v2:172.21.15.169:6816/3826842549 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+23999 (crc 0 0 0) 0x7f25940716f0 con 0x7f259406ec30 2023-12-16T21:36:54.612 INFO:teuthology.orchestra.run.smithi045.stderr:Inferring config /var/lib/ceph/88ff363c-9c5a-11ee-95a5-87774f69a715/mon.smithi045/config 2023-12-16T21:36:54.628 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:36:54.626+0000 7f25b0226700 1 -- 172.21.15.45:0/3763201941 --> [v2:172.21.15.169:6816/3826842549,v1:172.21.15.169:6817/3826842549] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7f25a8065de0 con 0x7f259406ec30 2023-12-16T21:36:54.628 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:36:54.626+0000 7f259e7fc700 1 -- 172.21.15.45:0/3763201941 <== osd.5 v2:172.21.15.169:6816/3826842549 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+12 (crc 0 0 0) 0x7f25a8065de0 con 0x7f259406ec30 2023-12-16T21:36:54.629 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:36:54.626+0000 7f25b0226700 1 -- 172.21.15.45:0/3763201941 >> [v2:172.21.15.169:6816/3826842549,v1:172.21.15.169:6817/3826842549] conn(0x7f259406ec30 msgr2=0x7f2594071040 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:36:54.630 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:36:54.626+0000 7f25b0226700 1 --2- 172.21.15.45:0/3763201941 >> [v2:172.21.15.169:6816/3826842549,v1:172.21.15.169:6817/3826842549] conn(0x7f259406ec30 0x7f2594071040 crc :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:36:54.630 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:36:54.627+0000 7f25b0226700 1 -- 172.21.15.45:0/3763201941 >> [v2:172.21.15.45:6800/2012105536,v1:172.21.15.45:6801/2012105536] conn(0x7f2594068f60 msgr2=0x7f259406b410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:36:54.630 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:36:54.627+0000 7f25b0226700 1 --2- 172.21.15.45:0/3763201941 >> [v2:172.21.15.45:6800/2012105536,v1:172.21.15.45:6801/2012105536] conn(0x7f2594068f60 0x7f259406b410 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f25a810ef00 tx=0x7f2598009e30 comp rx=0 tx=0).stop 2023-12-16T21:36:54.630 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:36:54.627+0000 7f25b0226700 1 -- 172.21.15.45:0/3763201941 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f25a80fbb40 msgr2=0x7f25a8113660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:36:54.630 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:36:54.627+0000 7f25b0226700 1 --2- 172.21.15.45:0/3763201941 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f25a80fbb40 0x7f25a8113660 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7f25a400bbb0 tx=0x7f25a4009840 comp rx=0 tx=0).stop 2023-12-16T21:36:54.630 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:36:54.627+0000 7f25b0226700 1 -- 172.21.15.45:0/3763201941 shutdown_connections 2023-12-16T21:36:54.630 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:36:54.627+0000 7f25b0226700 1 --2- 172.21.15.45:0/3763201941 >> [v2:172.21.15.169:6816/3826842549,v1:172.21.15.169:6817/3826842549] conn(0x7f259406ec30 0x7f2594071040 unknown :-1 s=CLOSED pgs=6 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:36:54.630 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:36:54.627+0000 7f25b0226700 1 --2- 172.21.15.45:0/3763201941 >> [v2:172.21.15.45:6800/2012105536,v1:172.21.15.45:6801/2012105536] conn(0x7f2594068f60 0x7f259406b410 unknown :-1 s=CLOSED pgs=70 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:36:54.631 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:36:54.627+0000 7f25b0226700 1 --2- 172.21.15.45:0/3763201941 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f25a80fbb40 0x7f25a8113660 unknown :-1 s=CLOSED pgs=203 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:36:54.631 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:36:54.627+0000 7f25b0226700 1 --2- 172.21.15.45:0/3763201941 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f25a8101b00 0x7f25a8113be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:36:54.631 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:36:54.627+0000 7f25b0226700 1 -- 172.21.15.45:0/3763201941 >> 172.21.15.45:0/3763201941 conn(0x7f25a80719b0 msgr2=0x7f25a8065410 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-16T21:36:54.631 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:36:54.627+0000 7f25b0226700 1 -- 172.21.15.45:0/3763201941 shutdown_connections 2023-12-16T21:36:54.631 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:36:54.627+0000 7f25b0226700 1 -- 172.21.15.45:0/3763201941 wait complete. 2023-12-16T21:36:54.896 INFO:teuthology.orchestra.run.smithi045.stdout:154618822662 2023-12-16T21:36:54.897 DEBUG:teuthology.orchestra.run.smithi045:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 88ff363c-9c5a-11ee-95a5-87774f69a715 -- ceph osd last-stat-seq osd.5 2023-12-16T21:36:55.600 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:36:55.596+0000 7f3606123700 1 -- 172.21.15.45:0/4206641582 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f3600065390 msgr2=0x7f3600065800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:36:55.601 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:36:55.596+0000 7f3606123700 1 --2- 172.21.15.45:0/4206641582 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f3600065390 0x7f3600065800 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7f35f4004940 tx=0x7f35f4009e30 comp rx=0 tx=0).stop 2023-12-16T21:36:55.601 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:36:55.596+0000 7f3606123700 1 -- 172.21.15.45:0/4206641582 shutdown_connections 2023-12-16T21:36:55.601 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:36:55.596+0000 7f3606123700 1 --2- 172.21.15.45:0/4206641582 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f3600065390 0x7f3600065800 unknown :-1 s=CLOSED pgs=204 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:36:55.601 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:36:55.596+0000 7f3606123700 1 --2- 172.21.15.45:0/4206641582 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f3600102c20 0x7f3600102ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:36:55.601 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:36:55.596+0000 7f3606123700 1 -- 172.21.15.45:0/4206641582 >> 172.21.15.45:0/4206641582 conn(0x7f36000719b0 msgr2=0x7f3600071db0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-16T21:36:55.601 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:36:55.596+0000 7f3606123700 1 -- 172.21.15.45:0/4206641582 shutdown_connections 2023-12-16T21:36:55.601 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:36:55.596+0000 7f3606123700 1 -- 172.21.15.45:0/4206641582 wait complete. 2023-12-16T21:36:55.601 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:36:55.597+0000 7f3606123700 1 Processor -- start 2023-12-16T21:36:55.601 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:36:55.597+0000 7f3606123700 1 -- start start 2023-12-16T21:36:55.601 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:36:55.597+0000 7f3606123700 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f3600065390 0x7f36000fbbf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:36:55.601 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:36:55.597+0000 7f3606123700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f3600102c20 0x7f36000fc130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:36:55.601 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:36:55.597+0000 7f3606123700 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f36000fc730 con 0x7f3600102c20 2023-12-16T21:36:55.601 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:36:55.597+0000 7f3606123700 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f36000fc8a0 con 0x7f3600065390 2023-12-16T21:36:55.602 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:36:55.597+0000 7f35feffd700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f3600102c20 0x7f36000fc130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:36:55.602 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:36:55.597+0000 7f35feffd700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f3600102c20 0x7f36000fc130 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:46770/0 (socket says 172.21.15.45:46770) 2023-12-16T21:36:55.602 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:36:55.597+0000 7f35feffd700 1 -- 172.21.15.45:0/1844410190 learned_addr learned my addr 172.21.15.45:0/1844410190 (peer_addr_for_me v2:172.21.15.45:0/0) 2023-12-16T21:36:55.602 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:36:55.598+0000 7f35ff7fe700 1 --2- 172.21.15.45:0/1844410190 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f3600065390 0x7f36000fbbf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:36:55.602 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:36:55.598+0000 7f35feffd700 1 -- 172.21.15.45:0/1844410190 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f3600065390 msgr2=0x7f36000fbbf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:36:55.602 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:36:55.598+0000 7f35feffd700 1 --2- 172.21.15.45:0/1844410190 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f3600065390 0x7f36000fbbf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:36:55.602 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:36:55.598+0000 7f35feffd700 1 -- 172.21.15.45:0/1844410190 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f35f4005040 con 0x7f3600102c20 2023-12-16T21:36:55.602 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:36:55.598+0000 7f35feffd700 1 --2- 172.21.15.45:0/1844410190 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f3600102c20 0x7f36000fc130 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x7f35f400b3b0 tx=0x7f35f4009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:36:55.602 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:36:55.598+0000 7f35e7fff700 1 -- 172.21.15.45:0/1844410190 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f35f400e040 con 0x7f3600102c20 2023-12-16T21:36:55.602 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:36:55.598+0000 7f35e7fff700 1 -- 172.21.15.45:0/1844410190 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f35f400bc50 con 0x7f3600102c20 2023-12-16T21:36:55.602 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:36:55.598+0000 7f3606123700 1 -- 172.21.15.45:0/1844410190 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3600119970 con 0x7f3600102c20 2023-12-16T21:36:55.602 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:36:55.598+0000 7f35e7fff700 1 -- 172.21.15.45:0/1844410190 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f35f4013420 con 0x7f3600102c20 2023-12-16T21:36:55.602 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:36:55.598+0000 7f3606123700 1 -- 172.21.15.45:0/1844410190 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3600119dd0 con 0x7f3600102c20 2023-12-16T21:36:55.603 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:36:55.599+0000 7f3606123700 1 -- 172.21.15.45:0/1844410190 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f360005e300 con 0x7f3600102c20 2023-12-16T21:36:55.603 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:36:55.600+0000 7f35e7fff700 1 -- 172.21.15.45:0/1844410190 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 19) v1 ==== 86447+0+0 (secure 0 0 0) 0x7f35f401c430 con 0x7f3600102c20 2023-12-16T21:36:55.604 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:36:55.600+0000 7f35e7fff700 1 --2- 172.21.15.45:0/1844410190 >> [v2:172.21.15.45:6800/2012105536,v1:172.21.15.45:6801/2012105536] conn(0x7f35e8068f60 0x7f35e806b410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:36:55.604 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:36:55.600+0000 7f35e7fff700 1 -- 172.21.15.45:0/1844410190 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(37..37 src has 1..37) v4 ==== 4634+0+0 (secure 0 0 0) 0x7f35f4087830 con 0x7f3600102c20 2023-12-16T21:36:55.604 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:36:55.600+0000 7f35ff7fe700 1 --2- 172.21.15.45:0/1844410190 >> [v2:172.21.15.45:6800/2012105536,v1:172.21.15.45:6801/2012105536] conn(0x7f35e8068f60 0x7f35e806b410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:36:55.604 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:36:55.600+0000 7f35e7fff700 1 --2- 172.21.15.45:0/1844410190 >> [v2:172.21.15.169:6808/3840494264,v1:172.21.15.169:6809/3840494264] conn(0x7f35e806ec30 0x7f35e8071040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:36:55.604 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:36:55.600+0000 7f35e7fff700 1 -- 172.21.15.45:0/1844410190 --> [v2:172.21.15.169:6808/3840494264,v1:172.21.15.169:6809/3840494264] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f35e80716f0 con 0x7f35e806ec30 2023-12-16T21:36:55.604 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:36:55.600+0000 7f35e7fff700 1 -- 172.21.15.45:0/1844410190 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_get_version_reply(handle=1 version=37) v2 ==== 24+0+0 (secure 0 0 0) 0x7f35f408c220 con 0x7f3600102c20 2023-12-16T21:36:55.604 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:36:55.600+0000 7f35ff7fe700 1 --2- 172.21.15.45:0/1844410190 >> [v2:172.21.15.45:6800/2012105536,v1:172.21.15.45:6801/2012105536] conn(0x7f35e8068f60 0x7f35e806b410 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7f35f0004990 tx=0x7f35f0009ee0 comp rx=0 tx=0).ready entity=mgr.14214 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:36:55.604 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:36:55.601+0000 7f35fffff700 1 --2- 172.21.15.45:0/1844410190 >> [v2:172.21.15.169:6808/3840494264,v1:172.21.15.169:6809/3840494264] conn(0x7f35e806ec30 0x7f35e8071040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:36:55.604 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:36:55.601+0000 7f35fffff700 1 --2- 172.21.15.45:0/1844410190 >> [v2:172.21.15.169:6808/3840494264,v1:172.21.15.169:6809/3840494264] conn(0x7f35e806ec30 0x7f35e8071040 crc :-1 s=READY pgs=5 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:36:55.609 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:36:55.602+0000 7f35e7fff700 1 -- 172.21.15.45:0/1844410190 <== osd.4 v2:172.21.15.169:6808/3840494264 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+23999 (crc 0 0 0) 0x7f35e80716f0 con 0x7f35e806ec30 2023-12-16T21:36:55.619 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:36:55.617+0000 7f3606123700 1 -- 172.21.15.45:0/1844410190 --> [v2:172.21.15.169:6808/3840494264,v1:172.21.15.169:6809/3840494264] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7f360005e730 con 0x7f35e806ec30 2023-12-16T21:36:55.620 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:36:55.618+0000 7f35e7fff700 1 -- 172.21.15.45:0/1844410190 <== osd.4 v2:172.21.15.169:6808/3840494264 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+12 (crc 0 0 0) 0x7f360005e730 con 0x7f35e806ec30 2023-12-16T21:36:55.621 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:36:55.618+0000 7f3606123700 1 -- 172.21.15.45:0/1844410190 >> [v2:172.21.15.169:6808/3840494264,v1:172.21.15.169:6809/3840494264] conn(0x7f35e806ec30 msgr2=0x7f35e8071040 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:36:55.621 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:36:55.618+0000 7f3606123700 1 --2- 172.21.15.45:0/1844410190 >> [v2:172.21.15.169:6808/3840494264,v1:172.21.15.169:6809/3840494264] conn(0x7f35e806ec30 0x7f35e8071040 crc :-1 s=READY pgs=5 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:36:55.621 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:36:55.618+0000 7f3606123700 1 -- 172.21.15.45:0/1844410190 >> [v2:172.21.15.45:6800/2012105536,v1:172.21.15.45:6801/2012105536] conn(0x7f35e8068f60 msgr2=0x7f35e806b410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:36:55.621 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:36:55.618+0000 7f3606123700 1 --2- 172.21.15.45:0/1844410190 >> [v2:172.21.15.45:6800/2012105536,v1:172.21.15.45:6801/2012105536] conn(0x7f35e8068f60 0x7f35e806b410 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7f35f0004990 tx=0x7f35f0009ee0 comp rx=0 tx=0).stop 2023-12-16T21:36:55.621 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:36:55.618+0000 7f3606123700 1 -- 172.21.15.45:0/1844410190 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f3600102c20 msgr2=0x7f36000fc130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:36:55.621 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:36:55.618+0000 7f3606123700 1 --2- 172.21.15.45:0/1844410190 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f3600102c20 0x7f36000fc130 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x7f35f400b3b0 tx=0x7f35f4009e30 comp rx=0 tx=0).stop 2023-12-16T21:36:55.621 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:36:55.618+0000 7f3606123700 1 -- 172.21.15.45:0/1844410190 shutdown_connections 2023-12-16T21:36:55.621 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:36:55.618+0000 7f3606123700 1 --2- 172.21.15.45:0/1844410190 >> [v2:172.21.15.169:6808/3840494264,v1:172.21.15.169:6809/3840494264] conn(0x7f35e806ec30 0x7f35e8071040 unknown :-1 s=CLOSED pgs=5 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:36:55.621 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:36:55.618+0000 7f3606123700 1 --2- 172.21.15.45:0/1844410190 >> [v2:172.21.15.45:6800/2012105536,v1:172.21.15.45:6801/2012105536] conn(0x7f35e8068f60 0x7f35e806b410 unknown :-1 s=CLOSED pgs=71 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:36:55.621 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:36:55.618+0000 7f3606123700 1 --2- 172.21.15.45:0/1844410190 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f3600065390 0x7f36000fbbf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:36:55.622 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:36:55.618+0000 7f3606123700 1 --2- 172.21.15.45:0/1844410190 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f3600102c20 0x7f36000fc130 unknown :-1 s=CLOSED pgs=205 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:36:55.622 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:36:55.618+0000 7f3606123700 1 -- 172.21.15.45:0/1844410190 >> 172.21.15.45:0/1844410190 conn(0x7f36000719b0 msgr2=0x7f36000fad10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-16T21:36:55.622 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:36:55.618+0000 7f3606123700 1 -- 172.21.15.45:0/1844410190 shutdown_connections 2023-12-16T21:36:55.622 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:36:55.618+0000 7f3606123700 1 -- 172.21.15.45:0/1844410190 wait complete. 2023-12-16T21:36:55.879 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Dec 16 21:36:55 smithi045 ceph-mon[80763]: pgmap v98: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2023-12-16T21:36:55.883 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Dec 16 21:36:55 smithi169 ceph-mon[88350]: pgmap v98: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2023-12-16T21:36:55.904 INFO:teuthology.orchestra.run.smithi045.stdout:137438953481 2023-12-16T21:36:55.905 DEBUG:teuthology.orchestra.run.smithi045:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 88ff363c-9c5a-11ee-95a5-87774f69a715 -- ceph osd last-stat-seq osd.4 2023-12-16T21:36:56.073 INFO:teuthology.orchestra.run.smithi045.stderr:Inferring config /var/lib/ceph/88ff363c-9c5a-11ee-95a5-87774f69a715/mon.smithi045/config 2023-12-16T21:36:56.325 INFO:teuthology.orchestra.run.smithi045.stderr:Inferring config /var/lib/ceph/88ff363c-9c5a-11ee-95a5-87774f69a715/mon.smithi045/config 2023-12-16T21:36:56.893 INFO:teuthology.orchestra.run.smithi045.stderr:Inferring config /var/lib/ceph/88ff363c-9c5a-11ee-95a5-87774f69a715/mon.smithi045/config 2023-12-16T21:36:56.997 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:36:56.993+0000 7f133d0fa700 1 -- 172.21.15.45:0/3927237670 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f1338103590 msgr2=0x7f13380fcc00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:36:56.997 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:36:56.993+0000 7f133d0fa700 1 --2- 172.21.15.45:0/3927237670 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f1338103590 0x7f13380fcc00 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7f132c004940 tx=0x7f132c009e30 comp rx=0 tx=0).stop 2023-12-16T21:36:56.997 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:36:56.993+0000 7f133d0fa700 1 -- 172.21.15.45:0/3927237670 shutdown_connections 2023-12-16T21:36:56.997 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:36:56.993+0000 7f133d0fa700 1 --2- 172.21.15.45:0/3927237670 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f1338103590 0x7f13380fcc00 unknown :-1 s=CLOSED pgs=206 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:36:56.997 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:36:56.993+0000 7f133d0fa700 1 --2- 172.21.15.45:0/3927237670 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f1338102bf0 0x7f1338102fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:36:56.997 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:36:56.993+0000 7f133d0fa700 1 -- 172.21.15.45:0/3927237670 >> 172.21.15.45:0/3927237670 conn(0x7f13380f87a0 msgr2=0x7f13380fabb0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-16T21:36:56.998 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:36:56.994+0000 7f133d0fa700 1 -- 172.21.15.45:0/3927237670 shutdown_connections 2023-12-16T21:36:56.998 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:36:56.994+0000 7f133d0fa700 1 -- 172.21.15.45:0/3927237670 wait complete. 2023-12-16T21:36:56.999 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:36:56.994+0000 7f133d0fa700 1 Processor -- start 2023-12-16T21:36:56.999 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:36:56.994+0000 7f133d0fa700 1 -- start start 2023-12-16T21:36:56.999 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:36:56.995+0000 7f133d0fa700 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f1338102bf0 0x7f13381136a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:36:56.999 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:36:56.995+0000 7f133d0fa700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f1338103590 0x7f1338113be0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:36:56.999 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:36:56.995+0000 7f133d0fa700 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1338114230 con 0x7f1338103590 2023-12-16T21:36:56.999 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:36:56.995+0000 7f133d0fa700 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f133810d580 con 0x7f1338102bf0 2023-12-16T21:36:56.999 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:36:56.995+0000 7f133659c700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f1338103590 0x7f1338113be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:36:56.999 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:36:56.995+0000 7f133659c700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f1338103590 0x7f1338113be0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:46776/0 (socket says 172.21.15.45:46776) 2023-12-16T21:36:56.999 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:36:56.995+0000 7f133659c700 1 -- 172.21.15.45:0/3527761043 learned_addr learned my addr 172.21.15.45:0/3527761043 (peer_addr_for_me v2:172.21.15.45:0/0) 2023-12-16T21:36:57.000 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:36:56.995+0000 7f1336d9d700 1 --2- 172.21.15.45:0/3527761043 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f1338102bf0 0x7f13381136a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:36:57.000 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:36:56.995+0000 7f133659c700 1 -- 172.21.15.45:0/3527761043 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f1338102bf0 msgr2=0x7f13381136a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:36:57.000 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:36:56.995+0000 7f133659c700 1 --2- 172.21.15.45:0/3527761043 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f1338102bf0 0x7f13381136a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:36:57.000 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:36:56.995+0000 7f133659c700 1 -- 172.21.15.45:0/3527761043 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f132c005040 con 0x7f1338103590 2023-12-16T21:36:57.000 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:36:56.995+0000 7f133659c700 1 --2- 172.21.15.45:0/3527761043 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f1338103590 0x7f1338113be0 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 crypto rx=0x7f132c009fd0 tx=0x7f132c009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:36:57.000 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:36:56.995+0000 7f131f7fe700 1 -- 172.21.15.45:0/3527761043 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f132c00e040 con 0x7f1338103590 2023-12-16T21:36:57.000 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:36:56.995+0000 7f131f7fe700 1 -- 172.21.15.45:0/3527761043 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f132c013420 con 0x7f1338103590 2023-12-16T21:36:57.000 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:36:56.995+0000 7f133d0fa700 1 -- 172.21.15.45:0/3527761043 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f133810d800 con 0x7f1338103590 2023-12-16T21:36:57.000 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:36:56.995+0000 7f131f7fe700 1 -- 172.21.15.45:0/3527761043 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f132c01c630 con 0x7f1338103590 2023-12-16T21:36:57.000 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:36:56.996+0000 7f133d0fa700 1 -- 172.21.15.45:0/3527761043 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f133810dd40 con 0x7f1338103590 2023-12-16T21:36:57.002 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:36:56.996+0000 7f133d0fa700 1 -- 172.21.15.45:0/3527761043 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f133810e1e0 con 0x7f1338103590 2023-12-16T21:36:57.002 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:36:56.997+0000 7f131f7fe700 1 -- 172.21.15.45:0/3527761043 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 19) v1 ==== 86447+0+0 (secure 0 0 0) 0x7f132c013a00 con 0x7f1338103590 2023-12-16T21:36:57.002 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:36:56.997+0000 7f131f7fe700 1 --2- 172.21.15.45:0/3527761043 >> [v2:172.21.15.45:6800/2012105536,v1:172.21.15.45:6801/2012105536] conn(0x7f1320068f60 0x7f132006b410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:36:57.002 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:36:56.997+0000 7f1336d9d700 1 --2- 172.21.15.45:0/3527761043 >> [v2:172.21.15.45:6800/2012105536,v1:172.21.15.45:6801/2012105536] conn(0x7f1320068f60 0x7f132006b410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:36:57.002 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:36:56.997+0000 7f131f7fe700 1 -- 172.21.15.45:0/3527761043 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(37..37 src has 1..37) v4 ==== 4634+0+0 (secure 0 0 0) 0x7f132c087ce0 con 0x7f1338103590 2023-12-16T21:36:57.002 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:36:56.998+0000 7f131f7fe700 1 --2- 172.21.15.45:0/3527761043 >> [v2:172.21.15.45:6802/1443487903,v1:172.21.15.45:6803/1443487903] conn(0x7f132006ec30 0x7f1320071040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:36:57.002 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:36:56.998+0000 7f131f7fe700 1 -- 172.21.15.45:0/3527761043 --> [v2:172.21.15.45:6802/1443487903,v1:172.21.15.45:6803/1443487903] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f13200716f0 con 0x7f132006ec30 2023-12-16T21:36:57.002 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:36:56.998+0000 7f131f7fe700 1 -- 172.21.15.45:0/3527761043 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_get_version_reply(handle=1 version=37) v2 ==== 24+0+0 (secure 0 0 0) 0x7f132c08d220 con 0x7f1338103590 2023-12-16T21:36:57.002 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:36:56.998+0000 7f133759e700 1 --2- 172.21.15.45:0/3527761043 >> [v2:172.21.15.45:6802/1443487903,v1:172.21.15.45:6803/1443487903] conn(0x7f132006ec30 0x7f1320071040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:36:57.002 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:36:56.998+0000 7f1336d9d700 1 --2- 172.21.15.45:0/3527761043 >> [v2:172.21.15.45:6800/2012105536,v1:172.21.15.45:6801/2012105536] conn(0x7f1320068f60 0x7f132006b410 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f1328004f40 tx=0x7f132800b040 comp rx=0 tx=0).ready entity=mgr.14214 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:36:57.002 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:36:56.998+0000 7f133759e700 1 --2- 172.21.15.45:0/3527761043 >> [v2:172.21.15.45:6802/1443487903,v1:172.21.15.45:6803/1443487903] conn(0x7f132006ec30 0x7f1320071040 crc :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:36:57.003 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:36:56.999+0000 7f131f7fe700 1 -- 172.21.15.45:0/3527761043 <== osd.0 v2:172.21.15.45:6802/1443487903 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+23999 (crc 0 0 0) 0x7f13200716f0 con 0x7f132006ec30 2023-12-16T21:36:57.017 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:36:57.014+0000 7f133d0fa700 1 -- 172.21.15.45:0/3527761043 --> [v2:172.21.15.45:6802/1443487903,v1:172.21.15.45:6803/1443487903] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7f133805e340 con 0x7f132006ec30 2023-12-16T21:36:57.017 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:36:57.014+0000 7f131f7fe700 1 -- 172.21.15.45:0/3527761043 <== osd.0 v2:172.21.15.45:6802/1443487903 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+11 (crc 0 0 0) 0x7f133805e340 con 0x7f132006ec30 2023-12-16T21:36:57.018 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:36:57.014+0000 7f133d0fa700 1 -- 172.21.15.45:0/3527761043 >> [v2:172.21.15.45:6802/1443487903,v1:172.21.15.45:6803/1443487903] conn(0x7f132006ec30 msgr2=0x7f1320071040 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:36:57.018 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:36:57.014+0000 7f133d0fa700 1 --2- 172.21.15.45:0/3527761043 >> [v2:172.21.15.45:6802/1443487903,v1:172.21.15.45:6803/1443487903] conn(0x7f132006ec30 0x7f1320071040 crc :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:36:57.018 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:36:57.015+0000 7f133d0fa700 1 -- 172.21.15.45:0/3527761043 >> [v2:172.21.15.45:6800/2012105536,v1:172.21.15.45:6801/2012105536] conn(0x7f1320068f60 msgr2=0x7f132006b410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:36:57.018 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:36:57.015+0000 7f133d0fa700 1 --2- 172.21.15.45:0/3527761043 >> [v2:172.21.15.45:6800/2012105536,v1:172.21.15.45:6801/2012105536] conn(0x7f1320068f60 0x7f132006b410 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f1328004f40 tx=0x7f132800b040 comp rx=0 tx=0).stop 2023-12-16T21:36:57.018 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:36:57.015+0000 7f133d0fa700 1 -- 172.21.15.45:0/3527761043 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f1338103590 msgr2=0x7f1338113be0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:36:57.019 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:36:57.015+0000 7f133d0fa700 1 --2- 172.21.15.45:0/3527761043 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f1338103590 0x7f1338113be0 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 crypto rx=0x7f132c009fd0 tx=0x7f132c009e30 comp rx=0 tx=0).stop 2023-12-16T21:36:57.019 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:36:57.015+0000 7f133d0fa700 1 -- 172.21.15.45:0/3527761043 shutdown_connections 2023-12-16T21:36:57.019 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:36:57.015+0000 7f133d0fa700 1 --2- 172.21.15.45:0/3527761043 >> [v2:172.21.15.45:6802/1443487903,v1:172.21.15.45:6803/1443487903] conn(0x7f132006ec30 0x7f1320071040 unknown :-1 s=CLOSED pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:36:57.019 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:36:57.015+0000 7f133d0fa700 1 --2- 172.21.15.45:0/3527761043 >> [v2:172.21.15.45:6800/2012105536,v1:172.21.15.45:6801/2012105536] conn(0x7f1320068f60 0x7f132006b410 unknown :-1 s=CLOSED pgs=72 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:36:57.019 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:36:57.015+0000 7f133d0fa700 1 --2- 172.21.15.45:0/3527761043 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f1338102bf0 0x7f13381136a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:36:57.019 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:36:57.015+0000 7f133d0fa700 1 --2- 172.21.15.45:0/3527761043 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f1338103590 0x7f1338113be0 unknown :-1 s=CLOSED pgs=207 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:36:57.019 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:36:57.015+0000 7f133d0fa700 1 -- 172.21.15.45:0/3527761043 >> 172.21.15.45:0/3527761043 conn(0x7f13380f87a0 msgr2=0x7f13380fa3d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-16T21:36:57.019 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:36:57.015+0000 7f133d0fa700 1 -- 172.21.15.45:0/3527761043 shutdown_connections 2023-12-16T21:36:57.019 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:36:57.015+0000 7f133d0fa700 1 -- 172.21.15.45:0/3527761043 wait complete. 2023-12-16T21:36:57.566 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Dec 16 21:36:57 smithi045 ceph-mon[80763]: pgmap v99: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2023-12-16T21:36:57.633 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Dec 16 21:36:57 smithi169 ceph-mon[88350]: pgmap v99: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2023-12-16T21:36:57.844 INFO:teuthology.orchestra.run.smithi045.stderr:Inferring config /var/lib/ceph/88ff363c-9c5a-11ee-95a5-87774f69a715/mon.smithi045/config 2023-12-16T21:36:57.864 INFO:teuthology.orchestra.run.smithi045.stderr:Inferring config /var/lib/ceph/88ff363c-9c5a-11ee-95a5-87774f69a715/mon.smithi045/config 2023-12-16T21:36:57.916 INFO:teuthology.orchestra.run.smithi045.stdout:38654705686 2023-12-16T21:36:57.949 DEBUG:teuthology.orchestra.run.smithi045:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 88ff363c-9c5a-11ee-95a5-87774f69a715 -- ceph osd last-stat-seq osd.0 2023-12-16T21:36:57.965 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:36:57.961+0000 7f11cde60700 1 -- 172.21.15.45:0/1476975181 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f11c81035a0 msgr2=0x7f11c80fcc10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:36:57.965 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:36:57.961+0000 7f11cde60700 1 --2- 172.21.15.45:0/1476975181 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f11c81035a0 0x7f11c80fcc10 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 crypto rx=0x7f11bc004940 tx=0x7f11bc009e30 comp rx=0 tx=0).stop 2023-12-16T21:36:57.965 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:36:57.961+0000 7f11cde60700 1 -- 172.21.15.45:0/1476975181 shutdown_connections 2023-12-16T21:36:57.965 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:36:57.961+0000 7f11cde60700 1 --2- 172.21.15.45:0/1476975181 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f11c81035a0 0x7f11c80fcc10 unknown :-1 s=CLOSED pgs=208 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:36:57.965 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:36:57.961+0000 7f11cde60700 1 --2- 172.21.15.45:0/1476975181 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f11c8102c00 0x7f11c8102fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:36:57.965 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:36:57.961+0000 7f11cde60700 1 -- 172.21.15.45:0/1476975181 >> 172.21.15.45:0/1476975181 conn(0x7f11c80f8790 msgr2=0x7f11c80faba0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-16T21:36:57.965 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:36:57.961+0000 7f11cde60700 1 -- 172.21.15.45:0/1476975181 shutdown_connections 2023-12-16T21:36:57.965 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:36:57.961+0000 7f11cde60700 1 -- 172.21.15.45:0/1476975181 wait complete. 2023-12-16T21:36:57.965 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:36:57.962+0000 7f11cde60700 1 Processor -- start 2023-12-16T21:36:57.967 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:36:57.962+0000 7f11cde60700 1 -- start start 2023-12-16T21:36:57.967 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:36:57.962+0000 7f11cde60700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f11c8102c00 0x7f11c8115830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:36:57.967 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:36:57.962+0000 7f11cde60700 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f11c81035a0 0x7f11c8115d70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:36:57.967 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:36:57.962+0000 7f11cde60700 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f11c81163c0 con 0x7f11c8102c00 2023-12-16T21:36:57.967 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:36:57.962+0000 7f11cde60700 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f11c810f6f0 con 0x7f11c81035a0 2023-12-16T21:36:57.967 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:36:57.962+0000 7f11c77fe700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f11c8102c00 0x7f11c8115830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:36:57.967 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:36:57.962+0000 7f11c77fe700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f11c8102c00 0x7f11c8115830 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:46780/0 (socket says 172.21.15.45:46780) 2023-12-16T21:36:57.967 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:36:57.962+0000 7f11c77fe700 1 -- 172.21.15.45:0/3674570399 learned_addr learned my addr 172.21.15.45:0/3674570399 (peer_addr_for_me v2:172.21.15.45:0/0) 2023-12-16T21:36:57.967 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:36:57.962+0000 7f11c6ffd700 1 --2- 172.21.15.45:0/3674570399 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f11c81035a0 0x7f11c8115d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:36:57.967 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:36:57.962+0000 7f11c77fe700 1 -- 172.21.15.45:0/3674570399 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f11c81035a0 msgr2=0x7f11c8115d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:36:57.967 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:36:57.962+0000 7f11c77fe700 1 --2- 172.21.15.45:0/3674570399 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f11c81035a0 0x7f11c8115d70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:36:57.968 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:36:57.962+0000 7f11c77fe700 1 -- 172.21.15.45:0/3674570399 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f11bc005040 con 0x7f11c8102c00 2023-12-16T21:36:57.968 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:36:57.962+0000 7f11c6ffd700 1 --2- 172.21.15.45:0/3674570399 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f11c81035a0 0x7f11c8115d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-12-16T21:36:57.968 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:36:57.963+0000 7f11c77fe700 1 --2- 172.21.15.45:0/3674570399 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f11c8102c00 0x7f11c8115830 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0x7f11b8002a50 tx=0x7f11b80046b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:36:57.968 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:36:57.963+0000 7f11affff700 1 -- 172.21.15.45:0/3674570399 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f11b8007cc0 con 0x7f11c8102c00 2023-12-16T21:36:57.968 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:36:57.963+0000 7f11cde60700 1 -- 172.21.15.45:0/3674570399 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f11c810f9d0 con 0x7f11c8102c00 2023-12-16T21:36:57.968 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:36:57.963+0000 7f11affff700 1 -- 172.21.15.45:0/3674570399 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f11b800f040 con 0x7f11c8102c00 2023-12-16T21:36:57.968 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:36:57.963+0000 7f11affff700 1 -- 172.21.15.45:0/3674570399 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f11b8009540 con 0x7f11c8102c00 2023-12-16T21:36:57.968 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:36:57.963+0000 7f11cde60700 1 -- 172.21.15.45:0/3674570399 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f11c810ff10 con 0x7f11c8102c00 2023-12-16T21:36:57.969 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:36:57.964+0000 7f11cde60700 1 -- 172.21.15.45:0/3674570399 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f11c80fc390 con 0x7f11c8102c00 2023-12-16T21:36:57.969 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:36:57.964+0000 7f11affff700 1 -- 172.21.15.45:0/3674570399 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 19) v1 ==== 86447+0+0 (secure 0 0 0) 0x7f11b80096c0 con 0x7f11c8102c00 2023-12-16T21:36:57.969 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:36:57.965+0000 7f11affff700 1 --2- 172.21.15.45:0/3674570399 >> [v2:172.21.15.45:6800/2012105536,v1:172.21.15.45:6801/2012105536] conn(0x7f11b0068f60 0x7f11b006b410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:36:57.969 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:36:57.965+0000 7f11affff700 1 -- 172.21.15.45:0/3674570399 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(37..37 src has 1..37) v4 ==== 4634+0+0 (secure 0 0 0) 0x7f11b8086860 con 0x7f11c8102c00 2023-12-16T21:36:57.969 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:36:57.965+0000 7f11c6ffd700 1 --2- 172.21.15.45:0/3674570399 >> [v2:172.21.15.45:6800/2012105536,v1:172.21.15.45:6801/2012105536] conn(0x7f11b0068f60 0x7f11b006b410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:36:57.969 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:36:57.965+0000 7f11c6ffd700 1 --2- 172.21.15.45:0/3674570399 >> [v2:172.21.15.45:6800/2012105536,v1:172.21.15.45:6801/2012105536] conn(0x7f11b0068f60 0x7f11b006b410 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f11bc00b3b0 tx=0x7f11bc009e30 comp rx=0 tx=0).ready entity=mgr.14214 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:36:57.973 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:36:57.969+0000 7f11affff700 1 -- 172.21.15.45:0/3674570399 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+174570 (secure 0 0 0) 0x7f11b8056a80 con 0x7f11c8102c00 2023-12-16T21:36:58.188 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:36:58.184+0000 7f11cde60700 1 -- 172.21.15.45:0/3674570399 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 3} v 0) v1 -- 0x7f11c8110e10 con 0x7f11c8102c00 2023-12-16T21:36:58.188 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:36:58.185+0000 7f11affff700 1 -- 172.21.15.45:0/3674570399 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 3}]=0 v0) v1 ==== 74+0+13 (secure 0 0 0) 0x7f11b8009970 con 0x7f11c8102c00 2023-12-16T21:36:58.188 INFO:teuthology.orchestra.run.smithi045.stdout:107374182412 2023-12-16T21:36:58.191 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:36:58.187+0000 7f11cde60700 1 -- 172.21.15.45:0/3674570399 >> [v2:172.21.15.45:6800/2012105536,v1:172.21.15.45:6801/2012105536] conn(0x7f11b0068f60 msgr2=0x7f11b006b410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:36:58.191 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:36:58.187+0000 7f11cde60700 1 --2- 172.21.15.45:0/3674570399 >> [v2:172.21.15.45:6800/2012105536,v1:172.21.15.45:6801/2012105536] conn(0x7f11b0068f60 0x7f11b006b410 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f11bc00b3b0 tx=0x7f11bc009e30 comp rx=0 tx=0).stop 2023-12-16T21:36:58.191 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:36:58.187+0000 7f11cde60700 1 -- 172.21.15.45:0/3674570399 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f11c8102c00 msgr2=0x7f11c8115830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:36:58.191 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:36:58.187+0000 7f11cde60700 1 --2- 172.21.15.45:0/3674570399 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f11c8102c00 0x7f11c8115830 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0x7f11b8002a50 tx=0x7f11b80046b0 comp rx=0 tx=0).stop 2023-12-16T21:36:58.191 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:36:58.187+0000 7f11cde60700 1 -- 172.21.15.45:0/3674570399 shutdown_connections 2023-12-16T21:36:58.191 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:36:58.187+0000 7f11cde60700 1 --2- 172.21.15.45:0/3674570399 >> [v2:172.21.15.45:6800/2012105536,v1:172.21.15.45:6801/2012105536] conn(0x7f11b0068f60 0x7f11b006b410 unknown :-1 s=CLOSED pgs=73 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:36:58.191 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:36:58.187+0000 7f11cde60700 1 --2- 172.21.15.45:0/3674570399 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f11c8102c00 0x7f11c8115830 unknown :-1 s=CLOSED pgs=209 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:36:58.191 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:36:58.187+0000 7f11cde60700 1 --2- 172.21.15.45:0/3674570399 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f11c81035a0 0x7f11c8115d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:36:58.191 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:36:58.187+0000 7f11cde60700 1 -- 172.21.15.45:0/3674570399 >> 172.21.15.45:0/3674570399 conn(0x7f11c80f8790 msgr2=0x7f11c80fa190 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-16T21:36:58.191 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:36:58.187+0000 7f11cde60700 1 -- 172.21.15.45:0/3674570399 shutdown_connections 2023-12-16T21:36:58.191 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:36:58.187+0000 7f11cde60700 1 -- 172.21.15.45:0/3674570399 wait complete. 2023-12-16T21:36:58.303 INFO:teuthology.orchestra.run.smithi045.stderr:Inferring config /var/lib/ceph/88ff363c-9c5a-11ee-95a5-87774f69a715/mon.smithi045/config 2023-12-16T21:36:58.376 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Dec 16 21:36:58 smithi045 ceph-mon[80763]: from='client.? 172.21.15.45:0/3674570399' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 3}]: dispatch 2023-12-16T21:36:58.633 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Dec 16 21:36:58 smithi169 ceph-mon[88350]: from='client.? 172.21.15.45:0/3674570399' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 3}]: dispatch 2023-12-16T21:36:58.929 INFO:tasks.cephadm.ceph_manager.ceph:need seq 107374182412 got 107374182412 for osd.3 2023-12-16T21:36:58.929 DEBUG:teuthology.parallel:result is None 2023-12-16T21:36:59.154 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:36:59.150+0000 7f995a326700 1 -- 172.21.15.45:0/159002190 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f9954102c20 msgr2=0x7f9954102ff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:36:59.154 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:36:59.150+0000 7f995a326700 1 --2- 172.21.15.45:0/159002190 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f9954102c20 0x7f9954102ff0 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7f9944004940 tx=0x7f9944009e30 comp rx=0 tx=0).stop 2023-12-16T21:36:59.154 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:36:59.150+0000 7f995a326700 1 -- 172.21.15.45:0/159002190 shutdown_connections 2023-12-16T21:36:59.154 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:36:59.150+0000 7f995a326700 1 --2- 172.21.15.45:0/159002190 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f9954065390 0x7f9954065800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:36:59.154 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:36:59.150+0000 7f995a326700 1 --2- 172.21.15.45:0/159002190 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f9954102c20 0x7f9954102ff0 unknown :-1 s=CLOSED pgs=210 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:36:59.155 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:36:59.150+0000 7f995a326700 1 -- 172.21.15.45:0/159002190 >> 172.21.15.45:0/159002190 conn(0x7f99540719b0 msgr2=0x7f9954071db0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-16T21:36:59.155 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:36:59.150+0000 7f995a326700 1 -- 172.21.15.45:0/159002190 shutdown_connections 2023-12-16T21:36:59.155 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:36:59.150+0000 7f995a326700 1 -- 172.21.15.45:0/159002190 wait complete. 2023-12-16T21:36:59.155 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:36:59.150+0000 7f995a326700 1 Processor -- start 2023-12-16T21:36:59.155 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:36:59.151+0000 7f995a326700 1 -- start start 2023-12-16T21:36:59.156 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:36:59.151+0000 7f995a326700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f9954065390 0x7f9954113680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:36:59.157 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:36:59.151+0000 7f995a326700 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f9954102c20 0x7f9954113c00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:36:59.157 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:36:59.151+0000 7f995a326700 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f995410d540 con 0x7f9954065390 2023-12-16T21:36:59.157 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:36:59.151+0000 7f995a326700 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f995410d6b0 con 0x7f9954102c20 2023-12-16T21:36:59.157 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:36:59.151+0000 7f9953fff700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f9954065390 0x7f9954113680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:36:59.157 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:36:59.151+0000 7f9953fff700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f9954065390 0x7f9954113680 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:46794/0 (socket says 172.21.15.45:46794) 2023-12-16T21:36:59.157 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:36:59.151+0000 7f9953fff700 1 -- 172.21.15.45:0/3062722676 learned_addr learned my addr 172.21.15.45:0/3062722676 (peer_addr_for_me v2:172.21.15.45:0/0) 2023-12-16T21:36:59.157 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:36:59.151+0000 7f99537fe700 1 --2- 172.21.15.45:0/3062722676 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f9954102c20 0x7f9954113c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:36:59.157 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:36:59.151+0000 7f9953fff700 1 -- 172.21.15.45:0/3062722676 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f9954102c20 msgr2=0x7f9954113c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:36:59.157 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:36:59.151+0000 7f9953fff700 1 --2- 172.21.15.45:0/3062722676 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f9954102c20 0x7f9954113c00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:36:59.157 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:36:59.151+0000 7f9953fff700 1 -- 172.21.15.45:0/3062722676 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9944005040 con 0x7f9954065390 2023-12-16T21:36:59.158 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:36:59.152+0000 7f9953fff700 1 --2- 172.21.15.45:0/3062722676 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f9954065390 0x7f9954113680 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 crypto rx=0x7f994400b3b0 tx=0x7f9944009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:36:59.158 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:36:59.152+0000 7f9950ff9700 1 -- 172.21.15.45:0/3062722676 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f994400e040 con 0x7f9954065390 2023-12-16T21:36:59.158 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:36:59.152+0000 7f9950ff9700 1 -- 172.21.15.45:0/3062722676 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f994401c430 con 0x7f9954065390 2023-12-16T21:36:59.158 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:36:59.152+0000 7f995a326700 1 -- 172.21.15.45:0/3062722676 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f995410d930 con 0x7f9954065390 2023-12-16T21:36:59.158 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:36:59.152+0000 7f9950ff9700 1 -- 172.21.15.45:0/3062722676 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f9944013640 con 0x7f9954065390 2023-12-16T21:36:59.158 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:36:59.152+0000 7f995a326700 1 -- 172.21.15.45:0/3062722676 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f995410de70 con 0x7f9954065390 2023-12-16T21:36:59.158 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:36:59.153+0000 7f995a326700 1 -- 172.21.15.45:0/3062722676 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f995410e300 con 0x7f9954065390 2023-12-16T21:36:59.158 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:36:59.153+0000 7f9950ff9700 1 -- 172.21.15.45:0/3062722676 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 19) v1 ==== 86447+0+0 (secure 0 0 0) 0x7f994400bb80 con 0x7f9954065390 2023-12-16T21:36:59.159 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:36:59.154+0000 7f9950ff9700 1 --2- 172.21.15.45:0/3062722676 >> [v2:172.21.15.45:6800/2012105536,v1:172.21.15.45:6801/2012105536] conn(0x7f993c068f60 0x7f993c06b410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:36:59.159 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:36:59.154+0000 7f9950ff9700 1 -- 172.21.15.45:0/3062722676 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(37..37 src has 1..37) v4 ==== 4634+0+0 (secure 0 0 0) 0x7f99440874d0 con 0x7f9954065390 2023-12-16T21:36:59.159 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:36:59.154+0000 7f99537fe700 1 --2- 172.21.15.45:0/3062722676 >> [v2:172.21.15.45:6800/2012105536,v1:172.21.15.45:6801/2012105536] conn(0x7f993c068f60 0x7f993c06b410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:36:59.160 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:36:59.154+0000 7f9950ff9700 1 --2- 172.21.15.45:0/3062722676 >> [v2:172.21.15.45:6810/3869108635,v1:172.21.15.45:6811/3869108635] conn(0x7f993c06ec30 0x7f993c071040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:36:59.160 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:36:59.154+0000 7f9950ff9700 1 -- 172.21.15.45:0/3062722676 --> [v2:172.21.15.45:6810/3869108635,v1:172.21.15.45:6811/3869108635] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f993c0716f0 con 0x7f993c06ec30 2023-12-16T21:36:59.160 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:36:59.154+0000 7f9950ff9700 1 -- 172.21.15.45:0/3062722676 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_get_version_reply(handle=1 version=37) v2 ==== 24+0+0 (secure 0 0 0) 0x7f9944089b40 con 0x7f9954065390 2023-12-16T21:36:59.160 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:36:59.154+0000 7f99588c3700 1 --2- 172.21.15.45:0/3062722676 >> [v2:172.21.15.45:6810/3869108635,v1:172.21.15.45:6811/3869108635] conn(0x7f993c06ec30 0x7f993c071040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:36:59.160 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:36:59.155+0000 7f99537fe700 1 --2- 172.21.15.45:0/3062722676 >> [v2:172.21.15.45:6800/2012105536,v1:172.21.15.45:6801/2012105536] conn(0x7f993c068f60 0x7f993c06b410 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7f995410ef20 tx=0x7f99480091a0 comp rx=0 tx=0).ready entity=mgr.14214 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:36:59.160 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:36:59.155+0000 7f99588c3700 1 --2- 172.21.15.45:0/3062722676 >> [v2:172.21.15.45:6810/3869108635,v1:172.21.15.45:6811/3869108635] conn(0x7f993c06ec30 0x7f993c071040 crc :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:36:59.160 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:36:59.156+0000 7f9950ff9700 1 -- 172.21.15.45:0/3062722676 <== osd.1 v2:172.21.15.45:6810/3869108635 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+23999 (crc 0 0 0) 0x7f993c0716f0 con 0x7f993c06ec30 2023-12-16T21:36:59.175 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:36:59.172+0000 7f995a326700 1 -- 172.21.15.45:0/3062722676 --> [v2:172.21.15.45:6810/3869108635,v1:172.21.15.45:6811/3869108635] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7f995405e2c0 con 0x7f993c06ec30 2023-12-16T21:36:59.176 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:36:59.172+0000 7f9950ff9700 1 -- 172.21.15.45:0/3062722676 <== osd.1 v2:172.21.15.45:6810/3869108635 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+11 (crc 0 0 0) 0x7f995405e2c0 con 0x7f993c06ec30 2023-12-16T21:36:59.176 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:36:59.172+0000 7f995a326700 1 -- 172.21.15.45:0/3062722676 >> [v2:172.21.15.45:6810/3869108635,v1:172.21.15.45:6811/3869108635] conn(0x7f993c06ec30 msgr2=0x7f993c071040 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:36:59.177 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:36:59.172+0000 7f995a326700 1 --2- 172.21.15.45:0/3062722676 >> [v2:172.21.15.45:6810/3869108635,v1:172.21.15.45:6811/3869108635] conn(0x7f993c06ec30 0x7f993c071040 crc :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:36:59.177 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:36:59.173+0000 7f995a326700 1 -- 172.21.15.45:0/3062722676 >> [v2:172.21.15.45:6800/2012105536,v1:172.21.15.45:6801/2012105536] conn(0x7f993c068f60 msgr2=0x7f993c06b410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:36:59.177 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:36:59.173+0000 7f995a326700 1 --2- 172.21.15.45:0/3062722676 >> [v2:172.21.15.45:6800/2012105536,v1:172.21.15.45:6801/2012105536] conn(0x7f993c068f60 0x7f993c06b410 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7f995410ef20 tx=0x7f99480091a0 comp rx=0 tx=0).stop 2023-12-16T21:36:59.177 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:36:59.173+0000 7f995a326700 1 -- 172.21.15.45:0/3062722676 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f9954065390 msgr2=0x7f9954113680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:36:59.177 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:36:59.173+0000 7f995a326700 1 --2- 172.21.15.45:0/3062722676 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f9954065390 0x7f9954113680 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 crypto rx=0x7f994400b3b0 tx=0x7f9944009e30 comp rx=0 tx=0).stop 2023-12-16T21:36:59.177 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:36:59.173+0000 7f995a326700 1 -- 172.21.15.45:0/3062722676 shutdown_connections 2023-12-16T21:36:59.177 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:36:59.173+0000 7f995a326700 1 --2- 172.21.15.45:0/3062722676 >> [v2:172.21.15.45:6810/3869108635,v1:172.21.15.45:6811/3869108635] conn(0x7f993c06ec30 0x7f993c071040 unknown :-1 s=CLOSED pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:36:59.177 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:36:59.173+0000 7f995a326700 1 --2- 172.21.15.45:0/3062722676 >> [v2:172.21.15.45:6800/2012105536,v1:172.21.15.45:6801/2012105536] conn(0x7f993c068f60 0x7f993c06b410 unknown :-1 s=CLOSED pgs=74 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:36:59.177 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:36:59.173+0000 7f995a326700 1 --2- 172.21.15.45:0/3062722676 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f9954065390 0x7f9954113680 unknown :-1 s=CLOSED pgs=211 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:36:59.177 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:36:59.173+0000 7f995a326700 1 --2- 172.21.15.45:0/3062722676 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f9954102c20 0x7f9954113c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:36:59.177 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:36:59.173+0000 7f995a326700 1 -- 172.21.15.45:0/3062722676 >> 172.21.15.45:0/3062722676 conn(0x7f99540719b0 msgr2=0x7f99540faaf0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-16T21:36:59.177 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:36:59.173+0000 7f995a326700 1 -- 172.21.15.45:0/3062722676 shutdown_connections 2023-12-16T21:36:59.177 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:36:59.173+0000 7f995a326700 1 -- 172.21.15.45:0/3062722676 wait complete. 2023-12-16T21:36:59.566 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Dec 16 21:36:59 smithi045 ceph-mon[80763]: pgmap v100: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2023-12-16T21:36:59.566 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Dec 16 21:36:59 smithi045 ceph-mon[80763]: from='mgr.14214 172.21.15.45:0/1227909437' entity='mgr.smithi045.nhtmwc' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2023-12-16T21:36:59.609 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:36:59.604+0000 7f57ac765700 1 -- 172.21.15.45:0/810891026 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f57a40fe5e0 msgr2=0x7f57a4106cb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:36:59.609 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:36:59.604+0000 7f57ac765700 1 --2- 172.21.15.45:0/810891026 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f57a40fe5e0 0x7f57a4106cb0 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 crypto rx=0x7f5794004940 tx=0x7f5794009e30 comp rx=0 tx=0).stop 2023-12-16T21:36:59.609 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:36:59.604+0000 7f57ac765700 1 -- 172.21.15.45:0/810891026 shutdown_connections 2023-12-16T21:36:59.609 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:36:59.604+0000 7f57ac765700 1 --2- 172.21.15.45:0/810891026 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f57a40fe5e0 0x7f57a4106cb0 unknown :-1 s=CLOSED pgs=212 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:36:59.609 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:36:59.604+0000 7f57ac765700 1 --2- 172.21.15.45:0/810891026 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f57a40fdc40 0x7f57a40fe010 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:36:59.609 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:36:59.604+0000 7f57ac765700 1 -- 172.21.15.45:0/810891026 >> 172.21.15.45:0/810891026 conn(0x7f57a40f76c0 msgr2=0x7f57a40f9ad0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-16T21:36:59.609 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:36:59.604+0000 7f57ac765700 1 -- 172.21.15.45:0/810891026 shutdown_connections 2023-12-16T21:36:59.609 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:36:59.604+0000 7f57ac765700 1 -- 172.21.15.45:0/810891026 wait complete. 2023-12-16T21:36:59.609 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:36:59.605+0000 7f57ac765700 1 Processor -- start 2023-12-16T21:36:59.610 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:36:59.605+0000 7f57ac765700 1 -- start start 2023-12-16T21:36:59.610 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:36:59.605+0000 7f57ac765700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f57a40fdc40 0x7f57a4113690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:36:59.610 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:36:59.605+0000 7f57ac765700 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f57a40fe5e0 0x7f57a4113bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:36:59.610 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:36:59.605+0000 7f57ac765700 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f57a41142b0 con 0x7f57a40fdc40 2023-12-16T21:36:59.610 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:36:59.605+0000 7f57ac765700 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f57a410d570 con 0x7f57a40fe5e0 2023-12-16T21:36:59.610 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:36:59.605+0000 7f57aa501700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f57a40fdc40 0x7f57a4113690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:36:59.610 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:36:59.605+0000 7f57aa501700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f57a40fdc40 0x7f57a4113690 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:46818/0 (socket says 172.21.15.45:46818) 2023-12-16T21:36:59.610 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:36:59.605+0000 7f57aa501700 1 -- 172.21.15.45:0/2454406044 learned_addr learned my addr 172.21.15.45:0/2454406044 (peer_addr_for_me v2:172.21.15.45:0/0) 2023-12-16T21:36:59.610 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:36:59.605+0000 7f57aa501700 1 -- 172.21.15.45:0/2454406044 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f57a40fe5e0 msgr2=0x7f57a4113bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:36:59.610 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:36:59.605+0000 7f57aa501700 1 --2- 172.21.15.45:0/2454406044 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f57a40fe5e0 0x7f57a4113bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:36:59.610 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:36:59.606+0000 7f57aa501700 1 -- 172.21.15.45:0/2454406044 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5794005040 con 0x7f57a40fdc40 2023-12-16T21:36:59.610 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:36:59.606+0000 7f57aa501700 1 --2- 172.21.15.45:0/2454406044 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f57a40fdc40 0x7f57a4113690 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x7f57a000b790 tx=0x7f57a0009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:36:59.612 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:36:59.606+0000 7f579affd700 1 -- 172.21.15.45:0/2454406044 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f57a000f770 con 0x7f57a40fdc40 2023-12-16T21:36:59.612 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:36:59.606+0000 7f57ac765700 1 -- 172.21.15.45:0/2454406044 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f57a410d850 con 0x7f57a40fdc40 2023-12-16T21:36:59.612 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:36:59.606+0000 7f579affd700 1 -- 172.21.15.45:0/2454406044 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f57a000fd80 con 0x7f57a40fdc40 2023-12-16T21:36:59.612 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:36:59.606+0000 7f579affd700 1 -- 172.21.15.45:0/2454406044 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f57a000d310 con 0x7f57a40fdc40 2023-12-16T21:36:59.612 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:36:59.606+0000 7f57ac765700 1 -- 172.21.15.45:0/2454406044 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f57a410dd90 con 0x7f57a40fdc40 2023-12-16T21:36:59.612 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:36:59.607+0000 7f57ac765700 1 -- 172.21.15.45:0/2454406044 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f57a4104370 con 0x7f57a40fdc40 2023-12-16T21:36:59.612 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:36:59.607+0000 7f579affd700 1 -- 172.21.15.45:0/2454406044 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 19) v1 ==== 86447+0+0 (secure 0 0 0) 0x7f57a0010420 con 0x7f57a40fdc40 2023-12-16T21:36:59.612 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:36:59.608+0000 7f579affd700 1 --2- 172.21.15.45:0/2454406044 >> [v2:172.21.15.45:6800/2012105536,v1:172.21.15.45:6801/2012105536] conn(0x7f5790068fc0 0x7f579006b470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:36:59.612 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:36:59.608+0000 7f579affd700 1 -- 172.21.15.45:0/2454406044 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(37..37 src has 1..37) v4 ==== 4634+0+0 (secure 0 0 0) 0x7f57a0086ae0 con 0x7f57a40fdc40 2023-12-16T21:36:59.612 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:36:59.608+0000 7f57a9d00700 1 --2- 172.21.15.45:0/2454406044 >> [v2:172.21.15.45:6800/2012105536,v1:172.21.15.45:6801/2012105536] conn(0x7f5790068fc0 0x7f579006b470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:36:59.612 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:36:59.608+0000 7f57a9d00700 1 --2- 172.21.15.45:0/2454406044 >> [v2:172.21.15.45:6800/2012105536,v1:172.21.15.45:6801/2012105536] conn(0x7f5790068fc0 0x7f579006b470 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7f5794004890 tx=0x7f5794009e30 comp rx=0 tx=0).ready entity=mgr.14214 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:36:59.615 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:36:59.612+0000 7f579affd700 1 -- 172.21.15.45:0/2454406044 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+174570 (secure 0 0 0) 0x7f57a0056ce0 con 0x7f57a40fdc40 2023-12-16T21:36:59.633 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Dec 16 21:36:59 smithi169 ceph-mon[88350]: pgmap v100: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2023-12-16T21:36:59.633 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Dec 16 21:36:59 smithi169 ceph-mon[88350]: from='mgr.14214 172.21.15.45:0/1227909437' entity='mgr.smithi045.nhtmwc' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2023-12-16T21:36:59.792 INFO:teuthology.orchestra.run.smithi045.stdout:60129542164 2023-12-16T21:36:59.792 DEBUG:teuthology.orchestra.run.smithi045:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 88ff363c-9c5a-11ee-95a5-87774f69a715 -- ceph osd last-stat-seq osd.1 2023-12-16T21:36:59.827 INFO:teuthology.orchestra.run.smithi045.stdout:154618822662 2023-12-16T21:36:59.827 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:36:59.822+0000 7f57ac765700 1 -- 172.21.15.45:0/2454406044 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 5} v 0) v1 -- 0x7f57a40fb2c0 con 0x7f57a40fdc40 2023-12-16T21:36:59.827 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:36:59.823+0000 7f579affd700 1 -- 172.21.15.45:0/2454406044 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 5}]=0 v0) v1 ==== 74+0+13 (secure 0 0 0) 0x7f57a0056870 con 0x7f57a40fdc40 2023-12-16T21:36:59.829 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:36:59.825+0000 7f57ac765700 1 -- 172.21.15.45:0/2454406044 >> [v2:172.21.15.45:6800/2012105536,v1:172.21.15.45:6801/2012105536] conn(0x7f5790068fc0 msgr2=0x7f579006b470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:36:59.829 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:36:59.825+0000 7f57ac765700 1 --2- 172.21.15.45:0/2454406044 >> [v2:172.21.15.45:6800/2012105536,v1:172.21.15.45:6801/2012105536] conn(0x7f5790068fc0 0x7f579006b470 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7f5794004890 tx=0x7f5794009e30 comp rx=0 tx=0).stop 2023-12-16T21:36:59.829 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:36:59.825+0000 7f57ac765700 1 -- 172.21.15.45:0/2454406044 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f57a40fdc40 msgr2=0x7f57a4113690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:36:59.829 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:36:59.825+0000 7f57ac765700 1 --2- 172.21.15.45:0/2454406044 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f57a40fdc40 0x7f57a4113690 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x7f57a000b790 tx=0x7f57a0009250 comp rx=0 tx=0).stop 2023-12-16T21:36:59.829 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:36:59.825+0000 7f57ac765700 1 -- 172.21.15.45:0/2454406044 shutdown_connections 2023-12-16T21:36:59.829 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:36:59.825+0000 7f57ac765700 1 --2- 172.21.15.45:0/2454406044 >> [v2:172.21.15.45:6800/2012105536,v1:172.21.15.45:6801/2012105536] conn(0x7f5790068fc0 0x7f579006b470 unknown :-1 s=CLOSED pgs=75 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:36:59.829 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:36:59.825+0000 7f57ac765700 1 --2- 172.21.15.45:0/2454406044 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f57a40fdc40 0x7f57a4113690 unknown :-1 s=CLOSED pgs=213 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:36:59.829 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:36:59.825+0000 7f57ac765700 1 --2- 172.21.15.45:0/2454406044 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f57a40fe5e0 0x7f57a4113bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:36:59.829 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:36:59.825+0000 7f57ac765700 1 -- 172.21.15.45:0/2454406044 >> 172.21.15.45:0/2454406044 conn(0x7f57a40f76c0 msgr2=0x7f57a41014f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-16T21:36:59.829 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:36:59.825+0000 7f57ac765700 1 -- 172.21.15.45:0/2454406044 shutdown_connections 2023-12-16T21:36:59.829 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:36:59.825+0000 7f57ac765700 1 -- 172.21.15.45:0/2454406044 wait complete. 2023-12-16T21:36:59.960 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:36:59.955+0000 7f3912e95700 1 -- 172.21.15.45:0/246817030 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f390c102c20 msgr2=0x7f390c102ff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:36:59.960 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:36:59.955+0000 7f3912e95700 1 --2- 172.21.15.45:0/246817030 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f390c102c20 0x7f390c102ff0 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 crypto rx=0x7f3900004940 tx=0x7f3900009e30 comp rx=0 tx=0).stop 2023-12-16T21:36:59.960 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:36:59.955+0000 7f3912e95700 1 -- 172.21.15.45:0/246817030 shutdown_connections 2023-12-16T21:36:59.961 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:36:59.955+0000 7f3912e95700 1 --2- 172.21.15.45:0/246817030 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f390c065390 0x7f390c065800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:36:59.961 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:36:59.955+0000 7f3912e95700 1 --2- 172.21.15.45:0/246817030 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f390c102c20 0x7f390c102ff0 unknown :-1 s=CLOSED pgs=214 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:36:59.961 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:36:59.955+0000 7f3912e95700 1 -- 172.21.15.45:0/246817030 >> 172.21.15.45:0/246817030 conn(0x7f390c0719b0 msgr2=0x7f390c071db0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-16T21:36:59.961 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:36:59.955+0000 7f3912e95700 1 -- 172.21.15.45:0/246817030 shutdown_connections 2023-12-16T21:36:59.961 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:36:59.955+0000 7f3912e95700 1 -- 172.21.15.45:0/246817030 wait complete. 2023-12-16T21:36:59.961 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:36:59.955+0000 7f3912e95700 1 Processor -- start 2023-12-16T21:36:59.961 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:36:59.955+0000 7f3912e95700 1 -- start start 2023-12-16T21:36:59.961 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:36:59.956+0000 7f3912e95700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f390c065390 0x7f390c0fbbf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:36:59.961 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:36:59.956+0000 7f3912e95700 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f390c102c20 0x7f390c0fc130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:36:59.961 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:36:59.956+0000 7f3912e95700 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f390c0fc730 con 0x7f390c065390 2023-12-16T21:36:59.961 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:36:59.956+0000 7f3912e95700 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f390c0fc8a0 con 0x7f390c102c20 2023-12-16T21:36:59.961 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:36:59.956+0000 7f3910c31700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f390c065390 0x7f390c0fbbf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:36:59.961 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:36:59.956+0000 7f3910c31700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f390c065390 0x7f390c0fbbf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:46828/0 (socket says 172.21.15.45:46828) 2023-12-16T21:36:59.961 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:36:59.956+0000 7f3910c31700 1 -- 172.21.15.45:0/1451534201 learned_addr learned my addr 172.21.15.45:0/1451534201 (peer_addr_for_me v2:172.21.15.45:0/0) 2023-12-16T21:36:59.962 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:36:59.956+0000 7f390bfff700 1 --2- 172.21.15.45:0/1451534201 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f390c102c20 0x7f390c0fc130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:36:59.962 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:36:59.956+0000 7f3910c31700 1 -- 172.21.15.45:0/1451534201 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f390c102c20 msgr2=0x7f390c0fc130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:36:59.962 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:36:59.956+0000 7f3910c31700 1 --2- 172.21.15.45:0/1451534201 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f390c102c20 0x7f390c0fc130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:36:59.962 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:36:59.956+0000 7f3910c31700 1 -- 172.21.15.45:0/1451534201 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3900005040 con 0x7f390c065390 2023-12-16T21:36:59.962 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:36:59.956+0000 7f3910c31700 1 --2- 172.21.15.45:0/1451534201 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f390c065390 0x7f390c0fbbf0 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 crypto rx=0x7f390000b3b0 tx=0x7f3900009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:36:59.962 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:36:59.957+0000 7f39097fa700 1 -- 172.21.15.45:0/1451534201 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f390000e040 con 0x7f390c065390 2023-12-16T21:36:59.963 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:36:59.957+0000 7f39097fa700 1 -- 172.21.15.45:0/1451534201 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f390000bc50 con 0x7f390c065390 2023-12-16T21:36:59.963 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:36:59.957+0000 7f3912e95700 1 -- 172.21.15.45:0/1451534201 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f390c119910 con 0x7f390c065390 2023-12-16T21:36:59.964 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:36:59.957+0000 7f39097fa700 1 -- 172.21.15.45:0/1451534201 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f3900013420 con 0x7f390c065390 2023-12-16T21:36:59.964 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:36:59.957+0000 7f3912e95700 1 -- 172.21.15.45:0/1451534201 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f390c119d70 con 0x7f390c065390 2023-12-16T21:36:59.964 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:36:59.958+0000 7f3912e95700 1 -- 172.21.15.45:0/1451534201 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f390c0fff40 con 0x7f390c065390 2023-12-16T21:36:59.964 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:36:59.958+0000 7f39097fa700 1 -- 172.21.15.45:0/1451534201 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 19) v1 ==== 86447+0+0 (secure 0 0 0) 0x7f390001c430 con 0x7f390c065390 2023-12-16T21:36:59.964 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:36:59.959+0000 7f39097fa700 1 --2- 172.21.15.45:0/1451534201 >> [v2:172.21.15.45:6800/2012105536,v1:172.21.15.45:6801/2012105536] conn(0x7f38fc068f60 0x7f38fc06b410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:36:59.964 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:36:59.959+0000 7f39097fa700 1 -- 172.21.15.45:0/1451534201 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(37..37 src has 1..37) v4 ==== 4634+0+0 (secure 0 0 0) 0x7f3900087870 con 0x7f390c065390 2023-12-16T21:36:59.964 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:36:59.959+0000 7f390bfff700 1 --2- 172.21.15.45:0/1451534201 >> [v2:172.21.15.45:6800/2012105536,v1:172.21.15.45:6801/2012105536] conn(0x7f38fc068f60 0x7f38fc06b410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:36:59.964 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:36:59.959+0000 7f390bfff700 1 --2- 172.21.15.45:0/1451534201 >> [v2:172.21.15.45:6800/2012105536,v1:172.21.15.45:6801/2012105536] conn(0x7f38fc068f60 0x7f38fc06b410 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7f390c0fd5f0 tx=0x7f38f8009ee0 comp rx=0 tx=0).ready entity=mgr.14214 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:36:59.966 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:36:59.963+0000 7f39097fa700 1 -- 172.21.15.45:0/1451534201 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+174570 (secure 0 0 0) 0x7f3900054130 con 0x7f390c065390 2023-12-16T21:37:00.176 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:37:00.172+0000 7f3912e95700 1 -- 172.21.15.45:0/1451534201 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 2} v 0) v1 -- 0x7f390c0734d0 con 0x7f390c065390 2023-12-16T21:37:00.176 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:37:00.173+0000 7f39097fa700 1 -- 172.21.15.45:0/1451534201 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 2}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7f3900057750 con 0x7f390c065390 2023-12-16T21:37:00.176 INFO:teuthology.orchestra.run.smithi045.stdout:77309411343 2023-12-16T21:37:00.178 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:37:00.176+0000 7f3912e95700 1 -- 172.21.15.45:0/1451534201 >> [v2:172.21.15.45:6800/2012105536,v1:172.21.15.45:6801/2012105536] conn(0x7f38fc068f60 msgr2=0x7f38fc06b410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:37:00.178 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:37:00.176+0000 7f3912e95700 1 --2- 172.21.15.45:0/1451534201 >> [v2:172.21.15.45:6800/2012105536,v1:172.21.15.45:6801/2012105536] conn(0x7f38fc068f60 0x7f38fc06b410 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7f390c0fd5f0 tx=0x7f38f8009ee0 comp rx=0 tx=0).stop 2023-12-16T21:37:00.178 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:37:00.176+0000 7f3912e95700 1 -- 172.21.15.45:0/1451534201 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f390c065390 msgr2=0x7f390c0fbbf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:37:00.178 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:37:00.176+0000 7f3912e95700 1 --2- 172.21.15.45:0/1451534201 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f390c065390 0x7f390c0fbbf0 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 crypto rx=0x7f390000b3b0 tx=0x7f3900009e30 comp rx=0 tx=0).stop 2023-12-16T21:37:00.178 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:37:00.176+0000 7f3912e95700 1 -- 172.21.15.45:0/1451534201 shutdown_connections 2023-12-16T21:37:00.179 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:37:00.176+0000 7f3912e95700 1 --2- 172.21.15.45:0/1451534201 >> [v2:172.21.15.45:6800/2012105536,v1:172.21.15.45:6801/2012105536] conn(0x7f38fc068f60 0x7f38fc06b410 unknown :-1 s=CLOSED pgs=76 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:37:00.179 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:37:00.176+0000 7f3912e95700 1 --2- 172.21.15.45:0/1451534201 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f390c065390 0x7f390c0fbbf0 unknown :-1 s=CLOSED pgs=215 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:37:00.179 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:37:00.176+0000 7f3912e95700 1 --2- 172.21.15.45:0/1451534201 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f390c102c20 0x7f390c0fc130 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:37:00.179 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:37:00.176+0000 7f3912e95700 1 -- 172.21.15.45:0/1451534201 >> 172.21.15.45:0/1451534201 conn(0x7f390c0719b0 msgr2=0x7f390c0fabf0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-16T21:37:00.179 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:37:00.176+0000 7f3912e95700 1 -- 172.21.15.45:0/1451534201 shutdown_connections 2023-12-16T21:37:00.179 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:37:00.176+0000 7f3912e95700 1 -- 172.21.15.45:0/1451534201 wait complete. 2023-12-16T21:37:00.275 INFO:tasks.cephadm.ceph_manager.ceph:need seq 154618822662 got 154618822662 for osd.5 2023-12-16T21:37:00.275 DEBUG:teuthology.parallel:result is None 2023-12-16T21:37:00.567 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Dec 16 21:37:00 smithi045 ceph-mon[80763]: from='client.? 172.21.15.45:0/2454406044' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 5}]: dispatch 2023-12-16T21:37:00.567 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Dec 16 21:37:00 smithi045 ceph-mon[80763]: from='client.? 172.21.15.45:0/1451534201' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 2}]: dispatch 2023-12-16T21:37:00.596 INFO:tasks.cephadm.ceph_manager.ceph:need seq 77309411342 got 77309411343 for osd.2 2023-12-16T21:37:00.596 DEBUG:teuthology.parallel:result is None 2023-12-16T21:37:00.633 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Dec 16 21:37:00 smithi169 ceph-mon[88350]: from='client.? 172.21.15.45:0/2454406044' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 5}]: dispatch 2023-12-16T21:37:00.633 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Dec 16 21:37:00 smithi169 ceph-mon[88350]: from='client.? 172.21.15.45:0/1451534201' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 2}]: dispatch 2023-12-16T21:37:00.756 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:37:00.747+0000 7f69a5f87700 1 -- 172.21.15.45:0/986647476 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f69a00fbb40 msgr2=0x7f69a00fbfb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:37:00.756 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:37:00.747+0000 7f69a5f87700 1 --2- 172.21.15.45:0/986647476 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f69a00fbb40 0x7f69a00fbfb0 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 crypto rx=0x7f6994004940 tx=0x7f6994009e30 comp rx=0 tx=0).stop 2023-12-16T21:37:00.756 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:37:00.747+0000 7f69a5f87700 1 -- 172.21.15.45:0/986647476 shutdown_connections 2023-12-16T21:37:00.756 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:37:00.747+0000 7f69a5f87700 1 --2- 172.21.15.45:0/986647476 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f69a00fbb40 0x7f69a00fbfb0 unknown :-1 s=CLOSED pgs=216 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:37:00.756 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:37:00.747+0000 7f69a5f87700 1 --2- 172.21.15.45:0/986647476 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f69a0101b00 0x7f69a0101ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:37:00.756 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:37:00.747+0000 7f69a5f87700 1 -- 172.21.15.45:0/986647476 >> 172.21.15.45:0/986647476 conn(0x7f69a00719b0 msgr2=0x7f69a0071db0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-16T21:37:00.756 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:37:00.747+0000 7f69a5f87700 1 -- 172.21.15.45:0/986647476 shutdown_connections 2023-12-16T21:37:00.756 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:37:00.747+0000 7f69a5f87700 1 -- 172.21.15.45:0/986647476 wait complete. 2023-12-16T21:37:00.756 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:37:00.748+0000 7f69a5f87700 1 Processor -- start 2023-12-16T21:37:00.756 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:37:00.748+0000 7f69a5f87700 1 -- start start 2023-12-16T21:37:00.757 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:37:00.748+0000 7f69a5f87700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f69a00fbb40 0x7f69a010c360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:37:00.757 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:37:00.748+0000 7f69a5f87700 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f69a0101b00 0x7f69a010c8a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:37:00.757 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:37:00.748+0000 7f69a5f87700 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f69a010cde0 con 0x7f69a00fbb40 2023-12-16T21:37:00.757 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:37:00.748+0000 7f69a5f87700 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f69a010cf50 con 0x7f69a0101b00 2023-12-16T21:37:00.757 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:37:00.748+0000 7f699f7fe700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f69a00fbb40 0x7f69a010c360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:37:00.757 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:37:00.748+0000 7f699f7fe700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f69a00fbb40 0x7f69a010c360 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:46838/0 (socket says 172.21.15.45:46838) 2023-12-16T21:37:00.757 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:37:00.748+0000 7f699f7fe700 1 -- 172.21.15.45:0/1565688129 learned_addr learned my addr 172.21.15.45:0/1565688129 (peer_addr_for_me v2:172.21.15.45:0/0) 2023-12-16T21:37:00.757 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:37:00.748+0000 7f699effd700 1 --2- 172.21.15.45:0/1565688129 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f69a0101b00 0x7f69a010c8a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:37:00.757 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:37:00.748+0000 7f699f7fe700 1 -- 172.21.15.45:0/1565688129 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f69a0101b00 msgr2=0x7f69a010c8a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:37:00.757 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:37:00.748+0000 7f699f7fe700 1 --2- 172.21.15.45:0/1565688129 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f69a0101b00 0x7f69a010c8a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:37:00.757 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:37:00.748+0000 7f699f7fe700 1 -- 172.21.15.45:0/1565688129 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6994005040 con 0x7f69a00fbb40 2023-12-16T21:37:00.757 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:37:00.749+0000 7f699f7fe700 1 --2- 172.21.15.45:0/1565688129 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f69a00fbb40 0x7f69a010c360 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 crypto rx=0x7f6990002a50 tx=0x7f69900046b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:37:00.757 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:37:00.749+0000 7f6987fff700 1 -- 172.21.15.45:0/1565688129 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f6990007cc0 con 0x7f69a00fbb40 2023-12-16T21:37:00.757 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:37:00.749+0000 7f6987fff700 1 -- 172.21.15.45:0/1565688129 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f699000f040 con 0x7f69a00fbb40 2023-12-16T21:37:00.758 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:37:00.749+0000 7f69a5f87700 1 -- 172.21.15.45:0/1565688129 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f69a01199d0 con 0x7f69a00fbb40 2023-12-16T21:37:00.758 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:37:00.749+0000 7f6987fff700 1 -- 172.21.15.45:0/1565688129 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f699000da70 con 0x7f69a00fbb40 2023-12-16T21:37:00.758 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:37:00.749+0000 7f69a5f87700 1 -- 172.21.15.45:0/1565688129 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f69a0119dd0 con 0x7f69a00fbb40 2023-12-16T21:37:00.758 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:37:00.750+0000 7f69a5f87700 1 -- 172.21.15.45:0/1565688129 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f69a0103fb0 con 0x7f69a00fbb40 2023-12-16T21:37:00.758 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:37:00.750+0000 7f6987fff700 1 -- 172.21.15.45:0/1565688129 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 19) v1 ==== 86447+0+0 (secure 0 0 0) 0x7f699000dc10 con 0x7f69a00fbb40 2023-12-16T21:37:00.758 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:37:00.751+0000 7f6987fff700 1 --2- 172.21.15.45:0/1565688129 >> [v2:172.21.15.45:6800/2012105536,v1:172.21.15.45:6801/2012105536] conn(0x7f6988068f60 0x7f698806b410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:37:00.758 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:37:00.751+0000 7f699effd700 1 --2- 172.21.15.45:0/1565688129 >> [v2:172.21.15.45:6800/2012105536,v1:172.21.15.45:6801/2012105536] conn(0x7f6988068f60 0x7f698806b410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:37:00.758 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:37:00.751+0000 7f6987fff700 1 -- 172.21.15.45:0/1565688129 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(37..37 src has 1..37) v4 ==== 4634+0+0 (secure 0 0 0) 0x7f69900861b0 con 0x7f69a00fbb40 2023-12-16T21:37:00.758 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:37:00.751+0000 7f699effd700 1 --2- 172.21.15.45:0/1565688129 >> [v2:172.21.15.45:6800/2012105536,v1:172.21.15.45:6801/2012105536] conn(0x7f6988068f60 0x7f698806b410 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7f699400b3b0 tx=0x7f6994009e30 comp rx=0 tx=0).ready entity=mgr.14214 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:37:00.759 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:37:00.755+0000 7f6987fff700 1 -- 172.21.15.45:0/1565688129 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+174570 (secure 0 0 0) 0x7f69900563d0 con 0x7f69a00fbb40 2023-12-16T21:37:00.828 INFO:teuthology.orchestra.run.smithi045.stderr:Inferring config /var/lib/ceph/88ff363c-9c5a-11ee-95a5-87774f69a715/mon.smithi045/config 2023-12-16T21:37:00.970 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:37:00.968+0000 7f69a5f87700 1 -- 172.21.15.45:0/1565688129 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 4} v 0) v1 -- 0x7f69a010d110 con 0x7f69a00fbb40 2023-12-16T21:37:00.971 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:37:00.968+0000 7f6987fff700 1 -- 172.21.15.45:0/1565688129 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 4}]=0 v0) v1 ==== 74+0+13 (secure 0 0 0) 0x7f69a010d110 con 0x7f69a00fbb40 2023-12-16T21:37:00.971 INFO:teuthology.orchestra.run.smithi045.stdout:137438953482 2023-12-16T21:37:00.973 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:37:00.970+0000 7f69a5f87700 1 -- 172.21.15.45:0/1565688129 >> [v2:172.21.15.45:6800/2012105536,v1:172.21.15.45:6801/2012105536] conn(0x7f6988068f60 msgr2=0x7f698806b410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:37:00.973 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:37:00.970+0000 7f69a5f87700 1 --2- 172.21.15.45:0/1565688129 >> [v2:172.21.15.45:6800/2012105536,v1:172.21.15.45:6801/2012105536] conn(0x7f6988068f60 0x7f698806b410 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7f699400b3b0 tx=0x7f6994009e30 comp rx=0 tx=0).stop 2023-12-16T21:37:00.973 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:37:00.970+0000 7f69a5f87700 1 -- 172.21.15.45:0/1565688129 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f69a00fbb40 msgr2=0x7f69a010c360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:37:00.973 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:37:00.970+0000 7f69a5f87700 1 --2- 172.21.15.45:0/1565688129 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f69a00fbb40 0x7f69a010c360 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 crypto rx=0x7f6990002a50 tx=0x7f69900046b0 comp rx=0 tx=0).stop 2023-12-16T21:37:00.973 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:37:00.970+0000 7f69a5f87700 1 -- 172.21.15.45:0/1565688129 shutdown_connections 2023-12-16T21:37:00.973 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:37:00.970+0000 7f69a5f87700 1 --2- 172.21.15.45:0/1565688129 >> [v2:172.21.15.45:6800/2012105536,v1:172.21.15.45:6801/2012105536] conn(0x7f6988068f60 0x7f698806b410 unknown :-1 s=CLOSED pgs=77 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:37:00.973 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:37:00.970+0000 7f69a5f87700 1 --2- 172.21.15.45:0/1565688129 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f69a00fbb40 0x7f69a010c360 unknown :-1 s=CLOSED pgs=217 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:37:00.973 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:37:00.970+0000 7f69a5f87700 1 --2- 172.21.15.45:0/1565688129 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f69a0101b00 0x7f69a010c8a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:37:00.973 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:37:00.970+0000 7f69a5f87700 1 -- 172.21.15.45:0/1565688129 >> 172.21.15.45:0/1565688129 conn(0x7f69a00719b0 msgr2=0x7f69a00faba0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-16T21:37:00.974 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:37:00.970+0000 7f69a5f87700 1 -- 172.21.15.45:0/1565688129 shutdown_connections 2023-12-16T21:37:00.974 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:37:00.970+0000 7f69a5f87700 1 -- 172.21.15.45:0/1565688129 wait complete. 2023-12-16T21:37:01.339 INFO:tasks.cephadm.ceph_manager.ceph:need seq 137438953481 got 137438953482 for osd.4 2023-12-16T21:37:01.339 DEBUG:teuthology.parallel:result is None 2023-12-16T21:37:01.566 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Dec 16 21:37:01 smithi045 ceph-mon[80763]: pgmap v101: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2023-12-16T21:37:01.566 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Dec 16 21:37:01 smithi045 ceph-mon[80763]: from='client.? 172.21.15.45:0/1565688129' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 4}]: dispatch 2023-12-16T21:37:01.569 INFO:teuthology.orchestra.run.smithi045.stderr:Inferring config /var/lib/ceph/88ff363c-9c5a-11ee-95a5-87774f69a715/mon.smithi045/config 2023-12-16T21:37:01.633 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Dec 16 21:37:01 smithi169 ceph-mon[88350]: pgmap v101: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2023-12-16T21:37:01.633 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Dec 16 21:37:01 smithi169 ceph-mon[88350]: from='client.? 172.21.15.45:0/1565688129' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 4}]: dispatch 2023-12-16T21:37:01.864 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:37:01.862+0000 7f80fca1a700 1 -- 172.21.15.45:0/679042064 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f80f80fbb40 msgr2=0x7f80f80fbfb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:37:01.864 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:37:01.862+0000 7f80fca1a700 1 --2- 172.21.15.45:0/679042064 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f80f80fbb40 0x7f80f80fbfb0 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 crypto rx=0x7f80ec004940 tx=0x7f80ec009e30 comp rx=0 tx=0).stop 2023-12-16T21:37:01.865 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:37:01.862+0000 7f80fca1a700 1 -- 172.21.15.45:0/679042064 shutdown_connections 2023-12-16T21:37:01.865 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:37:01.862+0000 7f80fca1a700 1 --2- 172.21.15.45:0/679042064 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f80f80fbb40 0x7f80f80fbfb0 unknown :-1 s=CLOSED pgs=218 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:37:01.865 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:37:01.862+0000 7f80fca1a700 1 --2- 172.21.15.45:0/679042064 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f80f8101b00 0x7f80f8101ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:37:01.865 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:37:01.862+0000 7f80fca1a700 1 -- 172.21.15.45:0/679042064 >> 172.21.15.45:0/679042064 conn(0x7f80f80f76c0 msgr2=0x7f80f80f9ad0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-16T21:37:01.865 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:37:01.862+0000 7f80fca1a700 1 -- 172.21.15.45:0/679042064 shutdown_connections 2023-12-16T21:37:01.865 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:37:01.862+0000 7f80fca1a700 1 -- 172.21.15.45:0/679042064 wait complete. 2023-12-16T21:37:01.867 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:37:01.862+0000 7f80fca1a700 1 Processor -- start 2023-12-16T21:37:01.867 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:37:01.863+0000 7f80fca1a700 1 -- start start 2023-12-16T21:37:01.867 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:37:01.863+0000 7f80fca1a700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f80f80fbb40 0x7f80f810e710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:37:01.867 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:37:01.863+0000 7f80fca1a700 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f80f8101b00 0x7f80f810ec50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:37:01.867 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:37:01.863+0000 7f80fca1a700 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f80f810f270 con 0x7f80f80fbb40 2023-12-16T21:37:01.867 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:37:01.863+0000 7f80fca1a700 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f80f810f3b0 con 0x7f80f8101b00 2023-12-16T21:37:01.867 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:37:01.863+0000 7f80f659c700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f80f80fbb40 0x7f80f810e710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:37:01.867 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:37:01.863+0000 7f80f659c700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f80f80fbb40 0x7f80f810e710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:46846/0 (socket says 172.21.15.45:46846) 2023-12-16T21:37:01.867 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:37:01.863+0000 7f80f659c700 1 -- 172.21.15.45:0/1711748589 learned_addr learned my addr 172.21.15.45:0/1711748589 (peer_addr_for_me v2:172.21.15.45:0/0) 2023-12-16T21:37:01.867 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:37:01.863+0000 7f80f5d9b700 1 --2- 172.21.15.45:0/1711748589 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f80f8101b00 0x7f80f810ec50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:37:01.867 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:37:01.863+0000 7f80f659c700 1 -- 172.21.15.45:0/1711748589 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f80f8101b00 msgr2=0x7f80f810ec50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:37:01.867 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:37:01.863+0000 7f80f659c700 1 --2- 172.21.15.45:0/1711748589 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f80f8101b00 0x7f80f810ec50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:37:01.867 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:37:01.863+0000 7f80f659c700 1 -- 172.21.15.45:0/1711748589 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f80ec005040 con 0x7f80f80fbb40 2023-12-16T21:37:01.868 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:37:01.863+0000 7f80f5d9b700 1 --2- 172.21.15.45:0/1711748589 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f80f8101b00 0x7f80f810ec50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-12-16T21:37:01.868 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:37:01.864+0000 7f80f659c700 1 --2- 172.21.15.45:0/1711748589 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f80f80fbb40 0x7f80f810e710 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 crypto rx=0x7f80e800b7c0 tx=0x7f80e8009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:37:01.868 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:37:01.864+0000 7f80e6ffd700 1 -- 172.21.15.45:0/1711748589 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f80e800f6f0 con 0x7f80f80fbb40 2023-12-16T21:37:01.868 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:37:01.864+0000 7f80e6ffd700 1 -- 172.21.15.45:0/1711748589 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f80e800fd00 con 0x7f80f80fbb40 2023-12-16T21:37:01.868 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:37:01.864+0000 7f80fca1a700 1 -- 172.21.15.45:0/1711748589 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f80f81176e0 con 0x7f80f80fbb40 2023-12-16T21:37:01.868 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:37:01.864+0000 7f80e6ffd700 1 -- 172.21.15.45:0/1711748589 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f80e8011e90 con 0x7f80f80fbb40 2023-12-16T21:37:01.868 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:37:01.864+0000 7f80fca1a700 1 -- 172.21.15.45:0/1711748589 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f80f8117c20 con 0x7f80f80fbb40 2023-12-16T21:37:01.869 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:37:01.865+0000 7f80fca1a700 1 -- 172.21.15.45:0/1711748589 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f80f8103fb0 con 0x7f80f80fbb40 2023-12-16T21:37:01.869 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:37:01.865+0000 7f80e6ffd700 1 -- 172.21.15.45:0/1711748589 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 19) v1 ==== 86447+0+0 (secure 0 0 0) 0x7f80e800f850 con 0x7f80f80fbb40 2023-12-16T21:37:01.869 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:37:01.866+0000 7f80e6ffd700 1 --2- 172.21.15.45:0/1711748589 >> [v2:172.21.15.45:6800/2012105536,v1:172.21.15.45:6801/2012105536] conn(0x7f80e0068f70 0x7f80e006b420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:37:01.869 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:37:01.866+0000 7f80e6ffd700 1 -- 172.21.15.45:0/1711748589 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(37..37 src has 1..37) v4 ==== 4634+0+0 (secure 0 0 0) 0x7f80e8086860 con 0x7f80f80fbb40 2023-12-16T21:37:01.869 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:37:01.866+0000 7f80f5d9b700 1 --2- 172.21.15.45:0/1711748589 >> [v2:172.21.15.45:6800/2012105536,v1:172.21.15.45:6801/2012105536] conn(0x7f80e0068f70 0x7f80e006b420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:37:01.869 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:37:01.866+0000 7f80f5d9b700 1 --2- 172.21.15.45:0/1711748589 >> [v2:172.21.15.45:6800/2012105536,v1:172.21.15.45:6801/2012105536] conn(0x7f80e0068f70 0x7f80e006b420 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7f80ec00b840 tx=0x7f80ec009e30 comp rx=0 tx=0).ready entity=mgr.14214 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:37:01.872 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:37:01.870+0000 7f80e6ffd700 1 -- 172.21.15.45:0/1711748589 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+174570 (secure 0 0 0) 0x7f80e8053110 con 0x7f80f80fbb40 2023-12-16T21:37:02.075 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:37:02.072+0000 7f80fca1a700 1 -- 172.21.15.45:0/1711748589 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 0} v 0) v1 -- 0x7f80f8066400 con 0x7f80f80fbb40 2023-12-16T21:37:02.075 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:37:02.073+0000 7f80e6ffd700 1 -- 172.21.15.45:0/1711748589 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 0}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7f80e8056730 con 0x7f80f80fbb40 2023-12-16T21:37:02.075 INFO:teuthology.orchestra.run.smithi045.stdout:38654705687 2023-12-16T21:37:02.077 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:37:02.074+0000 7f80fca1a700 1 -- 172.21.15.45:0/1711748589 >> [v2:172.21.15.45:6800/2012105536,v1:172.21.15.45:6801/2012105536] conn(0x7f80e0068f70 msgr2=0x7f80e006b420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:37:02.077 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:37:02.074+0000 7f80fca1a700 1 --2- 172.21.15.45:0/1711748589 >> [v2:172.21.15.45:6800/2012105536,v1:172.21.15.45:6801/2012105536] conn(0x7f80e0068f70 0x7f80e006b420 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7f80ec00b840 tx=0x7f80ec009e30 comp rx=0 tx=0).stop 2023-12-16T21:37:02.077 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:37:02.074+0000 7f80fca1a700 1 -- 172.21.15.45:0/1711748589 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f80f80fbb40 msgr2=0x7f80f810e710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:37:02.077 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:37:02.074+0000 7f80fca1a700 1 --2- 172.21.15.45:0/1711748589 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f80f80fbb40 0x7f80f810e710 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 crypto rx=0x7f80e800b7c0 tx=0x7f80e8009250 comp rx=0 tx=0).stop 2023-12-16T21:37:02.077 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:37:02.074+0000 7f80fca1a700 1 -- 172.21.15.45:0/1711748589 shutdown_connections 2023-12-16T21:37:02.077 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:37:02.074+0000 7f80fca1a700 1 --2- 172.21.15.45:0/1711748589 >> [v2:172.21.15.45:6800/2012105536,v1:172.21.15.45:6801/2012105536] conn(0x7f80e0068f70 0x7f80e006b420 unknown :-1 s=CLOSED pgs=78 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:37:02.077 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:37:02.074+0000 7f80fca1a700 1 --2- 172.21.15.45:0/1711748589 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f80f80fbb40 0x7f80f810e710 unknown :-1 s=CLOSED pgs=219 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:37:02.077 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:37:02.074+0000 7f80fca1a700 1 --2- 172.21.15.45:0/1711748589 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f80f8101b00 0x7f80f810ec50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:37:02.078 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:37:02.074+0000 7f80fca1a700 1 -- 172.21.15.45:0/1711748589 >> 172.21.15.45:0/1711748589 conn(0x7f80f80f76c0 msgr2=0x7f80f80f9150 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-16T21:37:02.078 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:37:02.075+0000 7f80fca1a700 1 -- 172.21.15.45:0/1711748589 shutdown_connections 2023-12-16T21:37:02.078 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:37:02.075+0000 7f80fca1a700 1 -- 172.21.15.45:0/1711748589 wait complete. 2023-12-16T21:37:02.542 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Dec 16 21:37:02 smithi045 ceph-mon[80763]: from='client.? 172.21.15.45:0/1711748589' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 0}]: dispatch 2023-12-16T21:37:02.543 INFO:tasks.cephadm.ceph_manager.ceph:need seq 38654705686 got 38654705687 for osd.0 2023-12-16T21:37:02.544 DEBUG:teuthology.parallel:result is None 2023-12-16T21:37:02.634 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Dec 16 21:37:02 smithi169 ceph-mon[88350]: from='client.? 172.21.15.45:0/1711748589' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 0}]: dispatch 2023-12-16T21:37:02.774 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:37:02.771+0000 7f152e204700 1 -- 172.21.15.45:0/2644478543 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f15281035a0 msgr2=0x7f15280fcc10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:37:02.774 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:37:02.771+0000 7f152e204700 1 --2- 172.21.15.45:0/2644478543 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f15281035a0 0x7f15280fcc10 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 crypto rx=0x7f1510004940 tx=0x7f1510009e30 comp rx=0 tx=0).stop 2023-12-16T21:37:02.774 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:37:02.771+0000 7f152e204700 1 -- 172.21.15.45:0/2644478543 shutdown_connections 2023-12-16T21:37:02.774 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:37:02.771+0000 7f152e204700 1 --2- 172.21.15.45:0/2644478543 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f15281035a0 0x7f15280fcc10 unknown :-1 s=CLOSED pgs=220 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:37:02.774 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:37:02.771+0000 7f152e204700 1 --2- 172.21.15.45:0/2644478543 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f1528102c00 0x7f1528102fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:37:02.774 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:37:02.771+0000 7f152e204700 1 -- 172.21.15.45:0/2644478543 >> 172.21.15.45:0/2644478543 conn(0x7f15280f8790 msgr2=0x7f15280faba0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-16T21:37:02.774 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:37:02.771+0000 7f152e204700 1 -- 172.21.15.45:0/2644478543 shutdown_connections 2023-12-16T21:37:02.774 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:37:02.771+0000 7f152e204700 1 -- 172.21.15.45:0/2644478543 wait complete. 2023-12-16T21:37:02.775 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:37:02.772+0000 7f152e204700 1 Processor -- start 2023-12-16T21:37:02.775 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:37:02.772+0000 7f152e204700 1 -- start start 2023-12-16T21:37:02.776 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:37:02.772+0000 7f152e204700 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f1528102c00 0x7f1528113660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:37:02.776 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:37:02.772+0000 7f152e204700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f15281035a0 0x7f1528113be0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:37:02.776 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:37:02.772+0000 7f152e204700 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f152810d520 con 0x7f15281035a0 2023-12-16T21:37:02.776 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:37:02.772+0000 7f152e204700 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f152810d690 con 0x7f1528102c00 2023-12-16T21:37:02.776 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:37:02.772+0000 7f1526ffd700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f15281035a0 0x7f1528113be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:37:02.776 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:37:02.772+0000 7f1526ffd700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f15281035a0 0x7f1528113be0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:60732/0 (socket says 172.21.15.45:60732) 2023-12-16T21:37:02.777 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:37:02.772+0000 7f1526ffd700 1 -- 172.21.15.45:0/402196203 learned_addr learned my addr 172.21.15.45:0/402196203 (peer_addr_for_me v2:172.21.15.45:0/0) 2023-12-16T21:37:02.777 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:37:02.772+0000 7f15277fe700 1 --2- 172.21.15.45:0/402196203 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f1528102c00 0x7f1528113660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:37:02.777 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:37:02.773+0000 7f1526ffd700 1 -- 172.21.15.45:0/402196203 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f1528102c00 msgr2=0x7f1528113660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:37:02.777 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:37:02.773+0000 7f1526ffd700 1 --2- 172.21.15.45:0/402196203 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f1528102c00 0x7f1528113660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:37:02.777 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:37:02.773+0000 7f1526ffd700 1 -- 172.21.15.45:0/402196203 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1510005040 con 0x7f15281035a0 2023-12-16T21:37:02.777 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:37:02.773+0000 7f1526ffd700 1 --2- 172.21.15.45:0/402196203 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f15281035a0 0x7f1528113be0 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 crypto rx=0x7f1510009380 tx=0x7f1510009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:37:02.777 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:37:02.773+0000 7f151ffff700 1 -- 172.21.15.45:0/402196203 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f151000e040 con 0x7f15281035a0 2023-12-16T21:37:02.777 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:37:02.773+0000 7f152e204700 1 -- 172.21.15.45:0/402196203 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f152810d910 con 0x7f15281035a0 2023-12-16T21:37:02.777 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:37:02.773+0000 7f151ffff700 1 -- 172.21.15.45:0/402196203 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f151000bc50 con 0x7f15281035a0 2023-12-16T21:37:02.777 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:37:02.773+0000 7f151ffff700 1 -- 172.21.15.45:0/402196203 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f151001c3f0 con 0x7f15281035a0 2023-12-16T21:37:02.777 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:37:02.773+0000 7f152e204700 1 -- 172.21.15.45:0/402196203 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f152810de50 con 0x7f15281035a0 2023-12-16T21:37:02.777 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:37:02.774+0000 7f152e204700 1 -- 172.21.15.45:0/402196203 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f15280fff10 con 0x7f15281035a0 2023-12-16T21:37:02.778 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:37:02.774+0000 7f151ffff700 1 -- 172.21.15.45:0/402196203 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 19) v1 ==== 86447+0+0 (secure 0 0 0) 0x7f1510013420 con 0x7f15281035a0 2023-12-16T21:37:02.778 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:37:02.775+0000 7f151ffff700 1 --2- 172.21.15.45:0/402196203 >> [v2:172.21.15.45:6800/2012105536,v1:172.21.15.45:6801/2012105536] conn(0x7f1514068f60 0x7f151406b410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:37:02.778 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:37:02.775+0000 7f151ffff700 1 -- 172.21.15.45:0/402196203 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(37..37 src has 1..37) v4 ==== 4634+0+0 (secure 0 0 0) 0x7f1510087e60 con 0x7f15281035a0 2023-12-16T21:37:02.778 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:37:02.775+0000 7f15277fe700 1 --2- 172.21.15.45:0/402196203 >> [v2:172.21.15.45:6800/2012105536,v1:172.21.15.45:6801/2012105536] conn(0x7f1514068f60 0x7f151406b410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:37:02.778 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:37:02.775+0000 7f15277fe700 1 --2- 172.21.15.45:0/402196203 >> [v2:172.21.15.45:6800/2012105536,v1:172.21.15.45:6801/2012105536] conn(0x7f1514068f60 0x7f151406b410 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7f1518006fd0 tx=0x7f1518008040 comp rx=0 tx=0).ready entity=mgr.14214 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:37:02.782 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:37:02.779+0000 7f151ffff700 1 -- 172.21.15.45:0/402196203 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+174570 (secure 0 0 0) 0x7f15100580e0 con 0x7f15281035a0 2023-12-16T21:37:02.980 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:37:02.977+0000 7f152e204700 1 -- 172.21.15.45:0/402196203 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 1} v 0) v1 -- 0x7f15280fc390 con 0x7f15281035a0 2023-12-16T21:37:02.980 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:37:02.977+0000 7f151ffff700 1 -- 172.21.15.45:0/402196203 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 1}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7f15280fc390 con 0x7f15281035a0 2023-12-16T21:37:02.980 INFO:teuthology.orchestra.run.smithi045.stdout:60129542164 2023-12-16T21:37:02.982 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:37:02.979+0000 7f152e204700 1 -- 172.21.15.45:0/402196203 >> [v2:172.21.15.45:6800/2012105536,v1:172.21.15.45:6801/2012105536] conn(0x7f1514068f60 msgr2=0x7f151406b410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:37:02.983 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:37:02.979+0000 7f152e204700 1 --2- 172.21.15.45:0/402196203 >> [v2:172.21.15.45:6800/2012105536,v1:172.21.15.45:6801/2012105536] conn(0x7f1514068f60 0x7f151406b410 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7f1518006fd0 tx=0x7f1518008040 comp rx=0 tx=0).stop 2023-12-16T21:37:02.983 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:37:02.979+0000 7f152e204700 1 -- 172.21.15.45:0/402196203 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f15281035a0 msgr2=0x7f1528113be0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:37:02.983 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:37:02.979+0000 7f152e204700 1 --2- 172.21.15.45:0/402196203 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f15281035a0 0x7f1528113be0 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 crypto rx=0x7f1510009380 tx=0x7f1510009e30 comp rx=0 tx=0).stop 2023-12-16T21:37:02.983 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:37:02.979+0000 7f152e204700 1 -- 172.21.15.45:0/402196203 shutdown_connections 2023-12-16T21:37:02.983 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:37:02.979+0000 7f152e204700 1 --2- 172.21.15.45:0/402196203 >> [v2:172.21.15.45:6800/2012105536,v1:172.21.15.45:6801/2012105536] conn(0x7f1514068f60 0x7f151406b410 unknown :-1 s=CLOSED pgs=79 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:37:02.983 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:37:02.980+0000 7f152e204700 1 --2- 172.21.15.45:0/402196203 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f1528102c00 0x7f1528113660 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:37:02.983 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:37:02.980+0000 7f152e204700 1 --2- 172.21.15.45:0/402196203 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f15281035a0 0x7f1528113be0 unknown :-1 s=CLOSED pgs=221 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:37:02.983 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:37:02.980+0000 7f152e204700 1 -- 172.21.15.45:0/402196203 >> 172.21.15.45:0/402196203 conn(0x7f15280f8790 msgr2=0x7f15280fa260 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-16T21:37:02.983 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:37:02.980+0000 7f152e204700 1 -- 172.21.15.45:0/402196203 shutdown_connections 2023-12-16T21:37:02.983 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:37:02.980+0000 7f152e204700 1 -- 172.21.15.45:0/402196203 wait complete. 2023-12-16T21:37:03.237 INFO:tasks.cephadm.ceph_manager.ceph:need seq 60129542164 got 60129542164 for osd.1 2023-12-16T21:37:03.237 DEBUG:teuthology.parallel:result is None 2023-12-16T21:37:03.237 INFO:tasks.cephadm.ceph_manager.ceph:waiting for clean 2023-12-16T21:37:03.238 DEBUG:teuthology.orchestra.run.smithi045:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 88ff363c-9c5a-11ee-95a5-87774f69a715 -- ceph pg dump --format=json 2023-12-16T21:37:03.506 INFO:teuthology.orchestra.run.smithi045.stderr:Inferring config /var/lib/ceph/88ff363c-9c5a-11ee-95a5-87774f69a715/mon.smithi045/config 2023-12-16T21:37:03.566 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Dec 16 21:37:03 smithi045 ceph-mon[80763]: pgmap v102: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2023-12-16T21:37:03.566 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Dec 16 21:37:03 smithi045 ceph-mon[80763]: from='client.? 172.21.15.45:0/402196203' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 1}]: dispatch 2023-12-16T21:37:03.633 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Dec 16 21:37:03 smithi169 ceph-mon[88350]: pgmap v102: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2023-12-16T21:37:03.634 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Dec 16 21:37:03 smithi169 ceph-mon[88350]: from='client.? 172.21.15.45:0/402196203' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 1}]: dispatch 2023-12-16T21:37:04.224 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:37:04.220+0000 7f77d8b22700 1 -- 172.21.15.45:0/3678062039 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f77d41035a0 msgr2=0x7f77d40fcc10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:37:04.224 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:37:04.220+0000 7f77d8b22700 1 --2- 172.21.15.45:0/3678062039 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f77d41035a0 0x7f77d40fcc10 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 crypto rx=0x7f77bc004940 tx=0x7f77bc009e30 comp rx=0 tx=0).stop 2023-12-16T21:37:04.224 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:37:04.220+0000 7f77d8b22700 1 -- 172.21.15.45:0/3678062039 shutdown_connections 2023-12-16T21:37:04.224 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:37:04.220+0000 7f77d8b22700 1 --2- 172.21.15.45:0/3678062039 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f77d41035a0 0x7f77d40fcc10 unknown :-1 s=CLOSED pgs=222 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:37:04.224 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:37:04.220+0000 7f77d8b22700 1 --2- 172.21.15.45:0/3678062039 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f77d4102c00 0x7f77d4102fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:37:04.224 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:37:04.220+0000 7f77d8b22700 1 -- 172.21.15.45:0/3678062039 >> 172.21.15.45:0/3678062039 conn(0x7f77d40f8790 msgr2=0x7f77d40faba0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-16T21:37:04.224 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:37:04.220+0000 7f77d8b22700 1 -- 172.21.15.45:0/3678062039 shutdown_connections 2023-12-16T21:37:04.224 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:37:04.220+0000 7f77d8b22700 1 -- 172.21.15.45:0/3678062039 wait complete. 2023-12-16T21:37:04.225 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:37:04.220+0000 7f77d8b22700 1 Processor -- start 2023-12-16T21:37:04.225 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:37:04.221+0000 7f77d8b22700 1 -- start start 2023-12-16T21:37:04.225 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:37:04.221+0000 7f77d8b22700 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f77d4102c00 0x7f77d4115830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:37:04.225 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:37:04.221+0000 7f77d8b22700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f77d41035a0 0x7f77d4115d70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:37:04.225 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:37:04.221+0000 7f77d8b22700 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f77d4116450 con 0x7f77d41035a0 2023-12-16T21:37:04.225 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:37:04.221+0000 7f77d8b22700 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f77d410f6f0 con 0x7f77d4102c00 2023-12-16T21:37:04.225 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:37:04.221+0000 7f77d1d9b700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f77d41035a0 0x7f77d4115d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:37:04.225 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:37:04.221+0000 7f77d1d9b700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f77d41035a0 0x7f77d4115d70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:60744/0 (socket says 172.21.15.45:60744) 2023-12-16T21:37:04.225 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:37:04.221+0000 7f77d1d9b700 1 -- 172.21.15.45:0/1594890189 learned_addr learned my addr 172.21.15.45:0/1594890189 (peer_addr_for_me v2:172.21.15.45:0/0) 2023-12-16T21:37:04.225 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:37:04.221+0000 7f77d259c700 1 --2- 172.21.15.45:0/1594890189 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f77d4102c00 0x7f77d4115830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:37:04.225 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:37:04.221+0000 7f77d1d9b700 1 -- 172.21.15.45:0/1594890189 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f77d4102c00 msgr2=0x7f77d4115830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:37:04.225 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:37:04.221+0000 7f77d1d9b700 1 --2- 172.21.15.45:0/1594890189 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f77d4102c00 0x7f77d4115830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:37:04.225 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:37:04.221+0000 7f77d1d9b700 1 -- 172.21.15.45:0/1594890189 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f77bc005040 con 0x7f77d41035a0 2023-12-16T21:37:04.226 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:37:04.221+0000 7f77d259c700 1 --2- 172.21.15.45:0/1594890189 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f77d4102c00 0x7f77d4115830 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-12-16T21:37:04.226 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:37:04.221+0000 7f77d1d9b700 1 --2- 172.21.15.45:0/1594890189 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f77d41035a0 0x7f77d4115d70 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 crypto rx=0x7f77bc009fd0 tx=0x7f77bc009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:37:04.226 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:37:04.222+0000 7f77caffd700 1 -- 172.21.15.45:0/1594890189 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f77bc00e040 con 0x7f77d41035a0 2023-12-16T21:37:04.228 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:37:04.222+0000 7f77caffd700 1 -- 172.21.15.45:0/1594890189 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f77bc00bc50 con 0x7f77d41035a0 2023-12-16T21:37:04.229 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:37:04.222+0000 7f77d8b22700 1 -- 172.21.15.45:0/1594890189 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f77d410f970 con 0x7f77d41035a0 2023-12-16T21:37:04.229 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:37:04.222+0000 7f77caffd700 1 -- 172.21.15.45:0/1594890189 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f77bc01c3f0 con 0x7f77d41035a0 2023-12-16T21:37:04.229 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:37:04.222+0000 7f77d8b22700 1 -- 172.21.15.45:0/1594890189 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f77d410feb0 con 0x7f77d41035a0 2023-12-16T21:37:04.229 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:37:04.223+0000 7f77d8b22700 1 -- 172.21.15.45:0/1594890189 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f77d40fc390 con 0x7f77d41035a0 2023-12-16T21:37:04.229 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:37:04.224+0000 7f77caffd700 1 -- 172.21.15.45:0/1594890189 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 19) v1 ==== 86447+0+0 (secure 0 0 0) 0x7f77bc013420 con 0x7f77d41035a0 2023-12-16T21:37:04.229 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:37:04.224+0000 7f77caffd700 1 --2- 172.21.15.45:0/1594890189 >> [v2:172.21.15.45:6800/2012105536,v1:172.21.15.45:6801/2012105536] conn(0x7f77c0068f60 0x7f77c006b410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:37:04.229 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:37:04.224+0000 7f77caffd700 1 -- 172.21.15.45:0/1594890189 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(37..37 src has 1..37) v4 ==== 4634+0+0 (secure 0 0 0) 0x7f77bc087e20 con 0x7f77d41035a0 2023-12-16T21:37:04.229 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:37:04.224+0000 7f77d259c700 1 --2- 172.21.15.45:0/1594890189 >> [v2:172.21.15.45:6800/2012105536,v1:172.21.15.45:6801/2012105536] conn(0x7f77c0068f60 0x7f77c006b410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:37:04.230 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:37:04.224+0000 7f77d259c700 1 --2- 172.21.15.45:0/1594890189 >> [v2:172.21.15.45:6800/2012105536,v1:172.21.15.45:6801/2012105536] conn(0x7f77c0068f60 0x7f77c006b410 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7f77c4007fd0 tx=0x7f77c40074f0 comp rx=0 tx=0).ready entity=mgr.14214 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:37:04.231 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:37:04.228+0000 7f77caffd700 1 -- 172.21.15.45:0/1594890189 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+174570 (secure 0 0 0) 0x7f77bc0580c0 con 0x7f77d41035a0 2023-12-16T21:37:04.419 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:37:04.416+0000 7f77d8b22700 1 -- 172.21.15.45:0/1594890189 --> [v2:172.21.15.45:6800/2012105536,v1:172.21.15.45:6801/2012105536] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f77d404bbf0 con 0x7f77c0068f60 2023-12-16T21:37:04.420 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:37:04.418+0000 7f77caffd700 1 -- 172.21.15.45:0/1594890189 <== mgr.14214 v2:172.21.15.45:6800/2012105536 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+21910 (secure 0 0 0) 0x7f77d404bbf0 con 0x7f77c0068f60 2023-12-16T21:37:04.420 INFO:teuthology.orchestra.run.smithi045.stdout: 2023-12-16T21:37:04.422 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:37:04.420+0000 7f77d8b22700 1 -- 172.21.15.45:0/1594890189 >> [v2:172.21.15.45:6800/2012105536,v1:172.21.15.45:6801/2012105536] conn(0x7f77c0068f60 msgr2=0x7f77c006b410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:37:04.423 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:37:04.420+0000 7f77d8b22700 1 --2- 172.21.15.45:0/1594890189 >> [v2:172.21.15.45:6800/2012105536,v1:172.21.15.45:6801/2012105536] conn(0x7f77c0068f60 0x7f77c006b410 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7f77c4007fd0 tx=0x7f77c40074f0 comp rx=0 tx=0).stop 2023-12-16T21:37:04.423 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:37:04.420+0000 7f77d8b22700 1 -- 172.21.15.45:0/1594890189 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f77d41035a0 msgr2=0x7f77d4115d70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:37:04.423 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:37:04.420+0000 7f77d8b22700 1 --2- 172.21.15.45:0/1594890189 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f77d41035a0 0x7f77d4115d70 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 crypto rx=0x7f77bc009fd0 tx=0x7f77bc009e30 comp rx=0 tx=0).stop 2023-12-16T21:37:04.423 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:37:04.420+0000 7f77d8b22700 1 -- 172.21.15.45:0/1594890189 shutdown_connections 2023-12-16T21:37:04.423 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:37:04.420+0000 7f77d8b22700 1 --2- 172.21.15.45:0/1594890189 >> [v2:172.21.15.45:6800/2012105536,v1:172.21.15.45:6801/2012105536] conn(0x7f77c0068f60 0x7f77c006b410 unknown :-1 s=CLOSED pgs=80 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:37:04.423 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:37:04.420+0000 7f77d8b22700 1 --2- 172.21.15.45:0/1594890189 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f77d4102c00 0x7f77d4115830 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:37:04.423 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:37:04.420+0000 7f77d8b22700 1 --2- 172.21.15.45:0/1594890189 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f77d41035a0 0x7f77d4115d70 unknown :-1 s=CLOSED pgs=223 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:37:04.423 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:37:04.420+0000 7f77d8b22700 1 -- 172.21.15.45:0/1594890189 >> 172.21.15.45:0/1594890189 conn(0x7f77d40f8790 msgr2=0x7f77d40fa040 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-16T21:37:04.423 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:37:04.420+0000 7f77d8b22700 1 -- 172.21.15.45:0/1594890189 shutdown_connections 2023-12-16T21:37:04.423 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:37:04.420+0000 7f77d8b22700 1 -- 172.21.15.45:0/1594890189 wait complete. 2023-12-16T21:37:04.424 INFO:teuthology.orchestra.run.smithi045.stderr:dumped all 2023-12-16T21:37:04.679 INFO:teuthology.orchestra.run.smithi045.stdout:{"pg_ready":true,"pg_map":{"version":102,"stamp":"2023-12-16T21:37:02.611483+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":3,"acting":3,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":3,"num_osds":6,"num_per_pool_osds":6,"num_per_pool_omap_osds":3,"kb":562470912,"kb_used":1787436,"kb_used_data":3708,"kb_used_omap":0,"kb_used_meta":1783680,"kb_avail":560683476,"statfs":{"total":575970213888,"available":574139879424,"internally_reserved":0,"allocated":3796992,"data_stored":2542338,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":1826488320},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001933"},"pg_stats":[{"pgid":"1.0","version":"22'117","reported_seq":188,"reported_epoch":36,"state":"active+clean","last_fresh":"2023-12-16T21:36:38.722612+0000","last_change":"2023-12-16T21:36:22.552736+0000","last_active":"2023-12-16T21:36:38.722612+0000","last_peered":"2023-12-16T21:36:38.722612+0000","last_clean":"2023-12-16T21:36:38.722612+0000","last_became_active":"2023-12-16T21:36:22.552458+0000","last_became_peered":"2023-12-16T21:36:22.552458+0000","last_unstale":"2023-12-16T21:36:38.722612+0000","last_undegraded":"2023-12-16T21:36:38.722612+0000","last_fullsized":"2023-12-16T21:36:38.722612+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:35:53.103103+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:35:53.103103+0000","last_clean_scrub_stamp":"2023-12-16T21:35:53.103103+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-18T01:22:13.951675+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":2375680,"data_stored":2361472,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":3,"acting":3,"num_store_stats":4}],"osd_stats":[{"osd":5,"up_from":36,"seq":154618822663,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":297616,"kb_used_data":328,"kb_used_omap":0,"kb_used_meta":297280,"kb_avail":93447536,"statfs":{"total":95995035648,"available":95690276864,"internally_reserved":0,"allocated":335872,"data_stored":128539,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304414720},"hb_peers":[0,1,2,3,4],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.56899999999999995}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.46100000000000002}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.61599999999999999}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.497}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.52700000000000002}]}]},{"osd":4,"up_from":32,"seq":137438953483,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":297616,"kb_used_data":328,"kb_used_omap":0,"kb_used_meta":297280,"kb_avail":93447536,"statfs":{"total":95995035648,"available":95690276864,"internally_reserved":0,"allocated":335872,"data_stored":128539,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304414720},"hb_peers":[0,1,2,3,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.56000000000000005}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.52400000000000002}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.54900000000000004}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.45500000000000002}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.47499999999999998}]}]},{"osd":3,"up_from":25,"seq":107374182413,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":298196,"kb_used_data":908,"kb_used_omap":0,"kb_used_meta":297280,"kb_avail":93446956,"statfs":{"total":95995035648,"available":95689682944,"internally_reserved":0,"allocated":929792,"data_stored":718907,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304414720},"hb_peers":[0,1,2,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.435}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.42099999999999999}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.45300000000000001}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.60299999999999998}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.57099999999999995}]}]},{"osd":2,"up_from":18,"seq":77309411344,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":297616,"kb_used_data":328,"kb_used_omap":0,"kb_used_meta":297280,"kb_avail":93447536,"statfs":{"total":95995035648,"available":95690276864,"internally_reserved":0,"allocated":335872,"data_stored":128539,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304414720},"hb_peers":[0,1,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Dec 16 21:36:52 2023","interfaces":[{"interface":"back","average":{"1min":0.32100000000000001,"5min":0.32100000000000001,"15min":0.32100000000000001},"min":{"1min":0.21299999999999999,"5min":0.21299999999999999,"15min":0.21299999999999999},"max":{"1min":0.42699999999999999,"5min":0.42699999999999999,"15min":0.42699999999999999},"last":0.245},{"interface":"front","average":{"1min":0.33900000000000002,"5min":0.33900000000000002,"15min":0.33900000000000002},"min":{"1min":0.26200000000000001,"5min":0.26200000000000001,"15min":0.26200000000000001},"max":{"1min":0.44900000000000001,"5min":0.44900000000000001,"15min":0.44900000000000001},"last":0.27500000000000002}]},{"osd":1,"last update":"Sat Dec 16 21:36:52 2023","interfaces":[{"interface":"back","average":{"1min":0.38700000000000001,"5min":0.38700000000000001,"15min":0.38700000000000001},"min":{"1min":0.27900000000000003,"5min":0.27900000000000003,"15min":0.27900000000000003},"max":{"1min":0.48599999999999999,"5min":0.48599999999999999,"15min":0.48599999999999999},"last":0.30499999999999999},{"interface":"front","average":{"1min":0.40000000000000002,"5min":0.40000000000000002,"15min":0.40000000000000002},"min":{"1min":0.30199999999999999,"5min":0.30199999999999999,"15min":0.30199999999999999},"max":{"1min":0.52000000000000002,"5min":0.52000000000000002,"15min":0.52000000000000002},"last":0.315}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.63500000000000001}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.621}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.53400000000000003}]}]},{"osd":0,"up_from":9,"seq":38654705687,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":298196,"kb_used_data":908,"kb_used_omap":0,"kb_used_meta":297280,"kb_avail":93446956,"statfs":{"total":95995035648,"available":95689682944,"internally_reserved":0,"allocated":929792,"data_stored":718907,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304414720},"hb_peers":[1,2,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Dec 16 21:36:38 2023","interfaces":[{"interface":"back","average":{"1min":0.34499999999999997,"5min":0.34499999999999997,"15min":0.34499999999999997},"min":{"1min":0.215,"5min":0.215,"15min":0.215},"max":{"1min":0.54700000000000004,"5min":0.54700000000000004,"15min":0.54700000000000004},"last":0.32600000000000001},{"interface":"front","average":{"1min":0.36399999999999999,"5min":0.36399999999999999,"15min":0.36399999999999999},"min":{"1min":0.25900000000000001,"5min":0.25900000000000001,"15min":0.25900000000000001},"max":{"1min":0.49099999999999999,"5min":0.49099999999999999,"15min":0.49099999999999999},"last":0.27400000000000002}]},{"osd":2,"last update":"Sat Dec 16 21:36:54 2023","interfaces":[{"interface":"back","average":{"1min":0.34799999999999998,"5min":0.34799999999999998,"15min":0.34799999999999998},"min":{"1min":0.27000000000000002,"5min":0.27000000000000002,"15min":0.27000000000000002},"max":{"1min":0.47799999999999998,"5min":0.47799999999999998,"15min":0.47799999999999998},"last":0.31},{"interface":"front","average":{"1min":0.36699999999999999,"5min":0.36699999999999999,"15min":0.36699999999999999},"min":{"1min":0.29199999999999998,"5min":0.29199999999999998,"15min":0.29199999999999998},"max":{"1min":0.45500000000000002,"5min":0.45500000000000002,"15min":0.45500000000000002},"last":0.372}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.59199999999999997}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.48699999999999999}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.54300000000000004}]}]},{"osd":1,"up_from":14,"seq":60129542165,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":298196,"kb_used_data":908,"kb_used_omap":0,"kb_used_meta":297280,"kb_avail":93446956,"statfs":{"total":95995035648,"available":95689682944,"internally_reserved":0,"allocated":929792,"data_stored":718907,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304414720},"hb_peers":[0,2,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Dec 16 21:36:37 2023","interfaces":[{"interface":"back","average":{"1min":0.36199999999999999,"5min":0.36199999999999999,"15min":0.36199999999999999},"min":{"1min":0.20999999999999999,"5min":0.20999999999999999,"15min":0.20999999999999999},"max":{"1min":0.51700000000000002,"5min":0.51700000000000002,"15min":0.51700000000000002},"last":0.29899999999999999},{"interface":"front","average":{"1min":0.36799999999999999,"5min":0.36799999999999999,"15min":0.36799999999999999},"min":{"1min":0.22600000000000001,"5min":0.22600000000000001,"15min":0.22600000000000001},"max":{"1min":0.59199999999999997,"5min":0.59199999999999997,"15min":0.59199999999999997},"last":0.317}]},{"osd":2,"last update":"Sat Dec 16 21:36:52 2023","interfaces":[{"interface":"back","average":{"1min":0.40100000000000002,"5min":0.40100000000000002,"15min":0.40100000000000002},"min":{"1min":0.252,"5min":0.252,"15min":0.252},"max":{"1min":0.627,"5min":0.627,"15min":0.627},"last":0.36099999999999999},{"interface":"front","average":{"1min":0.40300000000000002,"5min":0.40300000000000002,"15min":0.40300000000000002},"min":{"1min":0.22900000000000001,"5min":0.22900000000000001,"15min":0.22900000000000001},"max":{"1min":0.56799999999999995,"5min":0.56799999999999995,"15min":0.56799999999999995},"last":0.34699999999999998}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.58899999999999997}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.54900000000000004}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.53100000000000003}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-12-16T21:37:04.680 DEBUG:teuthology.orchestra.run.smithi045:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 88ff363c-9c5a-11ee-95a5-87774f69a715 -- ceph pg dump --format=json 2023-12-16T21:37:04.943 INFO:teuthology.orchestra.run.smithi045.stderr:Inferring config /var/lib/ceph/88ff363c-9c5a-11ee-95a5-87774f69a715/mon.smithi045/config 2023-12-16T21:37:05.695 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Dec 16 21:37:05 smithi045 ceph-mon[80763]: from='client.14424 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-12-16T21:37:05.695 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Dec 16 21:37:05 smithi045 ceph-mon[80763]: pgmap v103: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2023-12-16T21:37:05.753 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:37:05.748+0000 7f8ace1bd700 1 -- 172.21.15.45:0/2192140683 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f8ac8102c00 msgr2=0x7f8ac8102fd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:37:05.753 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:37:05.748+0000 7f8ace1bd700 1 --2- 172.21.15.45:0/2192140683 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f8ac8102c00 0x7f8ac8102fd0 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 crypto rx=0x7f8ab8004940 tx=0x7f8ab8009e30 comp rx=0 tx=0).stop 2023-12-16T21:37:05.753 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:37:05.749+0000 7f8ace1bd700 1 -- 172.21.15.45:0/2192140683 shutdown_connections 2023-12-16T21:37:05.753 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:37:05.749+0000 7f8ace1bd700 1 --2- 172.21.15.45:0/2192140683 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f8ac81035a0 0x7f8ac80fcc10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:37:05.753 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:37:05.749+0000 7f8ace1bd700 1 --2- 172.21.15.45:0/2192140683 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f8ac8102c00 0x7f8ac8102fd0 unknown :-1 s=CLOSED pgs=224 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:37:05.753 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:37:05.749+0000 7f8ace1bd700 1 -- 172.21.15.45:0/2192140683 >> 172.21.15.45:0/2192140683 conn(0x7f8ac80f8790 msgr2=0x7f8ac80faba0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-16T21:37:05.753 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:37:05.749+0000 7f8ace1bd700 1 -- 172.21.15.45:0/2192140683 shutdown_connections 2023-12-16T21:37:05.753 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:37:05.749+0000 7f8ace1bd700 1 -- 172.21.15.45:0/2192140683 wait complete. 2023-12-16T21:37:05.754 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:37:05.749+0000 7f8ace1bd700 1 Processor -- start 2023-12-16T21:37:05.754 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:37:05.749+0000 7f8ace1bd700 1 -- start start 2023-12-16T21:37:05.754 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:37:05.750+0000 7f8ace1bd700 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f8ac8102c00 0x7f8ac8113630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:37:05.754 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:37:05.750+0000 7f8ace1bd700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f8ac81035a0 0x7f8ac8113bb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:37:05.754 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:37:05.750+0000 7f8ace1bd700 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8ac8114200 con 0x7f8ac81035a0 2023-12-16T21:37:05.754 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:37:05.750+0000 7f8ace1bd700 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8ac810d4f0 con 0x7f8ac8102c00 2023-12-16T21:37:05.754 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:37:05.750+0000 7f8ac6ffd700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f8ac81035a0 0x7f8ac8113bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:37:05.754 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:37:05.750+0000 7f8ac6ffd700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f8ac81035a0 0x7f8ac8113bb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:60754/0 (socket says 172.21.15.45:60754) 2023-12-16T21:37:05.754 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:37:05.750+0000 7f8ac6ffd700 1 -- 172.21.15.45:0/3835014843 learned_addr learned my addr 172.21.15.45:0/3835014843 (peer_addr_for_me v2:172.21.15.45:0/0) 2023-12-16T21:37:05.754 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:37:05.750+0000 7f8ac77fe700 1 --2- 172.21.15.45:0/3835014843 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f8ac8102c00 0x7f8ac8113630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:37:05.754 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:37:05.750+0000 7f8ac6ffd700 1 -- 172.21.15.45:0/3835014843 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f8ac8102c00 msgr2=0x7f8ac8113630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:37:05.754 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:37:05.750+0000 7f8ac6ffd700 1 --2- 172.21.15.45:0/3835014843 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f8ac8102c00 0x7f8ac8113630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:37:05.754 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:37:05.750+0000 7f8ac6ffd700 1 -- 172.21.15.45:0/3835014843 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8ab8005040 con 0x7f8ac81035a0 2023-12-16T21:37:05.754 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:37:05.750+0000 7f8ac6ffd700 1 --2- 172.21.15.45:0/3835014843 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f8ac81035a0 0x7f8ac8113bb0 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 crypto rx=0x7f8abc002a50 tx=0x7f8abc004600 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:37:05.755 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:37:05.750+0000 7f8aaffff700 1 -- 172.21.15.45:0/3835014843 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f8abc007cc0 con 0x7f8ac81035a0 2023-12-16T21:37:05.755 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:37:05.750+0000 7f8aaffff700 1 -- 172.21.15.45:0/3835014843 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f8abc00f040 con 0x7f8ac81035a0 2023-12-16T21:37:05.755 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:37:05.750+0000 7f8ace1bd700 1 -- 172.21.15.45:0/3835014843 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8ac810d780 con 0x7f8ac81035a0 2023-12-16T21:37:05.755 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:37:05.750+0000 7f8aaffff700 1 -- 172.21.15.45:0/3835014843 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f8abc00d990 con 0x7f8ac81035a0 2023-12-16T21:37:05.755 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:37:05.750+0000 7f8ace1bd700 1 -- 172.21.15.45:0/3835014843 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8ac810dcc0 con 0x7f8ac81035a0 2023-12-16T21:37:05.756 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:37:05.751+0000 7f8ace1bd700 1 -- 172.21.15.45:0/3835014843 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8ac80fff10 con 0x7f8ac81035a0 2023-12-16T21:37:05.756 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:37:05.752+0000 7f8aaffff700 1 -- 172.21.15.45:0/3835014843 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 19) v1 ==== 86447+0+0 (secure 0 0 0) 0x7f8abc009710 con 0x7f8ac81035a0 2023-12-16T21:37:05.756 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:37:05.752+0000 7f8aaffff700 1 --2- 172.21.15.45:0/3835014843 >> [v2:172.21.15.45:6800/2012105536,v1:172.21.15.45:6801/2012105536] conn(0x7f8ab0068f60 0x7f8ab006b410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:37:05.756 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:37:05.752+0000 7f8ac77fe700 1 --2- 172.21.15.45:0/3835014843 >> [v2:172.21.15.45:6800/2012105536,v1:172.21.15.45:6801/2012105536] conn(0x7f8ab0068f60 0x7f8ab006b410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:37:05.756 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:37:05.753+0000 7f8aaffff700 1 -- 172.21.15.45:0/3835014843 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(37..37 src has 1..37) v4 ==== 4634+0+0 (secure 0 0 0) 0x7f8abc0868a0 con 0x7f8ac81035a0 2023-12-16T21:37:05.756 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:37:05.753+0000 7f8ac77fe700 1 --2- 172.21.15.45:0/3835014843 >> [v2:172.21.15.45:6800/2012105536,v1:172.21.15.45:6801/2012105536] conn(0x7f8ab0068f60 0x7f8ab006b410 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7f8ab8004f90 tx=0x7f8ab8009e30 comp rx=0 tx=0).ready entity=mgr.14214 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:37:05.759 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:37:05.757+0000 7f8aaffff700 1 -- 172.21.15.45:0/3835014843 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+174570 (secure 0 0 0) 0x7f8abc056ac0 con 0x7f8ac81035a0 2023-12-16T21:37:05.951 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:37:05.949+0000 7f8ace1bd700 1 -- 172.21.15.45:0/3835014843 --> [v2:172.21.15.45:6800/2012105536,v1:172.21.15.45:6801/2012105536] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f8ac810ec10 con 0x7f8ab0068f60 2023-12-16T21:37:05.952 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:37:05.950+0000 7f8aaffff700 1 -- 172.21.15.45:0/3835014843 <== mgr.14214 v2:172.21.15.45:6800/2012105536 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+21897 (secure 0 0 0) 0x7f8ac810ec10 con 0x7f8ab0068f60 2023-12-16T21:37:05.953 INFO:teuthology.orchestra.run.smithi045.stdout: 2023-12-16T21:37:05.955 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:37:05.952+0000 7f8ace1bd700 1 -- 172.21.15.45:0/3835014843 >> [v2:172.21.15.45:6800/2012105536,v1:172.21.15.45:6801/2012105536] conn(0x7f8ab0068f60 msgr2=0x7f8ab006b410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:37:05.955 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:37:05.952+0000 7f8ace1bd700 1 --2- 172.21.15.45:0/3835014843 >> [v2:172.21.15.45:6800/2012105536,v1:172.21.15.45:6801/2012105536] conn(0x7f8ab0068f60 0x7f8ab006b410 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7f8ab8004f90 tx=0x7f8ab8009e30 comp rx=0 tx=0).stop 2023-12-16T21:37:05.955 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:37:05.952+0000 7f8ace1bd700 1 -- 172.21.15.45:0/3835014843 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f8ac81035a0 msgr2=0x7f8ac8113bb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:37:05.955 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:37:05.952+0000 7f8ace1bd700 1 --2- 172.21.15.45:0/3835014843 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f8ac81035a0 0x7f8ac8113bb0 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 crypto rx=0x7f8abc002a50 tx=0x7f8abc004600 comp rx=0 tx=0).stop 2023-12-16T21:37:05.955 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:37:05.952+0000 7f8ace1bd700 1 -- 172.21.15.45:0/3835014843 shutdown_connections 2023-12-16T21:37:05.955 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:37:05.952+0000 7f8ace1bd700 1 --2- 172.21.15.45:0/3835014843 >> [v2:172.21.15.45:6800/2012105536,v1:172.21.15.45:6801/2012105536] conn(0x7f8ab0068f60 0x7f8ab006b410 unknown :-1 s=CLOSED pgs=81 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:37:05.955 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:37:05.952+0000 7f8ace1bd700 1 --2- 172.21.15.45:0/3835014843 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f8ac8102c00 0x7f8ac8113630 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:37:05.955 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:37:05.952+0000 7f8ace1bd700 1 --2- 172.21.15.45:0/3835014843 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f8ac81035a0 0x7f8ac8113bb0 unknown :-1 s=CLOSED pgs=225 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:37:05.955 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:37:05.952+0000 7f8ace1bd700 1 -- 172.21.15.45:0/3835014843 >> 172.21.15.45:0/3835014843 conn(0x7f8ac80f8790 msgr2=0x7f8ac80fa280 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-16T21:37:05.955 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:37:05.952+0000 7f8ace1bd700 1 -- 172.21.15.45:0/3835014843 shutdown_connections 2023-12-16T21:37:05.955 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:37:05.952+0000 7f8ace1bd700 1 -- 172.21.15.45:0/3835014843 wait complete. 2023-12-16T21:37:05.956 INFO:teuthology.orchestra.run.smithi045.stderr:dumped all 2023-12-16T21:37:06.133 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Dec 16 21:37:05 smithi169 ceph-mon[88350]: from='client.14424 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-12-16T21:37:06.133 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Dec 16 21:37:05 smithi169 ceph-mon[88350]: pgmap v103: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2023-12-16T21:37:06.238 INFO:teuthology.orchestra.run.smithi045.stdout:{"pg_ready":true,"pg_map":{"version":103,"stamp":"2023-12-16T21:37:04.611746+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":3,"acting":3,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":3,"num_osds":6,"num_per_pool_osds":6,"num_per_pool_omap_osds":3,"kb":562470912,"kb_used":1787436,"kb_used_data":3708,"kb_used_omap":0,"kb_used_meta":1783680,"kb_avail":560683476,"statfs":{"total":575970213888,"available":574139879424,"internally_reserved":0,"allocated":3796992,"data_stored":2542338,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":1826488320},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001927"},"pg_stats":[{"pgid":"1.0","version":"22'117","reported_seq":188,"reported_epoch":36,"state":"active+clean","last_fresh":"2023-12-16T21:36:38.722612+0000","last_change":"2023-12-16T21:36:22.552736+0000","last_active":"2023-12-16T21:36:38.722612+0000","last_peered":"2023-12-16T21:36:38.722612+0000","last_clean":"2023-12-16T21:36:38.722612+0000","last_became_active":"2023-12-16T21:36:22.552458+0000","last_became_peered":"2023-12-16T21:36:22.552458+0000","last_unstale":"2023-12-16T21:36:38.722612+0000","last_undegraded":"2023-12-16T21:36:38.722612+0000","last_fullsized":"2023-12-16T21:36:38.722612+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T21:35:53.103103+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T21:35:53.103103+0000","last_clean_scrub_stamp":"2023-12-16T21:35:53.103103+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-18T01:22:13.951675+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":2375680,"data_stored":2361472,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":3,"acting":3,"num_store_stats":4}],"osd_stats":[{"osd":5,"up_from":36,"seq":154618822664,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":297616,"kb_used_data":328,"kb_used_omap":0,"kb_used_meta":297280,"kb_avail":93447536,"statfs":{"total":95995035648,"available":95690276864,"internally_reserved":0,"allocated":335872,"data_stored":128539,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304414720},"hb_peers":[0,1,2,3,4],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.51900000000000002}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.433}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.40200000000000002}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.34499999999999997}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.42099999999999999}]}]},{"osd":4,"up_from":32,"seq":137438953483,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":297616,"kb_used_data":328,"kb_used_omap":0,"kb_used_meta":297280,"kb_avail":93447536,"statfs":{"total":95995035648,"available":95690276864,"internally_reserved":0,"allocated":335872,"data_stored":128539,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304414720},"hb_peers":[0,1,2,3,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.56000000000000005}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.52400000000000002}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.54900000000000004}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.45500000000000002}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.47499999999999998}]}]},{"osd":3,"up_from":25,"seq":107374182414,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":298196,"kb_used_data":908,"kb_used_omap":0,"kb_used_meta":297280,"kb_avail":93446956,"statfs":{"total":95995035648,"available":95689682944,"internally_reserved":0,"allocated":929792,"data_stored":718907,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304414720},"hb_peers":[0,1,2,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.46899999999999997}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":18.247}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.44}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.49299999999999999}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.55400000000000005}]}]},{"osd":2,"up_from":18,"seq":77309411344,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":297616,"kb_used_data":328,"kb_used_omap":0,"kb_used_meta":297280,"kb_avail":93447536,"statfs":{"total":95995035648,"available":95690276864,"internally_reserved":0,"allocated":335872,"data_stored":128539,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304414720},"hb_peers":[0,1,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Dec 16 21:36:52 2023","interfaces":[{"interface":"back","average":{"1min":0.32100000000000001,"5min":0.32100000000000001,"15min":0.32100000000000001},"min":{"1min":0.21299999999999999,"5min":0.21299999999999999,"15min":0.21299999999999999},"max":{"1min":0.42699999999999999,"5min":0.42699999999999999,"15min":0.42699999999999999},"last":0.245},{"interface":"front","average":{"1min":0.33900000000000002,"5min":0.33900000000000002,"15min":0.33900000000000002},"min":{"1min":0.26200000000000001,"5min":0.26200000000000001,"15min":0.26200000000000001},"max":{"1min":0.44900000000000001,"5min":0.44900000000000001,"15min":0.44900000000000001},"last":0.27500000000000002}]},{"osd":1,"last update":"Sat Dec 16 21:36:52 2023","interfaces":[{"interface":"back","average":{"1min":0.38700000000000001,"5min":0.38700000000000001,"15min":0.38700000000000001},"min":{"1min":0.27900000000000003,"5min":0.27900000000000003,"15min":0.27900000000000003},"max":{"1min":0.48599999999999999,"5min":0.48599999999999999,"15min":0.48599999999999999},"last":0.30499999999999999},{"interface":"front","average":{"1min":0.40000000000000002,"5min":0.40000000000000002,"15min":0.40000000000000002},"min":{"1min":0.30199999999999999,"5min":0.30199999999999999,"15min":0.30199999999999999},"max":{"1min":0.52000000000000002,"5min":0.52000000000000002,"15min":0.52000000000000002},"last":0.315}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.63500000000000001}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.621}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.53400000000000003}]}]},{"osd":0,"up_from":9,"seq":38654705688,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":298196,"kb_used_data":908,"kb_used_omap":0,"kb_used_meta":297280,"kb_avail":93446956,"statfs":{"total":95995035648,"available":95689682944,"internally_reserved":0,"allocated":929792,"data_stored":718907,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304414720},"hb_peers":[1,2,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Dec 16 21:36:38 2023","interfaces":[{"interface":"back","average":{"1min":0.34499999999999997,"5min":0.34499999999999997,"15min":0.34499999999999997},"min":{"1min":0.215,"5min":0.215,"15min":0.215},"max":{"1min":0.54700000000000004,"5min":0.54700000000000004,"15min":0.54700000000000004},"last":0.311},{"interface":"front","average":{"1min":0.36399999999999999,"5min":0.36399999999999999,"15min":0.36399999999999999},"min":{"1min":0.25900000000000001,"5min":0.25900000000000001,"15min":0.25900000000000001},"max":{"1min":0.49099999999999999,"5min":0.49099999999999999,"15min":0.49099999999999999},"last":0.378}]},{"osd":2,"last update":"Sat Dec 16 21:36:54 2023","interfaces":[{"interface":"back","average":{"1min":0.34799999999999998,"5min":0.34799999999999998,"15min":0.34799999999999998},"min":{"1min":0.27000000000000002,"5min":0.27000000000000002,"15min":0.27000000000000002},"max":{"1min":0.47799999999999998,"5min":0.47799999999999998,"15min":0.47799999999999998},"last":0.32800000000000001},{"interface":"front","average":{"1min":0.36699999999999999,"5min":0.36699999999999999,"15min":0.36699999999999999},"min":{"1min":0.29199999999999998,"5min":0.29199999999999998,"15min":0.29199999999999998},"max":{"1min":0.45500000000000002,"5min":0.45500000000000002,"15min":0.45500000000000002},"last":0.36199999999999999}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.76700000000000002}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.75700000000000001}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.80200000000000005}]}]},{"osd":1,"up_from":14,"seq":60129542165,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":298196,"kb_used_data":908,"kb_used_omap":0,"kb_used_meta":297280,"kb_avail":93446956,"statfs":{"total":95995035648,"available":95689682944,"internally_reserved":0,"allocated":929792,"data_stored":718907,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304414720},"hb_peers":[0,2,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Dec 16 21:36:37 2023","interfaces":[{"interface":"back","average":{"1min":0.36199999999999999,"5min":0.36199999999999999,"15min":0.36199999999999999},"min":{"1min":0.20999999999999999,"5min":0.20999999999999999,"15min":0.20999999999999999},"max":{"1min":0.51700000000000002,"5min":0.51700000000000002,"15min":0.51700000000000002},"last":0.29899999999999999},{"interface":"front","average":{"1min":0.36799999999999999,"5min":0.36799999999999999,"15min":0.36799999999999999},"min":{"1min":0.22600000000000001,"5min":0.22600000000000001,"15min":0.22600000000000001},"max":{"1min":0.59199999999999997,"5min":0.59199999999999997,"15min":0.59199999999999997},"last":0.317}]},{"osd":2,"last update":"Sat Dec 16 21:36:52 2023","interfaces":[{"interface":"back","average":{"1min":0.40100000000000002,"5min":0.40100000000000002,"15min":0.40100000000000002},"min":{"1min":0.252,"5min":0.252,"15min":0.252},"max":{"1min":0.627,"5min":0.627,"15min":0.627},"last":0.36099999999999999},{"interface":"front","average":{"1min":0.40300000000000002,"5min":0.40300000000000002,"15min":0.40300000000000002},"min":{"1min":0.22900000000000001,"5min":0.22900000000000001,"15min":0.22900000000000001},"max":{"1min":0.56799999999999995,"5min":0.56799999999999995,"15min":0.56799999999999995},"last":0.34699999999999998}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.58899999999999997}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.54900000000000004}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.53100000000000003}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-12-16T21:37:06.238 INFO:tasks.cephadm.ceph_manager.ceph:clean! 2023-12-16T21:37:06.239 INFO:tasks.ceph:Waiting until ceph cluster ceph is healthy... 2023-12-16T21:37:06.239 INFO:tasks.cephadm.ceph_manager.ceph:wait_until_healthy 2023-12-16T21:37:06.239 DEBUG:teuthology.orchestra.run.smithi045:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 88ff363c-9c5a-11ee-95a5-87774f69a715 -- ceph health --format=json 2023-12-16T21:37:06.501 INFO:teuthology.orchestra.run.smithi045.stderr:Inferring config /var/lib/ceph/88ff363c-9c5a-11ee-95a5-87774f69a715/mon.smithi045/config 2023-12-16T21:37:07.279 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:37:07.276+0000 7f2875a8c700 1 -- 172.21.15.45:0/1846645788 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f28700fdc50 msgr2=0x7f28700fe020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:37:07.279 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:37:07.276+0000 7f2875a8c700 1 --2- 172.21.15.45:0/1846645788 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f28700fdc50 0x7f28700fe020 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 crypto rx=0x7f2860004940 tx=0x7f2860009e30 comp rx=0 tx=0).stop 2023-12-16T21:37:07.279 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:37:07.276+0000 7f2875a8c700 1 -- 172.21.15.45:0/1846645788 shutdown_connections 2023-12-16T21:37:07.279 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:37:07.276+0000 7f2875a8c700 1 --2- 172.21.15.45:0/1846645788 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f28700fe5f0 0x7f2870106c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:37:07.280 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:37:07.276+0000 7f2875a8c700 1 --2- 172.21.15.45:0/1846645788 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f28700fdc50 0x7f28700fe020 unknown :-1 s=CLOSED pgs=226 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:37:07.280 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:37:07.276+0000 7f2875a8c700 1 -- 172.21.15.45:0/1846645788 >> 172.21.15.45:0/1846645788 conn(0x7f28700f7680 msgr2=0x7f28700f9a90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-16T21:37:07.280 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:37:07.276+0000 7f2875a8c700 1 -- 172.21.15.45:0/1846645788 shutdown_connections 2023-12-16T21:37:07.280 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:37:07.276+0000 7f2875a8c700 1 -- 172.21.15.45:0/1846645788 wait complete. 2023-12-16T21:37:07.280 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:37:07.276+0000 7f2875a8c700 1 Processor -- start 2023-12-16T21:37:07.280 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:37:07.276+0000 7f2875a8c700 1 -- start start 2023-12-16T21:37:07.280 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:37:07.277+0000 7f2875a8c700 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f28700fdc50 0x7f2870113660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:37:07.280 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:37:07.277+0000 7f2875a8c700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f28700fe5f0 0x7f2870113be0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:37:07.280 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:37:07.277+0000 7f2875a8c700 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2870114230 con 0x7f28700fe5f0 2023-12-16T21:37:07.280 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:37:07.277+0000 7f2875a8c700 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f287010d520 con 0x7f28700fdc50 2023-12-16T21:37:07.280 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:37:07.277+0000 7f286effd700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f28700fe5f0 0x7f2870113be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:37:07.280 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:37:07.277+0000 7f286effd700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f28700fe5f0 0x7f2870113be0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:60782/0 (socket says 172.21.15.45:60782) 2023-12-16T21:37:07.280 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:37:07.277+0000 7f286effd700 1 -- 172.21.15.45:0/1860227013 learned_addr learned my addr 172.21.15.45:0/1860227013 (peer_addr_for_me v2:172.21.15.45:0/0) 2023-12-16T21:37:07.281 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:37:07.277+0000 7f286f7fe700 1 --2- 172.21.15.45:0/1860227013 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f28700fdc50 0x7f2870113660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:37:07.282 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:37:07.277+0000 7f286effd700 1 -- 172.21.15.45:0/1860227013 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f28700fdc50 msgr2=0x7f2870113660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:37:07.282 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:37:07.277+0000 7f286effd700 1 --2- 172.21.15.45:0/1860227013 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f28700fdc50 0x7f2870113660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:37:07.282 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:37:07.277+0000 7f286effd700 1 -- 172.21.15.45:0/1860227013 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2860005040 con 0x7f28700fe5f0 2023-12-16T21:37:07.282 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:37:07.277+0000 7f286effd700 1 --2- 172.21.15.45:0/1860227013 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f28700fe5f0 0x7f2870113be0 secure :-1 s=READY pgs=227 cs=0 l=1 rev1=1 crypto rx=0x7f286400c830 tx=0x7f2864004470 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:37:07.282 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:37:07.278+0000 7f2857fff700 1 -- 172.21.15.45:0/1860227013 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f286400a760 con 0x7f28700fe5f0 2023-12-16T21:37:07.282 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:37:07.278+0000 7f2857fff700 1 -- 172.21.15.45:0/1860227013 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f286400ad70 con 0x7f28700fe5f0 2023-12-16T21:37:07.282 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:37:07.278+0000 7f2857fff700 1 -- 172.21.15.45:0/1860227013 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f28640113a0 con 0x7f28700fe5f0 2023-12-16T21:37:07.282 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:37:07.278+0000 7f2875a8c700 1 -- 172.21.15.45:0/1860227013 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f287010d7b0 con 0x7f28700fe5f0 2023-12-16T21:37:07.282 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:37:07.278+0000 7f2875a8c700 1 -- 172.21.15.45:0/1860227013 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f287010dca0 con 0x7f28700fe5f0 2023-12-16T21:37:07.282 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:37:07.279+0000 7f2875a8c700 1 -- 172.21.15.45:0/1860227013 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2870104240 con 0x7f28700fe5f0 2023-12-16T21:37:07.282 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:37:07.279+0000 7f2857fff700 1 -- 172.21.15.45:0/1860227013 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 19) v1 ==== 86447+0+0 (secure 0 0 0) 0x7f2864012400 con 0x7f28700fe5f0 2023-12-16T21:37:07.283 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:37:07.280+0000 7f2857fff700 1 --2- 172.21.15.45:0/1860227013 >> [v2:172.21.15.45:6800/2012105536,v1:172.21.15.45:6801/2012105536] conn(0x7f2858068f60 0x7f285806b410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:37:07.283 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:37:07.280+0000 7f2857fff700 1 -- 172.21.15.45:0/1860227013 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(37..37 src has 1..37) v4 ==== 4634+0+0 (secure 0 0 0) 0x7f2864086ad0 con 0x7f28700fe5f0 2023-12-16T21:37:07.283 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:37:07.280+0000 7f286f7fe700 1 --2- 172.21.15.45:0/1860227013 >> [v2:172.21.15.45:6800/2012105536,v1:172.21.15.45:6801/2012105536] conn(0x7f2858068f60 0x7f285806b410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:37:07.283 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:37:07.280+0000 7f286f7fe700 1 --2- 172.21.15.45:0/1860227013 >> [v2:172.21.15.45:6800/2012105536,v1:172.21.15.45:6801/2012105536] conn(0x7f2858068f60 0x7f285806b410 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7f286000b3b0 tx=0x7f2860009e30 comp rx=0 tx=0).ready entity=mgr.14214 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:37:07.286 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:37:07.284+0000 7f2857fff700 1 -- 172.21.15.45:0/1860227013 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+174570 (secure 0 0 0) 0x7f2864056d70 con 0x7f28700fe5f0 2023-12-16T21:37:07.528 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:37:07.525+0000 7f2875a8c700 1 -- 172.21.15.45:0/1860227013 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7f28700fb280 con 0x7f28700fe5f0 2023-12-16T21:37:07.528 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Dec 16 21:37:07 smithi045 ceph-mon[80763]: from='client.14428 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-12-16T21:37:07.528 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Dec 16 21:37:07 smithi045 ceph-mon[80763]: pgmap v104: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2023-12-16T21:37:07.529 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:37:07.526+0000 7f2857fff700 1 -- 172.21.15.45:0/1860227013 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+46 (secure 0 0 0) 0x7f2864056900 con 0x7f28700fe5f0 2023-12-16T21:37:07.529 INFO:teuthology.orchestra.run.smithi045.stdout: 2023-12-16T21:37:07.529 INFO:teuthology.orchestra.run.smithi045.stdout:{"status":"HEALTH_OK","checks":{},"mutes":[]} 2023-12-16T21:37:07.530 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:37:07.528+0000 7f2875a8c700 1 -- 172.21.15.45:0/1860227013 >> [v2:172.21.15.45:6800/2012105536,v1:172.21.15.45:6801/2012105536] conn(0x7f2858068f60 msgr2=0x7f285806b410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:37:07.531 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:37:07.528+0000 7f2875a8c700 1 --2- 172.21.15.45:0/1860227013 >> [v2:172.21.15.45:6800/2012105536,v1:172.21.15.45:6801/2012105536] conn(0x7f2858068f60 0x7f285806b410 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7f286000b3b0 tx=0x7f2860009e30 comp rx=0 tx=0).stop 2023-12-16T21:37:07.531 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:37:07.528+0000 7f2875a8c700 1 -- 172.21.15.45:0/1860227013 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f28700fe5f0 msgr2=0x7f2870113be0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:37:07.531 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:37:07.528+0000 7f2875a8c700 1 --2- 172.21.15.45:0/1860227013 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f28700fe5f0 0x7f2870113be0 secure :-1 s=READY pgs=227 cs=0 l=1 rev1=1 crypto rx=0x7f286400c830 tx=0x7f2864004470 comp rx=0 tx=0).stop 2023-12-16T21:37:07.531 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:37:07.528+0000 7f2875a8c700 1 -- 172.21.15.45:0/1860227013 shutdown_connections 2023-12-16T21:37:07.531 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:37:07.528+0000 7f2875a8c700 1 --2- 172.21.15.45:0/1860227013 >> [v2:172.21.15.45:6800/2012105536,v1:172.21.15.45:6801/2012105536] conn(0x7f2858068f60 0x7f285806b410 unknown :-1 s=CLOSED pgs=82 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:37:07.531 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:37:07.528+0000 7f2875a8c700 1 --2- 172.21.15.45:0/1860227013 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f28700fdc50 0x7f2870113660 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:37:07.531 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:37:07.528+0000 7f2875a8c700 1 --2- 172.21.15.45:0/1860227013 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f28700fe5f0 0x7f2870113be0 unknown :-1 s=CLOSED pgs=227 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:37:07.531 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:37:07.528+0000 7f2875a8c700 1 -- 172.21.15.45:0/1860227013 >> 172.21.15.45:0/1860227013 conn(0x7f28700f7680 msgr2=0x7f28701013c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-16T21:37:07.531 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:37:07.528+0000 7f2875a8c700 1 -- 172.21.15.45:0/1860227013 shutdown_connections 2023-12-16T21:37:07.531 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:37:07.528+0000 7f2875a8c700 1 -- 172.21.15.45:0/1860227013 wait complete. 2023-12-16T21:37:07.633 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Dec 16 21:37:07 smithi169 ceph-mon[88350]: from='client.14428 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-12-16T21:37:07.633 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Dec 16 21:37:07 smithi169 ceph-mon[88350]: pgmap v104: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2023-12-16T21:37:07.789 INFO:tasks.cephadm.ceph_manager.ceph:wait_until_healthy done 2023-12-16T21:37:07.789 INFO:tasks.cephadm:Setup complete, yielding 2023-12-16T21:37:07.790 INFO:teuthology.run_tasks:Running task print... 2023-12-16T21:37:07.799 INFO:teuthology.task.print:**** done end installing quincy cephadm ... 2023-12-16T21:37:07.800 INFO:teuthology.run_tasks:Running task cephadm.shell... 2023-12-16T21:37:07.825 INFO:tasks.cephadm:Running commands on role host.a host ubuntu@smithi045.front.sepia.ceph.com 2023-12-16T21:37:07.826 DEBUG:teuthology.orchestra.run.smithi045:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 88ff363c-9c5a-11ee-95a5-87774f69a715 -- bash -c 'ceph config set mgr mgr/cephadm/use_repo_digest true --force' 2023-12-16T21:37:08.095 INFO:teuthology.orchestra.run.smithi045.stderr:Inferring config /var/lib/ceph/88ff363c-9c5a-11ee-95a5-87774f69a715/mon.smithi045/config 2023-12-16T21:37:08.485 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Dec 16 21:37:08 smithi045 ceph-mon[80763]: from='client.? 172.21.15.45:0/1860227013' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2023-12-16T21:37:08.633 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Dec 16 21:37:08 smithi169 ceph-mon[88350]: from='client.? 172.21.15.45:0/1860227013' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2023-12-16T21:37:08.812 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:37:08.808+0000 7fa274177700 1 -- 172.21.15.45:0/3263624958 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fa26c0fde00 msgr2=0x7fa26c0fe250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:37:08.812 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:37:08.808+0000 7fa274177700 1 --2- 172.21.15.45:0/3263624958 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fa26c0fde00 0x7fa26c0fe250 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 crypto rx=0x7fa25c004940 tx=0x7fa25c009e30 comp rx=0 tx=0).stop 2023-12-16T21:37:08.813 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:37:08.808+0000 7fa274177700 1 -- 172.21.15.45:0/3263624958 shutdown_connections 2023-12-16T21:37:08.813 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:37:08.808+0000 7fa274177700 1 --2- 172.21.15.45:0/3263624958 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fa26c0fde00 0x7fa26c0fe250 unknown :-1 s=CLOSED pgs=228 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:37:08.813 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:37:08.808+0000 7fa274177700 1 --2- 172.21.15.45:0/3263624958 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fa26c0fcbf0 0x7fa26c0fd000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:37:08.813 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:37:08.808+0000 7fa274177700 1 -- 172.21.15.45:0/3263624958 >> 172.21.15.45:0/3263624958 conn(0x7fa26c0f82d0 msgr2=0x7fa26c0fa6e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-16T21:37:08.813 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:37:08.808+0000 7fa274177700 1 -- 172.21.15.45:0/3263624958 shutdown_connections 2023-12-16T21:37:08.813 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:37:08.808+0000 7fa274177700 1 -- 172.21.15.45:0/3263624958 wait complete. 2023-12-16T21:37:08.813 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:37:08.809+0000 7fa274177700 1 Processor -- start 2023-12-16T21:37:08.813 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:37:08.809+0000 7fa274177700 1 -- start start 2023-12-16T21:37:08.813 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:37:08.809+0000 7fa274177700 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fa26c0fcbf0 0x7fa26c071b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:37:08.813 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:37:08.809+0000 7fa274177700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fa26c0fde00 0x7fa26c06ff90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:37:08.813 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:37:08.809+0000 7fa274177700 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa26c070590 con 0x7fa26c0fde00 2023-12-16T21:37:08.813 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:37:08.809+0000 7fa274177700 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa26c070700 con 0x7fa26c0fcbf0 2023-12-16T21:37:08.814 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:37:08.809+0000 7fa271712700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fa26c0fde00 0x7fa26c06ff90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:37:08.814 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:37:08.809+0000 7fa271712700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fa26c0fde00 0x7fa26c06ff90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:60810/0 (socket says 172.21.15.45:60810) 2023-12-16T21:37:08.814 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:37:08.809+0000 7fa271712700 1 -- 172.21.15.45:0/1022190524 learned_addr learned my addr 172.21.15.45:0/1022190524 (peer_addr_for_me v2:172.21.15.45:0/0) 2023-12-16T21:37:08.814 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:37:08.809+0000 7fa271f13700 1 --2- 172.21.15.45:0/1022190524 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fa26c0fcbf0 0x7fa26c071b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:37:08.814 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:37:08.809+0000 7fa271712700 1 -- 172.21.15.45:0/1022190524 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fa26c0fcbf0 msgr2=0x7fa26c071b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:37:08.814 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:37:08.809+0000 7fa271712700 1 --2- 172.21.15.45:0/1022190524 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fa26c0fcbf0 0x7fa26c071b10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:37:08.814 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:37:08.809+0000 7fa271712700 1 -- 172.21.15.45:0/1022190524 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa25c005040 con 0x7fa26c0fde00 2023-12-16T21:37:08.814 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:37:08.810+0000 7fa271712700 1 --2- 172.21.15.45:0/1022190524 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fa26c0fde00 0x7fa26c06ff90 secure :-1 s=READY pgs=229 cs=0 l=1 rev1=1 crypto rx=0x7fa25c009fd0 tx=0x7fa25c009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:37:08.814 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:37:08.810+0000 7fa2627fc700 1 -- 172.21.15.45:0/1022190524 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fa25c00e040 con 0x7fa26c0fde00 2023-12-16T21:37:08.814 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:37:08.810+0000 7fa274177700 1 -- 172.21.15.45:0/1022190524 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa26c070980 con 0x7fa26c0fde00 2023-12-16T21:37:08.814 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:37:08.810+0000 7fa2627fc700 1 -- 172.21.15.45:0/1022190524 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fa25c004290 con 0x7fa26c0fde00 2023-12-16T21:37:08.814 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:37:08.810+0000 7fa2627fc700 1 -- 172.21.15.45:0/1022190524 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fa25c01c3f0 con 0x7fa26c0fde00 2023-12-16T21:37:08.814 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:37:08.810+0000 7fa274177700 1 -- 172.21.15.45:0/1022190524 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa26c115540 con 0x7fa26c0fde00 2023-12-16T21:37:08.815 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:37:08.811+0000 7fa274177700 1 -- 172.21.15.45:0/1022190524 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa26c04bb80 con 0x7fa26c0fde00 2023-12-16T21:37:08.815 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:37:08.811+0000 7fa2627fc700 1 -- 172.21.15.45:0/1022190524 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 19) v1 ==== 86447+0+0 (secure 0 0 0) 0x7fa25c013420 con 0x7fa26c0fde00 2023-12-16T21:37:08.815 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:37:08.812+0000 7fa2627fc700 1 --2- 172.21.15.45:0/1022190524 >> [v2:172.21.15.45:6800/2012105536,v1:172.21.15.45:6801/2012105536] conn(0x7fa258068f70 0x7fa25806b420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:37:08.815 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:37:08.812+0000 7fa2627fc700 1 -- 172.21.15.45:0/1022190524 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(37..37 src has 1..37) v4 ==== 4634+0+0 (secure 0 0 0) 0x7fa25c087ed0 con 0x7fa26c0fde00 2023-12-16T21:37:08.815 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:37:08.812+0000 7fa271f13700 1 --2- 172.21.15.45:0/1022190524 >> [v2:172.21.15.45:6800/2012105536,v1:172.21.15.45:6801/2012105536] conn(0x7fa258068f70 0x7fa25806b420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:37:08.816 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:37:08.812+0000 7fa271f13700 1 --2- 172.21.15.45:0/1022190524 >> [v2:172.21.15.45:6800/2012105536,v1:172.21.15.45:6801/2012105536] conn(0x7fa258068f70 0x7fa25806b420 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7fa26c0718a0 tx=0x7fa26800b040 comp rx=0 tx=0).ready entity=mgr.14214 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:37:08.818 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:37:08.816+0000 7fa2627fc700 1 -- 172.21.15.45:0/1022190524 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+174570 (secure 0 0 0) 0x7fa25c0580f0 con 0x7fa26c0fde00 2023-12-16T21:37:09.015 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:37:09.012+0000 7fa274177700 1 -- 172.21.15.45:0/1022190524 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command([{prefix=config set, name=mgr/cephadm/use_repo_digest}] v 0) v1 -- 0x7fa26c1157f0 con 0x7fa26c0fde00 2023-12-16T21:37:09.019 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:37:09.017+0000 7fa2627fc700 1 -- 172.21.15.45:0/1022190524 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/cephadm/use_repo_digest}]=0 v20)=0 v20) v1 ==== 143+0+0 (secure 0 0 0) 0x7fa25c057c80 con 0x7fa26c0fde00 2023-12-16T21:37:09.022 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:37:09.019+0000 7fa274177700 1 -- 172.21.15.45:0/1022190524 >> [v2:172.21.15.45:6800/2012105536,v1:172.21.15.45:6801/2012105536] conn(0x7fa258068f70 msgr2=0x7fa25806b420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:37:09.022 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:37:09.019+0000 7fa274177700 1 --2- 172.21.15.45:0/1022190524 >> [v2:172.21.15.45:6800/2012105536,v1:172.21.15.45:6801/2012105536] conn(0x7fa258068f70 0x7fa25806b420 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7fa26c0718a0 tx=0x7fa26800b040 comp rx=0 tx=0).stop 2023-12-16T21:37:09.022 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:37:09.019+0000 7fa274177700 1 -- 172.21.15.45:0/1022190524 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fa26c0fde00 msgr2=0x7fa26c06ff90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:37:09.022 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:37:09.019+0000 7fa274177700 1 --2- 172.21.15.45:0/1022190524 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fa26c0fde00 0x7fa26c06ff90 secure :-1 s=READY pgs=229 cs=0 l=1 rev1=1 crypto rx=0x7fa25c009fd0 tx=0x7fa25c009e30 comp rx=0 tx=0).stop 2023-12-16T21:37:09.022 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:37:09.019+0000 7fa274177700 1 -- 172.21.15.45:0/1022190524 shutdown_connections 2023-12-16T21:37:09.022 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:37:09.019+0000 7fa274177700 1 --2- 172.21.15.45:0/1022190524 >> [v2:172.21.15.45:6800/2012105536,v1:172.21.15.45:6801/2012105536] conn(0x7fa258068f70 0x7fa25806b420 unknown :-1 s=CLOSED pgs=83 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:37:09.022 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:37:09.019+0000 7fa274177700 1 --2- 172.21.15.45:0/1022190524 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fa26c0fcbf0 0x7fa26c071b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:37:09.022 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:37:09.019+0000 7fa274177700 1 --2- 172.21.15.45:0/1022190524 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fa26c0fde00 0x7fa26c06ff90 unknown :-1 s=CLOSED pgs=229 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:37:09.022 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:37:09.019+0000 7fa274177700 1 -- 172.21.15.45:0/1022190524 >> 172.21.15.45:0/1022190524 conn(0x7fa26c0f82d0 msgr2=0x7fa26c101030 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-16T21:37:09.022 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:37:09.019+0000 7fa274177700 1 -- 172.21.15.45:0/1022190524 shutdown_connections 2023-12-16T21:37:09.022 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:37:09.019+0000 7fa274177700 1 -- 172.21.15.45:0/1022190524 wait complete. 2023-12-16T21:37:09.278 INFO:teuthology.run_tasks:Running task print... 2023-12-16T21:37:09.287 INFO:teuthology.task.print:**** done cephadm.shell ceph config set mgr... 2023-12-16T21:37:09.287 INFO:teuthology.run_tasks:Running task cephadm.shell... 2023-12-16T21:37:09.302 INFO:tasks.cephadm:Running commands on role host.a host ubuntu@smithi045.front.sepia.ceph.com 2023-12-16T21:37:09.302 DEBUG:teuthology.orchestra.run.smithi045:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 88ff363c-9c5a-11ee-95a5-87774f69a715 -- bash -c 'ceph orch status' 2023-12-16T21:37:09.566 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Dec 16 21:37:09 smithi045 ceph-mon[80763]: pgmap v105: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2023-12-16T21:37:09.566 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Dec 16 21:37:09 smithi045 ceph-mon[80763]: from='client.? 172.21.15.45:0/1022190524' entity='client.admin' 2023-12-16T21:37:09.566 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Dec 16 21:37:09 smithi045 ceph-mon[80763]: from='mgr.14214 172.21.15.45:0/1227909437' entity='mgr.smithi045.nhtmwc' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-12-16T21:37:09.566 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Dec 16 21:37:09 smithi045 ceph-mon[80763]: from='mgr.14214 172.21.15.45:0/1227909437' entity='mgr.smithi045.nhtmwc' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-12-16T21:37:09.566 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Dec 16 21:37:09 smithi045 ceph-mon[80763]: from='mgr.14214 172.21.15.45:0/1227909437' entity='mgr.smithi045.nhtmwc' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-12-16T21:37:09.566 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Dec 16 21:37:09 smithi045 ceph-mon[80763]: from='mgr.14214 172.21.15.45:0/1227909437' entity='mgr.smithi045.nhtmwc' 2023-12-16T21:37:09.567 INFO:teuthology.orchestra.run.smithi045.stderr:Inferring config /var/lib/ceph/88ff363c-9c5a-11ee-95a5-87774f69a715/mon.smithi045/config 2023-12-16T21:37:09.633 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Dec 16 21:37:09 smithi169 ceph-mon[88350]: pgmap v105: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2023-12-16T21:37:09.634 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Dec 16 21:37:09 smithi169 ceph-mon[88350]: from='client.? 172.21.15.45:0/1022190524' entity='client.admin' 2023-12-16T21:37:09.634 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Dec 16 21:37:09 smithi169 ceph-mon[88350]: from='mgr.14214 172.21.15.45:0/1227909437' entity='mgr.smithi045.nhtmwc' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-12-16T21:37:09.634 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Dec 16 21:37:09 smithi169 ceph-mon[88350]: from='mgr.14214 172.21.15.45:0/1227909437' entity='mgr.smithi045.nhtmwc' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-12-16T21:37:09.634 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Dec 16 21:37:09 smithi169 ceph-mon[88350]: from='mgr.14214 172.21.15.45:0/1227909437' entity='mgr.smithi045.nhtmwc' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-12-16T21:37:09.634 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Dec 16 21:37:09 smithi169 ceph-mon[88350]: from='mgr.14214 172.21.15.45:0/1227909437' entity='mgr.smithi045.nhtmwc' 2023-12-16T21:37:10.424 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:37:10.420+0000 7fe5d87f7700 1 -- 172.21.15.45:0/4059104299 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fe5d00fde00 msgr2=0x7fe5d00fe250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:37:10.424 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:37:10.420+0000 7fe5d87f7700 1 --2- 172.21.15.45:0/4059104299 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fe5d00fde00 0x7fe5d00fe250 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 crypto rx=0x7fe5c0004940 tx=0x7fe5c0009e30 comp rx=0 tx=0).stop 2023-12-16T21:37:10.425 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:37:10.420+0000 7fe5d87f7700 1 -- 172.21.15.45:0/4059104299 shutdown_connections 2023-12-16T21:37:10.425 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:37:10.420+0000 7fe5d87f7700 1 --2- 172.21.15.45:0/4059104299 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fe5d00fde00 0x7fe5d00fe250 unknown :-1 s=CLOSED pgs=230 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:37:10.425 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:37:10.420+0000 7fe5d87f7700 1 --2- 172.21.15.45:0/4059104299 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fe5d00fcbf0 0x7fe5d00fd000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:37:10.425 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:37:10.420+0000 7fe5d87f7700 1 -- 172.21.15.45:0/4059104299 >> 172.21.15.45:0/4059104299 conn(0x7fe5d00f82d0 msgr2=0x7fe5d00fa6e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-16T21:37:10.425 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:37:10.420+0000 7fe5d87f7700 1 -- 172.21.15.45:0/4059104299 shutdown_connections 2023-12-16T21:37:10.425 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:37:10.420+0000 7fe5d87f7700 1 -- 172.21.15.45:0/4059104299 wait complete. 2023-12-16T21:37:10.425 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:37:10.421+0000 7fe5d87f7700 1 Processor -- start 2023-12-16T21:37:10.425 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:37:10.421+0000 7fe5d87f7700 1 -- start start 2023-12-16T21:37:10.425 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:37:10.421+0000 7fe5d87f7700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fe5d00fcbf0 0x7fe5d0071b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:37:10.425 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:37:10.421+0000 7fe5d87f7700 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fe5d00fde00 0x7fe5d006ff90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:37:10.425 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:37:10.421+0000 7fe5d87f7700 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe5d0070590 con 0x7fe5d00fcbf0 2023-12-16T21:37:10.425 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:37:10.421+0000 7fe5d87f7700 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe5d0070700 con 0x7fe5d00fde00 2023-12-16T21:37:10.425 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:37:10.421+0000 7fe5d6593700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fe5d00fcbf0 0x7fe5d0071b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:37:10.426 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:37:10.421+0000 7fe5d6593700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fe5d00fcbf0 0x7fe5d0071b10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:60830/0 (socket says 172.21.15.45:60830) 2023-12-16T21:37:10.426 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:37:10.421+0000 7fe5d6593700 1 -- 172.21.15.45:0/815833849 learned_addr learned my addr 172.21.15.45:0/815833849 (peer_addr_for_me v2:172.21.15.45:0/0) 2023-12-16T21:37:10.426 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:37:10.421+0000 7fe5d5d92700 1 --2- 172.21.15.45:0/815833849 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fe5d00fde00 0x7fe5d006ff90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:37:10.426 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:37:10.421+0000 7fe5d6593700 1 -- 172.21.15.45:0/815833849 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fe5d00fde00 msgr2=0x7fe5d006ff90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:37:10.426 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:37:10.421+0000 7fe5d6593700 1 --2- 172.21.15.45:0/815833849 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fe5d00fde00 0x7fe5d006ff90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:37:10.426 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:37:10.421+0000 7fe5d6593700 1 -- 172.21.15.45:0/815833849 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe5c0005040 con 0x7fe5d00fcbf0 2023-12-16T21:37:10.426 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:37:10.422+0000 7fe5d6593700 1 --2- 172.21.15.45:0/815833849 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fe5d00fcbf0 0x7fe5d0071b10 secure :-1 s=READY pgs=231 cs=0 l=1 rev1=1 crypto rx=0x7fe5cc00cae0 tx=0x7fe5cc007f60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:37:10.426 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:37:10.422+0000 7fe5c6ffd700 1 -- 172.21.15.45:0/815833849 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fe5cc00a980 con 0x7fe5d00fcbf0 2023-12-16T21:37:10.426 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:37:10.422+0000 7fe5d87f7700 1 -- 172.21.15.45:0/815833849 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe5d00709e0 con 0x7fe5d00fcbf0 2023-12-16T21:37:10.426 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:37:10.422+0000 7fe5c6ffd700 1 -- 172.21.15.45:0/815833849 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fe5cc012440 con 0x7fe5d00fcbf0 2023-12-16T21:37:10.426 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:37:10.422+0000 7fe5c6ffd700 1 -- 172.21.15.45:0/815833849 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fe5cc0115e0 con 0x7fe5d00fcbf0 2023-12-16T21:37:10.426 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:37:10.422+0000 7fe5d87f7700 1 -- 172.21.15.45:0/815833849 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe5d0115540 con 0x7fe5d00fcbf0 2023-12-16T21:37:10.427 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:37:10.423+0000 7fe5d87f7700 1 -- 172.21.15.45:0/815833849 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe5d004bb80 con 0x7fe5d00fcbf0 2023-12-16T21:37:10.427 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:37:10.423+0000 7fe5c6ffd700 1 -- 172.21.15.45:0/815833849 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 19) v1 ==== 86447+0+0 (secure 0 0 0) 0x7fe5cc012950 con 0x7fe5d00fcbf0 2023-12-16T21:37:10.427 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:37:10.424+0000 7fe5c6ffd700 1 --2- 172.21.15.45:0/815833849 >> [v2:172.21.15.45:6800/2012105536,v1:172.21.15.45:6801/2012105536] conn(0x7fe5bc068f60 0x7fe5bc06b410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:37:10.427 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:37:10.424+0000 7fe5c6ffd700 1 -- 172.21.15.45:0/815833849 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(37..37 src has 1..37) v4 ==== 4634+0+0 (secure 0 0 0) 0x7fe5cc0870e0 con 0x7fe5d00fcbf0 2023-12-16T21:37:10.428 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:37:10.424+0000 7fe5d5d92700 1 --2- 172.21.15.45:0/815833849 >> [v2:172.21.15.45:6800/2012105536,v1:172.21.15.45:6801/2012105536] conn(0x7fe5bc068f60 0x7fe5bc06b410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:37:10.428 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:37:10.424+0000 7fe5d5d92700 1 --2- 172.21.15.45:0/815833849 >> [v2:172.21.15.45:6800/2012105536,v1:172.21.15.45:6801/2012105536] conn(0x7fe5bc068f60 0x7fe5bc06b410 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7fe5c0004560 tx=0x7fe5c0009e30 comp rx=0 tx=0).ready entity=mgr.14214 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:37:10.431 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:37:10.428+0000 7fe5c6ffd700 1 -- 172.21.15.45:0/815833849 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+174570 (secure 0 0 0) 0x7fe5cc057300 con 0x7fe5d00fcbf0 2023-12-16T21:37:10.644 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:37:10.642+0000 7fe5d87f7700 1 -- 172.21.15.45:0/815833849 --> [v2:172.21.15.45:6800/2012105536,v1:172.21.15.45:6801/2012105536] -- mgr_command(tid 0: {"prefix": "orch status", "target": ["mon-mgr", ""]}) v1 -- 0x7fe5d0102750 con 0x7fe5bc068f60 2023-12-16T21:37:10.645 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:37:10.643+0000 7fe5c6ffd700 1 -- 172.21.15.45:0/815833849 <== mgr.14214 v2:172.21.15.45:6800/2012105536 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+43 (secure 0 0 0) 0x7fe5d0102750 con 0x7fe5bc068f60 2023-12-16T21:37:10.646 INFO:teuthology.orchestra.run.smithi045.stdout:Backend: cephadm 2023-12-16T21:37:10.646 INFO:teuthology.orchestra.run.smithi045.stdout:Available: Yes 2023-12-16T21:37:10.646 INFO:teuthology.orchestra.run.smithi045.stdout:Paused: No 2023-12-16T21:37:10.648 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:37:10.645+0000 7fe5d87f7700 1 -- 172.21.15.45:0/815833849 >> [v2:172.21.15.45:6800/2012105536,v1:172.21.15.45:6801/2012105536] conn(0x7fe5bc068f60 msgr2=0x7fe5bc06b410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:37:10.648 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:37:10.645+0000 7fe5d87f7700 1 --2- 172.21.15.45:0/815833849 >> [v2:172.21.15.45:6800/2012105536,v1:172.21.15.45:6801/2012105536] conn(0x7fe5bc068f60 0x7fe5bc06b410 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7fe5c0004560 tx=0x7fe5c0009e30 comp rx=0 tx=0).stop 2023-12-16T21:37:10.648 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:37:10.645+0000 7fe5d87f7700 1 -- 172.21.15.45:0/815833849 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fe5d00fcbf0 msgr2=0x7fe5d0071b10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:37:10.648 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:37:10.645+0000 7fe5d87f7700 1 --2- 172.21.15.45:0/815833849 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fe5d00fcbf0 0x7fe5d0071b10 secure :-1 s=READY pgs=231 cs=0 l=1 rev1=1 crypto rx=0x7fe5cc00cae0 tx=0x7fe5cc007f60 comp rx=0 tx=0).stop 2023-12-16T21:37:10.648 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:37:10.645+0000 7fe5d87f7700 1 -- 172.21.15.45:0/815833849 shutdown_connections 2023-12-16T21:37:10.648 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:37:10.645+0000 7fe5d87f7700 1 --2- 172.21.15.45:0/815833849 >> [v2:172.21.15.45:6800/2012105536,v1:172.21.15.45:6801/2012105536] conn(0x7fe5bc068f60 0x7fe5bc06b410 unknown :-1 s=CLOSED pgs=84 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:37:10.648 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:37:10.645+0000 7fe5d87f7700 1 --2- 172.21.15.45:0/815833849 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fe5d00fcbf0 0x7fe5d0071b10 unknown :-1 s=CLOSED pgs=231 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:37:10.649 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:37:10.645+0000 7fe5d87f7700 1 --2- 172.21.15.45:0/815833849 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fe5d00fde00 0x7fe5d006ff90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:37:10.649 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:37:10.645+0000 7fe5d87f7700 1 -- 172.21.15.45:0/815833849 >> 172.21.15.45:0/815833849 conn(0x7fe5d00f82d0 msgr2=0x7fe5d0101030 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-16T21:37:10.649 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:37:10.646+0000 7fe5d87f7700 1 -- 172.21.15.45:0/815833849 shutdown_connections 2023-12-16T21:37:10.649 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:37:10.646+0000 7fe5d87f7700 1 -- 172.21.15.45:0/815833849 wait complete. 2023-12-16T21:37:10.920 DEBUG:teuthology.orchestra.run.smithi045:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 88ff363c-9c5a-11ee-95a5-87774f69a715 -- bash -c 'ceph orch ps' 2023-12-16T21:37:11.187 INFO:teuthology.orchestra.run.smithi045.stderr:Inferring config /var/lib/ceph/88ff363c-9c5a-11ee-95a5-87774f69a715/mon.smithi045/config 2023-12-16T21:37:11.864 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Dec 16 21:37:11 smithi045 ceph-mon[80763]: pgmap v106: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2023-12-16T21:37:11.865 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Dec 16 21:37:11 smithi045 ceph-mon[80763]: from='client.14440 -' entity='client.admin' cmd=[{"prefix": "orch status", "target": ["mon-mgr", ""]}]: dispatch 2023-12-16T21:37:11.998 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:37:11.994+0000 7fa3bfa6d700 1 -- 172.21.15.45:0/3591885337 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fa3b80fde00 msgr2=0x7fa3b80fe250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:37:11.998 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:37:11.994+0000 7fa3bfa6d700 1 --2- 172.21.15.45:0/3591885337 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fa3b80fde00 0x7fa3b80fe250 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 crypto rx=0x7fa3b4004940 tx=0x7fa3b4009e30 comp rx=0 tx=0).stop 2023-12-16T21:37:11.998 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:37:11.994+0000 7fa3bfa6d700 1 -- 172.21.15.45:0/3591885337 shutdown_connections 2023-12-16T21:37:11.998 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:37:11.994+0000 7fa3bfa6d700 1 --2- 172.21.15.45:0/3591885337 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fa3b80fde00 0x7fa3b80fe250 unknown :-1 s=CLOSED pgs=232 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:37:11.998 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:37:11.994+0000 7fa3bfa6d700 1 --2- 172.21.15.45:0/3591885337 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fa3b80fcbf0 0x7fa3b80fd000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:37:11.998 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:37:11.994+0000 7fa3bfa6d700 1 -- 172.21.15.45:0/3591885337 >> 172.21.15.45:0/3591885337 conn(0x7fa3b80f82d0 msgr2=0x7fa3b80fa6e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-16T21:37:11.999 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:37:11.994+0000 7fa3bfa6d700 1 -- 172.21.15.45:0/3591885337 shutdown_connections 2023-12-16T21:37:11.999 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:37:11.994+0000 7fa3bfa6d700 1 -- 172.21.15.45:0/3591885337 wait complete. 2023-12-16T21:37:11.999 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:37:11.994+0000 7fa3bfa6d700 1 Processor -- start 2023-12-16T21:37:11.999 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:37:11.994+0000 7fa3bfa6d700 1 -- start start 2023-12-16T21:37:11.999 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:37:11.994+0000 7fa3bfa6d700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fa3b80fcbf0 0x7fa3b810f730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:37:11.999 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:37:11.994+0000 7fa3bfa6d700 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fa3b80fde00 0x7fa3b810fc70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:37:11.999 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:37:11.994+0000 7fa3bfa6d700 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa3b8110270 con 0x7fa3b80fcbf0 2023-12-16T21:37:11.999 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:37:11.994+0000 7fa3bfa6d700 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa3b8110b10 con 0x7fa3b80fde00 2023-12-16T21:37:11.999 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:37:11.995+0000 7fa3bd809700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fa3b80fcbf0 0x7fa3b810f730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:37:11.999 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:37:11.995+0000 7fa3bd809700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fa3b80fcbf0 0x7fa3b810f730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:60834/0 (socket says 172.21.15.45:60834) 2023-12-16T21:37:11.999 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:37:11.995+0000 7fa3bd809700 1 -- 172.21.15.45:0/2184894265 learned_addr learned my addr 172.21.15.45:0/2184894265 (peer_addr_for_me v2:172.21.15.45:0/0) 2023-12-16T21:37:11.999 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:37:11.995+0000 7fa3bd008700 1 --2- 172.21.15.45:0/2184894265 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fa3b80fde00 0x7fa3b810fc70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:37:11.999 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:37:11.995+0000 7fa3bd809700 1 -- 172.21.15.45:0/2184894265 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fa3b80fde00 msgr2=0x7fa3b810fc70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:37:12.000 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:37:11.995+0000 7fa3bd809700 1 --2- 172.21.15.45:0/2184894265 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fa3b80fde00 0x7fa3b810fc70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:37:12.000 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:37:11.995+0000 7fa3bd809700 1 -- 172.21.15.45:0/2184894265 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa3b4005040 con 0x7fa3b80fcbf0 2023-12-16T21:37:12.000 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:37:11.995+0000 7fa3bd809700 1 --2- 172.21.15.45:0/2184894265 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fa3b80fcbf0 0x7fa3b810f730 secure :-1 s=READY pgs=233 cs=0 l=1 rev1=1 crypto rx=0x7fa3a800c8e0 tx=0x7fa3a80044c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:37:12.000 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:37:11.995+0000 7fa3ae7fc700 1 -- 172.21.15.45:0/2184894265 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fa3a800a760 con 0x7fa3b80fcbf0 2023-12-16T21:37:12.000 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:37:11.995+0000 7fa3ae7fc700 1 -- 172.21.15.45:0/2184894265 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fa3a800ad70 con 0x7fa3b80fcbf0 2023-12-16T21:37:12.000 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:37:11.995+0000 7fa3ae7fc700 1 -- 172.21.15.45:0/2184894265 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fa3a80113a0 con 0x7fa3b80fcbf0 2023-12-16T21:37:12.000 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:37:11.995+0000 7fa3bfa6d700 1 -- 172.21.15.45:0/2184894265 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa3b8110df0 con 0x7fa3b80fcbf0 2023-12-16T21:37:12.000 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:37:11.995+0000 7fa3bfa6d700 1 -- 172.21.15.45:0/2184894265 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa3b8111330 con 0x7fa3b80fcbf0 2023-12-16T21:37:12.001 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:37:11.996+0000 7fa3bfa6d700 1 -- 172.21.15.45:0/2184894265 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa3b8101ea0 con 0x7fa3b80fcbf0 2023-12-16T21:37:12.001 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:37:11.997+0000 7fa3ae7fc700 1 -- 172.21.15.45:0/2184894265 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 19) v1 ==== 86447+0+0 (secure 0 0 0) 0x7fa3a8012400 con 0x7fa3b80fcbf0 2023-12-16T21:37:12.001 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:37:11.997+0000 7fa3ae7fc700 1 --2- 172.21.15.45:0/2184894265 >> [v2:172.21.15.45:6800/2012105536,v1:172.21.15.45:6801/2012105536] conn(0x7fa3a4068f60 0x7fa3a406b410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:37:12.001 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:37:11.997+0000 7fa3ae7fc700 1 -- 172.21.15.45:0/2184894265 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(37..37 src has 1..37) v4 ==== 4634+0+0 (secure 0 0 0) 0x7fa3a8086970 con 0x7fa3b80fcbf0 2023-12-16T21:37:12.001 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:37:11.997+0000 7fa3bd008700 1 --2- 172.21.15.45:0/2184894265 >> [v2:172.21.15.45:6800/2012105536,v1:172.21.15.45:6801/2012105536] conn(0x7fa3a4068f60 0x7fa3a406b410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:37:12.001 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:37:11.998+0000 7fa3bd008700 1 --2- 172.21.15.45:0/2184894265 >> [v2:172.21.15.45:6800/2012105536,v1:172.21.15.45:6801/2012105536] conn(0x7fa3a4068f60 0x7fa3a406b410 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7fa3b4004890 tx=0x7fa3b4009e30 comp rx=0 tx=0).ready entity=mgr.14214 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:37:12.004 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:37:12.002+0000 7fa3ae7fc700 1 -- 172.21.15.45:0/2184894265 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+174570 (secure 0 0 0) 0x7fa3a8056cb0 con 0x7fa3b80fcbf0 2023-12-16T21:37:12.133 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Dec 16 21:37:11 smithi169 ceph-mon[88350]: pgmap v106: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2023-12-16T21:37:12.134 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Dec 16 21:37:11 smithi169 ceph-mon[88350]: from='client.14440 -' entity='client.admin' cmd=[{"prefix": "orch status", "target": ["mon-mgr", ""]}]: dispatch 2023-12-16T21:37:12.205 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:37:12.203+0000 7fa3bfa6d700 1 -- 172.21.15.45:0/2184894265 --> [v2:172.21.15.45:6800/2012105536,v1:172.21.15.45:6801/2012105536] -- mgr_command(tid 0: {"prefix": "orch ps", "target": ["mon-mgr", ""]}) v1 -- 0x7fa3b8111600 con 0x7fa3a4068f60 2023-12-16T21:37:12.214 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:37:12.212+0000 7fa3ae7fc700 1 -- 172.21.15.45:0/2184894265 <== mgr.14214 v2:172.21.15.45:6800/2012105536 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+2718 (secure 0 0 0) 0x7fa3b8111600 con 0x7fa3a4068f60 2023-12-16T21:37:12.214 INFO:teuthology.orchestra.run.smithi045.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2023-12-16T21:37:12.214 INFO:teuthology.orchestra.run.smithi045.stdout:alertmanager.smithi045 smithi045 *:9093,9094 running (2m) 83s ago 3m 26.1M - 0.25.0 c8568f914cd2 d9982231d4a2 2023-12-16T21:37:12.214 INFO:teuthology.orchestra.run.smithi045.stdout:crash.smithi045 smithi045 running (3m) 83s ago 3m 6660k - 17.2.7-125-g1f2f5510 60cc83a9c8f1 fc997f70d946 2023-12-16T21:37:12.215 INFO:teuthology.orchestra.run.smithi045.stdout:crash.smithi169 smithi169 running (2m) 37s ago 2m 6665k - 17.2.7-125-g1f2f5510 60cc83a9c8f1 f29fac8b7321 2023-12-16T21:37:12.215 INFO:teuthology.orchestra.run.smithi045.stdout:grafana.smithi045 smithi045 *:3000 running (2m) 83s ago 3m 82.2M - 9.4.7 954c08fa6188 dddae79633f0 2023-12-16T21:37:12.215 INFO:teuthology.orchestra.run.smithi045.stdout:mgr.smithi045.nhtmwc smithi045 *:9283 running (4m) 83s ago 4m 467M - 17.2.7-125-g1f2f5510 60cc83a9c8f1 a4ec7bd78bc7 2023-12-16T21:37:12.215 INFO:teuthology.orchestra.run.smithi045.stdout:mgr.smithi169.cncbgh smithi169 *:8443,9283 running (2m) 37s ago 2m 431M - 17.2.7-125-g1f2f5510 60cc83a9c8f1 034b205acb79 2023-12-16T21:37:12.215 INFO:teuthology.orchestra.run.smithi045.stdout:mon.smithi045 smithi045 running (4m) 83s ago 4m 42.5M 2048M 17.2.7-125-g1f2f5510 60cc83a9c8f1 1cddbf5059f8 2023-12-16T21:37:12.215 INFO:teuthology.orchestra.run.smithi045.stdout:mon.smithi169 smithi169 running (2m) 37s ago 2m 40.9M 2048M 17.2.7-125-g1f2f5510 60cc83a9c8f1 f964ea9d81bd 2023-12-16T21:37:12.215 INFO:teuthology.orchestra.run.smithi045.stdout:node-exporter.smithi045 smithi045 *:9100 running (3m) 83s ago 3m 16.8M - 1.5.0 0da6a335fe13 c535836aef3c 2023-12-16T21:37:12.215 INFO:teuthology.orchestra.run.smithi045.stdout:node-exporter.smithi169 smithi169 *:9100 running (2m) 37s ago 2m 15.9M - 1.5.0 0da6a335fe13 f6778c39f7f1 2023-12-16T21:37:12.215 INFO:teuthology.orchestra.run.smithi045.stdout:osd.0 smithi045 running (2m) 83s ago 2m 34.9M 4289M 17.2.7-125-g1f2f5510 60cc83a9c8f1 eeea01051a62 2023-12-16T21:37:12.215 INFO:teuthology.orchestra.run.smithi045.stdout:osd.1 smithi045 running (103s) 83s ago 103s 35.9M 4289M 17.2.7-125-g1f2f5510 60cc83a9c8f1 94ded6ce4ce1 2023-12-16T21:37:12.215 INFO:teuthology.orchestra.run.smithi045.stdout:osd.2 smithi045 running (87s) 83s ago 87s 10.5M 4289M 17.2.7-125-g1f2f5510 60cc83a9c8f1 5fc744d14d11 2023-12-16T21:37:12.215 INFO:teuthology.orchestra.run.smithi045.stdout:osd.3 smithi169 running (71s) 37s ago 71s 38.9M 5312M 17.2.7-125-g1f2f5510 60cc83a9c8f1 f5ee44f8b3e2 2023-12-16T21:37:12.215 INFO:teuthology.orchestra.run.smithi045.stdout:osd.4 smithi169 running (55s) 37s ago 55s 35.2M 5312M 17.2.7-125-g1f2f5510 60cc83a9c8f1 8a74d1fa8b75 2023-12-16T21:37:12.215 INFO:teuthology.orchestra.run.smithi045.stdout:osd.5 smithi169 running (40s) 37s ago 40s 10.0M 5312M 17.2.7-125-g1f2f5510 60cc83a9c8f1 9c0a8a497615 2023-12-16T21:37:12.215 INFO:teuthology.orchestra.run.smithi045.stdout:prometheus.smithi045 smithi045 *:9095 running (2m) 83s ago 2m 32.9M - 2.43.0 a07b618ecd1d 1e1226889abe 2023-12-16T21:37:12.216 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:37:12.214+0000 7fa3bfa6d700 1 -- 172.21.15.45:0/2184894265 >> [v2:172.21.15.45:6800/2012105536,v1:172.21.15.45:6801/2012105536] conn(0x7fa3a4068f60 msgr2=0x7fa3a406b410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:37:12.216 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:37:12.214+0000 7fa3bfa6d700 1 --2- 172.21.15.45:0/2184894265 >> [v2:172.21.15.45:6800/2012105536,v1:172.21.15.45:6801/2012105536] conn(0x7fa3a4068f60 0x7fa3a406b410 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7fa3b4004890 tx=0x7fa3b4009e30 comp rx=0 tx=0).stop 2023-12-16T21:37:12.217 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:37:12.214+0000 7fa3bfa6d700 1 -- 172.21.15.45:0/2184894265 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fa3b80fcbf0 msgr2=0x7fa3b810f730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:37:12.217 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:37:12.214+0000 7fa3bfa6d700 1 --2- 172.21.15.45:0/2184894265 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fa3b80fcbf0 0x7fa3b810f730 secure :-1 s=READY pgs=233 cs=0 l=1 rev1=1 crypto rx=0x7fa3a800c8e0 tx=0x7fa3a80044c0 comp rx=0 tx=0).stop 2023-12-16T21:37:12.217 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:37:12.214+0000 7fa3bfa6d700 1 -- 172.21.15.45:0/2184894265 shutdown_connections 2023-12-16T21:37:12.217 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:37:12.214+0000 7fa3bfa6d700 1 --2- 172.21.15.45:0/2184894265 >> [v2:172.21.15.45:6800/2012105536,v1:172.21.15.45:6801/2012105536] conn(0x7fa3a4068f60 0x7fa3a406b410 unknown :-1 s=CLOSED pgs=85 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:37:12.217 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:37:12.214+0000 7fa3bfa6d700 1 --2- 172.21.15.45:0/2184894265 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fa3b80fcbf0 0x7fa3b810f730 unknown :-1 s=CLOSED pgs=233 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:37:12.217 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:37:12.214+0000 7fa3bfa6d700 1 --2- 172.21.15.45:0/2184894265 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fa3b80fde00 0x7fa3b810fc70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:37:12.217 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:37:12.214+0000 7fa3bfa6d700 1 -- 172.21.15.45:0/2184894265 >> 172.21.15.45:0/2184894265 conn(0x7fa3b80f82d0 msgr2=0x7fa3b80ff020 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-16T21:37:12.217 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:37:12.214+0000 7fa3bfa6d700 1 -- 172.21.15.45:0/2184894265 shutdown_connections 2023-12-16T21:37:12.217 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:37:12.214+0000 7fa3bfa6d700 1 -- 172.21.15.45:0/2184894265 wait complete. 2023-12-16T21:37:12.469 DEBUG:teuthology.orchestra.run.smithi045:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 88ff363c-9c5a-11ee-95a5-87774f69a715 -- bash -c 'ceph orch ls' 2023-12-16T21:37:12.734 INFO:teuthology.orchestra.run.smithi045.stderr:Inferring config /var/lib/ceph/88ff363c-9c5a-11ee-95a5-87774f69a715/mon.smithi045/config 2023-12-16T21:37:13.459 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:37:13.452+0000 7f6bab051700 1 -- 172.21.15.45:0/3668547047 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f6ba40ffd90 msgr2=0x7f6ba4102170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:37:13.459 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:37:13.452+0000 7f6bab051700 1 --2- 172.21.15.45:0/3668547047 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f6ba40ffd90 0x7f6ba4102170 secure :-1 s=READY pgs=234 cs=0 l=1 rev1=1 crypto rx=0x7f6b90004940 tx=0x7f6b90009e30 comp rx=0 tx=0).stop 2023-12-16T21:37:13.459 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:37:13.452+0000 7f6bab051700 1 -- 172.21.15.45:0/3668547047 shutdown_connections 2023-12-16T21:37:13.459 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:37:13.452+0000 7f6bab051700 1 --2- 172.21.15.45:0/3668547047 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f6ba40ffd90 0x7f6ba4102170 unknown :-1 s=CLOSED pgs=234 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:37:13.459 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:37:13.452+0000 7f6bab051700 1 --2- 172.21.15.45:0/3668547047 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f6ba40668c0 0x7f6ba40ff850 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:37:13.460 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:37:13.452+0000 7f6bab051700 1 -- 172.21.15.45:0/3668547047 >> 172.21.15.45:0/3668547047 conn(0x7f6ba40f7240 msgr2=0x7f6ba40f9670 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-16T21:37:13.460 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:37:13.452+0000 7f6bab051700 1 -- 172.21.15.45:0/3668547047 shutdown_connections 2023-12-16T21:37:13.460 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:37:13.452+0000 7f6bab051700 1 -- 172.21.15.45:0/3668547047 wait complete. 2023-12-16T21:37:13.460 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:37:13.452+0000 7f6bab051700 1 Processor -- start 2023-12-16T21:37:13.460 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:37:13.452+0000 7f6bab051700 1 -- start start 2023-12-16T21:37:13.460 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:37:13.453+0000 7f6bab051700 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f6ba40668c0 0x7f6ba40fd2d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:37:13.460 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:37:13.453+0000 7f6bab051700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f6ba40ffd90 0x7f6ba40fb750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:37:13.460 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:37:13.453+0000 7f6bab051700 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6ba40fbd50 con 0x7f6ba40ffd90 2023-12-16T21:37:13.460 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:37:13.453+0000 7f6bab051700 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6ba40fbec0 con 0x7f6ba40668c0 2023-12-16T21:37:13.460 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:37:13.453+0000 7f6ba3fff700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f6ba40ffd90 0x7f6ba40fb750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:37:13.461 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:37:13.453+0000 7f6ba3fff700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f6ba40ffd90 0x7f6ba40fb750 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:33254/0 (socket says 172.21.15.45:33254) 2023-12-16T21:37:13.461 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:37:13.453+0000 7f6ba3fff700 1 -- 172.21.15.45:0/840160300 learned_addr learned my addr 172.21.15.45:0/840160300 (peer_addr_for_me v2:172.21.15.45:0/0) 2023-12-16T21:37:13.461 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:37:13.453+0000 7f6ba8ded700 1 --2- 172.21.15.45:0/840160300 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f6ba40668c0 0x7f6ba40fd2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:37:13.461 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:37:13.453+0000 7f6ba3fff700 1 -- 172.21.15.45:0/840160300 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f6ba40668c0 msgr2=0x7f6ba40fd2d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:37:13.461 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:37:13.453+0000 7f6ba3fff700 1 --2- 172.21.15.45:0/840160300 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f6ba40668c0 0x7f6ba40fd2d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:37:13.461 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:37:13.453+0000 7f6ba3fff700 1 -- 172.21.15.45:0/840160300 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6b90005040 con 0x7f6ba40ffd90 2023-12-16T21:37:13.461 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:37:13.453+0000 7f6ba8ded700 1 --2- 172.21.15.45:0/840160300 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f6ba40668c0 0x7f6ba40fd2d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-12-16T21:37:13.461 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:37:13.453+0000 7f6ba3fff700 1 --2- 172.21.15.45:0/840160300 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f6ba40ffd90 0x7f6ba40fb750 secure :-1 s=READY pgs=235 cs=0 l=1 rev1=1 crypto rx=0x7f6b90009360 tx=0x7f6b90009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:37:13.461 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:37:13.454+0000 7f6ba17fa700 1 -- 172.21.15.45:0/840160300 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f6b9000e040 con 0x7f6ba40ffd90 2023-12-16T21:37:13.461 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:37:13.454+0000 7f6ba17fa700 1 -- 172.21.15.45:0/840160300 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f6b9000bcd0 con 0x7f6ba40ffd90 2023-12-16T21:37:13.461 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:37:13.454+0000 7f6bab051700 1 -- 172.21.15.45:0/840160300 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6ba40fc140 con 0x7f6ba40ffd90 2023-12-16T21:37:13.462 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:37:13.454+0000 7f6ba17fa700 1 -- 172.21.15.45:0/840160300 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f6b9001c630 con 0x7f6ba40ffd90 2023-12-16T21:37:13.462 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:37:13.454+0000 7f6bab051700 1 -- 172.21.15.45:0/840160300 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6ba4115540 con 0x7f6ba40ffd90 2023-12-16T21:37:13.462 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:37:13.455+0000 7f6bab051700 1 -- 172.21.15.45:0/840160300 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6ba404bb80 con 0x7f6ba40ffd90 2023-12-16T21:37:13.462 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:37:13.455+0000 7f6ba17fa700 1 -- 172.21.15.45:0/840160300 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 19) v1 ==== 86447+0+0 (secure 0 0 0) 0x7f6b90013960 con 0x7f6ba40ffd90 2023-12-16T21:37:13.462 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:37:13.456+0000 7f6ba17fa700 1 --2- 172.21.15.45:0/840160300 >> [v2:172.21.15.45:6800/2012105536,v1:172.21.15.45:6801/2012105536] conn(0x7f6b94068f60 0x7f6b9406b410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:37:13.462 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:37:13.456+0000 7f6ba8ded700 1 --2- 172.21.15.45:0/840160300 >> [v2:172.21.15.45:6800/2012105536,v1:172.21.15.45:6801/2012105536] conn(0x7f6b94068f60 0x7f6b9406b410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:37:13.462 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:37:13.456+0000 7f6ba17fa700 1 -- 172.21.15.45:0/840160300 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(37..37 src has 1..37) v4 ==== 4634+0+0 (secure 0 0 0) 0x7f6b90087f20 con 0x7f6ba40ffd90 2023-12-16T21:37:13.462 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:37:13.456+0000 7f6ba8ded700 1 --2- 172.21.15.45:0/840160300 >> [v2:172.21.15.45:6800/2012105536,v1:172.21.15.45:6801/2012105536] conn(0x7f6b94068f60 0x7f6b9406b410 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7f6b98009fd0 tx=0x7f6b98009f00 comp rx=0 tx=0).ready entity=mgr.14214 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:37:13.463 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:37:13.460+0000 7f6ba17fa700 1 -- 172.21.15.45:0/840160300 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+174570 (secure 0 0 0) 0x7f6b900b9a00 con 0x7f6ba40ffd90 2023-12-16T21:37:13.665 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:37:13.662+0000 7f6bab051700 1 -- 172.21.15.45:0/840160300 --> [v2:172.21.15.45:6800/2012105536,v1:172.21.15.45:6801/2012105536] -- mgr_command(tid 0: {"prefix": "orch ls", "target": ["mon-mgr", ""]}) v1 -- 0x7f6ba41009f0 con 0x7f6b94068f60 2023-12-16T21:37:13.669 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:37:13.667+0000 7f6ba17fa700 1 -- 172.21.15.45:0/840160300 <== mgr.14214 v2:172.21.15.45:6800/2012105536 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+1170 (secure 0 0 0) 0x7f6ba41009f0 con 0x7f6b94068f60 2023-12-16T21:37:13.669 INFO:teuthology.orchestra.run.smithi045.stdout:NAME PORTS RUNNING REFRESHED AGE PLACEMENT 2023-12-16T21:37:13.669 INFO:teuthology.orchestra.run.smithi045.stdout:alertmanager ?:9093,9094 1/1 85s ago 3m count:1 2023-12-16T21:37:13.669 INFO:teuthology.orchestra.run.smithi045.stdout:crash 2/2 85s ago 3m * 2023-12-16T21:37:13.669 INFO:teuthology.orchestra.run.smithi045.stdout:grafana ?:3000 1/1 85s ago 3m count:1 2023-12-16T21:37:13.670 INFO:teuthology.orchestra.run.smithi045.stdout:mgr 2/2 85s ago 3m count:2 2023-12-16T21:37:13.670 INFO:teuthology.orchestra.run.smithi045.stdout:mon 2/2 85s ago 3m smithi045:172.21.15.45=smithi045;smithi169:172.21.15.169=smithi169;count:2 2023-12-16T21:37:13.670 INFO:teuthology.orchestra.run.smithi045.stdout:node-exporter ?:9100 2/2 85s ago 3m * 2023-12-16T21:37:13.670 INFO:teuthology.orchestra.run.smithi045.stdout:osd 6 85s ago - 2023-12-16T21:37:13.670 INFO:teuthology.orchestra.run.smithi045.stdout:prometheus ?:9095 1/1 85s ago 3m count:1 2023-12-16T21:37:13.671 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:37:13.669+0000 7f6bab051700 1 -- 172.21.15.45:0/840160300 >> [v2:172.21.15.45:6800/2012105536,v1:172.21.15.45:6801/2012105536] conn(0x7f6b94068f60 msgr2=0x7f6b9406b410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:37:13.672 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:37:13.669+0000 7f6bab051700 1 --2- 172.21.15.45:0/840160300 >> [v2:172.21.15.45:6800/2012105536,v1:172.21.15.45:6801/2012105536] conn(0x7f6b94068f60 0x7f6b9406b410 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7f6b98009fd0 tx=0x7f6b98009f00 comp rx=0 tx=0).stop 2023-12-16T21:37:13.672 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:37:13.669+0000 7f6bab051700 1 -- 172.21.15.45:0/840160300 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f6ba40ffd90 msgr2=0x7f6ba40fb750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:37:13.672 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:37:13.669+0000 7f6bab051700 1 --2- 172.21.15.45:0/840160300 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f6ba40ffd90 0x7f6ba40fb750 secure :-1 s=READY pgs=235 cs=0 l=1 rev1=1 crypto rx=0x7f6b90009360 tx=0x7f6b90009e30 comp rx=0 tx=0).stop 2023-12-16T21:37:13.672 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:37:13.669+0000 7f6bab051700 1 -- 172.21.15.45:0/840160300 shutdown_connections 2023-12-16T21:37:13.672 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:37:13.669+0000 7f6bab051700 1 --2- 172.21.15.45:0/840160300 >> [v2:172.21.15.45:6800/2012105536,v1:172.21.15.45:6801/2012105536] conn(0x7f6b94068f60 0x7f6b9406b410 unknown :-1 s=CLOSED pgs=86 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:37:13.672 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:37:13.669+0000 7f6bab051700 1 --2- 172.21.15.45:0/840160300 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f6ba40668c0 0x7f6ba40fd2d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:37:13.672 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:37:13.669+0000 7f6bab051700 1 --2- 172.21.15.45:0/840160300 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f6ba40ffd90 0x7f6ba40fb750 unknown :-1 s=CLOSED pgs=235 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:37:13.672 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:37:13.669+0000 7f6bab051700 1 -- 172.21.15.45:0/840160300 >> 172.21.15.45:0/840160300 conn(0x7f6ba40f7240 msgr2=0x7f6ba40f9670 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-16T21:37:13.672 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:37:13.669+0000 7f6bab051700 1 -- 172.21.15.45:0/840160300 shutdown_connections 2023-12-16T21:37:13.672 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:37:13.669+0000 7f6bab051700 1 -- 172.21.15.45:0/840160300 wait complete. 2023-12-16T21:37:13.795 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Dec 16 21:37:13 smithi045 ceph-mon[80763]: from='client.14444 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2023-12-16T21:37:13.795 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Dec 16 21:37:13 smithi045 ceph-mon[80763]: pgmap v107: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2023-12-16T21:37:13.795 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Dec 16 21:37:13 smithi045 ceph-mon[80763]: from='mgr.14214 172.21.15.45:0/1227909437' entity='mgr.smithi045.nhtmwc' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2023-12-16T21:37:13.952 DEBUG:teuthology.orchestra.run.smithi045:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 88ff363c-9c5a-11ee-95a5-87774f69a715 -- bash -c 'ceph orch host ls' 2023-12-16T21:37:14.133 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Dec 16 21:37:13 smithi169 ceph-mon[88350]: from='client.14444 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2023-12-16T21:37:14.133 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Dec 16 21:37:13 smithi169 ceph-mon[88350]: pgmap v107: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2023-12-16T21:37:14.134 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Dec 16 21:37:13 smithi169 ceph-mon[88350]: from='mgr.14214 172.21.15.45:0/1227909437' entity='mgr.smithi045.nhtmwc' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2023-12-16T21:37:14.220 INFO:teuthology.orchestra.run.smithi045.stderr:Inferring config /var/lib/ceph/88ff363c-9c5a-11ee-95a5-87774f69a715/mon.smithi045/config 2023-12-16T21:37:14.850 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Dec 16 21:37:14 smithi045 ceph-mon[80763]: from='client.14448 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""]}]: dispatch 2023-12-16T21:37:14.994 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:37:14.990+0000 7fdc8d709700 1 -- 172.21.15.45:0/1817004703 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fdc880668e0 msgr2=0x7fdc880ff7f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:37:14.994 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:37:14.990+0000 7fdc8d709700 1 --2- 172.21.15.45:0/1817004703 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fdc880668e0 0x7fdc880ff7f0 secure :-1 s=READY pgs=236 cs=0 l=1 rev1=1 crypto rx=0x7fdc80009ce0 tx=0x7fdc80004d10 comp rx=0 tx=0).stop 2023-12-16T21:37:14.994 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:37:14.990+0000 7fdc8d709700 1 -- 172.21.15.45:0/1817004703 shutdown_connections 2023-12-16T21:37:14.994 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:37:14.990+0000 7fdc8d709700 1 --2- 172.21.15.45:0/1817004703 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fdc880ffdc0 0x7fdc881021e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:37:14.994 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:37:14.990+0000 7fdc8d709700 1 --2- 172.21.15.45:0/1817004703 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fdc880668e0 0x7fdc880ff7f0 unknown :-1 s=CLOSED pgs=236 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:37:14.994 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:37:14.990+0000 7fdc8d709700 1 -- 172.21.15.45:0/1817004703 >> 172.21.15.45:0/1817004703 conn(0x7fdc880f71c0 msgr2=0x7fdc880f95f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-16T21:37:14.994 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:37:14.990+0000 7fdc8d709700 1 -- 172.21.15.45:0/1817004703 shutdown_connections 2023-12-16T21:37:14.994 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:37:14.990+0000 7fdc8d709700 1 -- 172.21.15.45:0/1817004703 wait complete. 2023-12-16T21:37:14.995 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:37:14.991+0000 7fdc8d709700 1 Processor -- start 2023-12-16T21:37:14.995 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:37:14.991+0000 7fdc8d709700 1 -- start start 2023-12-16T21:37:14.995 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:37:14.991+0000 7fdc8d709700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fdc880668e0 0x7fdc8810d6e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:37:14.995 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:37:14.991+0000 7fdc8d709700 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fdc880ffdc0 0x7fdc8810dc20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:37:14.995 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:37:14.991+0000 7fdc8d709700 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdc8810e220 con 0x7fdc880668e0 2023-12-16T21:37:14.995 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:37:14.991+0000 7fdc8d709700 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdc8810eac0 con 0x7fdc880ffdc0 2023-12-16T21:37:14.995 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:37:14.991+0000 7fdc86ffd700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fdc880668e0 0x7fdc8810d6e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:37:14.995 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:37:14.991+0000 7fdc86ffd700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fdc880668e0 0x7fdc8810d6e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:33276/0 (socket says 172.21.15.45:33276) 2023-12-16T21:37:14.995 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:37:14.991+0000 7fdc86ffd700 1 -- 172.21.15.45:0/3187193087 learned_addr learned my addr 172.21.15.45:0/3187193087 (peer_addr_for_me v2:172.21.15.45:0/0) 2023-12-16T21:37:14.995 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:37:14.991+0000 7fdc867fc700 1 --2- 172.21.15.45:0/3187193087 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fdc880ffdc0 0x7fdc8810dc20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:37:14.995 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:37:14.991+0000 7fdc86ffd700 1 -- 172.21.15.45:0/3187193087 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fdc880ffdc0 msgr2=0x7fdc8810dc20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:37:14.995 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:37:14.991+0000 7fdc86ffd700 1 --2- 172.21.15.45:0/3187193087 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fdc880ffdc0 0x7fdc8810dc20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:37:14.995 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:37:14.991+0000 7fdc86ffd700 1 -- 172.21.15.45:0/3187193087 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdc80009940 con 0x7fdc880668e0 2023-12-16T21:37:14.995 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:37:14.991+0000 7fdc86ffd700 1 --2- 172.21.15.45:0/3187193087 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fdc880668e0 0x7fdc8810d6e0 secure :-1 s=READY pgs=237 cs=0 l=1 rev1=1 crypto rx=0x7fdc80015040 tx=0x7fdc80004d10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:37:14.996 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:37:14.992+0000 7fdc7f7fe700 1 -- 172.21.15.45:0/3187193087 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fdc8000e040 con 0x7fdc880668e0 2023-12-16T21:37:14.996 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:37:14.992+0000 7fdc7f7fe700 1 -- 172.21.15.45:0/3187193087 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fdc800128c0 con 0x7fdc880668e0 2023-12-16T21:37:14.996 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:37:14.992+0000 7fdc8d709700 1 -- 172.21.15.45:0/3187193087 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdc8810ed40 con 0x7fdc880668e0 2023-12-16T21:37:14.996 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:37:14.992+0000 7fdc7f7fe700 1 -- 172.21.15.45:0/3187193087 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fdc8001c7a0 con 0x7fdc880668e0 2023-12-16T21:37:14.996 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:37:14.992+0000 7fdc8d709700 1 -- 172.21.15.45:0/3187193087 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdc8810f280 con 0x7fdc880668e0 2023-12-16T21:37:14.997 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:37:14.993+0000 7fdc8d709700 1 -- 172.21.15.45:0/3187193087 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdc8804bb80 con 0x7fdc880668e0 2023-12-16T21:37:14.997 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:37:14.993+0000 7fdc7f7fe700 1 -- 172.21.15.45:0/3187193087 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 19) v1 ==== 86447+0+0 (secure 0 0 0) 0x7fdc80012a30 con 0x7fdc880668e0 2023-12-16T21:37:14.997 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:37:14.993+0000 7fdc7f7fe700 1 --2- 172.21.15.45:0/3187193087 >> [v2:172.21.15.45:6800/2012105536,v1:172.21.15.45:6801/2012105536] conn(0x7fdc74068f60 0x7fdc7406b410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:37:14.997 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:37:14.994+0000 7fdc7f7fe700 1 -- 172.21.15.45:0/3187193087 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(37..37 src has 1..37) v4 ==== 4634+0+0 (secure 0 0 0) 0x7fdc800887d0 con 0x7fdc880668e0 2023-12-16T21:37:14.997 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:37:14.994+0000 7fdc867fc700 1 --2- 172.21.15.45:0/3187193087 >> [v2:172.21.15.45:6800/2012105536,v1:172.21.15.45:6801/2012105536] conn(0x7fdc74068f60 0x7fdc7406b410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:37:14.997 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:37:14.994+0000 7fdc867fc700 1 --2- 172.21.15.45:0/3187193087 >> [v2:172.21.15.45:6800/2012105536,v1:172.21.15.45:6801/2012105536] conn(0x7fdc74068f60 0x7fdc7406b410 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7fdc700041d0 tx=0x7fdc700043c0 comp rx=0 tx=0).ready entity=mgr.14214 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:37:15.000 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:37:14.998+0000 7fdc7f7fe700 1 -- 172.21.15.45:0/3187193087 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+174570 (secure 0 0 0) 0x7fdc800589f0 con 0x7fdc880668e0 2023-12-16T21:37:15.035 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Dec 16 21:37:14 smithi169 ceph-mon[88350]: from='client.14448 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""]}]: dispatch 2023-12-16T21:37:15.201 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:37:15.198+0000 7fdc8d709700 1 -- 172.21.15.45:0/3187193087 --> [v2:172.21.15.45:6800/2012105536,v1:172.21.15.45:6801/2012105536] -- mgr_command(tid 0: {"prefix": "orch host ls", "target": ["mon-mgr", ""]}) v1 -- 0x7fdc8810f8f0 con 0x7fdc74068f60 2023-12-16T21:37:15.202 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:37:15.200+0000 7fdc7f7fe700 1 -- 172.21.15.45:0/3187193087 <== mgr.14214 v2:172.21.15.45:6800/2012105536 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+148 (secure 0 0 0) 0x7fdc8810f8f0 con 0x7fdc74068f60 2023-12-16T21:37:15.202 INFO:teuthology.orchestra.run.smithi045.stdout:HOST ADDR LABELS STATUS 2023-12-16T21:37:15.202 INFO:teuthology.orchestra.run.smithi045.stdout:smithi045 172.21.15.45 2023-12-16T21:37:15.202 INFO:teuthology.orchestra.run.smithi045.stdout:smithi169 172.21.15.169 2023-12-16T21:37:15.203 INFO:teuthology.orchestra.run.smithi045.stdout:2 hosts in cluster 2023-12-16T21:37:15.205 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:37:15.202+0000 7fdc8d709700 1 -- 172.21.15.45:0/3187193087 >> [v2:172.21.15.45:6800/2012105536,v1:172.21.15.45:6801/2012105536] conn(0x7fdc74068f60 msgr2=0x7fdc7406b410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:37:15.205 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:37:15.202+0000 7fdc8d709700 1 --2- 172.21.15.45:0/3187193087 >> [v2:172.21.15.45:6800/2012105536,v1:172.21.15.45:6801/2012105536] conn(0x7fdc74068f60 0x7fdc7406b410 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7fdc700041d0 tx=0x7fdc700043c0 comp rx=0 tx=0).stop 2023-12-16T21:37:15.205 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:37:15.202+0000 7fdc8d709700 1 -- 172.21.15.45:0/3187193087 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fdc880668e0 msgr2=0x7fdc8810d6e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:37:15.205 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:37:15.202+0000 7fdc8d709700 1 --2- 172.21.15.45:0/3187193087 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fdc880668e0 0x7fdc8810d6e0 secure :-1 s=READY pgs=237 cs=0 l=1 rev1=1 crypto rx=0x7fdc80015040 tx=0x7fdc80004d10 comp rx=0 tx=0).stop 2023-12-16T21:37:15.205 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:37:15.202+0000 7fdc8d709700 1 -- 172.21.15.45:0/3187193087 shutdown_connections 2023-12-16T21:37:15.205 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:37:15.202+0000 7fdc8d709700 1 --2- 172.21.15.45:0/3187193087 >> [v2:172.21.15.45:6800/2012105536,v1:172.21.15.45:6801/2012105536] conn(0x7fdc74068f60 0x7fdc7406b410 unknown :-1 s=CLOSED pgs=87 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:37:15.205 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:37:15.202+0000 7fdc8d709700 1 --2- 172.21.15.45:0/3187193087 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fdc880668e0 0x7fdc8810d6e0 unknown :-1 s=CLOSED pgs=237 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:37:15.205 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:37:15.202+0000 7fdc8d709700 1 --2- 172.21.15.45:0/3187193087 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fdc880ffdc0 0x7fdc8810dc20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:37:15.205 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:37:15.202+0000 7fdc8d709700 1 -- 172.21.15.45:0/3187193087 >> 172.21.15.45:0/3187193087 conn(0x7fdc880f71c0 msgr2=0x7fdc880f95f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-16T21:37:15.205 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:37:15.202+0000 7fdc8d709700 1 -- 172.21.15.45:0/3187193087 shutdown_connections 2023-12-16T21:37:15.206 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:37:15.202+0000 7fdc8d709700 1 -- 172.21.15.45:0/3187193087 wait complete. 2023-12-16T21:37:15.478 DEBUG:teuthology.orchestra.run.smithi045:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 88ff363c-9c5a-11ee-95a5-87774f69a715 -- bash -c 'ceph orch device ls' 2023-12-16T21:37:15.745 INFO:teuthology.orchestra.run.smithi045.stderr:Inferring config /var/lib/ceph/88ff363c-9c5a-11ee-95a5-87774f69a715/mon.smithi045/config 2023-12-16T21:37:15.816 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Dec 16 21:37:15 smithi045 ceph-mon[80763]: pgmap v108: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2023-12-16T21:37:16.040 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Dec 16 21:37:15 smithi169 ceph-mon[88350]: pgmap v108: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2023-12-16T21:37:16.519 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:37:16.515+0000 7f2393820700 1 -- 172.21.15.45:0/3357559921 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f238c0fde00 msgr2=0x7f238c0fe250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:37:16.519 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:37:16.515+0000 7f2393820700 1 --2- 172.21.15.45:0/3357559921 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f238c0fde00 0x7f238c0fe250 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 crypto rx=0x7f2380004940 tx=0x7f2380009e30 comp rx=0 tx=0).stop 2023-12-16T21:37:16.519 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:37:16.515+0000 7f2393820700 1 -- 172.21.15.45:0/3357559921 shutdown_connections 2023-12-16T21:37:16.519 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:37:16.515+0000 7f2393820700 1 --2- 172.21.15.45:0/3357559921 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f238c0fde00 0x7f238c0fe250 unknown :-1 s=CLOSED pgs=238 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:37:16.519 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:37:16.515+0000 7f2393820700 1 --2- 172.21.15.45:0/3357559921 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f238c0fcbf0 0x7f238c0fd000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:37:16.519 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:37:16.515+0000 7f2393820700 1 -- 172.21.15.45:0/3357559921 >> 172.21.15.45:0/3357559921 conn(0x7f238c0f82d0 msgr2=0x7f238c0fa6e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-16T21:37:16.519 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:37:16.515+0000 7f2393820700 1 -- 172.21.15.45:0/3357559921 shutdown_connections 2023-12-16T21:37:16.519 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:37:16.515+0000 7f2393820700 1 -- 172.21.15.45:0/3357559921 wait complete. 2023-12-16T21:37:16.519 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:37:16.515+0000 7f2393820700 1 Processor -- start 2023-12-16T21:37:16.519 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:37:16.516+0000 7f2393820700 1 -- start start 2023-12-16T21:37:16.519 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:37:16.516+0000 7f2393820700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f238c0fcbf0 0x7f238c10d580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:37:16.520 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:37:16.516+0000 7f2393820700 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f238c0fde00 0x7f238c10dac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:37:16.520 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:37:16.516+0000 7f2393820700 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f238c10e0c0 con 0x7f238c0fcbf0 2023-12-16T21:37:16.520 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:37:16.516+0000 7f2393820700 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f238c10e960 con 0x7f238c0fde00 2023-12-16T21:37:16.520 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:37:16.516+0000 7f23915bc700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f238c0fcbf0 0x7f238c10d580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:37:16.520 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:37:16.516+0000 7f23915bc700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f238c0fcbf0 0x7f238c10d580 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:33288/0 (socket says 172.21.15.45:33288) 2023-12-16T21:37:16.520 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:37:16.516+0000 7f23915bc700 1 -- 172.21.15.45:0/1255583793 learned_addr learned my addr 172.21.15.45:0/1255583793 (peer_addr_for_me v2:172.21.15.45:0/0) 2023-12-16T21:37:16.520 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:37:16.516+0000 7f2390dbb700 1 --2- 172.21.15.45:0/1255583793 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f238c0fde00 0x7f238c10dac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:37:16.520 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:37:16.516+0000 7f23915bc700 1 -- 172.21.15.45:0/1255583793 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f238c0fde00 msgr2=0x7f238c10dac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:37:16.520 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:37:16.516+0000 7f23915bc700 1 --2- 172.21.15.45:0/1255583793 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f238c0fde00 0x7f238c10dac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:37:16.520 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:37:16.516+0000 7f23915bc700 1 -- 172.21.15.45:0/1255583793 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2380005040 con 0x7f238c0fcbf0 2023-12-16T21:37:16.520 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:37:16.516+0000 7f23915bc700 1 --2- 172.21.15.45:0/1255583793 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f238c0fcbf0 0x7f238c10d580 secure :-1 s=READY pgs=239 cs=0 l=1 rev1=1 crypto rx=0x7f237c001ce0 tx=0x7f237c009e60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:37:16.520 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:37:16.517+0000 7f2389ffb700 1 -- 172.21.15.45:0/1255583793 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f237c007b30 con 0x7f238c0fcbf0 2023-12-16T21:37:16.520 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:37:16.517+0000 7f2393820700 1 -- 172.21.15.45:0/1255583793 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f238c10ec40 con 0x7f238c0fcbf0 2023-12-16T21:37:16.520 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:37:16.517+0000 7f2389ffb700 1 -- 172.21.15.45:0/1255583793 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f237c00f040 con 0x7f238c0fcbf0 2023-12-16T21:37:16.521 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:37:16.517+0000 7f2389ffb700 1 -- 172.21.15.45:0/1255583793 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f237c009620 con 0x7f238c0fcbf0 2023-12-16T21:37:16.521 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:37:16.517+0000 7f2393820700 1 -- 172.21.15.45:0/1255583793 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f238c10f180 con 0x7f238c0fcbf0 2023-12-16T21:37:16.522 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:37:16.518+0000 7f2393820700 1 -- 172.21.15.45:0/1255583793 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f238c04bb80 con 0x7f238c0fcbf0 2023-12-16T21:37:16.522 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:37:16.518+0000 7f2389ffb700 1 -- 172.21.15.45:0/1255583793 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 19) v1 ==== 86447+0+0 (secure 0 0 0) 0x7f237c007c90 con 0x7f238c0fcbf0 2023-12-16T21:37:16.522 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:37:16.519+0000 7f2389ffb700 1 --2- 172.21.15.45:0/1255583793 >> [v2:172.21.15.45:6800/2012105536,v1:172.21.15.45:6801/2012105536] conn(0x7f2374068f70 0x7f237406b420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:37:16.522 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:37:16.519+0000 7f2389ffb700 1 -- 172.21.15.45:0/1255583793 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(37..37 src has 1..37) v4 ==== 4634+0+0 (secure 0 0 0) 0x7f237c0863a0 con 0x7f238c0fcbf0 2023-12-16T21:37:16.522 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:37:16.519+0000 7f2390dbb700 1 --2- 172.21.15.45:0/1255583793 >> [v2:172.21.15.45:6800/2012105536,v1:172.21.15.45:6801/2012105536] conn(0x7f2374068f70 0x7f237406b420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:37:16.522 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:37:16.519+0000 7f2390dbb700 1 --2- 172.21.15.45:0/1255583793 >> [v2:172.21.15.45:6800/2012105536,v1:172.21.15.45:6801/2012105536] conn(0x7f2374068f70 0x7f237406b420 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7f23800043d0 tx=0x7f2380009e30 comp rx=0 tx=0).ready entity=mgr.14214 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:37:16.525 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:37:16.523+0000 7f2389ffb700 1 -- 172.21.15.45:0/1255583793 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+174570 (secure 0 0 0) 0x7f237c0566f0 con 0x7f238c0fcbf0 2023-12-16T21:37:16.729 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:37:16.727+0000 7f2393820700 1 -- 172.21.15.45:0/1255583793 --> [v2:172.21.15.45:6800/2012105536,v1:172.21.15.45:6801/2012105536] -- mgr_command(tid 0: {"prefix": "orch device ls", "target": ["mon-mgr", ""]}) v1 -- 0x7f238c10f620 con 0x7f2374068f70 2023-12-16T21:37:16.730 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Dec 16 21:37:16 smithi045 ceph-mon[80763]: from='client.14452 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""]}]: dispatch 2023-12-16T21:37:16.731 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:37:16.729+0000 7f2389ffb700 1 -- 172.21.15.45:0/1255583793 <== mgr.14214 v2:172.21.15.45:6800/2012105536 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+399 (secure 0 0 0) 0x7f238c10f620 con 0x7f2374068f70 2023-12-16T21:37:16.731 INFO:teuthology.orchestra.run.smithi045.stdout:HOST PATH TYPE DEVICE ID SIZE AVAILABLE REFRESHED REJECT REASONS 2023-12-16T21:37:16.731 INFO:teuthology.orchestra.run.smithi045.stdout:smithi045 /dev/nvme0n1 ssd INTEL_SSDPED1D480GA_PHMB75120098480DGN 447G No 86s ago Has a FileSystem, LVM detected 2023-12-16T21:37:16.731 INFO:teuthology.orchestra.run.smithi045.stdout:smithi169 /dev/nvme0n1 ssd INTEL_SSDPEDMD400G4_CVFT623300LN400BGN 372G No 40s ago Has a FileSystem, LVM detected 2023-12-16T21:37:16.733 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:37:16.731+0000 7f2393820700 1 -- 172.21.15.45:0/1255583793 >> [v2:172.21.15.45:6800/2012105536,v1:172.21.15.45:6801/2012105536] conn(0x7f2374068f70 msgr2=0x7f237406b420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:37:16.733 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:37:16.731+0000 7f2393820700 1 --2- 172.21.15.45:0/1255583793 >> [v2:172.21.15.45:6800/2012105536,v1:172.21.15.45:6801/2012105536] conn(0x7f2374068f70 0x7f237406b420 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7f23800043d0 tx=0x7f2380009e30 comp rx=0 tx=0).stop 2023-12-16T21:37:16.734 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:37:16.731+0000 7f2393820700 1 -- 172.21.15.45:0/1255583793 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f238c0fcbf0 msgr2=0x7f238c10d580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:37:16.734 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:37:16.731+0000 7f2393820700 1 --2- 172.21.15.45:0/1255583793 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f238c0fcbf0 0x7f238c10d580 secure :-1 s=READY pgs=239 cs=0 l=1 rev1=1 crypto rx=0x7f237c001ce0 tx=0x7f237c009e60 comp rx=0 tx=0).stop 2023-12-16T21:37:16.734 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:37:16.731+0000 7f2393820700 1 -- 172.21.15.45:0/1255583793 shutdown_connections 2023-12-16T21:37:16.734 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:37:16.731+0000 7f2393820700 1 --2- 172.21.15.45:0/1255583793 >> [v2:172.21.15.45:6800/2012105536,v1:172.21.15.45:6801/2012105536] conn(0x7f2374068f70 0x7f237406b420 unknown :-1 s=CLOSED pgs=88 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:37:16.734 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:37:16.731+0000 7f2393820700 1 --2- 172.21.15.45:0/1255583793 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f238c0fcbf0 0x7f238c10d580 unknown :-1 s=CLOSED pgs=239 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:37:16.734 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:37:16.731+0000 7f2393820700 1 --2- 172.21.15.45:0/1255583793 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f238c0fde00 0x7f238c10dac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:37:16.734 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:37:16.731+0000 7f2393820700 1 -- 172.21.15.45:0/1255583793 >> 172.21.15.45:0/1255583793 conn(0x7f238c0f82d0 msgr2=0x7f238c101030 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-16T21:37:16.734 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:37:16.731+0000 7f2393820700 1 -- 172.21.15.45:0/1255583793 shutdown_connections 2023-12-16T21:37:16.734 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:37:16.731+0000 7f2393820700 1 -- 172.21.15.45:0/1255583793 wait complete. 2023-12-16T21:37:17.044 INFO:teuthology.run_tasks:Running task cephadm.shell... 2023-12-16T21:37:17.054 INFO:tasks.cephadm:Running commands on role host.a host ubuntu@smithi045.front.sepia.ceph.com 2023-12-16T21:37:17.054 DEBUG:teuthology.orchestra.run.smithi045:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 88ff363c-9c5a-11ee-95a5-87774f69a715 -- bash -c 'ceph fs volume create cephfs --placement=4' 2023-12-16T21:37:17.133 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Dec 16 21:37:16 smithi169 ceph-mon[88350]: from='client.14452 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""]}]: dispatch 2023-12-16T21:37:17.319 INFO:teuthology.orchestra.run.smithi045.stderr:Inferring config /var/lib/ceph/88ff363c-9c5a-11ee-95a5-87774f69a715/mon.smithi045/config 2023-12-16T21:37:17.934 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Dec 16 21:37:17 smithi045 ceph-mon[80763]: pgmap v109: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2023-12-16T21:37:17.935 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Dec 16 21:37:17 smithi045 ceph-mon[80763]: from='client.14456 -' entity='client.admin' cmd=[{"prefix": "orch device ls", "target": ["mon-mgr", ""]}]: dispatch 2023-12-16T21:37:18.006 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:37:18.002+0000 7fdeaec0e700 1 -- 172.21.15.45:0/3628616220 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fdea8066920 msgr2=0x7fdea80ff7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:37:18.007 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:37:18.002+0000 7fdeaec0e700 1 --2- 172.21.15.45:0/3628616220 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fdea8066920 0x7fdea80ff7a0 secure :-1 s=READY pgs=240 cs=0 l=1 rev1=1 crypto rx=0x7fde9c004940 tx=0x7fde9c009e30 comp rx=0 tx=0).stop 2023-12-16T21:37:18.007 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:37:18.002+0000 7fdeaec0e700 1 -- 172.21.15.45:0/3628616220 shutdown_connections 2023-12-16T21:37:18.007 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:37:18.002+0000 7fdeaec0e700 1 --2- 172.21.15.45:0/3628616220 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fdea80ffd70 0x7fdea81021d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:37:18.007 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:37:18.002+0000 7fdeaec0e700 1 --2- 172.21.15.45:0/3628616220 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fdea8066920 0x7fdea80ff7a0 unknown :-1 s=CLOSED pgs=240 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:37:18.007 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:37:18.002+0000 7fdeaec0e700 1 -- 172.21.15.45:0/3628616220 >> 172.21.15.45:0/3628616220 conn(0x7fdea80f71d0 msgr2=0x7fdea80f95e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-16T21:37:18.007 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:37:18.003+0000 7fdeaec0e700 1 -- 172.21.15.45:0/3628616220 shutdown_connections 2023-12-16T21:37:18.007 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:37:18.003+0000 7fdeaec0e700 1 -- 172.21.15.45:0/3628616220 wait complete. 2023-12-16T21:37:18.007 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:37:18.003+0000 7fdeaec0e700 1 Processor -- start 2023-12-16T21:37:18.007 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:37:18.003+0000 7fdeaec0e700 1 -- start start 2023-12-16T21:37:18.007 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:37:18.003+0000 7fdeaec0e700 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fdea8066920 0x7fdea80fd220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:37:18.007 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:37:18.003+0000 7fdeaec0e700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fdea80ffd70 0x7fdea80fb6a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:37:18.007 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:37:18.003+0000 7fdeaec0e700 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdea80fbd30 con 0x7fdea80ffd70 2023-12-16T21:37:18.007 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:37:18.003+0000 7fdeaec0e700 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdea80fbea0 con 0x7fdea8066920 2023-12-16T21:37:18.007 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:37:18.003+0000 7fdea7fff700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fdea80ffd70 0x7fdea80fb6a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:37:18.008 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:37:18.004+0000 7fdea7fff700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fdea80ffd70 0x7fdea80fb6a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:33318/0 (socket says 172.21.15.45:33318) 2023-12-16T21:37:18.008 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:37:18.004+0000 7fdea7fff700 1 -- 172.21.15.45:0/3797626421 learned_addr learned my addr 172.21.15.45:0/3797626421 (peer_addr_for_me v2:172.21.15.45:0/0) 2023-12-16T21:37:18.008 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:37:18.004+0000 7fdeac9aa700 1 --2- 172.21.15.45:0/3797626421 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fdea8066920 0x7fdea80fd220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:37:18.008 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:37:18.004+0000 7fdea7fff700 1 -- 172.21.15.45:0/3797626421 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fdea8066920 msgr2=0x7fdea80fd220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:37:18.008 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:37:18.004+0000 7fdea7fff700 1 --2- 172.21.15.45:0/3797626421 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fdea8066920 0x7fdea80fd220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:37:18.008 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:37:18.004+0000 7fdea7fff700 1 -- 172.21.15.45:0/3797626421 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fde9c005040 con 0x7fdea80ffd70 2023-12-16T21:37:18.008 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:37:18.004+0000 7fdeac9aa700 1 --2- 172.21.15.45:0/3797626421 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fdea8066920 0x7fdea80fd220 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-12-16T21:37:18.008 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:37:18.004+0000 7fdea7fff700 1 --2- 172.21.15.45:0/3797626421 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fdea80ffd70 0x7fdea80fb6a0 secure :-1 s=READY pgs=241 cs=0 l=1 rev1=1 crypto rx=0x7fde9400b790 tx=0x7fde94009220 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:37:18.008 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:37:18.004+0000 7fdea57fa700 1 -- 172.21.15.45:0/3797626421 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fde9400f6f0 con 0x7fdea80ffd70 2023-12-16T21:37:18.008 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:37:18.004+0000 7fdeaec0e700 1 -- 172.21.15.45:0/3797626421 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdea80fc180 con 0x7fdea80ffd70 2023-12-16T21:37:18.008 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:37:18.004+0000 7fdea57fa700 1 -- 172.21.15.45:0/3797626421 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fde9400fd00 con 0x7fdea80ffd70 2023-12-16T21:37:18.009 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:37:18.004+0000 7fdea57fa700 1 -- 172.21.15.45:0/3797626421 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fde940103a0 con 0x7fdea80ffd70 2023-12-16T21:37:18.009 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:37:18.004+0000 7fdeaec0e700 1 -- 172.21.15.45:0/3797626421 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdea80fc500 con 0x7fdea80ffd70 2023-12-16T21:37:18.009 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:37:18.005+0000 7fdeaec0e700 1 -- 172.21.15.45:0/3797626421 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdea804bb80 con 0x7fdea80ffd70 2023-12-16T21:37:18.009 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:37:18.006+0000 7fdea57fa700 1 -- 172.21.15.45:0/3797626421 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 19) v1 ==== 86447+0+0 (secure 0 0 0) 0x7fde9400f850 con 0x7fdea80ffd70 2023-12-16T21:37:18.009 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:37:18.006+0000 7fdea57fa700 1 --2- 172.21.15.45:0/3797626421 >> [v2:172.21.15.45:6800/2012105536,v1:172.21.15.45:6801/2012105536] conn(0x7fde98068f60 0x7fde9806b410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:37:18.010 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:37:18.006+0000 7fdea57fa700 1 -- 172.21.15.45:0/3797626421 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(37..37 src has 1..37) v4 ==== 4634+0+0 (secure 0 0 0) 0x7fde940868c0 con 0x7fdea80ffd70 2023-12-16T21:37:18.010 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:37:18.006+0000 7fdeac9aa700 1 --2- 172.21.15.45:0/3797626421 >> [v2:172.21.15.45:6800/2012105536,v1:172.21.15.45:6801/2012105536] conn(0x7fde98068f60 0x7fde9806b410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:37:18.010 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:37:18.007+0000 7fdeac9aa700 1 --2- 172.21.15.45:0/3797626421 >> [v2:172.21.15.45:6800/2012105536,v1:172.21.15.45:6801/2012105536] conn(0x7fde98068f60 0x7fde9806b410 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7fde9c004890 tx=0x7fde9c009e30 comp rx=0 tx=0).ready entity=mgr.14214 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:37:18.013 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:37:18.011+0000 7fdea57fa700 1 -- 172.21.15.45:0/3797626421 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+174570 (secure 0 0 0) 0x7fde94056ac0 con 0x7fdea80ffd70 2023-12-16T21:37:18.133 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Dec 16 21:37:17 smithi169 ceph-mon[88350]: pgmap v109: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2023-12-16T21:37:18.136 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Dec 16 21:37:17 smithi169 ceph-mon[88350]: from='client.14456 -' entity='client.admin' cmd=[{"prefix": "orch device ls", "target": ["mon-mgr", ""]}]: dispatch 2023-12-16T21:37:18.237 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:37:18.234+0000 7fdeaec0e700 1 -- 172.21.15.45:0/3797626421 --> [v2:172.21.15.45:6800/2012105536,v1:172.21.15.45:6801/2012105536] -- mgr_command(tid 0: {"prefix": "fs volume create", "name": "cephfs", "placement": "4", "target": ["mon-mgr", ""]}) v1 -- 0x7fdea8100a50 con 0x7fde98068f60 2023-12-16T21:37:19.066 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Dec 16 21:37:18 smithi045 ceph-mon[80763]: from='mgr.14214 172.21.15.45:0/1227909437' entity='mgr.smithi045.nhtmwc' cmd=[{"prefix": "osd pool create", "pool": "cephfs.cephfs.meta"}]: dispatch 2023-12-16T21:37:19.133 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Dec 16 21:37:18 smithi169 ceph-mon[88350]: from='mgr.14214 172.21.15.45:0/1227909437' entity='mgr.smithi045.nhtmwc' cmd=[{"prefix": "osd pool create", "pool": "cephfs.cephfs.meta"}]: dispatch 2023-12-16T21:37:19.713 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:37:19.710+0000 7fdea57fa700 1 -- 172.21.15.45:0/3797626421 <== mgr.14214 v2:172.21.15.45:6800/2012105536 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+0 (secure 0 0 0) 0x7fdea8100a50 con 0x7fde98068f60 2023-12-16T21:37:19.716 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:37:19.714+0000 7fdeaec0e700 1 -- 172.21.15.45:0/3797626421 >> [v2:172.21.15.45:6800/2012105536,v1:172.21.15.45:6801/2012105536] conn(0x7fde98068f60 msgr2=0x7fde9806b410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:37:19.716 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:37:19.714+0000 7fdeaec0e700 1 --2- 172.21.15.45:0/3797626421 >> [v2:172.21.15.45:6800/2012105536,v1:172.21.15.45:6801/2012105536] conn(0x7fde98068f60 0x7fde9806b410 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7fde9c004890 tx=0x7fde9c009e30 comp rx=0 tx=0).stop 2023-12-16T21:37:19.716 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:37:19.714+0000 7fdeaec0e700 1 -- 172.21.15.45:0/3797626421 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fdea80ffd70 msgr2=0x7fdea80fb6a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:37:19.717 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:37:19.714+0000 7fdeaec0e700 1 --2- 172.21.15.45:0/3797626421 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fdea80ffd70 0x7fdea80fb6a0 secure :-1 s=READY pgs=241 cs=0 l=1 rev1=1 crypto rx=0x7fde9400b790 tx=0x7fde94009220 comp rx=0 tx=0).stop 2023-12-16T21:37:19.717 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:37:19.714+0000 7fdeaec0e700 1 -- 172.21.15.45:0/3797626421 shutdown_connections 2023-12-16T21:37:19.717 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:37:19.714+0000 7fdeaec0e700 1 --2- 172.21.15.45:0/3797626421 >> [v2:172.21.15.45:6800/2012105536,v1:172.21.15.45:6801/2012105536] conn(0x7fde98068f60 0x7fde9806b410 unknown :-1 s=CLOSED pgs=89 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:37:19.717 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:37:19.714+0000 7fdeaec0e700 1 --2- 172.21.15.45:0/3797626421 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fdea8066920 0x7fdea80fd220 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:37:19.717 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:37:19.714+0000 7fdeaec0e700 1 --2- 172.21.15.45:0/3797626421 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fdea80ffd70 0x7fdea80fb6a0 unknown :-1 s=CLOSED pgs=241 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:37:19.717 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:37:19.714+0000 7fdeaec0e700 1 -- 172.21.15.45:0/3797626421 >> 172.21.15.45:0/3797626421 conn(0x7fdea80f71d0 msgr2=0x7fdea80f95b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-16T21:37:19.718 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:37:19.715+0000 7fdeaec0e700 1 -- 172.21.15.45:0/3797626421 shutdown_connections 2023-12-16T21:37:19.718 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:37:19.715+0000 7fdeaec0e700 1 -- 172.21.15.45:0/3797626421 wait complete. 2023-12-16T21:37:19.959 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Dec 16 21:37:19 smithi045 ceph-mon[80763]: from='client.14460 -' entity='client.admin' cmd=[{"prefix": "fs volume create", "name": "cephfs", "placement": "4", "target": ["mon-mgr", ""]}]: dispatch 2023-12-16T21:37:19.960 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Dec 16 21:37:19 smithi045 ceph-mon[80763]: pgmap v110: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2023-12-16T21:37:19.960 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Dec 16 21:37:19 smithi045 ceph-mon[80763]: from='mgr.14214 172.21.15.45:0/1227909437' entity='mgr.smithi045.nhtmwc' cmd='[{"prefix": "osd pool create", "pool": "cephfs.cephfs.meta"}]': finished 2023-12-16T21:37:19.960 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Dec 16 21:37:19 smithi045 ceph-mon[80763]: osdmap e38: 6 total, 6 up, 6 in 2023-12-16T21:37:19.960 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Dec 16 21:37:19 smithi045 ceph-mon[80763]: from='mgr.14214 172.21.15.45:0/1227909437' entity='mgr.smithi045.nhtmwc' cmd=[{"bulk": true, "prefix": "osd pool create", "pool": "cephfs.cephfs.data"}]: dispatch 2023-12-16T21:37:19.960 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Dec 16 21:37:19 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-mon-smithi045[80759]: 2023-12-16T21:37:19.690+0000 7f459f78a700 -1 log_channel(cluster) log [ERR] : Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2023-12-16T21:37:19.989 DEBUG:teuthology.orchestra.run.smithi045:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 88ff363c-9c5a-11ee-95a5-87774f69a715 -- bash -c 'ceph fs dump' 2023-12-16T21:37:20.133 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Dec 16 21:37:19 smithi169 ceph-mon[88350]: from='client.14460 -' entity='client.admin' cmd=[{"prefix": "fs volume create", "name": "cephfs", "placement": "4", "target": ["mon-mgr", ""]}]: dispatch 2023-12-16T21:37:20.133 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Dec 16 21:37:19 smithi169 ceph-mon[88350]: pgmap v110: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2023-12-16T21:37:20.134 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Dec 16 21:37:19 smithi169 ceph-mon[88350]: from='mgr.14214 172.21.15.45:0/1227909437' entity='mgr.smithi045.nhtmwc' cmd='[{"prefix": "osd pool create", "pool": "cephfs.cephfs.meta"}]': finished 2023-12-16T21:37:20.134 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Dec 16 21:37:19 smithi169 ceph-mon[88350]: osdmap e38: 6 total, 6 up, 6 in 2023-12-16T21:37:20.134 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Dec 16 21:37:19 smithi169 ceph-mon[88350]: from='mgr.14214 172.21.15.45:0/1227909437' entity='mgr.smithi045.nhtmwc' cmd=[{"bulk": true, "prefix": "osd pool create", "pool": "cephfs.cephfs.data"}]: dispatch 2023-12-16T21:37:20.538 INFO:teuthology.orchestra.run.smithi045.stderr:Inferring config /var/lib/ceph/88ff363c-9c5a-11ee-95a5-87774f69a715/mon.smithi045/config 2023-12-16T21:37:20.878 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Dec 16 21:37:20 smithi045 ceph-mon[80763]: from='mgr.14214 172.21.15.45:0/1227909437' entity='mgr.smithi045.nhtmwc' cmd='[{"bulk": true, "prefix": "osd pool create", "pool": "cephfs.cephfs.data"}]': finished 2023-12-16T21:37:20.878 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Dec 16 21:37:20 smithi045 ceph-mon[80763]: osdmap e39: 6 total, 6 up, 6 in 2023-12-16T21:37:20.878 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Dec 16 21:37:20 smithi045 ceph-mon[80763]: from='mgr.14214 172.21.15.45:0/1227909437' entity='mgr.smithi045.nhtmwc' cmd=[{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs.cephfs.meta", "data": "cephfs.cephfs.data"}]: dispatch 2023-12-16T21:37:20.878 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Dec 16 21:37:20 smithi045 ceph-mon[80763]: Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2023-12-16T21:37:20.878 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Dec 16 21:37:20 smithi045 ceph-mon[80763]: Health check failed: 1 filesystem is online with fewer MDS than max_mds (MDS_UP_LESS_THAN_MAX) 2023-12-16T21:37:20.878 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Dec 16 21:37:20 smithi045 ceph-mon[80763]: osdmap e40: 6 total, 6 up, 6 in 2023-12-16T21:37:20.878 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Dec 16 21:37:20 smithi045 ceph-mon[80763]: from='mgr.14214 172.21.15.45:0/1227909437' entity='mgr.smithi045.nhtmwc' cmd='[{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs.cephfs.meta", "data": "cephfs.cephfs.data"}]': finished 2023-12-16T21:37:20.878 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Dec 16 21:37:20 smithi045 ceph-mon[80763]: fsmap cephfs:0 2023-12-16T21:37:20.878 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Dec 16 21:37:20 smithi045 ceph-mon[80763]: Saving service mds.cephfs spec with placement count:4 2023-12-16T21:37:20.878 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Dec 16 21:37:20 smithi045 ceph-mon[80763]: from='mgr.14214 172.21.15.45:0/1227909437' entity='mgr.smithi045.nhtmwc' 2023-12-16T21:37:20.878 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Dec 16 21:37:20 smithi045 ceph-mon[80763]: from='mgr.14214 172.21.15.45:0/1227909437' entity='mgr.smithi045.nhtmwc' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-12-16T21:37:20.879 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Dec 16 21:37:20 smithi045 ceph-mon[80763]: from='mgr.14214 172.21.15.45:0/1227909437' entity='mgr.smithi045.nhtmwc' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-12-16T21:37:20.879 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Dec 16 21:37:20 smithi045 ceph-mon[80763]: from='mgr.14214 172.21.15.45:0/1227909437' entity='mgr.smithi045.nhtmwc' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-12-16T21:37:20.879 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Dec 16 21:37:20 smithi045 ceph-mon[80763]: from='mgr.14214 172.21.15.45:0/1227909437' entity='mgr.smithi045.nhtmwc' 2023-12-16T21:37:20.879 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Dec 16 21:37:20 smithi045 ceph-mon[80763]: from='mgr.14214 172.21.15.45:0/1227909437' entity='mgr.smithi045.nhtmwc' cmd=[{"prefix": "auth get-or-create", "entity": "mds.cephfs.smithi045.fgsbgq", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2023-12-16T21:37:20.879 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Dec 16 21:37:20 smithi045 ceph-mon[80763]: from='mgr.14214 172.21.15.45:0/1227909437' entity='mgr.smithi045.nhtmwc' cmd='[{"prefix": "auth get-or-create", "entity": "mds.cephfs.smithi045.fgsbgq", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2023-12-16T21:37:20.879 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Dec 16 21:37:20 smithi045 ceph-mon[80763]: from='mgr.14214 172.21.15.45:0/1227909437' entity='mgr.smithi045.nhtmwc' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-12-16T21:37:20.879 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Dec 16 21:37:20 smithi045 ceph-mon[80763]: Deploying daemon mds.cephfs.smithi045.fgsbgq on smithi045 2023-12-16T21:37:21.133 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Dec 16 21:37:20 smithi169 ceph-mon[88350]: from='mgr.14214 172.21.15.45:0/1227909437' entity='mgr.smithi045.nhtmwc' cmd='[{"bulk": true, "prefix": "osd pool create", "pool": "cephfs.cephfs.data"}]': finished 2023-12-16T21:37:21.134 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Dec 16 21:37:20 smithi169 ceph-mon[88350]: osdmap e39: 6 total, 6 up, 6 in 2023-12-16T21:37:21.134 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Dec 16 21:37:20 smithi169 ceph-mon[88350]: from='mgr.14214 172.21.15.45:0/1227909437' entity='mgr.smithi045.nhtmwc' cmd=[{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs.cephfs.meta", "data": "cephfs.cephfs.data"}]: dispatch 2023-12-16T21:37:21.134 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Dec 16 21:37:20 smithi169 ceph-mon[88350]: Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2023-12-16T21:37:21.134 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Dec 16 21:37:20 smithi169 ceph-mon[88350]: Health check failed: 1 filesystem is online with fewer MDS than max_mds (MDS_UP_LESS_THAN_MAX) 2023-12-16T21:37:21.134 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Dec 16 21:37:20 smithi169 ceph-mon[88350]: osdmap e40: 6 total, 6 up, 6 in 2023-12-16T21:37:21.134 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Dec 16 21:37:20 smithi169 ceph-mon[88350]: from='mgr.14214 172.21.15.45:0/1227909437' entity='mgr.smithi045.nhtmwc' cmd='[{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs.cephfs.meta", "data": "cephfs.cephfs.data"}]': finished 2023-12-16T21:37:21.134 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Dec 16 21:37:20 smithi169 ceph-mon[88350]: fsmap cephfs:0 2023-12-16T21:37:21.134 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Dec 16 21:37:20 smithi169 ceph-mon[88350]: Saving service mds.cephfs spec with placement count:4 2023-12-16T21:37:21.134 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Dec 16 21:37:20 smithi169 ceph-mon[88350]: from='mgr.14214 172.21.15.45:0/1227909437' entity='mgr.smithi045.nhtmwc' 2023-12-16T21:37:21.134 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Dec 16 21:37:20 smithi169 ceph-mon[88350]: from='mgr.14214 172.21.15.45:0/1227909437' entity='mgr.smithi045.nhtmwc' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-12-16T21:37:21.134 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Dec 16 21:37:20 smithi169 ceph-mon[88350]: from='mgr.14214 172.21.15.45:0/1227909437' entity='mgr.smithi045.nhtmwc' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-12-16T21:37:21.134 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Dec 16 21:37:20 smithi169 ceph-mon[88350]: from='mgr.14214 172.21.15.45:0/1227909437' entity='mgr.smithi045.nhtmwc' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-12-16T21:37:21.134 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Dec 16 21:37:20 smithi169 ceph-mon[88350]: from='mgr.14214 172.21.15.45:0/1227909437' entity='mgr.smithi045.nhtmwc' 2023-12-16T21:37:21.135 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Dec 16 21:37:20 smithi169 ceph-mon[88350]: from='mgr.14214 172.21.15.45:0/1227909437' entity='mgr.smithi045.nhtmwc' cmd=[{"prefix": "auth get-or-create", "entity": "mds.cephfs.smithi045.fgsbgq", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2023-12-16T21:37:21.135 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Dec 16 21:37:20 smithi169 ceph-mon[88350]: from='mgr.14214 172.21.15.45:0/1227909437' entity='mgr.smithi045.nhtmwc' cmd='[{"prefix": "auth get-or-create", "entity": "mds.cephfs.smithi045.fgsbgq", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2023-12-16T21:37:21.135 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Dec 16 21:37:20 smithi169 ceph-mon[88350]: from='mgr.14214 172.21.15.45:0/1227909437' entity='mgr.smithi045.nhtmwc' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-12-16T21:37:21.135 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Dec 16 21:37:20 smithi169 ceph-mon[88350]: Deploying daemon mds.cephfs.smithi045.fgsbgq on smithi045 2023-12-16T21:37:21.676 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:37:21.671+0000 7f7ed42af700 1 -- 172.21.15.45:0/2338705330 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f7ecc0fee00 msgr2=0x7f7ecc0ff210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:37:21.677 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:37:21.671+0000 7f7ed42af700 1 --2- 172.21.15.45:0/2338705330 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f7ecc0fee00 0x7f7ecc0ff210 secure :-1 s=READY pgs=244 cs=0 l=1 rev1=1 crypto rx=0x7f7ec8004940 tx=0x7f7ec8009e30 comp rx=0 tx=0).stop 2023-12-16T21:37:21.677 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:37:21.672+0000 7f7ed42af700 1 -- 172.21.15.45:0/2338705330 shutdown_connections 2023-12-16T21:37:21.677 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:37:21.672+0000 7f7ed42af700 1 --2- 172.21.15.45:0/2338705330 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f7ecc0fff70 0x7f7ecc1003e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:37:21.677 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:37:21.672+0000 7f7ed42af700 1 --2- 172.21.15.45:0/2338705330 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f7ecc0fee00 0x7f7ecc0ff210 unknown :-1 s=CLOSED pgs=244 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:37:21.677 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:37:21.672+0000 7f7ed42af700 1 -- 172.21.15.45:0/2338705330 >> 172.21.15.45:0/2338705330 conn(0x7f7ecc0fa5d0 msgr2=0x7f7ecc0fc9b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-16T21:37:21.677 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:37:21.672+0000 7f7ed42af700 1 -- 172.21.15.45:0/2338705330 shutdown_connections 2023-12-16T21:37:21.677 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:37:21.672+0000 7f7ed42af700 1 -- 172.21.15.45:0/2338705330 wait complete. 2023-12-16T21:37:21.678 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:37:21.672+0000 7f7ed42af700 1 Processor -- start 2023-12-16T21:37:21.678 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:37:21.672+0000 7f7ed42af700 1 -- start start 2023-12-16T21:37:21.678 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:37:21.672+0000 7f7ed42af700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f7ecc0fee00 0x7f7ecc10f740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:37:21.678 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:37:21.672+0000 7f7ed42af700 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f7ecc0fff70 0x7f7ecc10fc80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:37:21.678 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:37:21.672+0000 7f7ed42af700 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7ecc110280 con 0x7f7ecc0fee00 2023-12-16T21:37:21.678 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:37:21.672+0000 7f7ed42af700 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7ecc110b20 con 0x7f7ecc0fff70 2023-12-16T21:37:21.678 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:37:21.672+0000 7f7ed204b700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f7ecc0fee00 0x7f7ecc10f740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:37:21.678 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:37:21.673+0000 7f7ed204b700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f7ecc0fee00 0x7f7ecc10f740 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:33344/0 (socket says 172.21.15.45:33344) 2023-12-16T21:37:21.678 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:37:21.673+0000 7f7ed204b700 1 -- 172.21.15.45:0/3946727668 learned_addr learned my addr 172.21.15.45:0/3946727668 (peer_addr_for_me v2:172.21.15.45:0/0) 2023-12-16T21:37:21.678 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:37:21.673+0000 7f7ed184a700 1 --2- 172.21.15.45:0/3946727668 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f7ecc0fff70 0x7f7ecc10fc80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:37:21.678 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:37:21.673+0000 7f7ed204b700 1 -- 172.21.15.45:0/3946727668 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f7ecc0fff70 msgr2=0x7f7ecc10fc80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:37:21.679 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:37:21.673+0000 7f7ed204b700 1 --2- 172.21.15.45:0/3946727668 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f7ecc0fff70 0x7f7ecc10fc80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:37:21.679 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:37:21.673+0000 7f7ed204b700 1 -- 172.21.15.45:0/3946727668 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7ec8005040 con 0x7f7ecc0fee00 2023-12-16T21:37:21.679 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:37:21.673+0000 7f7ed204b700 1 --2- 172.21.15.45:0/3946727668 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f7ecc0fee00 0x7f7ecc10f740 secure :-1 s=READY pgs=245 cs=0 l=1 rev1=1 crypto rx=0x7f7ec8009360 tx=0x7f7ec8009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:37:21.679 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:37:21.673+0000 7f7ec2ffd700 1 -- 172.21.15.45:0/3946727668 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f7ec800e040 con 0x7f7ecc0fee00 2023-12-16T21:37:21.679 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:37:21.673+0000 7f7ed42af700 1 -- 172.21.15.45:0/3946727668 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7ecc110da0 con 0x7f7ecc0fee00 2023-12-16T21:37:21.679 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:37:21.673+0000 7f7ec2ffd700 1 -- 172.21.15.45:0/3946727668 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f7ec800bc50 con 0x7f7ecc0fee00 2023-12-16T21:37:21.679 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:37:21.673+0000 7f7ec2ffd700 1 -- 172.21.15.45:0/3946727668 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f7ec801c3f0 con 0x7f7ecc0fee00 2023-12-16T21:37:21.679 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:37:21.673+0000 7f7ed42af700 1 -- 172.21.15.45:0/3946727668 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7ecc1112e0 con 0x7f7ecc0fee00 2023-12-16T21:37:21.679 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:37:21.674+0000 7f7ed42af700 1 -- 172.21.15.45:0/3946727668 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7ecc04bb80 con 0x7f7ecc0fee00 2023-12-16T21:37:21.680 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:37:21.675+0000 7f7ec2ffd700 1 -- 172.21.15.45:0/3946727668 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 19) v1 ==== 86447+0+0 (secure 0 0 0) 0x7f7ec8013420 con 0x7f7ecc0fee00 2023-12-16T21:37:21.680 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:37:21.675+0000 7f7ec2ffd700 1 --2- 172.21.15.45:0/3946727668 >> [v2:172.21.15.45:6800/2012105536,v1:172.21.15.45:6801/2012105536] conn(0x7f7eb8068f60 0x7f7eb806b410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:37:21.681 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:37:21.675+0000 7f7ec2ffd700 1 -- 172.21.15.45:0/3946727668 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(41..41 src has 1..41) v4 ==== 5464+0+0 (secure 0 0 0) 0x7f7ec8088110 con 0x7f7ecc0fee00 2023-12-16T21:37:21.681 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:37:21.675+0000 7f7ed184a700 1 --2- 172.21.15.45:0/3946727668 >> [v2:172.21.15.45:6800/2012105536,v1:172.21.15.45:6801/2012105536] conn(0x7f7eb8068f60 0x7f7eb806b410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:37:21.681 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:37:21.676+0000 7f7ed184a700 1 --2- 172.21.15.45:0/3946727668 >> [v2:172.21.15.45:6800/2012105536,v1:172.21.15.45:6801/2012105536] conn(0x7f7eb8068f60 0x7f7eb806b410 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7f7ebc004f40 tx=0x7f7ebc00b040 comp rx=0 tx=0).ready entity=mgr.14214 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:37:21.682 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:37:21.680+0000 7f7ec2ffd700 1 -- 172.21.15.45:0/3946727668 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+174570 (secure 0 0 0) 0x7f7ec8057ff0 con 0x7f7ecc0fee00 2023-12-16T21:37:21.816 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Dec 16 21:37:21 smithi045 ceph-mon[80763]: pgmap v114: 65 pgs: 5 creating+peering, 1 active+clean, 59 unknown; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2023-12-16T21:37:21.816 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Dec 16 21:37:21 smithi045 ceph-mon[80763]: osdmap e41: 6 total, 6 up, 6 in 2023-12-16T21:37:21.816 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Dec 16 21:37:21 smithi045 ceph-mon[80763]: from='mgr.14214 172.21.15.45:0/1227909437' entity='mgr.smithi045.nhtmwc' 2023-12-16T21:37:21.816 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Dec 16 21:37:21 smithi045 ceph-mon[80763]: from='mgr.14214 172.21.15.45:0/1227909437' entity='mgr.smithi045.nhtmwc' 2023-12-16T21:37:21.816 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Dec 16 21:37:21 smithi045 ceph-mon[80763]: from='mgr.14214 172.21.15.45:0/1227909437' entity='mgr.smithi045.nhtmwc' 2023-12-16T21:37:21.816 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Dec 16 21:37:21 smithi045 ceph-mon[80763]: from='mgr.14214 172.21.15.45:0/1227909437' entity='mgr.smithi045.nhtmwc' cmd=[{"prefix": "auth get-or-create", "entity": "mds.cephfs.smithi169.mokxju", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2023-12-16T21:37:21.817 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Dec 16 21:37:21 smithi045 ceph-mon[80763]: from='mgr.14214 172.21.15.45:0/1227909437' entity='mgr.smithi045.nhtmwc' cmd='[{"prefix": "auth get-or-create", "entity": "mds.cephfs.smithi169.mokxju", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2023-12-16T21:37:21.817 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Dec 16 21:37:21 smithi045 ceph-mon[80763]: from='mgr.14214 172.21.15.45:0/1227909437' entity='mgr.smithi045.nhtmwc' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-12-16T21:37:21.928 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:37:21.925+0000 7f7ed42af700 1 -- 172.21.15.45:0/3946727668 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "fs dump"} v 0) v1 -- 0x7f7ecc111950 con 0x7f7ecc0fee00 2023-12-16T21:37:21.928 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:37:21.925+0000 7f7ec2ffd700 1 -- 172.21.15.45:0/3946727668 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump"}]=0 dumped fsmap epoch 2 v2) v1 ==== 75+0+1076 (secure 0 0 0) 0x7f7ec8057b80 con 0x7f7ecc0fee00 2023-12-16T21:37:21.928 INFO:teuthology.orchestra.run.smithi045.stdout:e2 2023-12-16T21:37:21.928 INFO:teuthology.orchestra.run.smithi045.stdout:enable_multiple, ever_enabled_multiple: 1,1 2023-12-16T21:37:21.928 INFO:teuthology.orchestra.run.smithi045.stdout:default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2023-12-16T21:37:21.928 INFO:teuthology.orchestra.run.smithi045.stdout:legacy client fscid: 1 2023-12-16T21:37:21.928 INFO:teuthology.orchestra.run.smithi045.stdout: 2023-12-16T21:37:21.928 INFO:teuthology.orchestra.run.smithi045.stdout:Filesystem 'cephfs' (1) 2023-12-16T21:37:21.928 INFO:teuthology.orchestra.run.smithi045.stdout:fs_name cephfs 2023-12-16T21:37:21.929 INFO:teuthology.orchestra.run.smithi045.stdout:epoch 2 2023-12-16T21:37:21.929 INFO:teuthology.orchestra.run.smithi045.stdout:flags 12 joinable allow_snaps allow_multimds_snaps 2023-12-16T21:37:21.929 INFO:teuthology.orchestra.run.smithi045.stdout:created 2023-12-16T21:37:19.691154+0000 2023-12-16T21:37:21.929 INFO:teuthology.orchestra.run.smithi045.stdout:modified 2023-12-16T21:37:19.691209+0000 2023-12-16T21:37:21.929 INFO:teuthology.orchestra.run.smithi045.stdout:tableserver 0 2023-12-16T21:37:21.929 INFO:teuthology.orchestra.run.smithi045.stdout:root 0 2023-12-16T21:37:21.929 INFO:teuthology.orchestra.run.smithi045.stdout:session_timeout 60 2023-12-16T21:37:21.929 INFO:teuthology.orchestra.run.smithi045.stdout:session_autoclose 300 2023-12-16T21:37:21.929 INFO:teuthology.orchestra.run.smithi045.stdout:max_file_size 1099511627776 2023-12-16T21:37:21.929 INFO:teuthology.orchestra.run.smithi045.stdout:required_client_features {} 2023-12-16T21:37:21.929 INFO:teuthology.orchestra.run.smithi045.stdout:last_failure 0 2023-12-16T21:37:21.929 INFO:teuthology.orchestra.run.smithi045.stdout:last_failure_osd_epoch 0 2023-12-16T21:37:21.929 INFO:teuthology.orchestra.run.smithi045.stdout:compat compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,7=mds uses inline data,8=no anchor table,9=file layout v2,10=snaprealm v2} 2023-12-16T21:37:21.929 INFO:teuthology.orchestra.run.smithi045.stdout:max_mds 1 2023-12-16T21:37:21.930 INFO:teuthology.orchestra.run.smithi045.stdout:in 2023-12-16T21:37:21.930 INFO:teuthology.orchestra.run.smithi045.stdout:up {} 2023-12-16T21:37:21.930 INFO:teuthology.orchestra.run.smithi045.stdout:failed 2023-12-16T21:37:21.930 INFO:teuthology.orchestra.run.smithi045.stdout:damaged 2023-12-16T21:37:21.930 INFO:teuthology.orchestra.run.smithi045.stdout:stopped 2023-12-16T21:37:21.930 INFO:teuthology.orchestra.run.smithi045.stdout:data_pools [3] 2023-12-16T21:37:21.930 INFO:teuthology.orchestra.run.smithi045.stdout:metadata_pool 2 2023-12-16T21:37:21.930 INFO:teuthology.orchestra.run.smithi045.stdout:inline_data disabled 2023-12-16T21:37:21.930 INFO:teuthology.orchestra.run.smithi045.stdout:balancer 2023-12-16T21:37:21.930 INFO:teuthology.orchestra.run.smithi045.stdout:standby_count_wanted 0 2023-12-16T21:37:21.930 INFO:teuthology.orchestra.run.smithi045.stdout: 2023-12-16T21:37:21.930 INFO:teuthology.orchestra.run.smithi045.stdout: 2023-12-16T21:37:21.931 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:37:21.928+0000 7f7ed42af700 1 -- 172.21.15.45:0/3946727668 >> [v2:172.21.15.45:6800/2012105536,v1:172.21.15.45:6801/2012105536] conn(0x7f7eb8068f60 msgr2=0x7f7eb806b410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:37:21.931 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:37:21.928+0000 7f7ed42af700 1 --2- 172.21.15.45:0/3946727668 >> [v2:172.21.15.45:6800/2012105536,v1:172.21.15.45:6801/2012105536] conn(0x7f7eb8068f60 0x7f7eb806b410 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7f7ebc004f40 tx=0x7f7ebc00b040 comp rx=0 tx=0).stop 2023-12-16T21:37:21.931 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:37:21.928+0000 7f7ed42af700 1 -- 172.21.15.45:0/3946727668 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f7ecc0fee00 msgr2=0x7f7ecc10f740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:37:21.932 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:37:21.928+0000 7f7ed42af700 1 --2- 172.21.15.45:0/3946727668 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f7ecc0fee00 0x7f7ecc10f740 secure :-1 s=READY pgs=245 cs=0 l=1 rev1=1 crypto rx=0x7f7ec8009360 tx=0x7f7ec8009e30 comp rx=0 tx=0).stop 2023-12-16T21:37:21.932 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:37:21.928+0000 7f7ed42af700 1 -- 172.21.15.45:0/3946727668 shutdown_connections 2023-12-16T21:37:21.932 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:37:21.928+0000 7f7ed42af700 1 --2- 172.21.15.45:0/3946727668 >> [v2:172.21.15.45:6800/2012105536,v1:172.21.15.45:6801/2012105536] conn(0x7f7eb8068f60 0x7f7eb806b410 unknown :-1 s=CLOSED pgs=90 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:37:21.932 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:37:21.928+0000 7f7ed42af700 1 --2- 172.21.15.45:0/3946727668 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f7ecc0fee00 0x7f7ecc10f740 unknown :-1 s=CLOSED pgs=245 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:37:21.932 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:37:21.928+0000 7f7ed42af700 1 --2- 172.21.15.45:0/3946727668 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f7ecc0fff70 0x7f7ecc10fc80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:37:21.932 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:37:21.928+0000 7f7ed42af700 1 -- 172.21.15.45:0/3946727668 >> 172.21.15.45:0/3946727668 conn(0x7f7ecc0fa5d0 msgr2=0x7f7ecc103230 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-16T21:37:21.932 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:37:21.928+0000 7f7ed42af700 1 -- 172.21.15.45:0/3946727668 shutdown_connections 2023-12-16T21:37:21.932 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:37:21.928+0000 7f7ed42af700 1 -- 172.21.15.45:0/3946727668 wait complete. 2023-12-16T21:37:21.932 INFO:teuthology.orchestra.run.smithi045.stderr:dumped fsmap epoch 2 2023-12-16T21:37:21.991 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Dec 16 21:37:21 smithi169 ceph-mon[88350]: pgmap v114: 65 pgs: 5 creating+peering, 1 active+clean, 59 unknown; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2023-12-16T21:37:21.991 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Dec 16 21:37:21 smithi169 ceph-mon[88350]: osdmap e41: 6 total, 6 up, 6 in 2023-12-16T21:37:21.991 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Dec 16 21:37:21 smithi169 ceph-mon[88350]: from='mgr.14214 172.21.15.45:0/1227909437' entity='mgr.smithi045.nhtmwc' 2023-12-16T21:37:21.991 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Dec 16 21:37:21 smithi169 ceph-mon[88350]: from='mgr.14214 172.21.15.45:0/1227909437' entity='mgr.smithi045.nhtmwc' 2023-12-16T21:37:21.991 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Dec 16 21:37:21 smithi169 ceph-mon[88350]: from='mgr.14214 172.21.15.45:0/1227909437' entity='mgr.smithi045.nhtmwc' 2023-12-16T21:37:21.991 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Dec 16 21:37:21 smithi169 ceph-mon[88350]: from='mgr.14214 172.21.15.45:0/1227909437' entity='mgr.smithi045.nhtmwc' cmd=[{"prefix": "auth get-or-create", "entity": "mds.cephfs.smithi169.mokxju", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2023-12-16T21:37:21.991 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Dec 16 21:37:21 smithi169 ceph-mon[88350]: from='mgr.14214 172.21.15.45:0/1227909437' entity='mgr.smithi045.nhtmwc' cmd='[{"prefix": "auth get-or-create", "entity": "mds.cephfs.smithi169.mokxju", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2023-12-16T21:37:21.991 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Dec 16 21:37:21 smithi169 ceph-mon[88350]: from='mgr.14214 172.21.15.45:0/1227909437' entity='mgr.smithi045.nhtmwc' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-12-16T21:37:22.202 INFO:teuthology.run_tasks:Running task cephadm.shell... 2023-12-16T21:37:22.211 INFO:tasks.cephadm:Running commands on role host.a host ubuntu@smithi045.front.sepia.ceph.com 2023-12-16T21:37:22.211 DEBUG:teuthology.orchestra.run.smithi045:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 88ff363c-9c5a-11ee-95a5-87774f69a715 -- bash -c 'ceph fs set cephfs max_mds 1' 2023-12-16T21:37:22.487 INFO:teuthology.orchestra.run.smithi045.stderr:Inferring config /var/lib/ceph/88ff363c-9c5a-11ee-95a5-87774f69a715/mon.smithi045/config 2023-12-16T21:37:22.873 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Dec 16 21:37:22 smithi045 ceph-mon[80763]: Deploying daemon mds.cephfs.smithi169.mokxju on smithi169 2023-12-16T21:37:22.874 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Dec 16 21:37:22 smithi045 ceph-mon[80763]: osdmap e42: 6 total, 6 up, 6 in 2023-12-16T21:37:22.874 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Dec 16 21:37:22 smithi045 ceph-mon[80763]: from='client.? 172.21.15.45:0/3946727668' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2023-12-16T21:37:22.874 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Dec 16 21:37:22 smithi045 ceph-mon[80763]: mds.? [v2:172.21.15.45:6826/1230939770,v1:172.21.15.45:6827/1230939770] up:boot 2023-12-16T21:37:22.874 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Dec 16 21:37:22 smithi045 ceph-mon[80763]: daemon mds.cephfs.smithi045.fgsbgq assigned to filesystem cephfs as rank 0 (now has 1 ranks) 2023-12-16T21:37:22.874 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Dec 16 21:37:22 smithi045 ceph-mon[80763]: Health check cleared: MDS_ALL_DOWN (was: 1 filesystem is offline) 2023-12-16T21:37:22.874 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Dec 16 21:37:22 smithi045 ceph-mon[80763]: Health check cleared: MDS_UP_LESS_THAN_MAX (was: 1 filesystem is online with fewer MDS than max_mds) 2023-12-16T21:37:22.874 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Dec 16 21:37:22 smithi045 ceph-mon[80763]: Cluster is now healthy 2023-12-16T21:37:22.874 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Dec 16 21:37:22 smithi045 ceph-mon[80763]: fsmap cephfs:0 1 up:standby 2023-12-16T21:37:22.874 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Dec 16 21:37:22 smithi045 ceph-mon[80763]: from='mgr.14214 172.21.15.45:0/1227909437' entity='mgr.smithi045.nhtmwc' cmd=[{"prefix": "mds metadata", "who": "cephfs.smithi045.fgsbgq"}]: dispatch 2023-12-16T21:37:22.874 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Dec 16 21:37:22 smithi045 ceph-mon[80763]: fsmap cephfs:1 {0=cephfs.smithi045.fgsbgq=up:creating} 2023-12-16T21:37:22.874 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Dec 16 21:37:22 smithi045 ceph-mon[80763]: daemon mds.cephfs.smithi045.fgsbgq is now active in filesystem cephfs as rank 0 2023-12-16T21:37:22.957 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Dec 16 21:37:22 smithi169 ceph-mon[88350]: Deploying daemon mds.cephfs.smithi169.mokxju on smithi169 2023-12-16T21:37:22.957 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Dec 16 21:37:22 smithi169 ceph-mon[88350]: osdmap e42: 6 total, 6 up, 6 in 2023-12-16T21:37:22.957 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Dec 16 21:37:22 smithi169 ceph-mon[88350]: from='client.? 172.21.15.45:0/3946727668' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2023-12-16T21:37:22.958 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Dec 16 21:37:22 smithi169 ceph-mon[88350]: mds.? [v2:172.21.15.45:6826/1230939770,v1:172.21.15.45:6827/1230939770] up:boot 2023-12-16T21:37:22.958 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Dec 16 21:37:22 smithi169 ceph-mon[88350]: daemon mds.cephfs.smithi045.fgsbgq assigned to filesystem cephfs as rank 0 (now has 1 ranks) 2023-12-16T21:37:22.958 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Dec 16 21:37:22 smithi169 ceph-mon[88350]: Health check cleared: MDS_ALL_DOWN (was: 1 filesystem is offline) 2023-12-16T21:37:22.958 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Dec 16 21:37:22 smithi169 ceph-mon[88350]: Health check cleared: MDS_UP_LESS_THAN_MAX (was: 1 filesystem is online with fewer MDS than max_mds) 2023-12-16T21:37:22.958 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Dec 16 21:37:22 smithi169 ceph-mon[88350]: Cluster is now healthy 2023-12-16T21:37:22.958 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Dec 16 21:37:22 smithi169 ceph-mon[88350]: fsmap cephfs:0 1 up:standby 2023-12-16T21:37:22.958 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Dec 16 21:37:22 smithi169 ceph-mon[88350]: from='mgr.14214 172.21.15.45:0/1227909437' entity='mgr.smithi045.nhtmwc' cmd=[{"prefix": "mds metadata", "who": "cephfs.smithi045.fgsbgq"}]: dispatch 2023-12-16T21:37:22.958 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Dec 16 21:37:22 smithi169 ceph-mon[88350]: fsmap cephfs:1 {0=cephfs.smithi045.fgsbgq=up:creating} 2023-12-16T21:37:22.958 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Dec 16 21:37:22 smithi169 ceph-mon[88350]: daemon mds.cephfs.smithi045.fgsbgq is now active in filesystem cephfs as rank 0 2023-12-16T21:37:23.282 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:37:23.277+0000 7f2e1d77a700 1 -- 172.21.15.45:0/559479101 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f2e180fcbf0 msgr2=0x7f2e180fd000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:37:23.282 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:37:23.277+0000 7f2e1d77a700 1 --2- 172.21.15.45:0/559479101 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f2e180fcbf0 0x7f2e180fd000 secure :-1 s=READY pgs=246 cs=0 l=1 rev1=1 crypto rx=0x7f2e0c004940 tx=0x7f2e0c009e30 comp rx=0 tx=0).stop 2023-12-16T21:37:23.282 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:37:23.277+0000 7f2e1d77a700 1 -- 172.21.15.45:0/559479101 shutdown_connections 2023-12-16T21:37:23.282 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:37:23.277+0000 7f2e1d77a700 1 --2- 172.21.15.45:0/559479101 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f2e180fde00 0x7f2e180fe250 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:37:23.282 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:37:23.277+0000 7f2e1d77a700 1 --2- 172.21.15.45:0/559479101 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f2e180fcbf0 0x7f2e180fd000 unknown :-1 s=CLOSED pgs=246 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:37:23.282 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:37:23.277+0000 7f2e1d77a700 1 -- 172.21.15.45:0/559479101 >> 172.21.15.45:0/559479101 conn(0x7f2e180f82d0 msgr2=0x7f2e180fa6e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-16T21:37:23.282 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:37:23.277+0000 7f2e1d77a700 1 -- 172.21.15.45:0/559479101 shutdown_connections 2023-12-16T21:37:23.282 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:37:23.277+0000 7f2e1d77a700 1 -- 172.21.15.45:0/559479101 wait complete. 2023-12-16T21:37:23.282 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:37:23.278+0000 7f2e1d77a700 1 Processor -- start 2023-12-16T21:37:23.283 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:37:23.278+0000 7f2e1d77a700 1 -- start start 2023-12-16T21:37:23.283 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:37:23.278+0000 7f2e1d77a700 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f2e180fcbf0 0x7f2e1810d580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:37:23.283 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:37:23.278+0000 7f2e1d77a700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f2e180fde00 0x7f2e1810dac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:37:23.283 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:37:23.278+0000 7f2e1d77a700 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2e1810e0c0 con 0x7f2e180fde00 2023-12-16T21:37:23.283 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:37:23.278+0000 7f2e1d77a700 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2e1810e960 con 0x7f2e180fcbf0 2023-12-16T21:37:23.283 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:37:23.278+0000 7f2e167fc700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f2e180fde00 0x7f2e1810dac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:37:23.283 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:37:23.278+0000 7f2e167fc700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f2e180fde00 0x7f2e1810dac0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:39672/0 (socket says 172.21.15.45:39672) 2023-12-16T21:37:23.283 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:37:23.278+0000 7f2e167fc700 1 -- 172.21.15.45:0/3948923090 learned_addr learned my addr 172.21.15.45:0/3948923090 (peer_addr_for_me v2:172.21.15.45:0/0) 2023-12-16T21:37:23.283 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:37:23.278+0000 7f2e16ffd700 1 --2- 172.21.15.45:0/3948923090 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f2e180fcbf0 0x7f2e1810d580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:37:23.283 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:37:23.278+0000 7f2e167fc700 1 -- 172.21.15.45:0/3948923090 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f2e180fcbf0 msgr2=0x7f2e1810d580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:37:23.283 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:37:23.278+0000 7f2e167fc700 1 --2- 172.21.15.45:0/3948923090 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f2e180fcbf0 0x7f2e1810d580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:37:23.283 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:37:23.278+0000 7f2e167fc700 1 -- 172.21.15.45:0/3948923090 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2e0c005040 con 0x7f2e180fde00 2023-12-16T21:37:23.283 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:37:23.278+0000 7f2e16ffd700 1 --2- 172.21.15.45:0/3948923090 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f2e180fcbf0 0x7f2e1810d580 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-12-16T21:37:23.283 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:37:23.279+0000 7f2e167fc700 1 --2- 172.21.15.45:0/3948923090 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f2e180fde00 0x7f2e1810dac0 secure :-1 s=READY pgs=247 cs=0 l=1 rev1=1 crypto rx=0x7f2e08002a50 tx=0x7f2e080046b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:37:23.283 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:37:23.279+0000 7f2dff7fe700 1 -- 172.21.15.45:0/3948923090 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f2e08007cc0 con 0x7f2e180fde00 2023-12-16T21:37:23.284 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:37:23.279+0000 7f2e1d77a700 1 -- 172.21.15.45:0/3948923090 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2e1810ec40 con 0x7f2e180fde00 2023-12-16T21:37:23.284 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:37:23.279+0000 7f2dff7fe700 1 -- 172.21.15.45:0/3948923090 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f2e0800f040 con 0x7f2e180fde00 2023-12-16T21:37:23.284 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:37:23.279+0000 7f2dff7fe700 1 -- 172.21.15.45:0/3948923090 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f2e0800d9c0 con 0x7f2e180fde00 2023-12-16T21:37:23.284 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:37:23.279+0000 7f2e1d77a700 1 -- 172.21.15.45:0/3948923090 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2e1810f180 con 0x7f2e180fde00 2023-12-16T21:37:23.285 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:37:23.280+0000 7f2e1d77a700 1 -- 172.21.15.45:0/3948923090 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2e1804bb80 con 0x7f2e180fde00 2023-12-16T21:37:23.285 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:37:23.281+0000 7f2dff7fe700 1 -- 172.21.15.45:0/3948923090 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 19) v1 ==== 86447+0+0 (secure 0 0 0) 0x7f2e0800db40 con 0x7f2e180fde00 2023-12-16T21:37:23.285 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:37:23.281+0000 7f2dff7fe700 1 --2- 172.21.15.45:0/3948923090 >> [v2:172.21.15.45:6800/2012105536,v1:172.21.15.45:6801/2012105536] conn(0x7f2e00068f60 0x7f2e0006b410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:37:23.285 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:37:23.281+0000 7f2dff7fe700 1 -- 172.21.15.45:0/3948923090 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(42..42 src has 1..42) v4 ==== 5464+0+0 (secure 0 0 0) 0x7f2e08086ac0 con 0x7f2e180fde00 2023-12-16T21:37:23.285 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:37:23.281+0000 7f2e16ffd700 1 --2- 172.21.15.45:0/3948923090 >> [v2:172.21.15.45:6800/2012105536,v1:172.21.15.45:6801/2012105536] conn(0x7f2e00068f60 0x7f2e0006b410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:37:23.285 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:37:23.282+0000 7f2e16ffd700 1 --2- 172.21.15.45:0/3948923090 >> [v2:172.21.15.45:6800/2012105536,v1:172.21.15.45:6801/2012105536] conn(0x7f2e00068f60 0x7f2e0006b410 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7f2e0c004890 tx=0x7f2e0c009e30 comp rx=0 tx=0).ready entity=mgr.14214 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:37:23.288 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:37:23.286+0000 7f2dff7fe700 1 -- 172.21.15.45:0/3948923090 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+174570 (secure 0 0 0) 0x7f2e08056a50 con 0x7f2e180fde00 2023-12-16T21:37:23.545 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:37:23.543+0000 7f2e1d77a700 1 -- 172.21.15.45:0/3948923090 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "1"} v 0) v1 -- 0x7f2e1810f5b0 con 0x7f2e180fde00 2023-12-16T21:37:23.816 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Dec 16 21:37:23 smithi045 ceph-mon[80763]: pgmap v117: 65 pgs: 5 creating+peering, 28 active+clean, 32 unknown; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 260 B/s wr, 4 op/s 2023-12-16T21:37:23.816 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Dec 16 21:37:23 smithi045 ceph-mon[80763]: from='mgr.14214 172.21.15.45:0/1227909437' entity='mgr.smithi045.nhtmwc' 2023-12-16T21:37:23.816 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Dec 16 21:37:23 smithi045 ceph-mon[80763]: from='mgr.14214 172.21.15.45:0/1227909437' entity='mgr.smithi045.nhtmwc' 2023-12-16T21:37:23.816 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Dec 16 21:37:23 smithi045 ceph-mon[80763]: from='mgr.14214 172.21.15.45:0/1227909437' entity='mgr.smithi045.nhtmwc' 2023-12-16T21:37:23.817 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Dec 16 21:37:23 smithi045 ceph-mon[80763]: from='mgr.14214 172.21.15.45:0/1227909437' entity='mgr.smithi045.nhtmwc' cmd=[{"prefix": "auth get-or-create", "entity": "mds.cephfs.smithi045.hkjgzw", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2023-12-16T21:37:23.817 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Dec 16 21:37:23 smithi045 ceph-mon[80763]: from='mgr.14214 172.21.15.45:0/1227909437' entity='mgr.smithi045.nhtmwc' cmd='[{"prefix": "auth get-or-create", "entity": "mds.cephfs.smithi045.hkjgzw", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2023-12-16T21:37:23.817 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Dec 16 21:37:23 smithi045 ceph-mon[80763]: from='mgr.14214 172.21.15.45:0/1227909437' entity='mgr.smithi045.nhtmwc' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-12-16T21:37:23.817 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Dec 16 21:37:23 smithi045 ceph-mon[80763]: mds.? [v2:172.21.15.45:6826/1230939770,v1:172.21.15.45:6827/1230939770] up:active 2023-12-16T21:37:23.817 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Dec 16 21:37:23 smithi045 ceph-mon[80763]: mds.? [v2:172.21.15.169:6824/3648207429,v1:172.21.15.169:6825/3648207429] up:boot 2023-12-16T21:37:23.817 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Dec 16 21:37:23 smithi045 ceph-mon[80763]: fsmap cephfs:1 {0=cephfs.smithi045.fgsbgq=up:active} 1 up:standby 2023-12-16T21:37:23.817 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Dec 16 21:37:23 smithi045 ceph-mon[80763]: from='mgr.14214 172.21.15.45:0/1227909437' entity='mgr.smithi045.nhtmwc' cmd=[{"prefix": "mds metadata", "who": "cephfs.smithi169.mokxju"}]: dispatch 2023-12-16T21:37:23.817 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Dec 16 21:37:23 smithi045 ceph-mon[80763]: fsmap cephfs:1 {0=cephfs.smithi045.fgsbgq=up:active} 1 up:standby 2023-12-16T21:37:23.817 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Dec 16 21:37:23 smithi045 ceph-mon[80763]: from='client.? 172.21.15.45:0/3948923090' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "1"}]: dispatch 2023-12-16T21:37:24.133 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Dec 16 21:37:23 smithi169 ceph-mon[88350]: pgmap v117: 65 pgs: 5 creating+peering, 28 active+clean, 32 unknown; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 260 B/s wr, 4 op/s 2023-12-16T21:37:24.134 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Dec 16 21:37:23 smithi169 ceph-mon[88350]: from='mgr.14214 172.21.15.45:0/1227909437' entity='mgr.smithi045.nhtmwc' 2023-12-16T21:37:24.134 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Dec 16 21:37:23 smithi169 ceph-mon[88350]: from='mgr.14214 172.21.15.45:0/1227909437' entity='mgr.smithi045.nhtmwc' 2023-12-16T21:37:24.134 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Dec 16 21:37:23 smithi169 ceph-mon[88350]: from='mgr.14214 172.21.15.45:0/1227909437' entity='mgr.smithi045.nhtmwc' 2023-12-16T21:37:24.134 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Dec 16 21:37:23 smithi169 ceph-mon[88350]: from='mgr.14214 172.21.15.45:0/1227909437' entity='mgr.smithi045.nhtmwc' cmd=[{"prefix": "auth get-or-create", "entity": "mds.cephfs.smithi045.hkjgzw", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2023-12-16T21:37:24.134 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Dec 16 21:37:23 smithi169 ceph-mon[88350]: from='mgr.14214 172.21.15.45:0/1227909437' entity='mgr.smithi045.nhtmwc' cmd='[{"prefix": "auth get-or-create", "entity": "mds.cephfs.smithi045.hkjgzw", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2023-12-16T21:37:24.134 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Dec 16 21:37:23 smithi169 ceph-mon[88350]: from='mgr.14214 172.21.15.45:0/1227909437' entity='mgr.smithi045.nhtmwc' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-12-16T21:37:24.134 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Dec 16 21:37:23 smithi169 ceph-mon[88350]: mds.? [v2:172.21.15.45:6826/1230939770,v1:172.21.15.45:6827/1230939770] up:active 2023-12-16T21:37:24.134 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Dec 16 21:37:23 smithi169 ceph-mon[88350]: mds.? [v2:172.21.15.169:6824/3648207429,v1:172.21.15.169:6825/3648207429] up:boot 2023-12-16T21:37:24.134 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Dec 16 21:37:23 smithi169 ceph-mon[88350]: fsmap cephfs:1 {0=cephfs.smithi045.fgsbgq=up:active} 1 up:standby 2023-12-16T21:37:24.134 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Dec 16 21:37:23 smithi169 ceph-mon[88350]: from='mgr.14214 172.21.15.45:0/1227909437' entity='mgr.smithi045.nhtmwc' cmd=[{"prefix": "mds metadata", "who": "cephfs.smithi169.mokxju"}]: dispatch 2023-12-16T21:37:24.134 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Dec 16 21:37:23 smithi169 ceph-mon[88350]: fsmap cephfs:1 {0=cephfs.smithi045.fgsbgq=up:active} 1 up:standby 2023-12-16T21:37:24.134 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Dec 16 21:37:23 smithi169 ceph-mon[88350]: from='client.? 172.21.15.45:0/3948923090' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "1"}]: dispatch 2023-12-16T21:37:24.320 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:37:24.316+0000 7f2dff7fe700 1 -- 172.21.15.45:0/3948923090 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "1"}]=0 v7) v1 ==== 105+0+0 (secure 0 0 0) 0x7f2e0800ddf0 con 0x7f2e180fde00 2023-12-16T21:37:24.321 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:37:24.319+0000 7f2e1d77a700 1 -- 172.21.15.45:0/3948923090 >> [v2:172.21.15.45:6800/2012105536,v1:172.21.15.45:6801/2012105536] conn(0x7f2e00068f60 msgr2=0x7f2e0006b410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:37:24.321 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:37:24.319+0000 7f2e1d77a700 1 --2- 172.21.15.45:0/3948923090 >> [v2:172.21.15.45:6800/2012105536,v1:172.21.15.45:6801/2012105536] conn(0x7f2e00068f60 0x7f2e0006b410 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7f2e0c004890 tx=0x7f2e0c009e30 comp rx=0 tx=0).stop 2023-12-16T21:37:24.321 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:37:24.319+0000 7f2e1d77a700 1 -- 172.21.15.45:0/3948923090 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f2e180fde00 msgr2=0x7f2e1810dac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:37:24.321 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:37:24.319+0000 7f2e1d77a700 1 --2- 172.21.15.45:0/3948923090 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f2e180fde00 0x7f2e1810dac0 secure :-1 s=READY pgs=247 cs=0 l=1 rev1=1 crypto rx=0x7f2e08002a50 tx=0x7f2e080046b0 comp rx=0 tx=0).stop 2023-12-16T21:37:24.322 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:37:24.319+0000 7f2e1d77a700 1 -- 172.21.15.45:0/3948923090 shutdown_connections 2023-12-16T21:37:24.322 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:37:24.319+0000 7f2e1d77a700 1 --2- 172.21.15.45:0/3948923090 >> [v2:172.21.15.45:6800/2012105536,v1:172.21.15.45:6801/2012105536] conn(0x7f2e00068f60 0x7f2e0006b410 unknown :-1 s=CLOSED pgs=92 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:37:24.322 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:37:24.319+0000 7f2e1d77a700 1 --2- 172.21.15.45:0/3948923090 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f2e180fcbf0 0x7f2e1810d580 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:37:24.322 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:37:24.319+0000 7f2e1d77a700 1 --2- 172.21.15.45:0/3948923090 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f2e180fde00 0x7f2e1810dac0 unknown :-1 s=CLOSED pgs=247 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:37:24.322 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:37:24.319+0000 7f2e1d77a700 1 -- 172.21.15.45:0/3948923090 >> 172.21.15.45:0/3948923090 conn(0x7f2e180f82d0 msgr2=0x7f2e18101030 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-16T21:37:24.322 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:37:24.319+0000 7f2e1d77a700 1 -- 172.21.15.45:0/3948923090 shutdown_connections 2023-12-16T21:37:24.322 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:37:24.319+0000 7f2e1d77a700 1 -- 172.21.15.45:0/3948923090 wait complete. 2023-12-16T21:37:24.607 INFO:teuthology.run_tasks:Running task cephadm.shell... 2023-12-16T21:37:24.618 INFO:tasks.cephadm:Running commands on role host.a host ubuntu@smithi045.front.sepia.ceph.com 2023-12-16T21:37:24.618 DEBUG:teuthology.orchestra.run.smithi045:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 88ff363c-9c5a-11ee-95a5-87774f69a715 -- bash -c 'ceph fs set cephfs allow_standby_replay true' 2023-12-16T21:37:24.764 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Dec 16 21:37:24 smithi045 ceph-mon[80763]: Deploying daemon mds.cephfs.smithi045.hkjgzw on smithi045 2023-12-16T21:37:24.765 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Dec 16 21:37:24 smithi045 ceph-mon[80763]: from='client.? 172.21.15.45:0/3948923090' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "1"}]': finished 2023-12-16T21:37:24.765 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Dec 16 21:37:24 smithi045 ceph-mon[80763]: fsmap cephfs:1 {0=cephfs.smithi045.fgsbgq=up:active} 1 up:standby 2023-12-16T21:37:25.000 INFO:teuthology.orchestra.run.smithi045.stderr:Inferring config /var/lib/ceph/88ff363c-9c5a-11ee-95a5-87774f69a715/mon.smithi045/config 2023-12-16T21:37:25.133 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Dec 16 21:37:24 smithi169 ceph-mon[88350]: Deploying daemon mds.cephfs.smithi045.hkjgzw on smithi045 2023-12-16T21:37:25.133 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Dec 16 21:37:24 smithi169 ceph-mon[88350]: from='client.? 172.21.15.45:0/3948923090' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "1"}]': finished 2023-12-16T21:37:25.133 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Dec 16 21:37:24 smithi169 ceph-mon[88350]: fsmap cephfs:1 {0=cephfs.smithi045.fgsbgq=up:active} 1 up:standby 2023-12-16T21:37:25.708 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Dec 16 21:37:25 smithi169 ceph-mon[88350]: pgmap v118: 65 pgs: 65 active+clean; 579 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 2.2 KiB/s wr, 7 op/s 2023-12-16T21:37:25.708 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Dec 16 21:37:25 smithi169 ceph-mon[88350]: from='mgr.14214 172.21.15.45:0/1227909437' entity='mgr.smithi045.nhtmwc' 2023-12-16T21:37:25.708 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Dec 16 21:37:25 smithi169 ceph-mon[88350]: from='mgr.14214 172.21.15.45:0/1227909437' entity='mgr.smithi045.nhtmwc' 2023-12-16T21:37:25.783 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:37:25.779+0000 7fb21365e700 1 -- 172.21.15.45:0/1148632677 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fb20c0fde00 msgr2=0x7fb20c0fe250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:37:25.783 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:37:25.779+0000 7fb21365e700 1 --2- 172.21.15.45:0/1148632677 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fb20c0fde00 0x7fb20c0fe250 secure :-1 s=READY pgs=250 cs=0 l=1 rev1=1 crypto rx=0x7fb1fc004940 tx=0x7fb1fc009e30 comp rx=0 tx=0).stop 2023-12-16T21:37:25.783 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:37:25.779+0000 7fb21365e700 1 -- 172.21.15.45:0/1148632677 shutdown_connections 2023-12-16T21:37:25.783 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:37:25.779+0000 7fb21365e700 1 --2- 172.21.15.45:0/1148632677 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fb20c0fde00 0x7fb20c0fe250 unknown :-1 s=CLOSED pgs=250 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:37:25.783 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:37:25.779+0000 7fb21365e700 1 --2- 172.21.15.45:0/1148632677 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fb20c0fcbf0 0x7fb20c0fd000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:37:25.784 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:37:25.779+0000 7fb21365e700 1 -- 172.21.15.45:0/1148632677 >> 172.21.15.45:0/1148632677 conn(0x7fb20c0f82d0 msgr2=0x7fb20c0fa6e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-16T21:37:25.784 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:37:25.779+0000 7fb21365e700 1 -- 172.21.15.45:0/1148632677 shutdown_connections 2023-12-16T21:37:25.784 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:37:25.779+0000 7fb21365e700 1 -- 172.21.15.45:0/1148632677 wait complete. 2023-12-16T21:37:25.784 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:37:25.779+0000 7fb21365e700 1 Processor -- start 2023-12-16T21:37:25.784 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:37:25.779+0000 7fb21365e700 1 -- start start 2023-12-16T21:37:25.784 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:37:25.780+0000 7fb21365e700 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fb20c0fcbf0 0x7fb20c10d490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:37:25.784 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:37:25.780+0000 7fb21365e700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fb20c0fde00 0x7fb20c10d9d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:37:25.784 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:37:25.780+0000 7fb21365e700 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb20c10dfd0 con 0x7fb20c0fde00 2023-12-16T21:37:25.784 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:37:25.780+0000 7fb21365e700 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb20c10e870 con 0x7fb20c0fcbf0 2023-12-16T21:37:25.785 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:37:25.780+0000 7fb210bf9700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fb20c0fde00 0x7fb20c10d9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:37:25.785 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:37:25.780+0000 7fb210bf9700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fb20c0fde00 0x7fb20c10d9d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:39700/0 (socket says 172.21.15.45:39700) 2023-12-16T21:37:25.785 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:37:25.780+0000 7fb210bf9700 1 -- 172.21.15.45:0/2647252937 learned_addr learned my addr 172.21.15.45:0/2647252937 (peer_addr_for_me v2:172.21.15.45:0/0) 2023-12-16T21:37:25.785 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:37:25.780+0000 7fb2113fa700 1 --2- 172.21.15.45:0/2647252937 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fb20c0fcbf0 0x7fb20c10d490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:37:25.785 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:37:25.780+0000 7fb210bf9700 1 -- 172.21.15.45:0/2647252937 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fb20c0fcbf0 msgr2=0x7fb20c10d490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:37:25.785 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:37:25.780+0000 7fb210bf9700 1 --2- 172.21.15.45:0/2647252937 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fb20c0fcbf0 0x7fb20c10d490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:37:25.785 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:37:25.780+0000 7fb210bf9700 1 -- 172.21.15.45:0/2647252937 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb1fc005040 con 0x7fb20c0fde00 2023-12-16T21:37:25.785 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:37:25.780+0000 7fb210bf9700 1 --2- 172.21.15.45:0/2647252937 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fb20c0fde00 0x7fb20c10d9d0 secure :-1 s=READY pgs=251 cs=0 l=1 rev1=1 crypto rx=0x7fb1fc000c00 tx=0x7fb1fc009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:37:25.785 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:37:25.780+0000 7fb201ffb700 1 -- 172.21.15.45:0/2647252937 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fb1fc021070 con 0x7fb20c0fde00 2023-12-16T21:37:25.785 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:37:25.780+0000 7fb201ffb700 1 -- 172.21.15.45:0/2647252937 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fb1fc01c400 con 0x7fb20c0fde00 2023-12-16T21:37:25.785 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:37:25.780+0000 7fb21365e700 1 -- 172.21.15.45:0/2647252937 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb20c10eaf0 con 0x7fb20c0fde00 2023-12-16T21:37:25.785 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:37:25.780+0000 7fb201ffb700 1 -- 172.21.15.45:0/2647252937 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fb1fc00e070 con 0x7fb20c0fde00 2023-12-16T21:37:25.785 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:37:25.781+0000 7fb21365e700 1 -- 172.21.15.45:0/2647252937 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb20c10efd0 con 0x7fb20c0fde00 2023-12-16T21:37:25.787 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:37:25.782+0000 7fb21365e700 1 -- 172.21.15.45:0/2647252937 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb20c04bb80 con 0x7fb20c0fde00 2023-12-16T21:37:25.787 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:37:25.782+0000 7fb201ffb700 1 -- 172.21.15.45:0/2647252937 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 19) v1 ==== 86447+0+0 (secure 0 0 0) 0x7fb1fc01c9c0 con 0x7fb20c0fde00 2023-12-16T21:37:25.787 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:37:25.783+0000 7fb201ffb700 1 --2- 172.21.15.45:0/2647252937 >> [v2:172.21.15.45:6800/2012105536,v1:172.21.15.45:6801/2012105536] conn(0x7fb1f8068f60 0x7fb1f806b410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:37:25.787 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:37:25.783+0000 7fb201ffb700 1 -- 172.21.15.45:0/2647252937 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(42..42 src has 1..42) v4 ==== 5464+0+0 (secure 0 0 0) 0x7fb1fc087ff0 con 0x7fb20c0fde00 2023-12-16T21:37:25.787 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:37:25.783+0000 7fb2113fa700 1 --2- 172.21.15.45:0/2647252937 >> [v2:172.21.15.45:6800/2012105536,v1:172.21.15.45:6801/2012105536] conn(0x7fb1f8068f60 0x7fb1f806b410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:37:25.787 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:37:25.783+0000 7fb2113fa700 1 --2- 172.21.15.45:0/2647252937 >> [v2:172.21.15.45:6800/2012105536,v1:172.21.15.45:6801/2012105536] conn(0x7fb1f8068f60 0x7fb1f806b410 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7fb208009b80 tx=0x7fb20800b040 comp rx=0 tx=0).ready entity=mgr.14214 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:37:25.790 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:37:25.787+0000 7fb201ffb700 1 -- 172.21.15.45:0/2647252937 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+174570 (secure 0 0 0) 0x7fb1fc057f50 con 0x7fb20c0fde00 2023-12-16T21:37:26.033 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:37:26.030+0000 7fb21365e700 1 -- 172.21.15.45:0/2647252937 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "true"} v 0) v1 -- 0x7fb20c105590 con 0x7fb20c0fde00 2023-12-16T21:37:26.033 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Dec 16 21:37:25 smithi045 ceph-mon[80763]: pgmap v118: 65 pgs: 65 active+clean; 579 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 2.2 KiB/s wr, 7 op/s 2023-12-16T21:37:26.033 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Dec 16 21:37:25 smithi045 ceph-mon[80763]: from='mgr.14214 172.21.15.45:0/1227909437' entity='mgr.smithi045.nhtmwc' 2023-12-16T21:37:26.033 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Dec 16 21:37:25 smithi045 ceph-mon[80763]: from='mgr.14214 172.21.15.45:0/1227909437' entity='mgr.smithi045.nhtmwc' 2023-12-16T21:37:26.033 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Dec 16 21:37:25 smithi045 ceph-mon[80763]: from='mgr.14214 172.21.15.45:0/1227909437' entity='mgr.smithi045.nhtmwc' 2023-12-16T21:37:26.033 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Dec 16 21:37:25 smithi045 ceph-mon[80763]: from='mgr.14214 172.21.15.45:0/1227909437' entity='mgr.smithi045.nhtmwc' cmd=[{"prefix": "auth get-or-create", "entity": "mds.cephfs.smithi169.pihlib", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2023-12-16T21:37:26.033 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Dec 16 21:37:25 smithi045 ceph-mon[80763]: from='mgr.14214 172.21.15.45:0/1227909437' entity='mgr.smithi045.nhtmwc' cmd='[{"prefix": "auth get-or-create", "entity": "mds.cephfs.smithi169.pihlib", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2023-12-16T21:37:26.033 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Dec 16 21:37:25 smithi045 ceph-mon[80763]: from='mgr.14214 172.21.15.45:0/1227909437' entity='mgr.smithi045.nhtmwc' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-12-16T21:37:26.121 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Dec 16 21:37:25 smithi169 ceph-mon[88350]: from='mgr.14214 172.21.15.45:0/1227909437' entity='mgr.smithi045.nhtmwc' 2023-12-16T21:37:26.121 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Dec 16 21:37:25 smithi169 ceph-mon[88350]: from='mgr.14214 172.21.15.45:0/1227909437' entity='mgr.smithi045.nhtmwc' cmd=[{"prefix": "auth get-or-create", "entity": "mds.cephfs.smithi169.pihlib", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2023-12-16T21:37:26.121 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Dec 16 21:37:25 smithi169 ceph-mon[88350]: from='mgr.14214 172.21.15.45:0/1227909437' entity='mgr.smithi045.nhtmwc' cmd='[{"prefix": "auth get-or-create", "entity": "mds.cephfs.smithi169.pihlib", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2023-12-16T21:37:26.121 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Dec 16 21:37:25 smithi169 ceph-mon[88350]: from='mgr.14214 172.21.15.45:0/1227909437' entity='mgr.smithi045.nhtmwc' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-12-16T21:37:26.302 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:37:26.299+0000 7fb201ffb700 1 -- 172.21.15.45:0/2647252937 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "true"}]=0 v9) v1 ==== 121+0+0 (secure 0 0 0) 0x7fb1fc057ae0 con 0x7fb20c0fde00 2023-12-16T21:37:26.304 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:37:26.302+0000 7fb21365e700 1 -- 172.21.15.45:0/2647252937 >> [v2:172.21.15.45:6800/2012105536,v1:172.21.15.45:6801/2012105536] conn(0x7fb1f8068f60 msgr2=0x7fb1f806b410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:37:26.304 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:37:26.302+0000 7fb21365e700 1 --2- 172.21.15.45:0/2647252937 >> [v2:172.21.15.45:6800/2012105536,v1:172.21.15.45:6801/2012105536] conn(0x7fb1f8068f60 0x7fb1f806b410 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7fb208009b80 tx=0x7fb20800b040 comp rx=0 tx=0).stop 2023-12-16T21:37:26.305 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:37:26.302+0000 7fb21365e700 1 -- 172.21.15.45:0/2647252937 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fb20c0fde00 msgr2=0x7fb20c10d9d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:37:26.305 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:37:26.302+0000 7fb21365e700 1 --2- 172.21.15.45:0/2647252937 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fb20c0fde00 0x7fb20c10d9d0 secure :-1 s=READY pgs=251 cs=0 l=1 rev1=1 crypto rx=0x7fb1fc000c00 tx=0x7fb1fc009e30 comp rx=0 tx=0).stop 2023-12-16T21:37:26.305 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:37:26.302+0000 7fb21365e700 1 -- 172.21.15.45:0/2647252937 shutdown_connections 2023-12-16T21:37:26.305 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:37:26.302+0000 7fb21365e700 1 --2- 172.21.15.45:0/2647252937 >> [v2:172.21.15.45:6800/2012105536,v1:172.21.15.45:6801/2012105536] conn(0x7fb1f8068f60 0x7fb1f806b410 unknown :-1 s=CLOSED pgs=95 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:37:26.305 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:37:26.302+0000 7fb21365e700 1 --2- 172.21.15.45:0/2647252937 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fb20c0fcbf0 0x7fb20c10d490 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:37:26.305 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:37:26.302+0000 7fb21365e700 1 --2- 172.21.15.45:0/2647252937 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fb20c0fde00 0x7fb20c10d9d0 unknown :-1 s=CLOSED pgs=251 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:37:26.305 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:37:26.302+0000 7fb21365e700 1 -- 172.21.15.45:0/2647252937 >> 172.21.15.45:0/2647252937 conn(0x7fb20c0f82d0 msgr2=0x7fb20c101030 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-16T21:37:26.305 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:37:26.302+0000 7fb21365e700 1 -- 172.21.15.45:0/2647252937 shutdown_connections 2023-12-16T21:37:26.305 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:37:26.302+0000 7fb21365e700 1 -- 172.21.15.45:0/2647252937 wait complete. 2023-12-16T21:37:26.560 INFO:teuthology.run_tasks:Running task cephadm.shell... 2023-12-16T21:37:26.572 INFO:tasks.cephadm:Running commands on role host.a host ubuntu@smithi045.front.sepia.ceph.com 2023-12-16T21:37:26.572 DEBUG:teuthology.orchestra.run.smithi045:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 88ff363c-9c5a-11ee-95a5-87774f69a715 -- bash -c 'ceph fs set cephfs inline_data true --yes-i-really-really-mean-it' 2023-12-16T21:37:26.816 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Dec 16 21:37:26 smithi045 ceph-mon[80763]: Deploying daemon mds.cephfs.smithi169.pihlib on smithi169 2023-12-16T21:37:26.817 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Dec 16 21:37:26 smithi045 ceph-mon[80763]: mds.? [v2:172.21.15.45:6828/622084067,v1:172.21.15.45:6829/622084067] up:boot 2023-12-16T21:37:26.817 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Dec 16 21:37:26 smithi045 ceph-mon[80763]: fsmap cephfs:1 {0=cephfs.smithi045.fgsbgq=up:active} 2 up:standby 2023-12-16T21:37:26.817 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Dec 16 21:37:26 smithi045 ceph-mon[80763]: from='mgr.14214 172.21.15.45:0/1227909437' entity='mgr.smithi045.nhtmwc' cmd=[{"prefix": "mds metadata", "who": "cephfs.smithi045.hkjgzw"}]: dispatch 2023-12-16T21:37:26.817 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Dec 16 21:37:26 smithi045 ceph-mon[80763]: from='client.? 172.21.15.45:0/2647252937' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "true"}]: dispatch 2023-12-16T21:37:26.817 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Dec 16 21:37:26 smithi045 ceph-mon[80763]: from='client.? 172.21.15.45:0/2647252937' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "true"}]': finished 2023-12-16T21:37:26.818 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Dec 16 21:37:26 smithi045 ceph-mon[80763]: fsmap cephfs:1 {0=cephfs.smithi045.fgsbgq=up:active} 1 up:standby-replay 1 up:standby 2023-12-16T21:37:26.818 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Dec 16 21:37:26 smithi045 ceph-mon[80763]: from='mgr.14214 172.21.15.45:0/1227909437' entity='mgr.smithi045.nhtmwc' 2023-12-16T21:37:26.818 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Dec 16 21:37:26 smithi045 ceph-mon[80763]: from='mgr.14214 172.21.15.45:0/1227909437' entity='mgr.smithi045.nhtmwc' 2023-12-16T21:37:26.818 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Dec 16 21:37:26 smithi045 ceph-mon[80763]: from='mgr.14214 172.21.15.45:0/1227909437' entity='mgr.smithi045.nhtmwc' 2023-12-16T21:37:26.818 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Dec 16 21:37:26 smithi045 ceph-mon[80763]: from='mgr.14214 172.21.15.45:0/1227909437' entity='mgr.smithi045.nhtmwc' 2023-12-16T21:37:26.818 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Dec 16 21:37:26 smithi045 ceph-mon[80763]: from='mgr.14214 172.21.15.45:0/1227909437' entity='mgr.smithi045.nhtmwc' 2023-12-16T21:37:26.819 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Dec 16 21:37:26 smithi045 ceph-mon[80763]: from='mgr.14214 172.21.15.45:0/1227909437' entity='mgr.smithi045.nhtmwc' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-12-16T21:37:26.855 INFO:teuthology.orchestra.run.smithi045.stderr:Inferring config /var/lib/ceph/88ff363c-9c5a-11ee-95a5-87774f69a715/mon.smithi045/config 2023-12-16T21:37:27.060 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Dec 16 21:37:26 smithi169 ceph-mon[88350]: Deploying daemon mds.cephfs.smithi169.pihlib on smithi169 2023-12-16T21:37:27.061 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Dec 16 21:37:26 smithi169 ceph-mon[88350]: mds.? [v2:172.21.15.45:6828/622084067,v1:172.21.15.45:6829/622084067] up:boot 2023-12-16T21:37:27.061 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Dec 16 21:37:26 smithi169 ceph-mon[88350]: fsmap cephfs:1 {0=cephfs.smithi045.fgsbgq=up:active} 2 up:standby 2023-12-16T21:37:27.061 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Dec 16 21:37:26 smithi169 ceph-mon[88350]: from='mgr.14214 172.21.15.45:0/1227909437' entity='mgr.smithi045.nhtmwc' cmd=[{"prefix": "mds metadata", "who": "cephfs.smithi045.hkjgzw"}]: dispatch 2023-12-16T21:37:27.061 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Dec 16 21:37:26 smithi169 ceph-mon[88350]: from='client.? 172.21.15.45:0/2647252937' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "true"}]: dispatch 2023-12-16T21:37:27.061 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Dec 16 21:37:26 smithi169 ceph-mon[88350]: from='client.? 172.21.15.45:0/2647252937' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "true"}]': finished 2023-12-16T21:37:27.061 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Dec 16 21:37:26 smithi169 ceph-mon[88350]: fsmap cephfs:1 {0=cephfs.smithi045.fgsbgq=up:active} 1 up:standby-replay 1 up:standby 2023-12-16T21:37:27.061 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Dec 16 21:37:26 smithi169 ceph-mon[88350]: from='mgr.14214 172.21.15.45:0/1227909437' entity='mgr.smithi045.nhtmwc' 2023-12-16T21:37:27.061 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Dec 16 21:37:26 smithi169 ceph-mon[88350]: from='mgr.14214 172.21.15.45:0/1227909437' entity='mgr.smithi045.nhtmwc' 2023-12-16T21:37:27.061 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Dec 16 21:37:26 smithi169 ceph-mon[88350]: from='mgr.14214 172.21.15.45:0/1227909437' entity='mgr.smithi045.nhtmwc' 2023-12-16T21:37:27.061 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Dec 16 21:37:26 smithi169 ceph-mon[88350]: from='mgr.14214 172.21.15.45:0/1227909437' entity='mgr.smithi045.nhtmwc' 2023-12-16T21:37:27.061 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Dec 16 21:37:26 smithi169 ceph-mon[88350]: from='mgr.14214 172.21.15.45:0/1227909437' entity='mgr.smithi045.nhtmwc' 2023-12-16T21:37:27.061 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Dec 16 21:37:26 smithi169 ceph-mon[88350]: from='mgr.14214 172.21.15.45:0/1227909437' entity='mgr.smithi045.nhtmwc' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-12-16T21:37:27.923 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:37:27.918+0000 7f0af43d0700 1 -- 172.21.15.45:0/1806386813 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f0aec0fd740 msgr2=0x7f0aec0ffb20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:37:27.923 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:37:27.918+0000 7f0af43d0700 1 --2- 172.21.15.45:0/1806386813 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f0aec0fd740 0x7f0aec0ffb20 secure :-1 s=READY pgs=254 cs=0 l=1 rev1=1 crypto rx=0x7f0ae8004940 tx=0x7f0ae8009e30 comp rx=0 tx=0).stop 2023-12-16T21:37:27.923 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:37:27.918+0000 7f0af43d0700 1 -- 172.21.15.45:0/1806386813 shutdown_connections 2023-12-16T21:37:27.924 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:37:27.918+0000 7f0af43d0700 1 --2- 172.21.15.45:0/1806386813 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f0aec100060 0x7f0aec102440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:37:27.924 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:37:27.918+0000 7f0af43d0700 1 --2- 172.21.15.45:0/1806386813 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f0aec0fd740 0x7f0aec0ffb20 unknown :-1 s=CLOSED pgs=254 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:37:27.924 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:37:27.918+0000 7f0af43d0700 1 -- 172.21.15.45:0/1806386813 >> 172.21.15.45:0/1806386813 conn(0x7f0aec0f71c0 msgr2=0x7f0aec0f95f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-16T21:37:27.924 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:37:27.918+0000 7f0af43d0700 1 -- 172.21.15.45:0/1806386813 shutdown_connections 2023-12-16T21:37:27.924 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:37:27.918+0000 7f0af43d0700 1 -- 172.21.15.45:0/1806386813 wait complete. 2023-12-16T21:37:27.924 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:37:27.919+0000 7f0af43d0700 1 Processor -- start 2023-12-16T21:37:27.924 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:37:27.919+0000 7f0af43d0700 1 -- start start 2023-12-16T21:37:27.924 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:37:27.919+0000 7f0af43d0700 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f0aec0fd740 0x7f0aec0fd1d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:37:27.924 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:37:27.919+0000 7f0af43d0700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f0aec100060 0x7f0aec0fb650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:37:27.925 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:37:27.919+0000 7f0af43d0700 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0aec111160 con 0x7f0aec100060 2023-12-16T21:37:27.925 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:37:27.919+0000 7f0af43d0700 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0aec1112d0 con 0x7f0aec0fd740 2023-12-16T21:37:27.925 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:37:27.919+0000 7f0af196b700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f0aec100060 0x7f0aec0fb650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:37:27.925 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:37:27.919+0000 7f0af196b700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f0aec100060 0x7f0aec0fb650 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:39746/0 (socket says 172.21.15.45:39746) 2023-12-16T21:37:27.925 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:37:27.919+0000 7f0af196b700 1 -- 172.21.15.45:0/3294380997 learned_addr learned my addr 172.21.15.45:0/3294380997 (peer_addr_for_me v2:172.21.15.45:0/0) 2023-12-16T21:37:27.925 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:37:27.919+0000 7f0af216c700 1 --2- 172.21.15.45:0/3294380997 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f0aec0fd740 0x7f0aec0fd1d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:37:27.925 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:37:27.919+0000 7f0af196b700 1 -- 172.21.15.45:0/3294380997 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f0aec0fd740 msgr2=0x7f0aec0fd1d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:37:27.925 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:37:27.920+0000 7f0af196b700 1 --2- 172.21.15.45:0/3294380997 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f0aec0fd740 0x7f0aec0fd1d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:37:27.925 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:37:27.920+0000 7f0af196b700 1 -- 172.21.15.45:0/3294380997 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0ae8005040 con 0x7f0aec100060 2023-12-16T21:37:27.925 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:37:27.920+0000 7f0af196b700 1 --2- 172.21.15.45:0/3294380997 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f0aec100060 0x7f0aec0fb650 secure :-1 s=READY pgs=255 cs=0 l=1 rev1=1 crypto rx=0x7f0adc00eb40 tx=0x7f0adc00b040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:37:27.925 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:37:27.920+0000 7f0ae2ffd700 1 -- 172.21.15.45:0/3294380997 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f0adc007cc0 con 0x7f0aec100060 2023-12-16T21:37:27.926 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:37:27.920+0000 7f0ae2ffd700 1 -- 172.21.15.45:0/3294380997 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f0adc00f040 con 0x7f0aec100060 2023-12-16T21:37:27.926 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:37:27.920+0000 7f0ae2ffd700 1 -- 172.21.15.45:0/3294380997 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f0adc00c990 con 0x7f0aec100060 2023-12-16T21:37:27.926 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:37:27.920+0000 7f0af43d0700 1 -- 172.21.15.45:0/3294380997 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0aec0fbc50 con 0x7f0aec100060 2023-12-16T21:37:27.926 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:37:27.921+0000 7f0af43d0700 1 -- 172.21.15.45:0/3294380997 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0aec0fc140 con 0x7f0aec100060 2023-12-16T21:37:27.927 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:37:27.922+0000 7f0af43d0700 1 -- 172.21.15.45:0/3294380997 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0aec04bb80 con 0x7f0aec100060 2023-12-16T21:37:27.927 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:37:27.922+0000 7f0ae2ffd700 1 -- 172.21.15.45:0/3294380997 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 19) v1 ==== 86447+0+0 (secure 0 0 0) 0x7f0adc00cc00 con 0x7f0aec100060 2023-12-16T21:37:27.927 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:37:27.923+0000 7f0ae2ffd700 1 --2- 172.21.15.45:0/3294380997 >> [v2:172.21.15.45:6800/2012105536,v1:172.21.15.45:6801/2012105536] conn(0x7f0ad8068f60 0x7f0ad806b410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:37:27.927 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:37:27.923+0000 7f0ae2ffd700 1 -- 172.21.15.45:0/3294380997 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(43..43 src has 1..43) v4 ==== 5550+0+0 (secure 0 0 0) 0x7f0adc0869e0 con 0x7f0aec100060 2023-12-16T21:37:27.927 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:37:27.923+0000 7f0af216c700 1 --2- 172.21.15.45:0/3294380997 >> [v2:172.21.15.45:6800/2012105536,v1:172.21.15.45:6801/2012105536] conn(0x7f0ad8068f60 0x7f0ad806b410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:37:27.927 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:37:27.923+0000 7f0af216c700 1 --2- 172.21.15.45:0/3294380997 >> [v2:172.21.15.45:6800/2012105536,v1:172.21.15.45:6801/2012105536] conn(0x7f0ad8068f60 0x7f0ad806b410 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7f0ae8013680 tx=0x7f0ae8009e30 comp rx=0 tx=0).ready entity=mgr.14214 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:37:27.930 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:37:27.928+0000 7f0ae2ffd700 1 -- 172.21.15.45:0/3294380997 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+174570 (secure 0 0 0) 0x7f0adc04da00 con 0x7f0aec100060 2023-12-16T21:37:28.026 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Dec 16 21:37:27 smithi045 ceph-mon[80763]: pgmap v119: 65 pgs: 65 active+clean; 579 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 148 B/s rd, 1.9 KiB/s wr, 6 op/s 2023-12-16T21:37:28.027 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Dec 16 21:37:27 smithi045 ceph-mon[80763]: mds.? [v2:172.21.15.169:6826/4053377419,v1:172.21.15.169:6827/4053377419] up:boot 2023-12-16T21:37:28.027 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Dec 16 21:37:27 smithi045 ceph-mon[80763]: mds.? [v2:172.21.15.169:6824/3648207429,v1:172.21.15.169:6825/3648207429] up:standby-replay 2023-12-16T21:37:28.027 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Dec 16 21:37:27 smithi045 ceph-mon[80763]: Dropping low affinity active daemon mds.cephfs.smithi045.fgsbgq in favor of higher affinity standby. 2023-12-16T21:37:28.027 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Dec 16 21:37:27 smithi045 ceph-mon[80763]: Replacing daemon mds.cephfs.smithi045.fgsbgq as rank 0 with standby daemon mds.cephfs.smithi169.mokxju 2023-12-16T21:37:28.027 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Dec 16 21:37:27 smithi045 ceph-mon[80763]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2023-12-16T21:37:28.027 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Dec 16 21:37:27 smithi045 ceph-mon[80763]: fsmap cephfs:1 {0=cephfs.smithi045.fgsbgq=up:active} 1 up:standby-replay 2 up:standby 2023-12-16T21:37:28.027 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Dec 16 21:37:27 smithi045 ceph-mon[80763]: from='mgr.14214 172.21.15.45:0/1227909437' entity='mgr.smithi045.nhtmwc' cmd=[{"prefix": "mds metadata", "who": "cephfs.smithi169.pihlib"}]: dispatch 2023-12-16T21:37:28.027 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Dec 16 21:37:27 smithi045 ceph-mon[80763]: osdmap e43: 6 total, 6 up, 6 in 2023-12-16T21:37:28.027 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Dec 16 21:37:27 smithi045 ceph-mon[80763]: fsmap cephfs:1/1 {0=cephfs.smithi169.mokxju=up:replay} 2 up:standby 2023-12-16T21:37:28.027 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Dec 16 21:37:27 smithi045 ceph-mon[80763]: from='mgr.14214 172.21.15.45:0/1227909437' entity='mgr.smithi045.nhtmwc' cmd=[{"prefix": "mds metadata", "who": "cephfs.smithi045.fgsbgq"}]: dispatch 2023-12-16T21:37:28.133 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Dec 16 21:37:27 smithi169 ceph-mon[88350]: pgmap v119: 65 pgs: 65 active+clean; 579 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 148 B/s rd, 1.9 KiB/s wr, 6 op/s 2023-12-16T21:37:28.134 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Dec 16 21:37:27 smithi169 ceph-mon[88350]: mds.? [v2:172.21.15.169:6826/4053377419,v1:172.21.15.169:6827/4053377419] up:boot 2023-12-16T21:37:28.134 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Dec 16 21:37:27 smithi169 ceph-mon[88350]: mds.? [v2:172.21.15.169:6824/3648207429,v1:172.21.15.169:6825/3648207429] up:standby-replay 2023-12-16T21:37:28.134 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Dec 16 21:37:27 smithi169 ceph-mon[88350]: Dropping low affinity active daemon mds.cephfs.smithi045.fgsbgq in favor of higher affinity standby. 2023-12-16T21:37:28.134 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Dec 16 21:37:27 smithi169 ceph-mon[88350]: Replacing daemon mds.cephfs.smithi045.fgsbgq as rank 0 with standby daemon mds.cephfs.smithi169.mokxju 2023-12-16T21:37:28.134 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Dec 16 21:37:27 smithi169 ceph-mon[88350]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2023-12-16T21:37:28.134 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Dec 16 21:37:27 smithi169 ceph-mon[88350]: fsmap cephfs:1 {0=cephfs.smithi045.fgsbgq=up:active} 1 up:standby-replay 2 up:standby 2023-12-16T21:37:28.135 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Dec 16 21:37:27 smithi169 ceph-mon[88350]: from='mgr.14214 172.21.15.45:0/1227909437' entity='mgr.smithi045.nhtmwc' cmd=[{"prefix": "mds metadata", "who": "cephfs.smithi169.pihlib"}]: dispatch 2023-12-16T21:37:28.135 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Dec 16 21:37:27 smithi169 ceph-mon[88350]: osdmap e43: 6 total, 6 up, 6 in 2023-12-16T21:37:28.135 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Dec 16 21:37:27 smithi169 ceph-mon[88350]: fsmap cephfs:1/1 {0=cephfs.smithi169.mokxju=up:replay} 2 up:standby 2023-12-16T21:37:28.135 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Dec 16 21:37:27 smithi169 ceph-mon[88350]: from='mgr.14214 172.21.15.45:0/1227909437' entity='mgr.smithi045.nhtmwc' cmd=[{"prefix": "mds metadata", "who": "cephfs.smithi045.fgsbgq"}]: dispatch 2023-12-16T21:37:28.187 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:37:28.184+0000 7f0af43d0700 1 -- 172.21.15.45:0/3294380997 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "inline_data", "val": "true", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7f0aec0fc3b0 con 0x7f0aec100060 2023-12-16T21:37:28.312 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:37:28.309+0000 7f0ae2ffd700 1 -- 172.21.15.45:0/3294380997 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "inline_data", "val": "true", "yes_i_really_really_mean_it": true}]=0 inline data enabled v12) v1 ==== 168+0+0 (secure 0 0 0) 0x7f0adc056920 con 0x7f0aec100060 2023-12-16T21:37:28.314 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:37:28.311+0000 7f0af43d0700 1 -- 172.21.15.45:0/3294380997 >> [v2:172.21.15.45:6800/2012105536,v1:172.21.15.45:6801/2012105536] conn(0x7f0ad8068f60 msgr2=0x7f0ad806b410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:37:28.314 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:37:28.311+0000 7f0af43d0700 1 --2- 172.21.15.45:0/3294380997 >> [v2:172.21.15.45:6800/2012105536,v1:172.21.15.45:6801/2012105536] conn(0x7f0ad8068f60 0x7f0ad806b410 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7f0ae8013680 tx=0x7f0ae8009e30 comp rx=0 tx=0).stop 2023-12-16T21:37:28.314 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:37:28.311+0000 7f0af43d0700 1 -- 172.21.15.45:0/3294380997 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f0aec100060 msgr2=0x7f0aec0fb650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:37:28.314 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:37:28.311+0000 7f0af43d0700 1 --2- 172.21.15.45:0/3294380997 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f0aec100060 0x7f0aec0fb650 secure :-1 s=READY pgs=255 cs=0 l=1 rev1=1 crypto rx=0x7f0adc00eb40 tx=0x7f0adc00b040 comp rx=0 tx=0).stop 2023-12-16T21:37:28.314 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:37:28.312+0000 7f0af43d0700 1 -- 172.21.15.45:0/3294380997 shutdown_connections 2023-12-16T21:37:28.314 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:37:28.312+0000 7f0af43d0700 1 --2- 172.21.15.45:0/3294380997 >> [v2:172.21.15.45:6800/2012105536,v1:172.21.15.45:6801/2012105536] conn(0x7f0ad8068f60 0x7f0ad806b410 unknown :-1 s=CLOSED pgs=97 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:37:28.314 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:37:28.312+0000 7f0af43d0700 1 --2- 172.21.15.45:0/3294380997 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f0aec0fd740 0x7f0aec0fd1d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:37:28.315 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:37:28.312+0000 7f0af43d0700 1 --2- 172.21.15.45:0/3294380997 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f0aec100060 0x7f0aec0fb650 unknown :-1 s=CLOSED pgs=255 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:37:28.315 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:37:28.312+0000 7f0af43d0700 1 -- 172.21.15.45:0/3294380997 >> 172.21.15.45:0/3294380997 conn(0x7f0aec0f71c0 msgr2=0x7f0aec0f95f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-16T21:37:28.315 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:37:28.312+0000 7f0af43d0700 1 -- 172.21.15.45:0/3294380997 shutdown_connections 2023-12-16T21:37:28.315 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:37:28.312+0000 7f0af43d0700 1 -- 172.21.15.45:0/3294380997 wait complete. 2023-12-16T21:37:28.315 INFO:teuthology.orchestra.run.smithi045.stderr:inline data enabled 2023-12-16T21:37:28.773 INFO:teuthology.run_tasks:Running task cephadm.shell... 2023-12-16T21:37:28.784 INFO:tasks.cephadm:Running commands on role host.a host ubuntu@smithi045.front.sepia.ceph.com 2023-12-16T21:37:28.784 DEBUG:teuthology.orchestra.run.smithi045:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 88ff363c-9c5a-11ee-95a5-87774f69a715 -- bash -c 'ceph fs dump' 2023-12-16T21:37:29.066 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Dec 16 21:37:28 smithi045 ceph-mon[80763]: from='mgr.14214 172.21.15.45:0/1227909437' entity='mgr.smithi045.nhtmwc' 2023-12-16T21:37:29.066 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Dec 16 21:37:28 smithi045 ceph-mon[80763]: from='mgr.14214 172.21.15.45:0/1227909437' entity='mgr.smithi045.nhtmwc' 2023-12-16T21:37:29.066 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Dec 16 21:37:28 smithi045 ceph-mon[80763]: from='client.? 172.21.15.45:0/3294380997' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "inline_data", "val": "true", "yes_i_really_really_mean_it": true}]: dispatch 2023-12-16T21:37:29.066 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Dec 16 21:37:28 smithi045 ceph-mon[80763]: Health check failed: 1 filesystem with deprecated feature inline_data (FS_INLINE_DATA_DEPRECATED) 2023-12-16T21:37:29.066 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Dec 16 21:37:28 smithi045 ceph-mon[80763]: mds.? [v2:172.21.15.45:6826/2002585400,v1:172.21.15.45:6827/2002585400] up:boot 2023-12-16T21:37:29.067 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Dec 16 21:37:28 smithi045 ceph-mon[80763]: from='client.? 172.21.15.45:0/3294380997' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "inline_data", "val": "true", "yes_i_really_really_mean_it": true}]': finished 2023-12-16T21:37:29.067 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Dec 16 21:37:28 smithi045 ceph-mon[80763]: fsmap cephfs:1/1 {0=cephfs.smithi169.mokxju=up:replay} 3 up:standby 2023-12-16T21:37:29.067 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Dec 16 21:37:28 smithi045 ceph-mon[80763]: from='mgr.14214 172.21.15.45:0/1227909437' entity='mgr.smithi045.nhtmwc' cmd=[{"prefix": "mds metadata", "who": "cephfs.smithi045.fgsbgq"}]: dispatch 2023-12-16T21:37:29.067 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Dec 16 21:37:28 smithi045 ceph-mon[80763]: pgmap v121: 65 pgs: 65 active+clean; 579 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 1.3 KiB/s rd, 1.6 KiB/s wr, 7 op/s 2023-12-16T21:37:29.067 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Dec 16 21:37:28 smithi045 ceph-mon[80763]: from='mgr.14214 172.21.15.45:0/1227909437' entity='mgr.smithi045.nhtmwc' 2023-12-16T21:37:29.067 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Dec 16 21:37:28 smithi045 ceph-mon[80763]: from='mgr.14214 172.21.15.45:0/1227909437' entity='mgr.smithi045.nhtmwc' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2023-12-16T21:37:29.067 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Dec 16 21:37:28 smithi045 ceph-mon[80763]: from='mgr.14214 172.21.15.45:0/1227909437' entity='mgr.smithi045.nhtmwc' 2023-12-16T21:37:29.102 INFO:teuthology.orchestra.run.smithi045.stderr:Inferring config /var/lib/ceph/88ff363c-9c5a-11ee-95a5-87774f69a715/mon.smithi045/config 2023-12-16T21:37:29.384 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Dec 16 21:37:28 smithi169 ceph-mon[88350]: from='mgr.14214 172.21.15.45:0/1227909437' entity='mgr.smithi045.nhtmwc' 2023-12-16T21:37:29.384 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Dec 16 21:37:28 smithi169 ceph-mon[88350]: from='mgr.14214 172.21.15.45:0/1227909437' entity='mgr.smithi045.nhtmwc' 2023-12-16T21:37:29.384 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Dec 16 21:37:28 smithi169 ceph-mon[88350]: from='client.? 172.21.15.45:0/3294380997' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "inline_data", "val": "true", "yes_i_really_really_mean_it": true}]: dispatch 2023-12-16T21:37:29.384 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Dec 16 21:37:28 smithi169 ceph-mon[88350]: Health check failed: 1 filesystem with deprecated feature inline_data (FS_INLINE_DATA_DEPRECATED) 2023-12-16T21:37:29.384 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Dec 16 21:37:28 smithi169 ceph-mon[88350]: mds.? [v2:172.21.15.45:6826/2002585400,v1:172.21.15.45:6827/2002585400] up:boot 2023-12-16T21:37:29.385 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Dec 16 21:37:28 smithi169 ceph-mon[88350]: from='client.? 172.21.15.45:0/3294380997' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "inline_data", "val": "true", "yes_i_really_really_mean_it": true}]': finished 2023-12-16T21:37:29.385 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Dec 16 21:37:28 smithi169 ceph-mon[88350]: fsmap cephfs:1/1 {0=cephfs.smithi169.mokxju=up:replay} 3 up:standby 2023-12-16T21:37:29.385 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Dec 16 21:37:28 smithi169 ceph-mon[88350]: from='mgr.14214 172.21.15.45:0/1227909437' entity='mgr.smithi045.nhtmwc' cmd=[{"prefix": "mds metadata", "who": "cephfs.smithi045.fgsbgq"}]: dispatch 2023-12-16T21:37:29.385 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Dec 16 21:37:28 smithi169 ceph-mon[88350]: pgmap v121: 65 pgs: 65 active+clean; 579 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 1.3 KiB/s rd, 1.6 KiB/s wr, 7 op/s 2023-12-16T21:37:29.385 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Dec 16 21:37:28 smithi169 ceph-mon[88350]: from='mgr.14214 172.21.15.45:0/1227909437' entity='mgr.smithi045.nhtmwc' 2023-12-16T21:37:29.385 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Dec 16 21:37:28 smithi169 ceph-mon[88350]: from='mgr.14214 172.21.15.45:0/1227909437' entity='mgr.smithi045.nhtmwc' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2023-12-16T21:37:29.385 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Dec 16 21:37:28 smithi169 ceph-mon[88350]: from='mgr.14214 172.21.15.45:0/1227909437' entity='mgr.smithi045.nhtmwc' 2023-12-16T21:37:30.192 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:37:30.188+0000 7fbcb20fe700 1 -- 172.21.15.45:0/3157760512 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fbcac0fcd00 msgr2=0x7fbcac0fd150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:37:30.192 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:37:30.188+0000 7fbcb20fe700 1 --2- 172.21.15.45:0/3157760512 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fbcac0fcd00 0x7fbcac0fd150 secure :-1 s=READY pgs=256 cs=0 l=1 rev1=1 crypto rx=0x7fbca0004940 tx=0x7fbca0009e30 comp rx=0 tx=0).stop 2023-12-16T21:37:30.192 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:37:30.188+0000 7fbcb20fe700 1 -- 172.21.15.45:0/3157760512 shutdown_connections 2023-12-16T21:37:30.192 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:37:30.188+0000 7fbcb20fe700 1 --2- 172.21.15.45:0/3157760512 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fbcac0fcd00 0x7fbcac0fd150 unknown :-1 s=CLOSED pgs=256 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:37:30.192 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:37:30.188+0000 7fbcb20fe700 1 --2- 172.21.15.45:0/3157760512 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fbcac0fbaf0 0x7fbcac0fbf00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:37:30.193 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:37:30.188+0000 7fbcb20fe700 1 -- 172.21.15.45:0/3157760512 >> 172.21.15.45:0/3157760512 conn(0x7fbcac0f7190 msgr2=0x7fbcac0f95e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-16T21:37:30.193 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:37:30.188+0000 7fbcb20fe700 1 -- 172.21.15.45:0/3157760512 shutdown_connections 2023-12-16T21:37:30.193 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:37:30.188+0000 7fbcb20fe700 1 -- 172.21.15.45:0/3157760512 wait complete. 2023-12-16T21:37:30.193 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:37:30.189+0000 7fbcb20fe700 1 Processor -- start 2023-12-16T21:37:30.193 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:37:30.189+0000 7fbcb20fe700 1 -- start start 2023-12-16T21:37:30.193 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:37:30.189+0000 7fbcb20fe700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fbcac0fbaf0 0x7fbcac112690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:37:30.193 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:37:30.189+0000 7fbcb20fe700 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fbcac0fcd00 0x7fbcac114be0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:37:30.193 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:37:30.189+0000 7fbcb20fe700 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbcac066000 con 0x7fbcac0fbaf0 2023-12-16T21:37:30.193 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:37:30.189+0000 7fbcb20fe700 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbcac066140 con 0x7fbcac0fcd00 2023-12-16T21:37:30.193 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:37:30.189+0000 7fbcab7fe700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fbcac0fbaf0 0x7fbcac112690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:37:30.193 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:37:30.189+0000 7fbcab7fe700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fbcac0fbaf0 0x7fbcac112690 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:39762/0 (socket says 172.21.15.45:39762) 2023-12-16T21:37:30.193 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:37:30.189+0000 7fbcab7fe700 1 -- 172.21.15.45:0/3759026800 learned_addr learned my addr 172.21.15.45:0/3759026800 (peer_addr_for_me v2:172.21.15.45:0/0) 2023-12-16T21:37:30.193 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:37:30.189+0000 7fbcaaffd700 1 --2- 172.21.15.45:0/3759026800 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fbcac0fcd00 0x7fbcac114be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:37:30.193 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:37:30.190+0000 7fbcab7fe700 1 -- 172.21.15.45:0/3759026800 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fbcac0fcd00 msgr2=0x7fbcac114be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:37:30.194 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:37:30.190+0000 7fbcab7fe700 1 --2- 172.21.15.45:0/3759026800 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fbcac0fcd00 0x7fbcac114be0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:37:30.194 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:37:30.190+0000 7fbcab7fe700 1 -- 172.21.15.45:0/3759026800 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbca0005040 con 0x7fbcac0fbaf0 2023-12-16T21:37:30.194 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:37:30.190+0000 7fbcab7fe700 1 --2- 172.21.15.45:0/3759026800 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fbcac0fbaf0 0x7fbcac112690 secure :-1 s=READY pgs=257 cs=0 l=1 rev1=1 crypto rx=0x7fbc9c00b790 tx=0x7fbc9c009220 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:37:30.194 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:37:30.190+0000 7fbc93fff700 1 -- 172.21.15.45:0/3759026800 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fbc9c00f6f0 con 0x7fbcac0fbaf0 2023-12-16T21:37:30.195 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:37:30.190+0000 7fbcb20fe700 1 -- 172.21.15.45:0/3759026800 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbcac115120 con 0x7fbcac0fbaf0 2023-12-16T21:37:30.195 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:37:30.190+0000 7fbc93fff700 1 -- 172.21.15.45:0/3759026800 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fbc9c00fd00 con 0x7fbcac0fbaf0 2023-12-16T21:37:30.195 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:37:30.190+0000 7fbcb20fe700 1 -- 172.21.15.45:0/3759026800 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbcac1155b0 con 0x7fbcac0fbaf0 2023-12-16T21:37:30.195 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:37:30.190+0000 7fbc93fff700 1 -- 172.21.15.45:0/3759026800 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fbc9c011e90 con 0x7fbcac0fbaf0 2023-12-16T21:37:30.195 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:37:30.192+0000 7fbcb20fe700 1 -- 172.21.15.45:0/3759026800 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbcac04bb80 con 0x7fbcac0fbaf0 2023-12-16T21:37:30.195 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:37:30.192+0000 7fbc93fff700 1 -- 172.21.15.45:0/3759026800 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 19) v1 ==== 86447+0+0 (secure 0 0 0) 0x7fbc9c00f850 con 0x7fbcac0fbaf0 2023-12-16T21:37:30.195 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:37:30.192+0000 7fbc93fff700 1 --2- 172.21.15.45:0/3759026800 >> [v2:172.21.15.45:6800/2012105536,v1:172.21.15.45:6801/2012105536] conn(0x7fbc94068f60 0x7fbc9406b410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:37:30.195 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:37:30.193+0000 7fbc93fff700 1 -- 172.21.15.45:0/3759026800 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(43..43 src has 1..43) v4 ==== 5550+0+0 (secure 0 0 0) 0x7fbc9c086b80 con 0x7fbcac0fbaf0 2023-12-16T21:37:30.195 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:37:30.193+0000 7fbcaaffd700 1 --2- 172.21.15.45:0/3759026800 >> [v2:172.21.15.45:6800/2012105536,v1:172.21.15.45:6801/2012105536] conn(0x7fbc94068f60 0x7fbc9406b410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:37:30.196 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:37:30.193+0000 7fbcaaffd700 1 --2- 172.21.15.45:0/3759026800 >> [v2:172.21.15.45:6800/2012105536,v1:172.21.15.45:6801/2012105536] conn(0x7fbc94068f60 0x7fbc9406b410 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7fbca000b3b0 tx=0x7fbca0009e30 comp rx=0 tx=0).ready entity=mgr.14214 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:37:30.200 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:37:30.198+0000 7fbc93fff700 1 -- 172.21.15.45:0/3759026800 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+174570 (secure 0 0 0) 0x7fbc9c0530a0 con 0x7fbcac0fbaf0 2023-12-16T21:37:30.476 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:37:30.474+0000 7fbcb20fe700 1 -- 172.21.15.45:0/3759026800 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "fs dump"} v 0) v1 -- 0x7fbcac104530 con 0x7fbcac0fbaf0 2023-12-16T21:37:30.477 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:37:30.474+0000 7fbc93fff700 1 -- 172.21.15.45:0/3759026800 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump"}]=0 dumped fsmap epoch 14 v14) v1 ==== 76+0+1817 (secure 0 0 0) 0x7fbc9c0566c0 con 0x7fbcac0fbaf0 2023-12-16T21:37:30.477 INFO:teuthology.orchestra.run.smithi045.stdout:e14 2023-12-16T21:37:30.477 INFO:teuthology.orchestra.run.smithi045.stdout:enable_multiple, ever_enabled_multiple: 1,1 2023-12-16T21:37:30.477 INFO:teuthology.orchestra.run.smithi045.stdout:default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2023-12-16T21:37:30.477 INFO:teuthology.orchestra.run.smithi045.stdout:legacy client fscid: 1 2023-12-16T21:37:30.477 INFO:teuthology.orchestra.run.smithi045.stdout: 2023-12-16T21:37:30.477 INFO:teuthology.orchestra.run.smithi045.stdout:Filesystem 'cephfs' (1) 2023-12-16T21:37:30.477 INFO:teuthology.orchestra.run.smithi045.stdout:fs_name cephfs 2023-12-16T21:37:30.477 INFO:teuthology.orchestra.run.smithi045.stdout:epoch 14 2023-12-16T21:37:30.478 INFO:teuthology.orchestra.run.smithi045.stdout:flags 32 joinable allow_snaps allow_multimds_snaps allow_standby_replay 2023-12-16T21:37:30.478 INFO:teuthology.orchestra.run.smithi045.stdout:created 2023-12-16T21:37:19.691154+0000 2023-12-16T21:37:30.478 INFO:teuthology.orchestra.run.smithi045.stdout:modified 2023-12-16T21:37:30.311750+0000 2023-12-16T21:37:30.478 INFO:teuthology.orchestra.run.smithi045.stdout:tableserver 0 2023-12-16T21:37:30.478 INFO:teuthology.orchestra.run.smithi045.stdout:root 0 2023-12-16T21:37:30.478 INFO:teuthology.orchestra.run.smithi045.stdout:session_timeout 60 2023-12-16T21:37:30.478 INFO:teuthology.orchestra.run.smithi045.stdout:session_autoclose 300 2023-12-16T21:37:30.478 INFO:teuthology.orchestra.run.smithi045.stdout:max_file_size 1099511627776 2023-12-16T21:37:30.478 INFO:teuthology.orchestra.run.smithi045.stdout:required_client_features {} 2023-12-16T21:37:30.478 INFO:teuthology.orchestra.run.smithi045.stdout:last_failure 0 2023-12-16T21:37:30.478 INFO:teuthology.orchestra.run.smithi045.stdout:last_failure_osd_epoch 43 2023-12-16T21:37:30.478 INFO:teuthology.orchestra.run.smithi045.stdout:compat compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,7=mds uses inline data,8=no anchor table,9=file layout v2,10=snaprealm v2} 2023-12-16T21:37:30.478 INFO:teuthology.orchestra.run.smithi045.stdout:max_mds 1 2023-12-16T21:37:30.478 INFO:teuthology.orchestra.run.smithi045.stdout:in 0 2023-12-16T21:37:30.478 INFO:teuthology.orchestra.run.smithi045.stdout:up {0=24285} 2023-12-16T21:37:30.478 INFO:teuthology.orchestra.run.smithi045.stdout:failed 2023-12-16T21:37:30.479 INFO:teuthology.orchestra.run.smithi045.stdout:damaged 2023-12-16T21:37:30.479 INFO:teuthology.orchestra.run.smithi045.stdout:stopped 2023-12-16T21:37:30.479 INFO:teuthology.orchestra.run.smithi045.stdout:data_pools [3] 2023-12-16T21:37:30.479 INFO:teuthology.orchestra.run.smithi045.stdout:metadata_pool 2 2023-12-16T21:37:30.479 INFO:teuthology.orchestra.run.smithi045.stdout:inline_data enabled 2023-12-16T21:37:30.479 INFO:teuthology.orchestra.run.smithi045.stdout:balancer 2023-12-16T21:37:30.479 INFO:teuthology.orchestra.run.smithi045.stdout:standby_count_wanted 1 2023-12-16T21:37:30.479 INFO:teuthology.orchestra.run.smithi045.stdout:[mds.cephfs.smithi169.mokxju{0:24285} state up:rejoin seq 4 join_fscid=1 addr [v2:172.21.15.169:6824/3648207429,v1:172.21.15.169:6825/3648207429] compat {c=[1],r=[1],i=[7ff]}] 2023-12-16T21:37:30.479 INFO:teuthology.orchestra.run.smithi045.stdout: 2023-12-16T21:37:30.479 INFO:teuthology.orchestra.run.smithi045.stdout: 2023-12-16T21:37:30.479 INFO:teuthology.orchestra.run.smithi045.stdout:Standby daemons: 2023-12-16T21:37:30.479 INFO:teuthology.orchestra.run.smithi045.stdout: 2023-12-16T21:37:30.479 INFO:teuthology.orchestra.run.smithi045.stdout:[mds.cephfs.smithi045.hkjgzw{-1:14478} state up:standby seq 2 join_fscid=1 addr [v2:172.21.15.45:6828/622084067,v1:172.21.15.45:6829/622084067] compat {c=[1],r=[1],i=[7ff]}] 2023-12-16T21:37:30.479 INFO:teuthology.orchestra.run.smithi045.stdout:[mds.cephfs.smithi045.fgsbgq{-1:14488} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.45:6826/2002585400,v1:172.21.15.45:6827/2002585400] compat {c=[1],r=[1],i=[7ff]}] 2023-12-16T21:37:30.479 INFO:teuthology.orchestra.run.smithi045.stdout:[mds.cephfs.smithi169.pihlib{-1:24301} state up:standby seq 1 addr [v2:172.21.15.169:6826/4053377419,v1:172.21.15.169:6827/4053377419] compat {c=[1],r=[1],i=[7ff]}] 2023-12-16T21:37:30.480 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:37:30.477+0000 7fbcb20fe700 1 -- 172.21.15.45:0/3759026800 >> [v2:172.21.15.45:6800/2012105536,v1:172.21.15.45:6801/2012105536] conn(0x7fbc94068f60 msgr2=0x7fbc9406b410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:37:30.480 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:37:30.477+0000 7fbcb20fe700 1 --2- 172.21.15.45:0/3759026800 >> [v2:172.21.15.45:6800/2012105536,v1:172.21.15.45:6801/2012105536] conn(0x7fbc94068f60 0x7fbc9406b410 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7fbca000b3b0 tx=0x7fbca0009e30 comp rx=0 tx=0).stop 2023-12-16T21:37:30.480 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:37:30.477+0000 7fbcb20fe700 1 -- 172.21.15.45:0/3759026800 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fbcac0fbaf0 msgr2=0x7fbcac112690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:37:30.480 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:37:30.477+0000 7fbcb20fe700 1 --2- 172.21.15.45:0/3759026800 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fbcac0fbaf0 0x7fbcac112690 secure :-1 s=READY pgs=257 cs=0 l=1 rev1=1 crypto rx=0x7fbc9c00b790 tx=0x7fbc9c009220 comp rx=0 tx=0).stop 2023-12-16T21:37:30.480 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:37:30.477+0000 7fbcb20fe700 1 -- 172.21.15.45:0/3759026800 shutdown_connections 2023-12-16T21:37:30.480 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:37:30.477+0000 7fbcb20fe700 1 --2- 172.21.15.45:0/3759026800 >> [v2:172.21.15.45:6800/2012105536,v1:172.21.15.45:6801/2012105536] conn(0x7fbc94068f60 0x7fbc9406b410 unknown :-1 s=CLOSED pgs=99 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:37:30.480 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:37:30.477+0000 7fbcb20fe700 1 --2- 172.21.15.45:0/3759026800 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fbcac0fbaf0 0x7fbcac112690 unknown :-1 s=CLOSED pgs=257 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:37:30.480 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:37:30.477+0000 7fbcb20fe700 1 --2- 172.21.15.45:0/3759026800 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fbcac0fcd00 0x7fbcac114be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:37:30.481 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:37:30.477+0000 7fbcb20fe700 1 -- 172.21.15.45:0/3759026800 >> 172.21.15.45:0/3759026800 conn(0x7fbcac0f7190 msgr2=0x7fbcac0fff30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-16T21:37:30.481 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:37:30.477+0000 7fbcb20fe700 1 -- 172.21.15.45:0/3759026800 shutdown_connections 2023-12-16T21:37:30.481 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:37:30.477+0000 7fbcb20fe700 1 -- 172.21.15.45:0/3759026800 wait complete. 2023-12-16T21:37:30.481 INFO:teuthology.orchestra.run.smithi045.stderr:dumped fsmap epoch 14 2023-12-16T21:37:30.566 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Dec 16 21:37:30 smithi045 ceph-mon[80763]: mds.? [v2:172.21.15.169:6824/3648207429,v1:172.21.15.169:6825/3648207429] up:reconnect 2023-12-16T21:37:30.566 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Dec 16 21:37:30 smithi045 ceph-mon[80763]: mds.? [v2:172.21.15.45:6828/622084067,v1:172.21.15.45:6829/622084067] up:standby 2023-12-16T21:37:30.566 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Dec 16 21:37:30 smithi045 ceph-mon[80763]: fsmap cephfs:1/1 {0=cephfs.smithi169.mokxju=up:reconnect} 3 up:standby 2023-12-16T21:37:30.633 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Dec 16 21:37:30 smithi169 ceph-mon[88350]: mds.? [v2:172.21.15.169:6824/3648207429,v1:172.21.15.169:6825/3648207429] up:reconnect 2023-12-16T21:37:30.633 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Dec 16 21:37:30 smithi169 ceph-mon[88350]: mds.? [v2:172.21.15.45:6828/622084067,v1:172.21.15.45:6829/622084067] up:standby 2023-12-16T21:37:30.633 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Dec 16 21:37:30 smithi169 ceph-mon[88350]: fsmap cephfs:1/1 {0=cephfs.smithi169.mokxju=up:reconnect} 3 up:standby 2023-12-16T21:37:30.730 DEBUG:teuthology.orchestra.run.smithi045:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 88ff363c-9c5a-11ee-95a5-87774f69a715 -- bash -c 'ceph --format=json fs dump | jq -e ".filesystems | length == 1"' 2023-12-16T21:37:31.273 INFO:teuthology.orchestra.run.smithi045.stderr:Inferring config /var/lib/ceph/88ff363c-9c5a-11ee-95a5-87774f69a715/mon.smithi045/config 2023-12-16T21:37:31.591 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Dec 16 21:37:31 smithi045 ceph-mon[80763]: mds.? [v2:172.21.15.169:6824/3648207429,v1:172.21.15.169:6825/3648207429] up:rejoin 2023-12-16T21:37:31.592 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Dec 16 21:37:31 smithi045 ceph-mon[80763]: fsmap cephfs:1/1 {0=cephfs.smithi169.mokxju=up:rejoin} 3 up:standby 2023-12-16T21:37:31.592 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Dec 16 21:37:31 smithi045 ceph-mon[80763]: daemon mds.cephfs.smithi169.mokxju is now active in filesystem cephfs as rank 0 2023-12-16T21:37:31.592 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Dec 16 21:37:31 smithi045 ceph-mon[80763]: from='mgr.14214 172.21.15.45:0/1227909437' entity='mgr.smithi045.nhtmwc' 2023-12-16T21:37:31.592 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Dec 16 21:37:31 smithi045 ceph-mon[80763]: from='mgr.14214 172.21.15.45:0/1227909437' entity='mgr.smithi045.nhtmwc' 2023-12-16T21:37:31.592 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Dec 16 21:37:31 smithi045 ceph-mon[80763]: from='mgr.14214 172.21.15.45:0/1227909437' entity='mgr.smithi045.nhtmwc' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-12-16T21:37:31.592 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Dec 16 21:37:31 smithi045 ceph-mon[80763]: from='mgr.14214 172.21.15.45:0/1227909437' entity='mgr.smithi045.nhtmwc' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-12-16T21:37:31.593 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Dec 16 21:37:31 smithi045 ceph-mon[80763]: from='mgr.14214 172.21.15.45:0/1227909437' entity='mgr.smithi045.nhtmwc' 2023-12-16T21:37:31.593 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Dec 16 21:37:31 smithi045 ceph-mon[80763]: from='mgr.14214 172.21.15.45:0/1227909437' entity='mgr.smithi045.nhtmwc' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-12-16T21:37:31.593 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Dec 16 21:37:31 smithi045 ceph-mon[80763]: from='client.? 172.21.15.45:0/3759026800' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2023-12-16T21:37:31.593 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Dec 16 21:37:31 smithi045 ceph-mon[80763]: pgmap v122: 65 pgs: 65 active+clean; 579 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 1.1 KiB/s rd, 1.5 KiB/s wr, 6 op/s 2023-12-16T21:37:31.593 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Dec 16 21:37:31 smithi045 ceph-mon[80763]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2023-12-16T21:37:31.593 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Dec 16 21:37:31 smithi045 ceph-mon[80763]: mds.? [v2:172.21.15.169:6824/3648207429,v1:172.21.15.169:6825/3648207429] up:active 2023-12-16T21:37:31.593 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Dec 16 21:37:31 smithi045 ceph-mon[80763]: mds.? [v2:172.21.15.169:6826/4053377419,v1:172.21.15.169:6827/4053377419] up:standby 2023-12-16T21:37:31.593 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Dec 16 21:37:31 smithi045 ceph-mon[80763]: fsmap cephfs:1 {0=cephfs.smithi169.mokxju=up:active} 1 up:standby-replay 2 up:standby 2023-12-16T21:37:31.633 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Dec 16 21:37:31 smithi169 ceph-mon[88350]: mds.? [v2:172.21.15.169:6824/3648207429,v1:172.21.15.169:6825/3648207429] up:rejoin 2023-12-16T21:37:31.633 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Dec 16 21:37:31 smithi169 ceph-mon[88350]: fsmap cephfs:1/1 {0=cephfs.smithi169.mokxju=up:rejoin} 3 up:standby 2023-12-16T21:37:31.634 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Dec 16 21:37:31 smithi169 ceph-mon[88350]: daemon mds.cephfs.smithi169.mokxju is now active in filesystem cephfs as rank 0 2023-12-16T21:37:31.634 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Dec 16 21:37:31 smithi169 ceph-mon[88350]: from='mgr.14214 172.21.15.45:0/1227909437' entity='mgr.smithi045.nhtmwc' 2023-12-16T21:37:31.634 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Dec 16 21:37:31 smithi169 ceph-mon[88350]: from='mgr.14214 172.21.15.45:0/1227909437' entity='mgr.smithi045.nhtmwc' 2023-12-16T21:37:31.634 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Dec 16 21:37:31 smithi169 ceph-mon[88350]: from='mgr.14214 172.21.15.45:0/1227909437' entity='mgr.smithi045.nhtmwc' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-12-16T21:37:31.634 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Dec 16 21:37:31 smithi169 ceph-mon[88350]: from='mgr.14214 172.21.15.45:0/1227909437' entity='mgr.smithi045.nhtmwc' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-12-16T21:37:31.634 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Dec 16 21:37:31 smithi169 ceph-mon[88350]: from='mgr.14214 172.21.15.45:0/1227909437' entity='mgr.smithi045.nhtmwc' 2023-12-16T21:37:31.634 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Dec 16 21:37:31 smithi169 ceph-mon[88350]: from='mgr.14214 172.21.15.45:0/1227909437' entity='mgr.smithi045.nhtmwc' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-12-16T21:37:31.634 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Dec 16 21:37:31 smithi169 ceph-mon[88350]: from='client.? 172.21.15.45:0/3759026800' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2023-12-16T21:37:31.634 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Dec 16 21:37:31 smithi169 ceph-mon[88350]: pgmap v122: 65 pgs: 65 active+clean; 579 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 1.1 KiB/s rd, 1.5 KiB/s wr, 6 op/s 2023-12-16T21:37:31.635 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Dec 16 21:37:31 smithi169 ceph-mon[88350]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2023-12-16T21:37:31.635 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Dec 16 21:37:31 smithi169 ceph-mon[88350]: mds.? [v2:172.21.15.169:6824/3648207429,v1:172.21.15.169:6825/3648207429] up:active 2023-12-16T21:37:31.635 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Dec 16 21:37:31 smithi169 ceph-mon[88350]: mds.? [v2:172.21.15.169:6826/4053377419,v1:172.21.15.169:6827/4053377419] up:standby 2023-12-16T21:37:31.635 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Dec 16 21:37:31 smithi169 ceph-mon[88350]: fsmap cephfs:1 {0=cephfs.smithi169.mokxju=up:active} 1 up:standby-replay 2 up:standby 2023-12-16T21:37:32.401 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:37:32.397+0000 7f3fe8822700 1 -- 172.21.15.45:0/2539184708 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f3fe4100050 msgr2=0x7f3fe4102430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:37:32.402 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:37:32.397+0000 7f3fe8822700 1 --2- 172.21.15.45:0/2539184708 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f3fe4100050 0x7f3fe4102430 secure :-1 s=READY pgs=258 cs=0 l=1 rev1=1 crypto rx=0x7f3fd8004940 tx=0x7f3fd8009e30 comp rx=0 tx=0).stop 2023-12-16T21:37:32.402 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:37:32.397+0000 7f3fe8822700 1 -- 172.21.15.45:0/2539184708 shutdown_connections 2023-12-16T21:37:32.402 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:37:32.397+0000 7f3fe8822700 1 --2- 172.21.15.45:0/2539184708 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f3fe4100050 0x7f3fe4102430 unknown :-1 s=CLOSED pgs=258 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:37:32.402 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:37:32.397+0000 7f3fe8822700 1 --2- 172.21.15.45:0/2539184708 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f3fe40fd730 0x7f3fe40ffb10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:37:32.402 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:37:32.397+0000 7f3fe8822700 1 -- 172.21.15.45:0/2539184708 >> 172.21.15.45:0/2539184708 conn(0x7f3fe40f71d0 msgr2=0x7f3fe40f95e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-16T21:37:32.402 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:37:32.397+0000 7f3fe8822700 1 -- 172.21.15.45:0/2539184708 shutdown_connections 2023-12-16T21:37:32.402 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:37:32.397+0000 7f3fe8822700 1 -- 172.21.15.45:0/2539184708 wait complete. 2023-12-16T21:37:32.402 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:37:32.398+0000 7f3fe8822700 1 Processor -- start 2023-12-16T21:37:32.402 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:37:32.398+0000 7f3fe8822700 1 -- start start 2023-12-16T21:37:32.402 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:37:32.398+0000 7f3fe8822700 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f3fe40fd730 0x7f3fe410d510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:37:32.402 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:37:32.398+0000 7f3fe8822700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f3fe4100050 0x7f3fe410da50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:37:32.402 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:37:32.398+0000 7f3fe8822700 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3fe410e0e0 con 0x7f3fe4100050 2023-12-16T21:37:32.402 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:37:32.398+0000 7f3fe8822700 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3fe410e920 con 0x7f3fe40fd730 2023-12-16T21:37:32.403 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:37:32.398+0000 7f3fe1d9b700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f3fe4100050 0x7f3fe410da50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:37:32.403 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:37:32.398+0000 7f3fe1d9b700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f3fe4100050 0x7f3fe410da50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:39770/0 (socket says 172.21.15.45:39770) 2023-12-16T21:37:32.403 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:37:32.398+0000 7f3fe1d9b700 1 -- 172.21.15.45:0/288246851 learned_addr learned my addr 172.21.15.45:0/288246851 (peer_addr_for_me v2:172.21.15.45:0/0) 2023-12-16T21:37:32.403 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:37:32.398+0000 7f3fe259c700 1 --2- 172.21.15.45:0/288246851 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f3fe40fd730 0x7f3fe410d510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:37:32.403 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:37:32.399+0000 7f3fe1d9b700 1 -- 172.21.15.45:0/288246851 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f3fe40fd730 msgr2=0x7f3fe410d510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:37:32.403 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:37:32.399+0000 7f3fe1d9b700 1 --2- 172.21.15.45:0/288246851 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f3fe40fd730 0x7f3fe410d510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:37:32.403 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:37:32.399+0000 7f3fe1d9b700 1 -- 172.21.15.45:0/288246851 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3fd8005040 con 0x7f3fe4100050 2023-12-16T21:37:32.403 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:37:32.399+0000 7f3fe1d9b700 1 --2- 172.21.15.45:0/288246851 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f3fe4100050 0x7f3fe410da50 secure :-1 s=READY pgs=259 cs=0 l=1 rev1=1 crypto rx=0x7f3fd8009380 tx=0x7f3fd8009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:37:32.403 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:37:32.399+0000 7f3fd2ffd700 1 -- 172.21.15.45:0/288246851 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f3fd800e040 con 0x7f3fe4100050 2023-12-16T21:37:32.403 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:37:32.399+0000 7f3fd2ffd700 1 -- 172.21.15.45:0/288246851 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f3fd800bd20 con 0x7f3fe4100050 2023-12-16T21:37:32.403 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:37:32.399+0000 7f3fd2ffd700 1 -- 172.21.15.45:0/288246851 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f3fd801c630 con 0x7f3fe4100050 2023-12-16T21:37:32.403 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:37:32.399+0000 7f3fe8822700 1 -- 172.21.15.45:0/288246851 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3fe410eba0 con 0x7f3fe4100050 2023-12-16T21:37:32.403 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:37:32.399+0000 7f3fe8822700 1 -- 172.21.15.45:0/288246851 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3fe410f0e0 con 0x7f3fe4100050 2023-12-16T21:37:32.404 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:37:32.400+0000 7f3fe8822700 1 -- 172.21.15.45:0/288246851 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3fe404bb80 con 0x7f3fe4100050 2023-12-16T21:37:32.404 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:37:32.400+0000 7f3fd2ffd700 1 -- 172.21.15.45:0/288246851 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 19) v1 ==== 86447+0+0 (secure 0 0 0) 0x7f3fd8013420 con 0x7f3fe4100050 2023-12-16T21:37:32.404 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:37:32.401+0000 7f3fd2ffd700 1 --2- 172.21.15.45:0/288246851 >> [v2:172.21.15.45:6800/2012105536,v1:172.21.15.45:6801/2012105536] conn(0x7f3fcc068f60 0x7f3fcc06b410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:37:32.404 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:37:32.401+0000 7f3fd2ffd700 1 -- 172.21.15.45:0/288246851 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(43..43 src has 1..43) v4 ==== 5550+0+0 (secure 0 0 0) 0x7f3fd80880c0 con 0x7f3fe4100050 2023-12-16T21:37:32.404 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:37:32.401+0000 7f3fe259c700 1 --2- 172.21.15.45:0/288246851 >> [v2:172.21.15.45:6800/2012105536,v1:172.21.15.45:6801/2012105536] conn(0x7f3fcc068f60 0x7f3fcc06b410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:37:32.404 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:37:32.401+0000 7f3fe259c700 1 --2- 172.21.15.45:0/288246851 >> [v2:172.21.15.45:6800/2012105536,v1:172.21.15.45:6801/2012105536] conn(0x7f3fcc068f60 0x7f3fcc06b410 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7f3fd4009bf0 tx=0x7f3fd40091a0 comp rx=0 tx=0).ready entity=mgr.14214 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:37:32.408 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:37:32.405+0000 7f3fd2ffd700 1 -- 172.21.15.45:0/288246851 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+174570 (secure 0 0 0) 0x7f3fd8058070 con 0x7f3fe4100050 2023-12-16T21:37:32.664 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:37:32.662+0000 7f3fe8822700 1 -- 172.21.15.45:0/288246851 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f3fe410f5a0 con 0x7f3fe4100050 2023-12-16T21:37:32.665 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:37:32.662+0000 7f3fd2ffd700 1 -- 172.21.15.45:0/288246851 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 15 v15) v1 ==== 94+0+4699 (secure 0 0 0) 0x7f3fd8057c00 con 0x7f3fe4100050 2023-12-16T21:37:32.667 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:37:32.665+0000 7f3fe8822700 1 -- 172.21.15.45:0/288246851 >> [v2:172.21.15.45:6800/2012105536,v1:172.21.15.45:6801/2012105536] conn(0x7f3fcc068f60 msgr2=0x7f3fcc06b410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:37:32.667 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:37:32.665+0000 7f3fe8822700 1 --2- 172.21.15.45:0/288246851 >> [v2:172.21.15.45:6800/2012105536,v1:172.21.15.45:6801/2012105536] conn(0x7f3fcc068f60 0x7f3fcc06b410 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7f3fd4009bf0 tx=0x7f3fd40091a0 comp rx=0 tx=0).stop 2023-12-16T21:37:32.667 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:37:32.665+0000 7f3fe8822700 1 -- 172.21.15.45:0/288246851 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f3fe4100050 msgr2=0x7f3fe410da50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:37:32.667 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:37:32.665+0000 7f3fe8822700 1 --2- 172.21.15.45:0/288246851 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f3fe4100050 0x7f3fe410da50 secure :-1 s=READY pgs=259 cs=0 l=1 rev1=1 crypto rx=0x7f3fd8009380 tx=0x7f3fd8009e30 comp rx=0 tx=0).stop 2023-12-16T21:37:32.667 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:37:32.665+0000 7f3fe8822700 1 -- 172.21.15.45:0/288246851 shutdown_connections 2023-12-16T21:37:32.667 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:37:32.665+0000 7f3fe8822700 1 --2- 172.21.15.45:0/288246851 >> [v2:172.21.15.45:6800/2012105536,v1:172.21.15.45:6801/2012105536] conn(0x7f3fcc068f60 0x7f3fcc06b410 unknown :-1 s=CLOSED pgs=100 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:37:32.668 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:37:32.665+0000 7f3fe8822700 1 --2- 172.21.15.45:0/288246851 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f3fe40fd730 0x7f3fe410d510 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:37:32.668 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:37:32.665+0000 7f3fe8822700 1 --2- 172.21.15.45:0/288246851 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f3fe4100050 0x7f3fe410da50 unknown :-1 s=CLOSED pgs=259 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:37:32.668 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:37:32.665+0000 7f3fe8822700 1 -- 172.21.15.45:0/288246851 >> 172.21.15.45:0/288246851 conn(0x7f3fe40f71d0 msgr2=0x7f3fe40f95b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-16T21:37:32.668 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:37:32.665+0000 7f3fe8822700 1 -- 172.21.15.45:0/288246851 shutdown_connections 2023-12-16T21:37:32.668 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:37:32.665+0000 7f3fe8822700 1 -- 172.21.15.45:0/288246851 wait complete. 2023-12-16T21:37:32.668 INFO:teuthology.orchestra.run.smithi045.stderr:dumped fsmap epoch 15 2023-12-16T21:37:32.678 INFO:teuthology.orchestra.run.smithi045.stdout:true 2023-12-16T21:37:32.996 DEBUG:teuthology.orchestra.run.smithi045:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 88ff363c-9c5a-11ee-95a5-87774f69a715 -- bash -c 'while ! ceph --format=json mds versions | jq -e ". | add == 4"; do sleep 1; done' 2023-12-16T21:37:33.316 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Dec 16 21:37:32 smithi045 ceph-mon[80763]: from='mgr.14214 172.21.15.45:0/1227909437' entity='mgr.smithi045.nhtmwc' 2023-12-16T21:37:33.316 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Dec 16 21:37:32 smithi045 ceph-mon[80763]: from='mgr.14214 172.21.15.45:0/1227909437' entity='mgr.smithi045.nhtmwc' 2023-12-16T21:37:33.316 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Dec 16 21:37:32 smithi045 ceph-mon[80763]: pgmap v123: 65 pgs: 65 active+clean; 579 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 3.6 KiB/s rd, 1.2 KiB/s wr, 6 op/s 2023-12-16T21:37:33.316 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Dec 16 21:37:32 smithi045 ceph-mon[80763]: from='client.? 172.21.15.45:0/288246851' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2023-12-16T21:37:33.319 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Dec 16 21:37:32 smithi169 ceph-mon[88350]: from='mgr.14214 172.21.15.45:0/1227909437' entity='mgr.smithi045.nhtmwc' 2023-12-16T21:37:33.319 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Dec 16 21:37:32 smithi169 ceph-mon[88350]: from='mgr.14214 172.21.15.45:0/1227909437' entity='mgr.smithi045.nhtmwc' 2023-12-16T21:37:33.319 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Dec 16 21:37:32 smithi169 ceph-mon[88350]: pgmap v123: 65 pgs: 65 active+clean; 579 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 3.6 KiB/s rd, 1.2 KiB/s wr, 6 op/s 2023-12-16T21:37:33.319 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Dec 16 21:37:32 smithi169 ceph-mon[88350]: from='client.? 172.21.15.45:0/288246851' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2023-12-16T21:37:33.401 INFO:teuthology.orchestra.run.smithi045.stderr:Inferring config /var/lib/ceph/88ff363c-9c5a-11ee-95a5-87774f69a715/mon.smithi045/config 2023-12-16T21:37:34.176 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:37:34.173+0000 7ffa00721700 1 -- 172.21.15.45:0/1896031482 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff9f80fcd00 msgr2=0x7ff9f80fd150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:37:34.176 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:37:34.173+0000 7ffa00721700 1 --2- 172.21.15.45:0/1896031482 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff9f80fcd00 0x7ff9f80fd150 secure :-1 s=READY pgs=260 cs=0 l=1 rev1=1 crypto rx=0x7ff9ec004940 tx=0x7ff9ec009e30 comp rx=0 tx=0).stop 2023-12-16T21:37:34.176 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:37:34.173+0000 7ffa00721700 1 -- 172.21.15.45:0/1896031482 shutdown_connections 2023-12-16T21:37:34.176 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:37:34.173+0000 7ffa00721700 1 --2- 172.21.15.45:0/1896031482 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff9f80fcd00 0x7ff9f80fd150 unknown :-1 s=CLOSED pgs=260 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:37:34.176 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:37:34.173+0000 7ffa00721700 1 --2- 172.21.15.45:0/1896031482 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7ff9f80fbaf0 0x7ff9f80fbf00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:37:34.176 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:37:34.173+0000 7ffa00721700 1 -- 172.21.15.45:0/1896031482 >> 172.21.15.45:0/1896031482 conn(0x7ff9f80f7190 msgr2=0x7ff9f80f95e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-16T21:37:34.176 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:37:34.173+0000 7ffa00721700 1 -- 172.21.15.45:0/1896031482 shutdown_connections 2023-12-16T21:37:34.177 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:37:34.173+0000 7ffa00721700 1 -- 172.21.15.45:0/1896031482 wait complete. 2023-12-16T21:37:34.177 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:37:34.174+0000 7ffa00721700 1 Processor -- start 2023-12-16T21:37:34.177 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:37:34.174+0000 7ffa00721700 1 -- start start 2023-12-16T21:37:34.178 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:37:34.174+0000 7ffa00721700 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7ff9f80fbaf0 0x7ff9f810f760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:37:34.178 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:37:34.174+0000 7ffa00721700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff9f80fcd00 0x7ff9f810fca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:37:34.178 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:37:34.174+0000 7ffa00721700 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff9f8110330 con 0x7ff9f80fcd00 2023-12-16T21:37:34.178 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:37:34.174+0000 7ffa00721700 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff9f8110b70 con 0x7ff9f80fbaf0 2023-12-16T21:37:34.178 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:37:34.174+0000 7ff9fdcbc700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff9f80fcd00 0x7ff9f810fca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:37:34.178 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:37:34.174+0000 7ff9fdcbc700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff9f80fcd00 0x7ff9f810fca0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:33644/0 (socket says 172.21.15.45:33644) 2023-12-16T21:37:34.179 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:37:34.174+0000 7ff9fdcbc700 1 -- 172.21.15.45:0/2775586378 learned_addr learned my addr 172.21.15.45:0/2775586378 (peer_addr_for_me v2:172.21.15.45:0/0) 2023-12-16T21:37:34.179 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:37:34.174+0000 7ff9fe4bd700 1 --2- 172.21.15.45:0/2775586378 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7ff9f80fbaf0 0x7ff9f810f760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:37:34.179 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:37:34.174+0000 7ff9fdcbc700 1 -- 172.21.15.45:0/2775586378 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7ff9f80fbaf0 msgr2=0x7ff9f810f760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:37:34.179 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:37:34.174+0000 7ff9fdcbc700 1 --2- 172.21.15.45:0/2775586378 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7ff9f80fbaf0 0x7ff9f810f760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:37:34.179 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:37:34.174+0000 7ff9fdcbc700 1 -- 172.21.15.45:0/2775586378 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff9ec005040 con 0x7ff9f80fcd00 2023-12-16T21:37:34.179 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:37:34.174+0000 7ff9fe4bd700 1 --2- 172.21.15.45:0/2775586378 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7ff9f80fbaf0 0x7ff9f810f760 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-12-16T21:37:34.179 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:37:34.175+0000 7ff9fdcbc700 1 --2- 172.21.15.45:0/2775586378 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff9f80fcd00 0x7ff9f810fca0 secure :-1 s=READY pgs=261 cs=0 l=1 rev1=1 crypto rx=0x7ff9ec009360 tx=0x7ff9ec009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:37:34.179 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:37:34.175+0000 7ff9f6ffd700 1 -- 172.21.15.45:0/2775586378 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7ff9ec00e040 con 0x7ff9f80fcd00 2023-12-16T21:37:34.179 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:37:34.175+0000 7ffa00721700 1 -- 172.21.15.45:0/2775586378 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff9f8110df0 con 0x7ff9f80fcd00 2023-12-16T21:37:34.179 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:37:34.175+0000 7ff9f6ffd700 1 -- 172.21.15.45:0/2775586378 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7ff9ec00bcd0 con 0x7ff9f80fcd00 2023-12-16T21:37:34.179 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:37:34.175+0000 7ffa00721700 1 -- 172.21.15.45:0/2775586378 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff9f8111330 con 0x7ff9f80fcd00 2023-12-16T21:37:34.179 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:37:34.175+0000 7ff9f6ffd700 1 -- 172.21.15.45:0/2775586378 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7ff9ec0255e0 con 0x7ff9f80fcd00 2023-12-16T21:37:34.179 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:37:34.176+0000 7ffa00721700 1 -- 172.21.15.45:0/2775586378 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff9f804bb80 con 0x7ff9f80fcd00 2023-12-16T21:37:34.180 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:37:34.176+0000 7ff9f6ffd700 1 -- 172.21.15.45:0/2775586378 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 19) v1 ==== 86447+0+0 (secure 0 0 0) 0x7ff9ec013960 con 0x7ff9f80fcd00 2023-12-16T21:37:34.180 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:37:34.177+0000 7ff9f6ffd700 1 --2- 172.21.15.45:0/2775586378 >> [v2:172.21.15.45:6800/2012105536,v1:172.21.15.45:6801/2012105536] conn(0x7ff9e0068f60 0x7ff9e006b410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:37:34.180 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:37:34.177+0000 7ff9f6ffd700 1 -- 172.21.15.45:0/2775586378 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(43..43 src has 1..43) v4 ==== 5550+0+0 (secure 0 0 0) 0x7ff9ec088260 con 0x7ff9f80fcd00 2023-12-16T21:37:34.180 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:37:34.177+0000 7ff9fe4bd700 1 --2- 172.21.15.45:0/2775586378 >> [v2:172.21.15.45:6800/2012105536,v1:172.21.15.45:6801/2012105536] conn(0x7ff9e0068f60 0x7ff9e006b410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:37:34.180 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:37:34.177+0000 7ff9fe4bd700 1 --2- 172.21.15.45:0/2775586378 >> [v2:172.21.15.45:6800/2012105536,v1:172.21.15.45:6801/2012105536] conn(0x7ff9e0068f60 0x7ff9e006b410 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7ff9f0006fd0 tx=0x7ff9f0008040 comp rx=0 tx=0).ready entity=mgr.14214 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:37:34.183 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:37:34.181+0000 7ff9f6ffd700 1 -- 172.21.15.45:0/2775586378 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+174570 (secure 0 0 0) 0x7ff9ec058220 con 0x7ff9f80fcd00 2023-12-16T21:37:34.464 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:37:34.462+0000 7ffa00721700 1 -- 172.21.15.45:0/2775586378 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "mds versions", "format": "json"} v 0) v1 -- 0x7ff9f8104530 con 0x7ff9f80fcd00 2023-12-16T21:37:34.465 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:37:34.462+0000 7ff9f6ffd700 1 -- 172.21.15.45:0/2775586378 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mon_command_ack([{"prefix": "mds versions", "format": "json"}]=0 v15) v1 ==== 78+0+99 (secure 0 0 0) 0x7ff9ec057db0 con 0x7ff9f80fcd00 2023-12-16T21:37:34.467 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:37:34.464+0000 7ffa00721700 1 -- 172.21.15.45:0/2775586378 >> [v2:172.21.15.45:6800/2012105536,v1:172.21.15.45:6801/2012105536] conn(0x7ff9e0068f60 msgr2=0x7ff9e006b410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:37:34.467 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:37:34.464+0000 7ffa00721700 1 --2- 172.21.15.45:0/2775586378 >> [v2:172.21.15.45:6800/2012105536,v1:172.21.15.45:6801/2012105536] conn(0x7ff9e0068f60 0x7ff9e006b410 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7ff9f0006fd0 tx=0x7ff9f0008040 comp rx=0 tx=0).stop 2023-12-16T21:37:34.467 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:37:34.464+0000 7ffa00721700 1 -- 172.21.15.45:0/2775586378 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff9f80fcd00 msgr2=0x7ff9f810fca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:37:34.467 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:37:34.464+0000 7ffa00721700 1 --2- 172.21.15.45:0/2775586378 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff9f80fcd00 0x7ff9f810fca0 secure :-1 s=READY pgs=261 cs=0 l=1 rev1=1 crypto rx=0x7ff9ec009360 tx=0x7ff9ec009e30 comp rx=0 tx=0).stop 2023-12-16T21:37:34.467 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:37:34.464+0000 7ffa00721700 1 -- 172.21.15.45:0/2775586378 shutdown_connections 2023-12-16T21:37:34.467 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:37:34.464+0000 7ffa00721700 1 --2- 172.21.15.45:0/2775586378 >> [v2:172.21.15.45:6800/2012105536,v1:172.21.15.45:6801/2012105536] conn(0x7ff9e0068f60 0x7ff9e006b410 unknown :-1 s=CLOSED pgs=101 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:37:34.467 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:37:34.464+0000 7ffa00721700 1 --2- 172.21.15.45:0/2775586378 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7ff9f80fbaf0 0x7ff9f810f760 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:37:34.467 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:37:34.464+0000 7ffa00721700 1 --2- 172.21.15.45:0/2775586378 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff9f80fcd00 0x7ff9f810fca0 unknown :-1 s=CLOSED pgs=261 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:37:34.468 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:37:34.464+0000 7ffa00721700 1 -- 172.21.15.45:0/2775586378 >> 172.21.15.45:0/2775586378 conn(0x7ff9f80f7190 msgr2=0x7ff9f8065370 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-16T21:37:34.468 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:37:34.465+0000 7ffa00721700 1 -- 172.21.15.45:0/2775586378 shutdown_connections 2023-12-16T21:37:34.468 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:37:34.465+0000 7ffa00721700 1 -- 172.21.15.45:0/2775586378 wait complete. 2023-12-16T21:37:34.477 INFO:teuthology.orchestra.run.smithi045.stdout:true 2023-12-16T21:37:34.715 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Dec 16 21:37:34 smithi045 ceph-mon[80763]: from='mgr.14214 172.21.15.45:0/1227909437' entity='mgr.smithi045.nhtmwc' 2023-12-16T21:37:34.715 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Dec 16 21:37:34 smithi045 ceph-mon[80763]: from='mgr.14214 172.21.15.45:0/1227909437' entity='mgr.smithi045.nhtmwc' 2023-12-16T21:37:34.716 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Dec 16 21:37:34 smithi045 ceph-mon[80763]: from='mgr.14214 172.21.15.45:0/1227909437' entity='mgr.smithi045.nhtmwc' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-12-16T21:37:34.716 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Dec 16 21:37:34 smithi045 ceph-mon[80763]: from='mgr.14214 172.21.15.45:0/1227909437' entity='mgr.smithi045.nhtmwc' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-12-16T21:37:34.716 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Dec 16 21:37:34 smithi045 ceph-mon[80763]: from='mgr.14214 172.21.15.45:0/1227909437' entity='mgr.smithi045.nhtmwc' 2023-12-16T21:37:34.716 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Dec 16 21:37:34 smithi045 ceph-mon[80763]: from='mgr.14214 172.21.15.45:0/1227909437' entity='mgr.smithi045.nhtmwc' 2023-12-16T21:37:34.716 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Dec 16 21:37:34 smithi045 ceph-mon[80763]: from='client.? 172.21.15.45:0/2775586378' entity='client.admin' cmd=[{"prefix": "mds versions", "format": "json"}]: dispatch 2023-12-16T21:37:34.718 INFO:teuthology.run_tasks:Running task fs.pre_upgrade_save... 2023-12-16T21:37:34.735 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-12-16T21:37:34.847 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:37:34.843+0000 7f8a51618640 1 -- 172.21.15.45:0/2731760931 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f8a4c0ff070 msgr2=0x7f8a4c101450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:37:34.847 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:37:34.843+0000 7f8a51618640 1 --2- 172.21.15.45:0/2731760931 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f8a4c0ff070 0x7f8a4c101450 secure :-1 s=READY pgs=262 cs=0 l=1 rev1=1 crypto rx=0x7f8a400048d0 tx=0x7f8a40009da0 comp rx=0 tx=0).stop 2023-12-16T21:37:34.847 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:37:34.843+0000 7f8a51618640 1 -- 172.21.15.45:0/2731760931 shutdown_connections 2023-12-16T21:37:34.847 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:37:34.843+0000 7f8a51618640 1 --2- 172.21.15.45:0/2731760931 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f8a4c0ff070 0x7f8a4c101450 unknown :-1 s=CLOSED pgs=262 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:37:34.848 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:37:34.843+0000 7f8a51618640 1 --2- 172.21.15.45:0/2731760931 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f8a4c066080 0x7f8a4c0feb30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:37:34.848 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:37:34.843+0000 7f8a51618640 1 -- 172.21.15.45:0/2731760931 >> 172.21.15.45:0/2731760931 conn(0x7f8a4c0f65c0 msgr2=0x7f8a4c0f8a10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-16T21:37:34.848 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:37:34.843+0000 7f8a51618640 1 -- 172.21.15.45:0/2731760931 shutdown_connections 2023-12-16T21:37:34.848 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:37:34.843+0000 7f8a51618640 1 -- 172.21.15.45:0/2731760931 wait complete. 2023-12-16T21:37:34.848 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:37:34.844+0000 7f8a51618640 1 Processor -- start 2023-12-16T21:37:34.848 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:37:34.844+0000 7f8a51618640 1 -- start start 2023-12-16T21:37:34.848 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:37:34.844+0000 7f8a51618640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f8a4c066080 0x7f8a4c0fc3b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:37:34.848 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:37:34.844+0000 7f8a51618640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f8a4c0ff070 0x7f8a4c0faa30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:37:34.848 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:37:34.844+0000 7f8a51618640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8a4c0faf70 con 0x7f8a4c066080 2023-12-16T21:37:34.848 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:37:34.844+0000 7f8a51618640 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8a4c0fb0e0 con 0x7f8a4c0ff070 2023-12-16T21:37:34.848 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:37:34.844+0000 7f8a4affd640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f8a4c066080 0x7f8a4c0fc3b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:37:34.848 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:37:34.844+0000 7f8a4affd640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f8a4c066080 0x7f8a4c0fc3b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:33668/0 (socket says 172.21.15.45:33668) 2023-12-16T21:37:34.848 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:37:34.844+0000 7f8a4affd640 1 -- 172.21.15.45:0/1920767365 learned_addr learned my addr 172.21.15.45:0/1920767365 (peer_addr_for_me v2:172.21.15.45:0/0) 2023-12-16T21:37:34.848 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:37:34.844+0000 7f8a4a7fc640 1 --2- 172.21.15.45:0/1920767365 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f8a4c0ff070 0x7f8a4c0faa30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:37:34.849 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:37:34.844+0000 7f8a4affd640 1 -- 172.21.15.45:0/1920767365 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f8a4c0ff070 msgr2=0x7f8a4c0faa30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:37:34.849 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:37:34.844+0000 7f8a4affd640 1 --2- 172.21.15.45:0/1920767365 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f8a4c0ff070 0x7f8a4c0faa30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:37:34.849 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:37:34.844+0000 7f8a4affd640 1 -- 172.21.15.45:0/1920767365 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8a40005040 con 0x7f8a4c066080 2023-12-16T21:37:34.849 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:37:34.844+0000 7f8a4affd640 1 --2- 172.21.15.45:0/1920767365 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f8a4c066080 0x7f8a4c0fc3b0 secure :-1 s=READY pgs=263 cs=0 l=1 rev1=1 crypto rx=0x7f8a3c00db00 tx=0x7f8a3c004e80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:37:34.849 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:37:34.845+0000 7f8a337fe640 1 -- 172.21.15.45:0/1920767365 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f8a3c00b910 con 0x7f8a4c066080 2023-12-16T21:37:34.849 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:37:34.845+0000 7f8a337fe640 1 -- 172.21.15.45:0/1920767365 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f8a3c00f040 con 0x7f8a4c066080 2023-12-16T21:37:34.849 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:37:34.845+0000 7f8a51618640 1 -- 172.21.15.45:0/1920767365 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8a4c0fb3c0 con 0x7f8a4c066080 2023-12-16T21:37:34.849 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:37:34.845+0000 7f8a337fe640 1 -- 172.21.15.45:0/1920767365 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f8a3c012390 con 0x7f8a4c066080 2023-12-16T21:37:34.849 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:37:34.845+0000 7f8a51618640 1 -- 172.21.15.45:0/1920767365 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8a4c114860 con 0x7f8a4c066080 2023-12-16T21:37:34.850 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:37:34.846+0000 7f8a51618640 1 -- 172.21.15.45:0/1920767365 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8a4c064130 con 0x7f8a4c066080 2023-12-16T21:37:34.850 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:37:34.846+0000 7f8a337fe640 1 -- 172.21.15.45:0/1920767365 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 19) v1 ==== 86447+0+0 (secure 0 0 0) 0x7f8a3c013430 con 0x7f8a4c066080 2023-12-16T21:37:34.850 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:37:34.847+0000 7f8a337fe640 1 --2- 172.21.15.45:0/1920767365 >> [v2:172.21.15.45:6800/2012105536,v1:172.21.15.45:6801/2012105536] conn(0x7f8a34068ed0 0x7f8a3406b380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:37:34.850 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:37:34.847+0000 7f8a4a7fc640 1 --2- 172.21.15.45:0/1920767365 >> [v2:172.21.15.45:6800/2012105536,v1:172.21.15.45:6801/2012105536] conn(0x7f8a34068ed0 0x7f8a3406b380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:37:34.850 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:37:34.847+0000 7f8a337fe640 1 -- 172.21.15.45:0/1920767365 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(43..43 src has 1..43) v4 ==== 5550+0+0 (secure 0 0 0) 0x7f8a3c087340 con 0x7f8a4c066080 2023-12-16T21:37:34.850 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:37:34.847+0000 7f8a4a7fc640 1 --2- 172.21.15.45:0/1920767365 >> [v2:172.21.15.45:6800/2012105536,v1:172.21.15.45:6801/2012105536] conn(0x7f8a34068ed0 0x7f8a3406b380 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7f8a4002bda0 tx=0x7f8a40009da0 comp rx=0 tx=0).ready entity=mgr.14214 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:37:34.853 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:37:34.851+0000 7f8a337fe640 1 -- 172.21.15.45:0/1920767365 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+174570 (secure 0 0 0) 0x7f8a3c053740 con 0x7f8a4c066080 2023-12-16T21:37:34.883 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Dec 16 21:37:34 smithi169 ceph-mon[88350]: from='mgr.14214 172.21.15.45:0/1227909437' entity='mgr.smithi045.nhtmwc' 2023-12-16T21:37:34.883 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Dec 16 21:37:34 smithi169 ceph-mon[88350]: from='mgr.14214 172.21.15.45:0/1227909437' entity='mgr.smithi045.nhtmwc' 2023-12-16T21:37:34.883 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Dec 16 21:37:34 smithi169 ceph-mon[88350]: from='mgr.14214 172.21.15.45:0/1227909437' entity='mgr.smithi045.nhtmwc' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-12-16T21:37:34.883 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Dec 16 21:37:34 smithi169 ceph-mon[88350]: from='mgr.14214 172.21.15.45:0/1227909437' entity='mgr.smithi045.nhtmwc' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-12-16T21:37:34.884 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Dec 16 21:37:34 smithi169 ceph-mon[88350]: from='mgr.14214 172.21.15.45:0/1227909437' entity='mgr.smithi045.nhtmwc' 2023-12-16T21:37:34.884 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Dec 16 21:37:34 smithi169 ceph-mon[88350]: from='mgr.14214 172.21.15.45:0/1227909437' entity='mgr.smithi045.nhtmwc' 2023-12-16T21:37:34.884 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Dec 16 21:37:34 smithi169 ceph-mon[88350]: from='client.? 172.21.15.45:0/2775586378' entity='client.admin' cmd=[{"prefix": "mds versions", "format": "json"}]: dispatch 2023-12-16T21:37:35.026 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:37:35.024+0000 7f8a51618640 1 -- 172.21.15.45:0/1920767365 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f8a4c104930 con 0x7f8a4c066080 2023-12-16T21:37:35.027 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:37:35.025+0000 7f8a337fe640 1 -- 172.21.15.45:0/1920767365 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v43) v1 ==== 74+0+14385 (secure 0 0 0) 0x7f8a3c056d60 con 0x7f8a4c066080 2023-12-16T21:37:35.027 INFO:teuthology.orchestra.run.smithi045.stdout: 2023-12-16T21:37:35.027 INFO:teuthology.orchestra.run.smithi045.stdout:{"epoch":43,"fsid":"88ff363c-9c5a-11ee-95a5-87774f69a715","created":"2023-12-16T21:32:48.502711+0000","modified":"2023-12-16T21:37:27.305049+0000","last_up_change":"2023-12-16T21:36:38.234094+0000","last_in_change":"2023-12-16T21:36:23.065916+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":14,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":6,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-12-16T21:35:53.102992+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}},{"pool":2,"pool_name":"cephfs.cephfs.meta","create_time":"2023-12-16T21:37:18.673419+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"41","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}}},{"pool":3,"pool_name":"cephfs.cephfs.data","create_time":"2023-12-16T21:37:19.119658+0000","flags":131073,"flags_names":"hashpspool,bulk","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"42","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"bc814a3e-c753-4948-b9eb-0d9d2d30904c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":39,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6802","nonce":1443487903},{"type":"v1","addr":"172.21.15.45:6803","nonce":1443487903}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6804","nonce":1443487903},{"type":"v1","addr":"172.21.15.45:6805","nonce":1443487903}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6808","nonce":1443487903},{"type":"v1","addr":"172.21.15.45:6809","nonce":1443487903}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6806","nonce":1443487903},{"type":"v1","addr":"172.21.15.45:6807","nonce":1443487903}]},"public_addr":"172.21.15.45:6803/1443487903","cluster_addr":"172.21.15.45:6805/1443487903","heartbeat_back_addr":"172.21.15.45:6809/1443487903","heartbeat_front_addr":"172.21.15.45:6807/1443487903","state":["exists","up"]},{"osd":1,"uuid":"66f2184c-26d0-492c-ae32-ece7b19ce7bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":39,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6810","nonce":3869108635},{"type":"v1","addr":"172.21.15.45:6811","nonce":3869108635}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6812","nonce":3869108635},{"type":"v1","addr":"172.21.15.45:6813","nonce":3869108635}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6816","nonce":3869108635},{"type":"v1","addr":"172.21.15.45:6817","nonce":3869108635}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6814","nonce":3869108635},{"type":"v1","addr":"172.21.15.45:6815","nonce":3869108635}]},"public_addr":"172.21.15.45:6811/3869108635","cluster_addr":"172.21.15.45:6813/3869108635","heartbeat_back_addr":"172.21.15.45:6817/3869108635","heartbeat_front_addr":"172.21.15.45:6815/3869108635","state":["exists","up"]},{"osd":2,"uuid":"e4c3ac33-08a6-439a-9a83-31488cc35202","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":39,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6818","nonce":844221874},{"type":"v1","addr":"172.21.15.45:6819","nonce":844221874}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6820","nonce":844221874},{"type":"v1","addr":"172.21.15.45:6821","nonce":844221874}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6824","nonce":844221874},{"type":"v1","addr":"172.21.15.45:6825","nonce":844221874}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6822","nonce":844221874},{"type":"v1","addr":"172.21.15.45:6823","nonce":844221874}]},"public_addr":"172.21.15.45:6819/844221874","cluster_addr":"172.21.15.45:6821/844221874","heartbeat_back_addr":"172.21.15.45:6825/844221874","heartbeat_front_addr":"172.21.15.45:6823/844221874","state":["exists","up"]},{"osd":3,"uuid":"957064f1-6dda-475b-903e-130134845a90","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":39,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6800","nonce":201299104},{"type":"v1","addr":"172.21.15.169:6801","nonce":201299104}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6802","nonce":201299104},{"type":"v1","addr":"172.21.15.169:6803","nonce":201299104}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6806","nonce":201299104},{"type":"v1","addr":"172.21.15.169:6807","nonce":201299104}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6804","nonce":201299104},{"type":"v1","addr":"172.21.15.169:6805","nonce":201299104}]},"public_addr":"172.21.15.169:6801/201299104","cluster_addr":"172.21.15.169:6803/201299104","heartbeat_back_addr":"172.21.15.169:6807/201299104","heartbeat_front_addr":"172.21.15.169:6805/201299104","state":["exists","up"]},{"osd":4,"uuid":"0926f0d3-f077-4045-9d5a-e10a94421e9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":32,"up_thru":39,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6808","nonce":3840494264},{"type":"v1","addr":"172.21.15.169:6809","nonce":3840494264}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6810","nonce":3840494264},{"type":"v1","addr":"172.21.15.169:6811","nonce":3840494264}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6814","nonce":3840494264},{"type":"v1","addr":"172.21.15.169:6815","nonce":3840494264}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6812","nonce":3840494264},{"type":"v1","addr":"172.21.15.169:6813","nonce":3840494264}]},"public_addr":"172.21.15.169:6809/3840494264","cluster_addr":"172.21.15.169:6811/3840494264","heartbeat_back_addr":"172.21.15.169:6815/3840494264","heartbeat_front_addr":"172.21.15.169:6813/3840494264","state":["exists","up"]},{"osd":5,"uuid":"363ba5fe-e249-4935-a370-6bc071a2f405","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":36,"up_thru":39,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6816","nonce":3826842549},{"type":"v1","addr":"172.21.15.169:6817","nonce":3826842549}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6818","nonce":3826842549},{"type":"v1","addr":"172.21.15.169:6819","nonce":3826842549}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6822","nonce":3826842549},{"type":"v1","addr":"172.21.15.169:6823","nonce":3826842549}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6820","nonce":3826842549},{"type":"v1","addr":"172.21.15.169:6821","nonce":3826842549}]},"public_addr":"172.21.15.169:6817/3826842549","cluster_addr":"172.21.15.169:6819/3826842549","heartbeat_back_addr":"172.21.15.169:6823/3826842549","heartbeat_front_addr":"172.21.15.169:6821/3826842549","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-16T21:35:15.986469+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-16T21:35:33.469306+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-16T21:35:50.142969+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-16T21:36:05.641289+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-16T21:36:21.464972+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-12-16T21:36:36.899654+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.45:6827/1230939770":"2023-12-17T21:37:27.305029+0000","172.21.15.45:0/3764934134":"2023-12-17T21:33:16.105866+0000","172.21.15.45:6801/2502560408":"2023-12-17T21:33:38.101959+0000","172.21.15.45:6800/1656764601":"2023-12-17T21:33:16.105866+0000","172.21.15.45:6800/2502560408":"2023-12-17T21:33:38.101959+0000","172.21.15.45:0/989600782":"2023-12-17T21:33:38.101959+0000","172.21.15.45:6801/1656764601":"2023-12-17T21:33:16.105866+0000","172.21.15.45:0/3248165816":"2023-12-17T21:34:28.582912+0000","172.21.15.45:0/1828324795":"2023-12-17T21:34:28.582912+0000","172.21.15.45:0/2329702517":"2023-12-17T21:33:16.105866+0000","172.21.15.45:0/1866561551":"2023-12-17T21:33:16.105866+0000","172.21.15.45:0/566984549":"2023-12-17T21:33:38.101959+0000","172.21.15.45:6800/2215417738":"2023-12-17T21:34:28.582912+0000","172.21.15.45:0/3536329610":"2023-12-17T21:33:38.101959+0000","172.21.15.45:6801/2215417738":"2023-12-17T21:34:28.582912+0000","172.21.15.45:0/103340221":"2023-12-17T21:34:28.582912+0000","172.21.15.45:6826/1230939770":"2023-12-17T21:37:27.305029+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-12-16T21:37:35.029 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:37:35.027+0000 7f8a51618640 1 -- 172.21.15.45:0/1920767365 >> [v2:172.21.15.45:6800/2012105536,v1:172.21.15.45:6801/2012105536] conn(0x7f8a34068ed0 msgr2=0x7f8a3406b380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:37:35.029 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:37:35.027+0000 7f8a51618640 1 --2- 172.21.15.45:0/1920767365 >> [v2:172.21.15.45:6800/2012105536,v1:172.21.15.45:6801/2012105536] conn(0x7f8a34068ed0 0x7f8a3406b380 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7f8a4002bda0 tx=0x7f8a40009da0 comp rx=0 tx=0).stop 2023-12-16T21:37:35.029 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:37:35.027+0000 7f8a51618640 1 -- 172.21.15.45:0/1920767365 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f8a4c066080 msgr2=0x7f8a4c0fc3b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:37:35.029 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:37:35.027+0000 7f8a51618640 1 --2- 172.21.15.45:0/1920767365 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f8a4c066080 0x7f8a4c0fc3b0 secure :-1 s=READY pgs=263 cs=0 l=1 rev1=1 crypto rx=0x7f8a3c00db00 tx=0x7f8a3c004e80 comp rx=0 tx=0).stop 2023-12-16T21:37:35.029 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:37:35.027+0000 7f8a51618640 1 -- 172.21.15.45:0/1920767365 shutdown_connections 2023-12-16T21:37:35.029 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:37:35.027+0000 7f8a51618640 1 --2- 172.21.15.45:0/1920767365 >> [v2:172.21.15.45:6800/2012105536,v1:172.21.15.45:6801/2012105536] conn(0x7f8a34068ed0 0x7f8a3406b380 unknown :-1 s=CLOSED pgs=102 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:37:35.030 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:37:35.027+0000 7f8a51618640 1 --2- 172.21.15.45:0/1920767365 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f8a4c0ff070 0x7f8a4c0faa30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:37:35.030 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:37:35.027+0000 7f8a51618640 1 --2- 172.21.15.45:0/1920767365 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f8a4c066080 0x7f8a4c0fc3b0 unknown :-1 s=CLOSED pgs=263 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:37:35.030 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:37:35.027+0000 7f8a51618640 1 -- 172.21.15.45:0/1920767365 >> 172.21.15.45:0/1920767365 conn(0x7f8a4c0f65c0 msgr2=0x7f8a4c0f8a10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-16T21:37:35.030 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:37:35.027+0000 7f8a51618640 1 -- 172.21.15.45:0/1920767365 shutdown_connections 2023-12-16T21:37:35.030 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:37:35.027+0000 7f8a51618640 1 -- 172.21.15.45:0/1920767365 wait complete. 2023-12-16T21:37:35.041 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2023-12-16T21:35:53.102992+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '22', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}}, {'pool': 2, 'pool_name': 'cephfs.cephfs.meta', 'create_time': '2023-12-16T21:37:18.673419+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '41', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}}, {'pool': 3, 'pool_name': 'cephfs.cephfs.data', 'create_time': '2023-12-16T21:37:19.119658+0000', 'flags': 131073, 'flags_names': 'hashpspool,bulk', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '42', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2023-12-16T21:37:35.041 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2023-12-16T21:37:35.149 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:37:35.145+0000 7f1b38d1d640 1 -- 172.21.15.45:0/1845689798 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f1b340fd310 msgr2=0x7f1b340fd780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:37:35.149 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:37:35.145+0000 7f1b38d1d640 1 --2- 172.21.15.45:0/1845689798 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f1b340fd310 0x7f1b340fd780 secure :-1 s=READY pgs=264 cs=0 l=1 rev1=1 crypto rx=0x7f1b1c0048d0 tx=0x7f1b1c009da0 comp rx=0 tx=0).stop 2023-12-16T21:37:35.149 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:37:35.145+0000 7f1b38d1d640 1 -- 172.21.15.45:0/1845689798 shutdown_connections 2023-12-16T21:37:35.149 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:37:35.145+0000 7f1b38d1d640 1 --2- 172.21.15.45:0/1845689798 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f1b340fd310 0x7f1b340fd780 unknown :-1 s=CLOSED pgs=264 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:37:35.149 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:37:35.145+0000 7f1b38d1d640 1 --2- 172.21.15.45:0/1845689798 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f1b340fa260 0x7f1b340fa670 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:37:35.149 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:37:35.145+0000 7f1b38d1d640 1 -- 172.21.15.45:0/1845689798 >> 172.21.15.45:0/1845689798 conn(0x7f1b340f5d40 msgr2=0x7f1b340f8170 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-16T21:37:35.150 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:37:35.145+0000 7f1b38d1d640 1 -- 172.21.15.45:0/1845689798 shutdown_connections 2023-12-16T21:37:35.150 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:37:35.145+0000 7f1b38d1d640 1 -- 172.21.15.45:0/1845689798 wait complete. 2023-12-16T21:37:35.150 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:37:35.145+0000 7f1b38d1d640 1 Processor -- start 2023-12-16T21:37:35.150 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:37:35.145+0000 7f1b38d1d640 1 -- start start 2023-12-16T21:37:35.150 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:37:35.146+0000 7f1b38d1d640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f1b340fa260 0x7f1b3410c7e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:37:35.150 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:37:35.146+0000 7f1b38d1d640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f1b340fd310 0x7f1b3410cd20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:37:35.150 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:37:35.146+0000 7f1b38d1d640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1b3410d370 con 0x7f1b340fa260 2023-12-16T21:37:35.150 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:37:35.146+0000 7f1b38d1d640 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1b34111dc0 con 0x7f1b340fd310 2023-12-16T21:37:35.150 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:37:35.146+0000 7f1b3259c640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f1b340fa260 0x7f1b3410c7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:37:35.150 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:37:35.146+0000 7f1b3259c640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f1b340fa260 0x7f1b3410c7e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:33698/0 (socket says 172.21.15.45:33698) 2023-12-16T21:37:35.150 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:37:35.146+0000 7f1b3259c640 1 -- 172.21.15.45:0/3725347644 learned_addr learned my addr 172.21.15.45:0/3725347644 (peer_addr_for_me v2:172.21.15.45:0/0) 2023-12-16T21:37:35.150 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:37:35.146+0000 7f1b29d9b640 1 --2- 172.21.15.45:0/3725347644 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f1b340fd310 0x7f1b3410cd20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:37:35.150 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:37:35.146+0000 7f1b3259c640 1 -- 172.21.15.45:0/3725347644 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f1b340fd310 msgr2=0x7f1b3410cd20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:37:35.150 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:37:35.146+0000 7f1b3259c640 1 --2- 172.21.15.45:0/3725347644 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f1b340fd310 0x7f1b3410cd20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:37:35.151 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:37:35.146+0000 7f1b3259c640 1 -- 172.21.15.45:0/3725347644 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1b1c005040 con 0x7f1b340fa260 2023-12-16T21:37:35.151 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:37:35.146+0000 7f1b3259c640 1 --2- 172.21.15.45:0/3725347644 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f1b340fa260 0x7f1b3410c7e0 secure :-1 s=READY pgs=265 cs=0 l=1 rev1=1 crypto rx=0x7f1b24002840 tx=0x7f1b24004570 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:37:35.151 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:37:35.146+0000 7f1b2b7fe640 1 -- 172.21.15.45:0/3725347644 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f1b24007c30 con 0x7f1b340fa260 2023-12-16T21:37:35.151 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:37:35.146+0000 7f1b2b7fe640 1 -- 172.21.15.45:0/3725347644 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f1b2400f040 con 0x7f1b340fa260 2023-12-16T21:37:35.151 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:37:35.146+0000 7f1b38d1d640 1 -- 172.21.15.45:0/3725347644 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1b34111f60 con 0x7f1b340fa260 2023-12-16T21:37:35.151 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:37:35.146+0000 7f1b2b7fe640 1 -- 172.21.15.45:0/3725347644 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f1b24009430 con 0x7f1b340fa260 2023-12-16T21:37:35.151 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:37:35.146+0000 7f1b38d1d640 1 -- 172.21.15.45:0/3725347644 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1b34112470 con 0x7f1b340fa260 2023-12-16T21:37:35.152 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:37:35.147+0000 7f1b38d1d640 1 -- 172.21.15.45:0/3725347644 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1b341049c0 con 0x7f1b340fa260 2023-12-16T21:37:35.152 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:37:35.148+0000 7f1b2b7fe640 1 -- 172.21.15.45:0/3725347644 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 19) v1 ==== 86447+0+0 (secure 0 0 0) 0x7f1b240096f0 con 0x7f1b340fa260 2023-12-16T21:37:35.152 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:37:35.148+0000 7f1b2b7fe640 1 --2- 172.21.15.45:0/3725347644 >> [v2:172.21.15.45:6800/2012105536,v1:172.21.15.45:6801/2012105536] conn(0x7f1b20068ed0 0x7f1b2006b380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:37:35.152 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:37:35.148+0000 7f1b2b7fe640 1 -- 172.21.15.45:0/3725347644 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(43..43 src has 1..43) v4 ==== 5550+0+0 (secure 0 0 0) 0x7f1b24086db0 con 0x7f1b340fa260 2023-12-16T21:37:35.152 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:37:35.148+0000 7f1b29d9b640 1 --2- 172.21.15.45:0/3725347644 >> [v2:172.21.15.45:6800/2012105536,v1:172.21.15.45:6801/2012105536] conn(0x7f1b20068ed0 0x7f1b2006b380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:37:35.152 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:37:35.149+0000 7f1b29d9b640 1 --2- 172.21.15.45:0/3725347644 >> [v2:172.21.15.45:6800/2012105536,v1:172.21.15.45:6801/2012105536] conn(0x7f1b20068ed0 0x7f1b2006b380 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7f1b3410dd30 tx=0x7f1b1c009da0 comp rx=0 tx=0).ready entity=mgr.14214 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:37:35.155 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:37:35.153+0000 7f1b2b7fe640 1 -- 172.21.15.45:0/3725347644 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+174570 (secure 0 0 0) 0x7f1b24053260 con 0x7f1b340fa260 2023-12-16T21:37:35.325 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:37:35.322+0000 7f1b38d1d640 1 -- 172.21.15.45:0/3725347644 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f1b34112970 con 0x7f1b340fa260 2023-12-16T21:37:35.325 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:37:35.323+0000 7f1b2b7fe640 1 -- 172.21.15.45:0/3725347644 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v43) v1 ==== 93+0+10 (secure 0 0 0) 0x7f1b24056880 con 0x7f1b340fa260 2023-12-16T21:37:35.325 INFO:teuthology.orchestra.run.smithi045.stdout:pg_num: 1 2023-12-16T21:37:35.327 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:37:35.325+0000 7f1b38d1d640 1 -- 172.21.15.45:0/3725347644 >> [v2:172.21.15.45:6800/2012105536,v1:172.21.15.45:6801/2012105536] conn(0x7f1b20068ed0 msgr2=0x7f1b2006b380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:37:35.327 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:37:35.325+0000 7f1b38d1d640 1 --2- 172.21.15.45:0/3725347644 >> [v2:172.21.15.45:6800/2012105536,v1:172.21.15.45:6801/2012105536] conn(0x7f1b20068ed0 0x7f1b2006b380 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7f1b3410dd30 tx=0x7f1b1c009da0 comp rx=0 tx=0).stop 2023-12-16T21:37:35.327 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:37:35.325+0000 7f1b38d1d640 1 -- 172.21.15.45:0/3725347644 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f1b340fa260 msgr2=0x7f1b3410c7e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:37:35.327 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:37:35.325+0000 7f1b38d1d640 1 --2- 172.21.15.45:0/3725347644 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f1b340fa260 0x7f1b3410c7e0 secure :-1 s=READY pgs=265 cs=0 l=1 rev1=1 crypto rx=0x7f1b24002840 tx=0x7f1b24004570 comp rx=0 tx=0).stop 2023-12-16T21:37:35.328 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:37:35.325+0000 7f1b38d1d640 1 -- 172.21.15.45:0/3725347644 shutdown_connections 2023-12-16T21:37:35.328 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:37:35.325+0000 7f1b38d1d640 1 --2- 172.21.15.45:0/3725347644 >> [v2:172.21.15.45:6800/2012105536,v1:172.21.15.45:6801/2012105536] conn(0x7f1b20068ed0 0x7f1b2006b380 unknown :-1 s=CLOSED pgs=103 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:37:35.328 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:37:35.325+0000 7f1b38d1d640 1 --2- 172.21.15.45:0/3725347644 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f1b340fd310 0x7f1b3410cd20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:37:35.328 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:37:35.325+0000 7f1b38d1d640 1 --2- 172.21.15.45:0/3725347644 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f1b340fa260 0x7f1b3410c7e0 unknown :-1 s=CLOSED pgs=265 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:37:35.328 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:37:35.325+0000 7f1b38d1d640 1 -- 172.21.15.45:0/3725347644 >> 172.21.15.45:0/3725347644 conn(0x7f1b340f5d40 msgr2=0x7f1b34100540 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-16T21:37:35.328 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:37:35.325+0000 7f1b38d1d640 1 -- 172.21.15.45:0/3725347644 shutdown_connections 2023-12-16T21:37:35.328 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:37:35.325+0000 7f1b38d1d640 1 -- 172.21.15.45:0/3725347644 wait complete. 2023-12-16T21:37:35.338 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs.cephfs.meta pg_num 2023-12-16T21:37:35.447 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:37:35.442+0000 7f41e6e14640 1 -- 172.21.15.45:0/1633217511 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f41e00fc080 msgr2=0x7f41e00fc4f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:37:35.447 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:37:35.442+0000 7f41e6e14640 1 --2- 172.21.15.45:0/1633217511 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f41e00fc080 0x7f41e00fc4f0 secure :-1 s=READY pgs=266 cs=0 l=1 rev1=1 crypto rx=0x7f41d0009c50 tx=0x7f41d0004d10 comp rx=0 tx=0).stop 2023-12-16T21:37:35.447 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:37:35.442+0000 7f41e6e14640 1 -- 172.21.15.45:0/1633217511 shutdown_connections 2023-12-16T21:37:35.447 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:37:35.442+0000 7f41e6e14640 1 --2- 172.21.15.45:0/1633217511 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f41e00fc080 0x7f41e00fc4f0 unknown :-1 s=CLOSED pgs=266 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:37:35.447 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:37:35.443+0000 7f41e6e14640 1 --2- 172.21.15.45:0/1633217511 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f41e00faef0 0x7f41e00fb300 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:37:35.447 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:37:35.443+0000 7f41e6e14640 1 -- 172.21.15.45:0/1633217511 >> 172.21.15.45:0/1633217511 conn(0x7f41e00f6610 msgr2=0x7f41e00f8a40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-16T21:37:35.447 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:37:35.443+0000 7f41e6e14640 1 -- 172.21.15.45:0/1633217511 shutdown_connections 2023-12-16T21:37:35.447 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:37:35.443+0000 7f41e6e14640 1 -- 172.21.15.45:0/1633217511 wait complete. 2023-12-16T21:37:35.447 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:37:35.443+0000 7f41e6e14640 1 Processor -- start 2023-12-16T21:37:35.447 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:37:35.443+0000 7f41e6e14640 1 -- start start 2023-12-16T21:37:35.447 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:37:35.443+0000 7f41e6e14640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f41e00faef0 0x7f41e010c7c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:37:35.448 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:37:35.443+0000 7f41e6e14640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f41e00fc080 0x7f41e010cd00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:37:35.448 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:37:35.443+0000 7f41e6e14640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f41e010d350 con 0x7f41e00fc080 2023-12-16T21:37:35.448 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:37:35.443+0000 7f41e6e14640 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f41e0111da0 con 0x7f41e00faef0 2023-12-16T21:37:35.448 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:37:35.444+0000 7f41dffff640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f41e00fc080 0x7f41e010cd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:37:35.448 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:37:35.444+0000 7f41dffff640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f41e00fc080 0x7f41e010cd00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:33716/0 (socket says 172.21.15.45:33716) 2023-12-16T21:37:35.448 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:37:35.444+0000 7f41dffff640 1 -- 172.21.15.45:0/2645797946 learned_addr learned my addr 172.21.15.45:0/2645797946 (peer_addr_for_me v2:172.21.15.45:0/0) 2023-12-16T21:37:35.448 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:37:35.444+0000 7f41e4bb0640 1 --2- 172.21.15.45:0/2645797946 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f41e00faef0 0x7f41e010c7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:37:35.448 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:37:35.444+0000 7f41dffff640 1 -- 172.21.15.45:0/2645797946 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f41e00faef0 msgr2=0x7f41e010c7c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:37:35.448 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:37:35.444+0000 7f41dffff640 1 --2- 172.21.15.45:0/2645797946 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f41e00faef0 0x7f41e010c7c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:37:35.448 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:37:35.444+0000 7f41dffff640 1 -- 172.21.15.45:0/2645797946 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f41d00098b0 con 0x7f41e00fc080 2023-12-16T21:37:35.448 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:37:35.444+0000 7f41dffff640 1 --2- 172.21.15.45:0/2645797946 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f41e00fc080 0x7f41e010cd00 secure :-1 s=READY pgs=267 cs=0 l=1 rev1=1 crypto rx=0x7f41d002c8d0 tx=0x7f41d0004d10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:37:35.448 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:37:35.444+0000 7f41dd7fa640 1 -- 172.21.15.45:0/2645797946 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f41d003a070 con 0x7f41e00fc080 2023-12-16T21:37:35.448 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:37:35.444+0000 7f41dd7fa640 1 -- 172.21.15.45:0/2645797946 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f41d00094c0 con 0x7f41e00fc080 2023-12-16T21:37:35.448 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:37:35.444+0000 7f41e6e14640 1 -- 172.21.15.45:0/2645797946 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f41e0111ee0 con 0x7f41e00fc080 2023-12-16T21:37:35.449 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:37:35.444+0000 7f41dd7fa640 1 -- 172.21.15.45:0/2645797946 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f41d003e650 con 0x7f41e00fc080 2023-12-16T21:37:35.449 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:37:35.444+0000 7f41e6e14640 1 -- 172.21.15.45:0/2645797946 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f41e0112450 con 0x7f41e00fc080 2023-12-16T21:37:35.449 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:37:35.445+0000 7f41e6e14640 1 -- 172.21.15.45:0/2645797946 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f41e0103880 con 0x7f41e00fc080 2023-12-16T21:37:35.450 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:37:35.446+0000 7f41dd7fa640 1 -- 172.21.15.45:0/2645797946 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 19) v1 ==== 86447+0+0 (secure 0 0 0) 0x7f41d0032070 con 0x7f41e00fc080 2023-12-16T21:37:35.450 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:37:35.446+0000 7f41dd7fa640 1 --2- 172.21.15.45:0/2645797946 >> [v2:172.21.15.45:6800/2012105536,v1:172.21.15.45:6801/2012105536] conn(0x7f41cc068ed0 0x7f41cc06b380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:37:35.450 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:37:35.446+0000 7f41dd7fa640 1 -- 172.21.15.45:0/2645797946 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(43..43 src has 1..43) v4 ==== 5550+0+0 (secure 0 0 0) 0x7f41d00a8bf0 con 0x7f41e00fc080 2023-12-16T21:37:35.450 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:37:35.446+0000 7f41e4bb0640 1 --2- 172.21.15.45:0/2645797946 >> [v2:172.21.15.45:6800/2012105536,v1:172.21.15.45:6801/2012105536] conn(0x7f41cc068ed0 0x7f41cc06b380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:37:35.450 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:37:35.447+0000 7f41e4bb0640 1 --2- 172.21.15.45:0/2645797946 >> [v2:172.21.15.45:6800/2012105536,v1:172.21.15.45:6801/2012105536] conn(0x7f41cc068ed0 0x7f41cc06b380 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7f41d8009af0 tx=0x7f41d800b040 comp rx=0 tx=0).ready entity=mgr.14214 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:37:35.453 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:37:35.450+0000 7f41dd7fa640 1 -- 172.21.15.45:0/2645797946 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+174570 (secure 0 0 0) 0x7f41d0074ff0 con 0x7f41e00fc080 2023-12-16T21:37:35.626 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:37:35.623+0000 7f41e6e14640 1 -- 172.21.15.45:0/2645797946 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs.cephfs.meta", "var": "pg_num"} v 0) v1 -- 0x7f41e0112850 con 0x7f41e00fc080 2023-12-16T21:37:35.626 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:37:35.623+0000 7f41dd7fa640 1 -- 172.21.15.45:0/2645797946 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs.cephfs.meta", "var": "pg_num"}]=0 v43) v1 ==== 107+0+11 (secure 0 0 0) 0x7f41d0035e60 con 0x7f41e00fc080 2023-12-16T21:37:35.626 INFO:teuthology.orchestra.run.smithi045.stdout:pg_num: 32 2023-12-16T21:37:35.628 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:37:35.625+0000 7f41e6e14640 1 -- 172.21.15.45:0/2645797946 >> [v2:172.21.15.45:6800/2012105536,v1:172.21.15.45:6801/2012105536] conn(0x7f41cc068ed0 msgr2=0x7f41cc06b380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:37:35.628 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:37:35.625+0000 7f41e6e14640 1 --2- 172.21.15.45:0/2645797946 >> [v2:172.21.15.45:6800/2012105536,v1:172.21.15.45:6801/2012105536] conn(0x7f41cc068ed0 0x7f41cc06b380 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7f41d8009af0 tx=0x7f41d800b040 comp rx=0 tx=0).stop 2023-12-16T21:37:35.628 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:37:35.625+0000 7f41e6e14640 1 -- 172.21.15.45:0/2645797946 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f41e00fc080 msgr2=0x7f41e010cd00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:37:35.628 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:37:35.625+0000 7f41e6e14640 1 --2- 172.21.15.45:0/2645797946 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f41e00fc080 0x7f41e010cd00 secure :-1 s=READY pgs=267 cs=0 l=1 rev1=1 crypto rx=0x7f41d002c8d0 tx=0x7f41d0004d10 comp rx=0 tx=0).stop 2023-12-16T21:37:35.628 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:37:35.626+0000 7f41e6e14640 1 -- 172.21.15.45:0/2645797946 shutdown_connections 2023-12-16T21:37:35.628 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:37:35.626+0000 7f41e6e14640 1 --2- 172.21.15.45:0/2645797946 >> [v2:172.21.15.45:6800/2012105536,v1:172.21.15.45:6801/2012105536] conn(0x7f41cc068ed0 0x7f41cc06b380 unknown :-1 s=CLOSED pgs=104 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:37:35.628 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:37:35.626+0000 7f41e6e14640 1 --2- 172.21.15.45:0/2645797946 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f41e00fc080 0x7f41e010cd00 unknown :-1 s=CLOSED pgs=267 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:37:35.629 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:37:35.626+0000 7f41e6e14640 1 --2- 172.21.15.45:0/2645797946 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f41e00faef0 0x7f41e010c7c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:37:35.629 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:37:35.626+0000 7f41e6e14640 1 -- 172.21.15.45:0/2645797946 >> 172.21.15.45:0/2645797946 conn(0x7f41e00f6610 msgr2=0x7f41e00652e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-16T21:37:35.629 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:37:35.626+0000 7f41e6e14640 1 -- 172.21.15.45:0/2645797946 shutdown_connections 2023-12-16T21:37:35.629 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:37:35.626+0000 7f41e6e14640 1 -- 172.21.15.45:0/2645797946 wait complete. 2023-12-16T21:37:35.637 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Dec 16 21:37:35 smithi045 ceph-mon[80763]: pgmap v124: 65 pgs: 65 active+clean; 579 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 6.4 KiB/s rd, 204 B/s wr, 7 op/s 2023-12-16T21:37:35.637 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Dec 16 21:37:35 smithi045 ceph-mon[80763]: from='client.? 172.21.15.45:0/1920767365' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2023-12-16T21:37:35.638 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Dec 16 21:37:35 smithi045 ceph-mon[80763]: from='client.? 172.21.15.45:0/3725347644' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2023-12-16T21:37:35.639 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs.cephfs.data pg_num 2023-12-16T21:37:35.747 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:37:35.745+0000 7fcb6691d640 1 -- 172.21.15.45:0/2412720258 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fcb600fd200 msgr2=0x7fcb600fd670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:37:35.748 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:37:35.745+0000 7fcb6691d640 1 --2- 172.21.15.45:0/2412720258 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fcb600fd200 0x7fcb600fd670 secure :-1 s=READY pgs=268 cs=0 l=1 rev1=1 crypto rx=0x7fcb540048d0 tx=0x7fcb54009da0 comp rx=0 tx=0).stop 2023-12-16T21:37:35.748 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:37:35.745+0000 7fcb6691d640 1 -- 172.21.15.45:0/2412720258 shutdown_connections 2023-12-16T21:37:35.748 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:37:35.745+0000 7fcb6691d640 1 --2- 172.21.15.45:0/2412720258 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fcb600fd200 0x7fcb600fd670 unknown :-1 s=CLOSED pgs=268 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:37:35.748 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:37:35.745+0000 7fcb6691d640 1 --2- 172.21.15.45:0/2412720258 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fcb600fbf30 0x7fcb600fc340 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:37:35.748 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:37:35.745+0000 7fcb6691d640 1 -- 172.21.15.45:0/2412720258 >> 172.21.15.45:0/2412720258 conn(0x7fcb600f7710 msgr2=0x7fcb600f9b20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-16T21:37:35.748 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:37:35.745+0000 7fcb6691d640 1 -- 172.21.15.45:0/2412720258 shutdown_connections 2023-12-16T21:37:35.748 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:37:35.745+0000 7fcb6691d640 1 -- 172.21.15.45:0/2412720258 wait complete. 2023-12-16T21:37:35.748 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:37:35.745+0000 7fcb6691d640 1 Processor -- start 2023-12-16T21:37:35.748 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:37:35.745+0000 7fcb6691d640 1 -- start start 2023-12-16T21:37:35.750 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:37:35.746+0000 7fcb6691d640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fcb600fbf30 0x7fcb60070e20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:37:35.750 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:37:35.746+0000 7fcb6691d640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fcb600fd200 0x7fcb6006f4a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:37:35.750 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:37:35.746+0000 7fcb6691d640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcb6006f9e0 con 0x7fcb600fbf30 2023-12-16T21:37:35.751 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:37:35.746+0000 7fcb6691d640 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcb6006fb50 con 0x7fcb600fd200 2023-12-16T21:37:35.751 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:37:35.746+0000 7fcb5ffff640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fcb600fbf30 0x7fcb60070e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:37:35.751 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:37:35.746+0000 7fcb5ffff640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fcb600fbf30 0x7fcb60070e20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:33740/0 (socket says 172.21.15.45:33740) 2023-12-16T21:37:35.751 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:37:35.746+0000 7fcb5ffff640 1 -- 172.21.15.45:0/717472542 learned_addr learned my addr 172.21.15.45:0/717472542 (peer_addr_for_me v2:172.21.15.45:0/0) 2023-12-16T21:37:35.751 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:37:35.746+0000 7fcb5f7fe640 1 --2- 172.21.15.45:0/717472542 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fcb600fd200 0x7fcb6006f4a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:37:35.751 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:37:35.746+0000 7fcb5ffff640 1 -- 172.21.15.45:0/717472542 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fcb600fd200 msgr2=0x7fcb6006f4a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:37:35.751 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:37:35.746+0000 7fcb5ffff640 1 --2- 172.21.15.45:0/717472542 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fcb600fd200 0x7fcb6006f4a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:37:35.751 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:37:35.746+0000 7fcb5ffff640 1 -- 172.21.15.45:0/717472542 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcb54005040 con 0x7fcb600fbf30 2023-12-16T21:37:35.751 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:37:35.746+0000 7fcb5ffff640 1 --2- 172.21.15.45:0/717472542 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fcb600fbf30 0x7fcb60070e20 secure :-1 s=READY pgs=269 cs=0 l=1 rev1=1 crypto rx=0x7fcb5000b790 tx=0x7fcb500091d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:37:35.751 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:37:35.746+0000 7fcb5cff9640 1 -- 172.21.15.45:0/717472542 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fcb5000c700 con 0x7fcb600fbf30 2023-12-16T21:37:35.751 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:37:35.746+0000 7fcb5cff9640 1 -- 172.21.15.45:0/717472542 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fcb5000cd10 con 0x7fcb600fbf30 2023-12-16T21:37:35.751 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:37:35.746+0000 7fcb6691d640 1 -- 172.21.15.45:0/717472542 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcb6006fe30 con 0x7fcb600fbf30 2023-12-16T21:37:35.751 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:37:35.746+0000 7fcb5cff9640 1 -- 172.21.15.45:0/717472542 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fcb50013370 con 0x7fcb600fbf30 2023-12-16T21:37:35.751 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:37:35.746+0000 7fcb6691d640 1 -- 172.21.15.45:0/717472542 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcb60070320 con 0x7fcb600fbf30 2023-12-16T21:37:35.751 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:37:35.747+0000 7fcb6691d640 1 -- 172.21.15.45:0/717472542 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcb601049c0 con 0x7fcb600fbf30 2023-12-16T21:37:35.752 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:37:35.748+0000 7fcb5cff9640 1 -- 172.21.15.45:0/717472542 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 19) v1 ==== 86447+0+0 (secure 0 0 0) 0x7fcb50012070 con 0x7fcb600fbf30 2023-12-16T21:37:35.752 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:37:35.748+0000 7fcb5cff9640 1 --2- 172.21.15.45:0/717472542 >> [v2:172.21.15.45:6800/2012105536,v1:172.21.15.45:6801/2012105536] conn(0x7fcb48068ee0 0x7fcb4806b390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:37:35.752 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:37:35.748+0000 7fcb5cff9640 1 -- 172.21.15.45:0/717472542 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(43..43 src has 1..43) v4 ==== 5550+0+0 (secure 0 0 0) 0x7fcb50086870 con 0x7fcb600fbf30 2023-12-16T21:37:35.752 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:37:35.748+0000 7fcb5f7fe640 1 --2- 172.21.15.45:0/717472542 >> [v2:172.21.15.45:6800/2012105536,v1:172.21.15.45:6801/2012105536] conn(0x7fcb48068ee0 0x7fcb4806b390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:37:35.752 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:37:35.748+0000 7fcb5f7fe640 1 --2- 172.21.15.45:0/717472542 >> [v2:172.21.15.45:6800/2012105536,v1:172.21.15.45:6801/2012105536] conn(0x7fcb48068ee0 0x7fcb4806b390 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7fcb54034040 tx=0x7fcb54009da0 comp rx=0 tx=0).ready entity=mgr.14214 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:37:35.754 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:37:35.752+0000 7fcb5cff9640 1 -- 172.21.15.45:0/717472542 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+174570 (secure 0 0 0) 0x7fcb50052c70 con 0x7fcb600fbf30 2023-12-16T21:37:35.883 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Dec 16 21:37:35 smithi169 ceph-mon[88350]: pgmap v124: 65 pgs: 65 active+clean; 579 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 6.4 KiB/s rd, 204 B/s wr, 7 op/s 2023-12-16T21:37:35.883 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Dec 16 21:37:35 smithi169 ceph-mon[88350]: from='client.? 172.21.15.45:0/1920767365' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2023-12-16T21:37:35.883 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Dec 16 21:37:35 smithi169 ceph-mon[88350]: from='client.? 172.21.15.45:0/3725347644' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2023-12-16T21:37:35.926 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:37:35.924+0000 7fcb6691d640 1 -- 172.21.15.45:0/717472542 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs.cephfs.data", "var": "pg_num"} v 0) v1 -- 0x7fcb60114ae0 con 0x7fcb600fbf30 2023-12-16T21:37:35.926 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:37:35.924+0000 7fcb5cff9640 1 -- 172.21.15.45:0/717472542 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs.cephfs.data", "var": "pg_num"}]=0 v43) v1 ==== 107+0+11 (secure 0 0 0) 0x7fcb50056290 con 0x7fcb600fbf30 2023-12-16T21:37:35.927 INFO:teuthology.orchestra.run.smithi045.stdout:pg_num: 32 2023-12-16T21:37:35.928 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:37:35.926+0000 7fcb6691d640 1 -- 172.21.15.45:0/717472542 >> [v2:172.21.15.45:6800/2012105536,v1:172.21.15.45:6801/2012105536] conn(0x7fcb48068ee0 msgr2=0x7fcb4806b390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:37:35.929 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:37:35.926+0000 7fcb6691d640 1 --2- 172.21.15.45:0/717472542 >> [v2:172.21.15.45:6800/2012105536,v1:172.21.15.45:6801/2012105536] conn(0x7fcb48068ee0 0x7fcb4806b390 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7fcb54034040 tx=0x7fcb54009da0 comp rx=0 tx=0).stop 2023-12-16T21:37:35.929 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:37:35.926+0000 7fcb6691d640 1 -- 172.21.15.45:0/717472542 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fcb600fbf30 msgr2=0x7fcb60070e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:37:35.929 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:37:35.926+0000 7fcb6691d640 1 --2- 172.21.15.45:0/717472542 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fcb600fbf30 0x7fcb60070e20 secure :-1 s=READY pgs=269 cs=0 l=1 rev1=1 crypto rx=0x7fcb5000b790 tx=0x7fcb500091d0 comp rx=0 tx=0).stop 2023-12-16T21:37:35.929 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:37:35.926+0000 7fcb6691d640 1 -- 172.21.15.45:0/717472542 shutdown_connections 2023-12-16T21:37:35.929 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:37:35.926+0000 7fcb6691d640 1 --2- 172.21.15.45:0/717472542 >> [v2:172.21.15.45:6800/2012105536,v1:172.21.15.45:6801/2012105536] conn(0x7fcb48068ee0 0x7fcb4806b390 unknown :-1 s=CLOSED pgs=105 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:37:35.929 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:37:35.926+0000 7fcb6691d640 1 --2- 172.21.15.45:0/717472542 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fcb600fd200 0x7fcb6006f4a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:37:35.929 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:37:35.926+0000 7fcb6691d640 1 --2- 172.21.15.45:0/717472542 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fcb600fbf30 0x7fcb60070e20 unknown :-1 s=CLOSED pgs=269 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:37:35.929 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:37:35.926+0000 7fcb6691d640 1 -- 172.21.15.45:0/717472542 >> 172.21.15.45:0/717472542 conn(0x7fcb600f7710 msgr2=0x7fcb60100430 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-16T21:37:35.929 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:37:35.926+0000 7fcb6691d640 1 -- 172.21.15.45:0/717472542 shutdown_connections 2023-12-16T21:37:35.929 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:37:35.926+0000 7fcb6691d640 1 -- 172.21.15.45:0/717472542 wait complete. 2023-12-16T21:37:35.939 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-12-16T21:37:36.045 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:37:36.040+0000 7fd49b91d640 1 -- 172.21.15.45:0/3200189141 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fd4940fa240 msgr2=0x7fd4940fa650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:37:36.045 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:37:36.040+0000 7fd49b91d640 1 --2- 172.21.15.45:0/3200189141 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fd4940fa240 0x7fd4940fa650 secure :-1 s=READY pgs=270 cs=0 l=1 rev1=1 crypto rx=0x7fd48c0048d0 tx=0x7fd48c009da0 comp rx=0 tx=0).stop 2023-12-16T21:37:36.045 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:37:36.040+0000 7fd49b91d640 1 -- 172.21.15.45:0/3200189141 shutdown_connections 2023-12-16T21:37:36.045 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:37:36.040+0000 7fd49b91d640 1 --2- 172.21.15.45:0/3200189141 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fd4940fd2f0 0x7fd4940fd760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:37:36.045 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:37:36.040+0000 7fd49b91d640 1 --2- 172.21.15.45:0/3200189141 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fd4940fa240 0x7fd4940fa650 unknown :-1 s=CLOSED pgs=270 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:37:36.045 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:37:36.041+0000 7fd49b91d640 1 -- 172.21.15.45:0/3200189141 >> 172.21.15.45:0/3200189141 conn(0x7fd4940f5d40 msgr2=0x7fd4940f8150 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-16T21:37:36.045 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:37:36.041+0000 7fd49b91d640 1 -- 172.21.15.45:0/3200189141 shutdown_connections 2023-12-16T21:37:36.045 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:37:36.041+0000 7fd49b91d640 1 -- 172.21.15.45:0/3200189141 wait complete. 2023-12-16T21:37:36.045 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:37:36.041+0000 7fd49b91d640 1 Processor -- start 2023-12-16T21:37:36.045 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:37:36.041+0000 7fd49b91d640 1 -- start start 2023-12-16T21:37:36.046 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:37:36.041+0000 7fd49b91d640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fd4940fa240 0x7fd4940fcb80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:37:36.046 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:37:36.041+0000 7fd49b91d640 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fd4940fd2f0 0x7fd4940fb200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:37:36.046 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:37:36.041+0000 7fd49b91d640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd494110450 con 0x7fd4940fa240 2023-12-16T21:37:36.046 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:37:36.041+0000 7fd49b91d640 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd4941105c0 con 0x7fd4940fd2f0 2023-12-16T21:37:36.046 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:37:36.041+0000 7fd4996b9640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fd4940fa240 0x7fd4940fcb80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:37:36.046 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:37:36.041+0000 7fd4996b9640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fd4940fa240 0x7fd4940fcb80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:33744/0 (socket says 172.21.15.45:33744) 2023-12-16T21:37:36.046 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:37:36.041+0000 7fd4996b9640 1 -- 172.21.15.45:0/3085332034 learned_addr learned my addr 172.21.15.45:0/3085332034 (peer_addr_for_me v2:172.21.15.45:0/0) 2023-12-16T21:37:36.046 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:37:36.041+0000 7fd498eb8640 1 --2- 172.21.15.45:0/3085332034 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fd4940fd2f0 0x7fd4940fb200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:37:36.046 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:37:36.042+0000 7fd4996b9640 1 -- 172.21.15.45:0/3085332034 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fd4940fd2f0 msgr2=0x7fd4940fb200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:37:36.046 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:37:36.042+0000 7fd4996b9640 1 --2- 172.21.15.45:0/3085332034 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fd4940fd2f0 0x7fd4940fb200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:37:36.046 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:37:36.042+0000 7fd4996b9640 1 -- 172.21.15.45:0/3085332034 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd48c005040 con 0x7fd4940fa240 2023-12-16T21:37:36.046 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:37:36.042+0000 7fd4996b9640 1 --2- 172.21.15.45:0/3085332034 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fd4940fa240 0x7fd4940fcb80 secure :-1 s=READY pgs=271 cs=0 l=1 rev1=1 crypto rx=0x7fd48c02bda0 tx=0x7fd48c009da0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:37:36.046 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:37:36.042+0000 7fd491ffb640 1 -- 172.21.15.45:0/3085332034 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fd48c0094c0 con 0x7fd4940fa240 2023-12-16T21:37:36.046 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:37:36.042+0000 7fd49b91d640 1 -- 172.21.15.45:0/3085332034 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd4940fb740 con 0x7fd4940fa240 2023-12-16T21:37:36.047 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:37:36.042+0000 7fd491ffb640 1 -- 172.21.15.45:0/3085332034 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fd48c009620 con 0x7fd4940fa240 2023-12-16T21:37:36.047 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:37:36.042+0000 7fd491ffb640 1 -- 172.21.15.45:0/3085332034 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fd48c03b400 con 0x7fd4940fa240 2023-12-16T21:37:36.047 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:37:36.042+0000 7fd49b91d640 1 -- 172.21.15.45:0/3085332034 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd4940fbce0 con 0x7fd4940fa240 2023-12-16T21:37:36.048 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:37:36.043+0000 7fd49b91d640 1 -- 172.21.15.45:0/3085332034 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd494064130 con 0x7fd4940fa240 2023-12-16T21:37:36.048 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:37:36.043+0000 7fd491ffb640 1 -- 172.21.15.45:0/3085332034 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 19) v1 ==== 86447+0+0 (secure 0 0 0) 0x7fd48c02f070 con 0x7fd4940fa240 2023-12-16T21:37:36.048 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:37:36.044+0000 7fd491ffb640 1 --2- 172.21.15.45:0/3085332034 >> [v2:172.21.15.45:6800/2012105536,v1:172.21.15.45:6801/2012105536] conn(0x7fd47c068ed0 0x7fd47c06b380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:37:36.048 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:37:36.044+0000 7fd491ffb640 1 -- 172.21.15.45:0/3085332034 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(43..43 src has 1..43) v4 ==== 5550+0+0 (secure 0 0 0) 0x7fd48c0a9360 con 0x7fd4940fa240 2023-12-16T21:37:36.048 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:37:36.044+0000 7fd498eb8640 1 --2- 172.21.15.45:0/3085332034 >> [v2:172.21.15.45:6800/2012105536,v1:172.21.15.45:6801/2012105536] conn(0x7fd47c068ed0 0x7fd47c06b380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:37:36.048 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:37:36.044+0000 7fd498eb8640 1 --2- 172.21.15.45:0/3085332034 >> [v2:172.21.15.45:6800/2012105536,v1:172.21.15.45:6801/2012105536] conn(0x7fd47c068ed0 0x7fd47c06b380 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7fd4940fc930 tx=0x7fd48800b040 comp rx=0 tx=0).ready entity=mgr.14214 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:37:36.050 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:37:36.048+0000 7fd491ffb640 1 -- 172.21.15.45:0/3085332034 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+174570 (secure 0 0 0) 0x7fd48c075760 con 0x7fd4940fa240 2023-12-16T21:37:36.257 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:37:36.255+0000 7fd49b91d640 1 -- 172.21.15.45:0/3085332034 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fd4940fbfc0 con 0x7fd4940fa240 2023-12-16T21:37:36.258 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:37:36.256+0000 7fd491ffb640 1 -- 172.21.15.45:0/3085332034 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 15 v15) v1 ==== 94+0+4699 (secure 0 0 0) 0x7fd48c078d80 con 0x7fd4940fa240 2023-12-16T21:37:36.258 INFO:teuthology.orchestra.run.smithi045.stdout: 2023-12-16T21:37:36.258 INFO:teuthology.orchestra.run.smithi045.stdout:{"epoch":15,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14488,"name":"cephfs.smithi045.fgsbgq","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.45:6827/2002585400","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6826","nonce":2002585400},{"type":"v1","addr":"172.21.15.45:6827","nonce":2002585400}]},"join_fscid":1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":12},{"gid":24301,"name":"cephfs.smithi169.pihlib","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.169:6827/4053377419","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6826","nonce":4053377419},{"type":"v1","addr":"172.21.15.169:6827","nonce":4053377419}]},"join_fscid":1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":15}],"filesystems":[{"mdsmap":{"epoch":15,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2023-12-16T21:37:19.691154+0000","modified":"2023-12-16T21:37:31.298350+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":43,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":24285},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14478":{"gid":14478,"name":"cephfs.smithi045.hkjgzw","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":2,"addr":"172.21.15.45:6829/622084067","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6828","nonce":622084067},{"type":"v1","addr":"172.21.15.45:6829","nonce":622084067}]},"join_fscid":1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24285":{"gid":24285,"name":"cephfs.smithi169.mokxju","rank":0,"incarnation":11,"state":"up:active","state_seq":5,"addr":"172.21.15.169:6825/3648207429","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6824","nonce":3648207429},{"type":"v1","addr":"172.21.15.169:6825","nonce":3648207429}]},"join_fscid":1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":1},"id":1}]} 2023-12-16T21:37:36.259 INFO:teuthology.orchestra.run.smithi045.stderr:dumped fsmap epoch 15 2023-12-16T21:37:36.260 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:37:36.257+0000 7fd49b91d640 1 -- 172.21.15.45:0/3085332034 >> [v2:172.21.15.45:6800/2012105536,v1:172.21.15.45:6801/2012105536] conn(0x7fd47c068ed0 msgr2=0x7fd47c06b380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:37:36.260 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:37:36.257+0000 7fd49b91d640 1 --2- 172.21.15.45:0/3085332034 >> [v2:172.21.15.45:6800/2012105536,v1:172.21.15.45:6801/2012105536] conn(0x7fd47c068ed0 0x7fd47c06b380 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7fd4940fc930 tx=0x7fd48800b040 comp rx=0 tx=0).stop 2023-12-16T21:37:36.260 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:37:36.257+0000 7fd49b91d640 1 -- 172.21.15.45:0/3085332034 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fd4940fa240 msgr2=0x7fd4940fcb80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:37:36.260 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:37:36.257+0000 7fd49b91d640 1 --2- 172.21.15.45:0/3085332034 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fd4940fa240 0x7fd4940fcb80 secure :-1 s=READY pgs=271 cs=0 l=1 rev1=1 crypto rx=0x7fd48c02bda0 tx=0x7fd48c009da0 comp rx=0 tx=0).stop 2023-12-16T21:37:36.260 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:37:36.257+0000 7fd49b91d640 1 -- 172.21.15.45:0/3085332034 shutdown_connections 2023-12-16T21:37:36.260 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:37:36.257+0000 7fd49b91d640 1 --2- 172.21.15.45:0/3085332034 >> [v2:172.21.15.45:6800/2012105536,v1:172.21.15.45:6801/2012105536] conn(0x7fd47c068ed0 0x7fd47c06b380 unknown :-1 s=CLOSED pgs=106 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:37:36.260 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:37:36.257+0000 7fd49b91d640 1 --2- 172.21.15.45:0/3085332034 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fd4940fd2f0 0x7fd4940fb200 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:37:36.260 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:37:36.257+0000 7fd49b91d640 1 --2- 172.21.15.45:0/3085332034 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fd4940fa240 0x7fd4940fcb80 unknown :-1 s=CLOSED pgs=271 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:37:36.261 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:37:36.257+0000 7fd49b91d640 1 -- 172.21.15.45:0/3085332034 >> 172.21.15.45:0/3085332034 conn(0x7fd4940f5d40 msgr2=0x7fd494100520 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-16T21:37:36.261 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:37:36.257+0000 7fd49b91d640 1 -- 172.21.15.45:0/3085332034 shutdown_connections 2023-12-16T21:37:36.261 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:37:36.257+0000 7fd49b91d640 1 -- 172.21.15.45:0/3085332034 wait complete. 2023-12-16T21:37:36.270 DEBUG:tasks.fs:fs fscid=1,name=cephfs state = {'epoch': 15, 'max_mds': 1, 'flags': 50} 2023-12-16T21:37:36.271 INFO:teuthology.run_tasks:Running task kclient... 2023-12-16T21:37:36.299 INFO:tasks.kclient:Mounting kernel clients... 2023-12-16T21:37:36.299 INFO:tasks.kclient:config is {'client.0': {}, 'client.1': {}} 2023-12-16T21:37:36.299 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-12-16T21:37:36.299 DEBUG:teuthology.orchestra.run.smithi045:> ip netns list 2023-12-16T21:37:36.315 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-12-16T21:37:36.315 DEBUG:teuthology.orchestra.run.smithi045:> sudo ip link delete ceph-brx 2023-12-16T21:37:36.388 INFO:teuthology.orchestra.run.smithi045.stderr:Cannot find device "ceph-brx" 2023-12-16T21:37:36.389 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-12-16T21:37:36.389 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-12-16T21:37:36.389 DEBUG:teuthology.orchestra.run.smithi169:> ip netns list 2023-12-16T21:37:36.410 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-12-16T21:37:36.410 DEBUG:teuthology.orchestra.run.smithi169:> sudo ip link delete ceph-brx 2023-12-16T21:37:36.487 INFO:teuthology.orchestra.run.smithi169.stderr:Cannot find device "ceph-brx" 2023-12-16T21:37:36.488 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-12-16T21:37:36.489 INFO:tasks.kclient:client.0 config is {'syntax': 'v1'} 2023-12-16T21:37:36.489 INFO:tasks.cephfs.mount:cephfs_mntpt = None 2023-12-16T21:37:36.489 INFO:tasks.cephfs.mount:hostfs_mntpt = /home/ubuntu/cephtest/mnt.0 2023-12-16T21:37:36.489 DEBUG:teuthology.orchestra.run.smithi045:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 88ff363c-9c5a-11ee-95a5-87774f69a715 -- ceph fs ls 2023-12-16T21:37:36.559 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Dec 16 21:37:36 smithi045 ceph-mon[80763]: from='client.? 172.21.15.45:0/2645797946' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs.cephfs.meta", "var": "pg_num"}]: dispatch 2023-12-16T21:37:36.560 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Dec 16 21:37:36 smithi045 ceph-mon[80763]: from='client.? 172.21.15.45:0/717472542' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs.cephfs.data", "var": "pg_num"}]: dispatch 2023-12-16T21:37:36.560 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Dec 16 21:37:36 smithi045 ceph-mon[80763]: from='client.? 172.21.15.45:0/3085332034' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2023-12-16T21:37:36.758 INFO:teuthology.orchestra.run.smithi045.stderr:Inferring config /var/lib/ceph/88ff363c-9c5a-11ee-95a5-87774f69a715/mon.smithi045/config 2023-12-16T21:37:36.883 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Dec 16 21:37:36 smithi169 ceph-mon[88350]: from='client.? 172.21.15.45:0/2645797946' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs.cephfs.meta", "var": "pg_num"}]: dispatch 2023-12-16T21:37:36.884 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Dec 16 21:37:36 smithi169 ceph-mon[88350]: from='client.? 172.21.15.45:0/717472542' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs.cephfs.data", "var": "pg_num"}]: dispatch 2023-12-16T21:37:36.884 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Dec 16 21:37:36 smithi169 ceph-mon[88350]: from='client.? 172.21.15.45:0/3085332034' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2023-12-16T21:37:37.519 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:37:37.514+0000 7f1ba2b60700 1 -- 172.21.15.45:0/2072991233 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f1b9c0fe5b0 msgr2=0x7f1b9c106bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:37:37.519 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:37:37.514+0000 7f1ba2b60700 1 --2- 172.21.15.45:0/2072991233 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f1b9c0fe5b0 0x7f1b9c106bf0 secure :-1 s=READY pgs=272 cs=0 l=1 rev1=1 crypto rx=0x7f1b88004940 tx=0x7f1b88009e30 comp rx=0 tx=0).stop 2023-12-16T21:37:37.519 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:37:37.514+0000 7f1ba2b60700 1 -- 172.21.15.45:0/2072991233 shutdown_connections 2023-12-16T21:37:37.519 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:37:37.514+0000 7f1ba2b60700 1 --2- 172.21.15.45:0/2072991233 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f1b9c0fe5b0 0x7f1b9c106bf0 unknown :-1 s=CLOSED pgs=272 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:37:37.519 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:37:37.514+0000 7f1ba2b60700 1 --2- 172.21.15.45:0/2072991233 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f1b9c0fdc10 0x7f1b9c0fdfe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:37:37.519 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:37:37.514+0000 7f1ba2b60700 1 -- 172.21.15.45:0/2072991233 >> 172.21.15.45:0/2072991233 conn(0x7f1b9c0f76c0 msgr2=0x7f1b9c0f9ad0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-16T21:37:37.519 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:37:37.514+0000 7f1ba2b60700 1 -- 172.21.15.45:0/2072991233 shutdown_connections 2023-12-16T21:37:37.519 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:37:37.514+0000 7f1ba2b60700 1 -- 172.21.15.45:0/2072991233 wait complete. 2023-12-16T21:37:37.519 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:37:37.515+0000 7f1ba2b60700 1 Processor -- start 2023-12-16T21:37:37.519 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:37:37.515+0000 7f1ba2b60700 1 -- start start 2023-12-16T21:37:37.520 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:37:37.515+0000 7f1ba2b60700 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f1b9c0fdc10 0x7f1b9c1136b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:37:37.520 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:37:37.515+0000 7f1ba2b60700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f1b9c0fe5b0 0x7f1b9c113c30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:37:37.520 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:37:37.515+0000 7f1ba2b60700 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1b9c114280 con 0x7f1b9c0fe5b0 2023-12-16T21:37:37.520 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:37:37.515+0000 7f1ba2b60700 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1b9c10d570 con 0x7f1b9c0fdc10 2023-12-16T21:37:37.520 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:37:37.515+0000 7f1b9bfff700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f1b9c0fe5b0 0x7f1b9c113c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:37:37.520 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:37:37.515+0000 7f1b9bfff700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f1b9c0fe5b0 0x7f1b9c113c30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:33766/0 (socket says 172.21.15.45:33766) 2023-12-16T21:37:37.520 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:37:37.515+0000 7f1ba08fc700 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f1b9c0fdc10 0x7f1b9c1136b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:37:37.520 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:37:37.515+0000 7f1b9bfff700 1 -- 172.21.15.45:0/2687851165 learned_addr learned my addr 172.21.15.45:0/2687851165 (peer_addr_for_me v2:172.21.15.45:0/0) 2023-12-16T21:37:37.520 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:37:37.515+0000 7f1b9bfff700 1 -- 172.21.15.45:0/2687851165 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f1b9c0fdc10 msgr2=0x7f1b9c1136b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:37:37.520 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:37:37.515+0000 7f1b9bfff700 1 --2- 172.21.15.45:0/2687851165 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f1b9c0fdc10 0x7f1b9c1136b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:37:37.520 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:37:37.515+0000 7f1b9bfff700 1 -- 172.21.15.45:0/2687851165 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1b88005040 con 0x7f1b9c0fe5b0 2023-12-16T21:37:37.520 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:37:37.516+0000 7f1b9bfff700 1 --2- 172.21.15.45:0/2687851165 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f1b9c0fe5b0 0x7f1b9c113c30 secure :-1 s=READY pgs=273 cs=0 l=1 rev1=1 crypto rx=0x7f1b8800b3b0 tx=0x7f1b88009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:37:37.521 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:37:37.516+0000 7f1b997fa700 1 -- 172.21.15.45:0/2687851165 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f1b8800e040 con 0x7f1b9c0fe5b0 2023-12-16T21:37:37.521 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:37:37.516+0000 7f1b997fa700 1 -- 172.21.15.45:0/2687851165 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f1b88004290 con 0x7f1b9c0fe5b0 2023-12-16T21:37:37.521 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:37:37.516+0000 7f1b997fa700 1 -- 172.21.15.45:0/2687851165 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f1b8801c3f0 con 0x7f1b9c0fe5b0 2023-12-16T21:37:37.521 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:37:37.516+0000 7f1ba2b60700 1 -- 172.21.15.45:0/2687851165 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1b9c10d7a0 con 0x7f1b9c0fe5b0 2023-12-16T21:37:37.521 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:37:37.516+0000 7f1ba2b60700 1 -- 172.21.15.45:0/2687851165 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1b9c10dce0 con 0x7f1b9c0fe5b0 2023-12-16T21:37:37.522 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:37:37.517+0000 7f1ba2b60700 1 -- 172.21.15.45:0/2687851165 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1b9c104200 con 0x7f1b9c0fe5b0 2023-12-16T21:37:37.522 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:37:37.517+0000 7f1b997fa700 1 -- 172.21.15.45:0/2687851165 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 19) v1 ==== 86447+0+0 (secure 0 0 0) 0x7f1b88013820 con 0x7f1b9c0fe5b0 2023-12-16T21:37:37.522 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:37:37.518+0000 7f1b997fa700 1 --2- 172.21.15.45:0/2687851165 >> [v2:172.21.15.45:6800/2012105536,v1:172.21.15.45:6801/2012105536] conn(0x7f1b8c068f60 0x7f1b8c06b410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:37:37.522 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:37:37.518+0000 7f1b997fa700 1 -- 172.21.15.45:0/2687851165 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(43..43 src has 1..43) v4 ==== 5550+0+0 (secure 0 0 0) 0x7f1b880880c0 con 0x7f1b9c0fe5b0 2023-12-16T21:37:37.522 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:37:37.518+0000 7f1ba08fc700 1 --2- 172.21.15.45:0/2687851165 >> [v2:172.21.15.45:6800/2012105536,v1:172.21.15.45:6801/2012105536] conn(0x7f1b8c068f60 0x7f1b8c06b410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:37:37.522 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:37:37.518+0000 7f1ba08fc700 1 --2- 172.21.15.45:0/2687851165 >> [v2:172.21.15.45:6800/2012105536,v1:172.21.15.45:6801/2012105536] conn(0x7f1b8c068f60 0x7f1b8c06b410 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7f1b9c0f9f90 tx=0x7f1b9000b040 comp rx=0 tx=0).ready entity=mgr.14214 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:37:37.525 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:37:37.522+0000 7f1b997fa700 1 -- 172.21.15.45:0/2687851165 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+174570 (secure 0 0 0) 0x7f1b88057fd0 con 0x7f1b9c0fe5b0 2023-12-16T21:37:37.766 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:37:37.764+0000 7f1ba2b60700 1 -- 172.21.15.45:0/2687851165 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "fs ls"} v 0) v1 -- 0x7f1b9c04bbb0 con 0x7f1b9c0fe5b0 2023-12-16T21:37:37.767 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Dec 16 21:37:37 smithi045 ceph-mon[80763]: pgmap v125: 65 pgs: 65 active+clean; 579 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 6.3 KiB/s rd, 0 B/s wr, 6 op/s 2023-12-16T21:37:37.767 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:37:37.764+0000 7f1b997fa700 1 -- 172.21.15.45:0/2687851165 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v15) v1 ==== 53+0+83 (secure 0 0 0) 0x7f1b88057b60 con 0x7f1b9c0fe5b0 2023-12-16T21:37:37.767 INFO:teuthology.orchestra.run.smithi045.stdout:name: cephfs, metadata pool: cephfs.cephfs.meta, data pools: [cephfs.cephfs.data ] 2023-12-16T21:37:37.769 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:37:37.766+0000 7f1ba2b60700 1 -- 172.21.15.45:0/2687851165 >> [v2:172.21.15.45:6800/2012105536,v1:172.21.15.45:6801/2012105536] conn(0x7f1b8c068f60 msgr2=0x7f1b8c06b410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:37:37.769 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:37:37.766+0000 7f1ba2b60700 1 --2- 172.21.15.45:0/2687851165 >> [v2:172.21.15.45:6800/2012105536,v1:172.21.15.45:6801/2012105536] conn(0x7f1b8c068f60 0x7f1b8c06b410 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7f1b9c0f9f90 tx=0x7f1b9000b040 comp rx=0 tx=0).stop 2023-12-16T21:37:37.769 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:37:37.766+0000 7f1ba2b60700 1 -- 172.21.15.45:0/2687851165 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f1b9c0fe5b0 msgr2=0x7f1b9c113c30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:37:37.769 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:37:37.766+0000 7f1ba2b60700 1 --2- 172.21.15.45:0/2687851165 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f1b9c0fe5b0 0x7f1b9c113c30 secure :-1 s=READY pgs=273 cs=0 l=1 rev1=1 crypto rx=0x7f1b8800b3b0 tx=0x7f1b88009e30 comp rx=0 tx=0).stop 2023-12-16T21:37:37.770 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:37:37.766+0000 7f1ba2b60700 1 -- 172.21.15.45:0/2687851165 shutdown_connections 2023-12-16T21:37:37.770 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:37:37.766+0000 7f1ba2b60700 1 --2- 172.21.15.45:0/2687851165 >> [v2:172.21.15.45:6800/2012105536,v1:172.21.15.45:6801/2012105536] conn(0x7f1b8c068f60 0x7f1b8c06b410 unknown :-1 s=CLOSED pgs=107 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:37:37.770 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:37:37.766+0000 7f1ba2b60700 1 --2- 172.21.15.45:0/2687851165 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f1b9c0fdc10 0x7f1b9c1136b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:37:37.770 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:37:37.766+0000 7f1ba2b60700 1 --2- 172.21.15.45:0/2687851165 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f1b9c0fe5b0 0x7f1b9c113c30 unknown :-1 s=CLOSED pgs=273 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:37:37.770 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:37:37.766+0000 7f1ba2b60700 1 -- 172.21.15.45:0/2687851165 >> 172.21.15.45:0/2687851165 conn(0x7f1b9c0f76c0 msgr2=0x7f1b9c101380 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-16T21:37:37.770 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:37:37.766+0000 7f1ba2b60700 1 -- 172.21.15.45:0/2687851165 shutdown_connections 2023-12-16T21:37:37.770 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:37:37.766+0000 7f1ba2b60700 1 -- 172.21.15.45:0/2687851165 wait complete. 2023-12-16T21:37:37.883 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Dec 16 21:37:37 smithi169 ceph-mon[88350]: pgmap v125: 65 pgs: 65 active+clean; 579 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 6.3 KiB/s rd, 0 B/s wr, 6 op/s 2023-12-16T21:37:38.073 INFO:tasks.cephfs.mount:Mounting default Ceph FS; just confirmed its presence on cluster 2023-12-16T21:37:38.074 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2023-12-16T21:37:38.074 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi045.front.sepia.ceph.com 2023-12-16T21:37:38.074 INFO:tasks.cephfs.mount:self.client.name = client.0 2023-12-16T21:37:38.074 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.0 2023-12-16T21:37:38.074 INFO:tasks.cephfs.mount:self.cephfs_name = None 2023-12-16T21:37:38.074 INFO:tasks.cephfs.mount:self.cephfs_mntpt = None 2023-12-16T21:37:38.074 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2023-12-16T21:37:38.074 INFO:tasks.cephfs.mount:Setting the 'None' netns for '/home/ubuntu/cephtest/mnt.0' 2023-12-16T21:37:38.074 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-12-16T21:37:38.074 DEBUG:teuthology.orchestra.run.smithi045:> ip addr 2023-12-16T21:37:38.090 INFO:teuthology.orchestra.run.smithi045.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2023-12-16T21:37:38.090 INFO:teuthology.orchestra.run.smithi045.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2023-12-16T21:37:38.090 INFO:teuthology.orchestra.run.smithi045.stdout: inet 127.0.0.1/8 scope host lo 2023-12-16T21:37:38.090 INFO:teuthology.orchestra.run.smithi045.stdout: valid_lft forever preferred_lft forever 2023-12-16T21:37:38.090 INFO:teuthology.orchestra.run.smithi045.stdout: inet6 ::1/128 scope host 2023-12-16T21:37:38.091 INFO:teuthology.orchestra.run.smithi045.stdout: valid_lft forever preferred_lft forever 2023-12-16T21:37:38.091 INFO:teuthology.orchestra.run.smithi045.stdout:2: eno1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2023-12-16T21:37:38.091 INFO:teuthology.orchestra.run.smithi045.stdout: link/ether 0c:c4:7a:6b:fd:a6 brd ff:ff:ff:ff:ff:ff 2023-12-16T21:37:38.091 INFO:teuthology.orchestra.run.smithi045.stdout: altname enp7s0f0 2023-12-16T21:37:38.091 INFO:teuthology.orchestra.run.smithi045.stdout:3: eno2: mtu 1500 qdisc mq state DOWN group default qlen 1000 2023-12-16T21:37:38.091 INFO:teuthology.orchestra.run.smithi045.stdout: link/ether 0c:c4:7a:6b:fd:a7 brd ff:ff:ff:ff:ff:ff 2023-12-16T21:37:38.091 INFO:teuthology.orchestra.run.smithi045.stdout: altname enp7s0f1 2023-12-16T21:37:38.091 INFO:teuthology.orchestra.run.smithi045.stdout:4: ens1f0: mtu 1500 qdisc mq state UP group default qlen 1000 2023-12-16T21:37:38.091 INFO:teuthology.orchestra.run.smithi045.stdout: link/ether 0c:c4:7a:bd:12:be brd ff:ff:ff:ff:ff:ff 2023-12-16T21:37:38.091 INFO:teuthology.orchestra.run.smithi045.stdout: altname enp1s0f0 2023-12-16T21:37:38.091 INFO:teuthology.orchestra.run.smithi045.stdout: inet 172.21.15.45/20 brd 172.21.15.255 scope global dynamic noprefixroute ens1f0 2023-12-16T21:37:38.091 INFO:teuthology.orchestra.run.smithi045.stdout: valid_lft 42331sec preferred_lft 42331sec 2023-12-16T21:37:38.092 INFO:teuthology.orchestra.run.smithi045.stdout: inet6 fe80::ec4:7aff:febd:12be/64 scope link noprefixroute 2023-12-16T21:37:38.092 INFO:teuthology.orchestra.run.smithi045.stdout: valid_lft forever preferred_lft forever 2023-12-16T21:37:38.092 INFO:teuthology.orchestra.run.smithi045.stdout:5: ens1f1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2023-12-16T21:37:38.092 INFO:teuthology.orchestra.run.smithi045.stdout: link/ether 0c:c4:7a:bd:12:bf brd ff:ff:ff:ff:ff:ff 2023-12-16T21:37:38.092 INFO:teuthology.orchestra.run.smithi045.stdout: altname enp1s0f1 2023-12-16T21:37:38.093 INFO:tasks.cephfs.mount:Setuping the 'ceph-brx' with 192.168.255.254/16 2023-12-16T21:37:38.093 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-12-16T21:37:38.093 DEBUG:teuthology.orchestra.run.smithi045:> (cd / && exec bash -c ' 2023-12-16T21:37:38.093 DEBUG:teuthology.orchestra.run.smithi045:> set -e 2023-12-16T21:37:38.093 DEBUG:teuthology.orchestra.run.smithi045:> sudo ip link add name ceph-brx type bridge 2023-12-16T21:37:38.093 DEBUG:teuthology.orchestra.run.smithi045:> sudo ip addr flush dev ceph-brx 2023-12-16T21:37:38.093 DEBUG:teuthology.orchestra.run.smithi045:> sudo ip link set ceph-brx up 2023-12-16T21:37:38.093 DEBUG:teuthology.orchestra.run.smithi045:> sudo ip addr add 192.168.255.254/16 brd 192.168.255.255 dev ceph-brx 2023-12-16T21:37:38.094 DEBUG:teuthology.orchestra.run.smithi045:> ') 2023-12-16T21:37:38.279 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-12-16T21:37:38.279 DEBUG:teuthology.orchestra.run.smithi045:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2023-12-16T21:37:38.312 INFO:teuthology.orchestra.run.smithi045.stdout:1 2023-12-16T21:37:38.314 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-12-16T21:37:38.314 DEBUG:teuthology.orchestra.run.smithi045:> route 2023-12-16T21:37:38.386 INFO:teuthology.orchestra.run.smithi045.stdout:Kernel IP routing table 2023-12-16T21:37:38.386 INFO:teuthology.orchestra.run.smithi045.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2023-12-16T21:37:38.386 INFO:teuthology.orchestra.run.smithi045.stdout:default _gateway 0.0.0.0 UG 100 0 0 ens1f0 2023-12-16T21:37:38.386 INFO:teuthology.orchestra.run.smithi045.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 ens1f0 2023-12-16T21:37:38.386 INFO:teuthology.orchestra.run.smithi045.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2023-12-16T21:37:38.386 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-12-16T21:37:38.387 DEBUG:teuthology.orchestra.run.smithi045:> (cd / && exec bash -c ' 2023-12-16T21:37:38.387 DEBUG:teuthology.orchestra.run.smithi045:> set -e 2023-12-16T21:37:38.387 DEBUG:teuthology.orchestra.run.smithi045:> sudo iptables -A FORWARD -o ens1f0 -i ceph-brx -j ACCEPT 2023-12-16T21:37:38.387 DEBUG:teuthology.orchestra.run.smithi045:> sudo iptables -A FORWARD -i ens1f0 -o ceph-brx -j ACCEPT 2023-12-16T21:37:38.387 DEBUG:teuthology.orchestra.run.smithi045:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o ens1f0 -j MASQUERADE 2023-12-16T21:37:38.387 DEBUG:teuthology.orchestra.run.smithi045:> ') 2023-12-16T21:37:38.633 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Dec 16 21:37:38 smithi169 ceph-mon[88350]: from='client.? 172.21.15.45:0/2687851165' entity='client.admin' cmd=[{"prefix": "fs ls"}]: dispatch 2023-12-16T21:37:38.641 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-12-16T21:37:38.642 DEBUG:teuthology.orchestra.run.smithi045:> ip netns list 2023-12-16T21:37:38.657 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-12-16T21:37:38.657 DEBUG:teuthology.orchestra.run.smithi045:> ip netns list-id 2023-12-16T21:37:38.714 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-12-16T21:37:38.714 DEBUG:teuthology.orchestra.run.smithi045:> (cd / && exec bash -c ' 2023-12-16T21:37:38.714 DEBUG:teuthology.orchestra.run.smithi045:> set -e 2023-12-16T21:37:38.715 DEBUG:teuthology.orchestra.run.smithi045:> sudo ip netns add ceph-ns--home-ubuntu-cephtest-mnt.0 2023-12-16T21:37:38.715 DEBUG:teuthology.orchestra.run.smithi045:> sudo ip netns set ceph-ns--home-ubuntu-cephtest-mnt.0 0 2023-12-16T21:37:38.715 DEBUG:teuthology.orchestra.run.smithi045:> ') 2023-12-16T21:37:38.807 INFO:tasks.cephfs.mount:Setuping the netns 'ceph-ns--home-ubuntu-cephtest-mnt.0' with 192.168.0.1/16 2023-12-16T21:37:38.807 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-12-16T21:37:38.807 DEBUG:teuthology.orchestra.run.smithi045:> (cd / && exec bash -c ' 2023-12-16T21:37:38.808 DEBUG:teuthology.orchestra.run.smithi045:> set -e 2023-12-16T21:37:38.808 DEBUG:teuthology.orchestra.run.smithi045:> sudo ip link add veth0 netns ceph-ns--home-ubuntu-cephtest-mnt.0 type veth peer name brx.0 2023-12-16T21:37:38.808 DEBUG:teuthology.orchestra.run.smithi045:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.0 ip addr add 192.168.0.1/16 brd 192.168.255.255 dev veth0 2023-12-16T21:37:38.808 DEBUG:teuthology.orchestra.run.smithi045:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.0 ip link set veth0 up 2023-12-16T21:37:38.808 DEBUG:teuthology.orchestra.run.smithi045:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.0 ip link set lo up 2023-12-16T21:37:38.808 DEBUG:teuthology.orchestra.run.smithi045:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.0 ip route add default via 192.168.255.254 2023-12-16T21:37:38.808 DEBUG:teuthology.orchestra.run.smithi045:> ') 2023-12-16T21:37:38.816 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Dec 16 21:37:38 smithi045 ceph-mon[80763]: from='client.? 172.21.15.45:0/2687851165' entity='client.admin' cmd=[{"prefix": "fs ls"}]: dispatch 2023-12-16T21:37:38.925 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-12-16T21:37:38.925 DEBUG:teuthology.orchestra.run.smithi045:> (cd / && exec bash -c ' 2023-12-16T21:37:38.925 DEBUG:teuthology.orchestra.run.smithi045:> set -e 2023-12-16T21:37:38.925 DEBUG:teuthology.orchestra.run.smithi045:> sudo ip link set brx.0 up 2023-12-16T21:37:38.925 DEBUG:teuthology.orchestra.run.smithi045:> sudo ip link set dev brx.0 master ceph-brx 2023-12-16T21:37:38.925 DEBUG:teuthology.orchestra.run.smithi045:> ') 2023-12-16T21:37:39.018 INFO:teuthology.orchestra.run:Running command with timeout 60 2023-12-16T21:37:39.019 DEBUG:teuthology.orchestra.run.smithi045:> mkdir -p -v /home/ubuntu/cephtest/mnt.0 2023-12-16T21:37:39.034 INFO:teuthology.orchestra.run.smithi045.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.0' 2023-12-16T21:37:39.034 INFO:teuthology.orchestra.run:Running command with timeout 60 2023-12-16T21:37:39.034 DEBUG:teuthology.orchestra.run.smithi045:> chmod 0000 /home/ubuntu/cephtest/mnt.0 2023-12-16T21:37:39.093 INFO:tasks.cephfs.kernel_mount:mounting using device: :/ 2023-12-16T21:37:39.093 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-12-16T21:37:39.094 DEBUG:teuthology.orchestra.run.smithi045:> sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage /bin/mount -t ceph :/ /home/ubuntu/cephtest/mnt.0 -v -o norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=0,mds_namespace=cephfs 2023-12-16T21:37:39.193 INFO:teuthology.orchestra.run.smithi045.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=0,mds_namespace=cephfs 2023-12-16T21:37:39.193 INFO:teuthology.orchestra.run.smithi045.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,mds_namespace=cephfs". 2023-12-16T21:37:39.193 INFO:teuthology.orchestra.run.smithi045.stdout:invalid new device string format 2023-12-16T21:37:39.299 INFO:teuthology.orchestra.run.smithi045.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=0,mds_namespace=cephfs 2023-12-16T21:37:39.299 INFO:teuthology.orchestra.run.smithi045.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,mds_namespace=cephfs". 2023-12-16T21:37:39.299 INFO:teuthology.orchestra.run.smithi045.stdout:invalid new device string format 2023-12-16T21:37:39.299 INFO:teuthology.orchestra.run.smithi045.stdout:mount.ceph: resolved to: "172.21.15.45:3300,172.21.15.169:3300" 2023-12-16T21:37:39.300 INFO:teuthology.orchestra.run.smithi045.stdout:mount.ceph: switching to using v1 address with old syntax 2023-12-16T21:37:39.305 INFO:teuthology.orchestra.run.smithi045.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=0,mds_namespace=cephfs 2023-12-16T21:37:39.305 INFO:teuthology.orchestra.run.smithi045.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,mds_namespace=cephfs". 2023-12-16T21:37:39.305 INFO:teuthology.orchestra.run.smithi045.stdout:invalid new device string format 2023-12-16T21:37:39.305 INFO:teuthology.orchestra.run.smithi045.stdout:mount.ceph: resolved to: "172.21.15.45:3300,172.21.15.169:3300" 2023-12-16T21:37:39.305 INFO:teuthology.orchestra.run.smithi045.stdout:mount.ceph: switching to using v1 address with old syntax 2023-12-16T21:37:39.305 INFO:teuthology.orchestra.run.smithi045.stdout:mount.ceph: resolved to: "172.21.15.45:6789,172.21.15.169:6789" 2023-12-16T21:37:39.305 INFO:teuthology.orchestra.run.smithi045.stdout:mount.ceph: trying mount with old device syntax: 172.21.15.45:6789,172.21.15.169:6789:/ 2023-12-16T21:37:39.305 INFO:teuthology.orchestra.run.smithi045.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,mds_namespace=cephfs,key=0,fsid=88ff363c-9c5a-11ee-95a5-87774f69a715" will pass to kernel 2023-12-16T21:37:39.306 INFO:teuthology.orchestra.run.smithi045.stdout:mount: /home/ubuntu/cephtest/mnt.0 does not contain SELinux labels. 2023-12-16T21:37:39.306 INFO:teuthology.orchestra.run.smithi045.stdout: You just mounted a file system that supports labels which does not 2023-12-16T21:37:39.306 INFO:teuthology.orchestra.run.smithi045.stdout: contain labels, onto an SELinux box. It is likely that confined 2023-12-16T21:37:39.306 INFO:teuthology.orchestra.run.smithi045.stdout: applications will generate AVC messages and not be allowed access to 2023-12-16T21:37:39.306 INFO:teuthology.orchestra.run.smithi045.stdout: this file system. For more details see restorecon(8) and mount(8). 2023-12-16T21:37:39.308 INFO:tasks.cephfs.kernel_mount:mount command passed 2023-12-16T21:37:39.308 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-12-16T21:37:39.308 DEBUG:teuthology.orchestra.run.smithi045:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.0 2023-12-16T21:37:39.345 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-12-16T21:37:39.346 DEBUG:teuthology.orchestra.run.smithi045:> (cd /home/ubuntu/cephtest/mnt.0 && exec bash -c 'getfattr --only-values -n ceph.client_id .') 2023-12-16T21:37:39.410 INFO:teuthology.orchestra.run.smithi045.stdout:client14530 2023-12-16T21:37:39.410 DEBUG:teuthology.orchestra.run.smithi045:> set -ex 2023-12-16T21:37:39.410 DEBUG:teuthology.orchestra.run.smithi045:> dd if=/proc/self/mounts of=/dev/stdout 2023-12-16T21:37:39.467 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-12-16T21:37:39.467 DEBUG:teuthology.orchestra.run.smithi045:> (cd /home/ubuntu/cephtest/mnt.0 && exec bash -c 'getfattr --only-values -n ceph.client_id .') 2023-12-16T21:37:39.530 INFO:teuthology.orchestra.run.smithi045.stdout:client14530 2023-12-16T21:37:39.531 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-12-16T21:37:39.531 DEBUG:teuthology.orchestra.run.smithi045:> (cd /home/ubuntu/cephtest/mnt.0 && exec bash -c 'getfattr --only-values -n ceph.client_id .') 2023-12-16T21:37:39.594 INFO:teuthology.orchestra.run.smithi045.stdout:client14530 2023-12-16T21:37:39.594 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-12-16T21:37:39.594 DEBUG:teuthology.orchestra.run.smithi045:> (cd /home/ubuntu/cephtest/mnt.0 && exec bash -c 'sudo dd if=/sys/kernel/debug/ceph/88ff363c-9c5a-11ee-95a5-87774f69a715.client14530/status') 2023-12-16T21:37:39.670 INFO:teuthology.orchestra.run.smithi045.stdout:instance: client.14530 (3)192.168.0.1:0/1058601342 2023-12-16T21:37:39.670 INFO:teuthology.orchestra.run.smithi045.stdout:blocklisted: false 2023-12-16T21:37:39.672 INFO:teuthology.orchestra.run.smithi045.stderr:0+1 records in 2023-12-16T21:37:39.672 INFO:teuthology.orchestra.run.smithi045.stderr:0+1 records out 2023-12-16T21:37:39.672 INFO:teuthology.orchestra.run.smithi045.stderr:70 bytes copied, 3.0202e-05 s, 2.3 MB/s 2023-12-16T21:37:39.673 INFO:tasks.kclient:client.1 config is {'syntax': 'v1'} 2023-12-16T21:37:39.673 INFO:tasks.cephfs.mount:cephfs_mntpt = None 2023-12-16T21:37:39.673 INFO:tasks.cephfs.mount:hostfs_mntpt = /home/ubuntu/cephtest/mnt.1 2023-12-16T21:37:39.673 DEBUG:teuthology.orchestra.run.smithi045:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 88ff363c-9c5a-11ee-95a5-87774f69a715 -- ceph fs ls 2023-12-16T21:37:39.883 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Dec 16 21:37:39 smithi169 ceph-mon[88350]: pgmap v126: 65 pgs: 65 active+clean; 579 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 6.0 KiB/s rd, 0 B/s wr, 6 op/s 2023-12-16T21:37:39.981 INFO:teuthology.orchestra.run.smithi045.stderr:Inferring config /var/lib/ceph/88ff363c-9c5a-11ee-95a5-87774f69a715/mon.smithi045/config 2023-12-16T21:37:40.048 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Dec 16 21:37:39 smithi045 ceph-mon[80763]: pgmap v126: 65 pgs: 65 active+clean; 579 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 6.0 KiB/s rd, 0 B/s wr, 6 op/s 2023-12-16T21:37:40.733 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:37:40.728+0000 7f12e7e2b700 1 -- 172.21.15.45:0/2083678871 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f12e00fbb30 msgr2=0x7f12e00fbfa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:37:40.733 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:37:40.728+0000 7f12e7e2b700 1 --2- 172.21.15.45:0/2083678871 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f12e00fbb30 0x7f12e00fbfa0 secure :-1 s=READY pgs=275 cs=0 l=1 rev1=1 crypto rx=0x7f12d4004940 tx=0x7f12d4009e30 comp rx=0 tx=0).stop 2023-12-16T21:37:40.733 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:37:40.728+0000 7f12e7e2b700 1 -- 172.21.15.45:0/2083678871 shutdown_connections 2023-12-16T21:37:40.734 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:37:40.728+0000 7f12e7e2b700 1 --2- 172.21.15.45:0/2083678871 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f12e00fbb30 0x7f12e00fbfa0 unknown :-1 s=CLOSED pgs=275 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:37:40.734 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:37:40.728+0000 7f12e7e2b700 1 --2- 172.21.15.45:0/2083678871 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f12e0101af0 0x7f12e0101ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:37:40.734 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:37:40.728+0000 7f12e7e2b700 1 -- 172.21.15.45:0/2083678871 >> 172.21.15.45:0/2083678871 conn(0x7f12e00f7690 msgr2=0x7f12e00f9aa0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-16T21:37:40.734 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:37:40.728+0000 7f12e7e2b700 1 -- 172.21.15.45:0/2083678871 shutdown_connections 2023-12-16T21:37:40.734 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:37:40.728+0000 7f12e7e2b700 1 -- 172.21.15.45:0/2083678871 wait complete. 2023-12-16T21:37:40.734 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:37:40.729+0000 7f12e7e2b700 1 Processor -- start 2023-12-16T21:37:40.734 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:37:40.729+0000 7f12e7e2b700 1 -- start start 2023-12-16T21:37:40.734 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:37:40.729+0000 7f12e7e2b700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f12e00fbb30 0x7f12e0115880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:37:40.734 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:37:40.729+0000 7f12e7e2b700 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f12e0101af0 0x7f12e0115dc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:37:40.734 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:37:40.729+0000 7f12e7e2b700 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f12e0116410 con 0x7f12e00fbb30 2023-12-16T21:37:40.734 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:37:40.729+0000 7f12e7e2b700 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f12e010f740 con 0x7f12e0101af0 2023-12-16T21:37:40.734 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:37:40.729+0000 7f12e5bc7700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f12e00fbb30 0x7f12e0115880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:37:40.734 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:37:40.729+0000 7f12e5bc7700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f12e00fbb30 0x7f12e0115880 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:33778/0 (socket says 172.21.15.45:33778) 2023-12-16T21:37:40.734 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:37:40.729+0000 7f12e5bc7700 1 -- 172.21.15.45:0/823991253 learned_addr learned my addr 172.21.15.45:0/823991253 (peer_addr_for_me v2:172.21.15.45:0/0) 2023-12-16T21:37:40.735 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:37:40.729+0000 7f12e5bc7700 1 -- 172.21.15.45:0/823991253 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f12e0101af0 msgr2=0x7f12e0115dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:37:40.735 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:37:40.729+0000 7f12e53c6700 1 --2- 172.21.15.45:0/823991253 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f12e0101af0 0x7f12e0115dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:37:40.735 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:37:40.729+0000 7f12e5bc7700 1 --2- 172.21.15.45:0/823991253 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f12e0101af0 0x7f12e0115dc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:37:40.735 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:37:40.729+0000 7f12e5bc7700 1 -- 172.21.15.45:0/823991253 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f12d4005040 con 0x7f12e00fbb30 2023-12-16T21:37:40.735 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:37:40.729+0000 7f12e53c6700 1 --2- 172.21.15.45:0/823991253 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f12e0101af0 0x7f12e0115dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-12-16T21:37:40.735 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:37:40.730+0000 7f12e5bc7700 1 --2- 172.21.15.45:0/823991253 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f12e00fbb30 0x7f12e0115880 secure :-1 s=READY pgs=276 cs=0 l=1 rev1=1 crypto rx=0x7f12d800cb10 tx=0x7f12d8007f90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:37:40.735 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:37:40.730+0000 7f12de7fc700 1 -- 172.21.15.45:0/823991253 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f12d8004200 con 0x7f12e00fbb30 2023-12-16T21:37:40.735 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:37:40.730+0000 7f12de7fc700 1 -- 172.21.15.45:0/823991253 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f12d800aca0 con 0x7f12e00fbb30 2023-12-16T21:37:40.735 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:37:40.730+0000 7f12e7e2b700 1 -- 172.21.15.45:0/823991253 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f12e010fa20 con 0x7f12e00fbb30 2023-12-16T21:37:40.735 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:37:40.730+0000 7f12de7fc700 1 -- 172.21.15.45:0/823991253 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f12d8003ce0 con 0x7f12e00fbb30 2023-12-16T21:37:40.735 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:37:40.730+0000 7f12e7e2b700 1 -- 172.21.15.45:0/823991253 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f12e010ff60 con 0x7f12e00fbb30 2023-12-16T21:37:40.735 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:37:40.731+0000 7f12e7e2b700 1 -- 172.21.15.45:0/823991253 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f12e0103fa0 con 0x7f12e00fbb30 2023-12-16T21:37:40.736 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:37:40.731+0000 7f12de7fc700 1 -- 172.21.15.45:0/823991253 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 19) v1 ==== 86447+0+0 (secure 0 0 0) 0x7f12d800a7f0 con 0x7f12e00fbb30 2023-12-16T21:37:40.736 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:37:40.732+0000 7f12de7fc700 1 --2- 172.21.15.45:0/823991253 >> [v2:172.21.15.45:6800/2012105536,v1:172.21.15.45:6801/2012105536] conn(0x7f12c8068f70 0x7f12c806b420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:37:40.736 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:37:40.732+0000 7f12de7fc700 1 -- 172.21.15.45:0/823991253 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(43..43 src has 1..43) v4 ==== 5550+0+0 (secure 0 0 0) 0x7f12d8087340 con 0x7f12e00fbb30 2023-12-16T21:37:40.736 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:37:40.732+0000 7f12e53c6700 1 --2- 172.21.15.45:0/823991253 >> [v2:172.21.15.45:6800/2012105536,v1:172.21.15.45:6801/2012105536] conn(0x7f12c8068f70 0x7f12c806b420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:37:40.736 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:37:40.732+0000 7f12e53c6700 1 --2- 172.21.15.45:0/823991253 >> [v2:172.21.15.45:6800/2012105536,v1:172.21.15.45:6801/2012105536] conn(0x7f12c8068f70 0x7f12c806b420 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7f12d4009fd0 tx=0x7f12d4009e30 comp rx=0 tx=0).ready entity=mgr.14214 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:37:40.738 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:37:40.736+0000 7f12de7fc700 1 -- 172.21.15.45:0/823991253 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+174570 (secure 0 0 0) 0x7f12d80571d0 con 0x7f12e00fbb30 2023-12-16T21:37:40.979 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:37:40.976+0000 7f12e7e2b700 1 -- 172.21.15.45:0/823991253 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "fs ls"} v 0) v1 -- 0x7f12e004bb80 con 0x7f12e00fbb30 2023-12-16T21:37:40.979 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:37:40.977+0000 7f12de7fc700 1 -- 172.21.15.45:0/823991253 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v15) v1 ==== 53+0+83 (secure 0 0 0) 0x7f12d8003e80 con 0x7f12e00fbb30 2023-12-16T21:37:40.979 INFO:teuthology.orchestra.run.smithi045.stdout:name: cephfs, metadata pool: cephfs.cephfs.meta, data pools: [cephfs.cephfs.data ] 2023-12-16T21:37:40.982 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:37:40.979+0000 7f12e7e2b700 1 -- 172.21.15.45:0/823991253 >> [v2:172.21.15.45:6800/2012105536,v1:172.21.15.45:6801/2012105536] conn(0x7f12c8068f70 msgr2=0x7f12c806b420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:37:40.982 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:37:40.979+0000 7f12e7e2b700 1 --2- 172.21.15.45:0/823991253 >> [v2:172.21.15.45:6800/2012105536,v1:172.21.15.45:6801/2012105536] conn(0x7f12c8068f70 0x7f12c806b420 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7f12d4009fd0 tx=0x7f12d4009e30 comp rx=0 tx=0).stop 2023-12-16T21:37:40.982 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:37:40.979+0000 7f12e7e2b700 1 -- 172.21.15.45:0/823991253 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f12e00fbb30 msgr2=0x7f12e0115880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:37:40.982 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:37:40.979+0000 7f12e7e2b700 1 --2- 172.21.15.45:0/823991253 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f12e00fbb30 0x7f12e0115880 secure :-1 s=READY pgs=276 cs=0 l=1 rev1=1 crypto rx=0x7f12d800cb10 tx=0x7f12d8007f90 comp rx=0 tx=0).stop 2023-12-16T21:37:40.982 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:37:40.979+0000 7f12e7e2b700 1 -- 172.21.15.45:0/823991253 shutdown_connections 2023-12-16T21:37:40.982 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:37:40.979+0000 7f12e7e2b700 1 --2- 172.21.15.45:0/823991253 >> [v2:172.21.15.45:6800/2012105536,v1:172.21.15.45:6801/2012105536] conn(0x7f12c8068f70 0x7f12c806b420 unknown :-1 s=CLOSED pgs=108 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:37:40.982 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:37:40.979+0000 7f12e7e2b700 1 --2- 172.21.15.45:0/823991253 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f12e00fbb30 0x7f12e0115880 unknown :-1 s=CLOSED pgs=276 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:37:40.982 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:37:40.979+0000 7f12e7e2b700 1 --2- 172.21.15.45:0/823991253 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f12e0101af0 0x7f12e0115dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:37:40.982 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:37:40.979+0000 7f12e7e2b700 1 -- 172.21.15.45:0/823991253 >> 172.21.15.45:0/823991253 conn(0x7f12e00f7690 msgr2=0x7f12e00f92d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-16T21:37:40.982 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:37:40.979+0000 7f12e7e2b700 1 -- 172.21.15.45:0/823991253 shutdown_connections 2023-12-16T21:37:40.982 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:37:40.979+0000 7f12e7e2b700 1 -- 172.21.15.45:0/823991253 wait complete. 2023-12-16T21:37:41.239 INFO:tasks.cephfs.mount:Mounting default Ceph FS; just confirmed its presence on cluster 2023-12-16T21:37:41.239 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2023-12-16T21:37:41.239 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi169.front.sepia.ceph.com 2023-12-16T21:37:41.239 INFO:tasks.cephfs.mount:self.client.name = client.1 2023-12-16T21:37:41.239 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.1 2023-12-16T21:37:41.239 INFO:tasks.cephfs.mount:self.cephfs_name = None 2023-12-16T21:37:41.239 INFO:tasks.cephfs.mount:self.cephfs_mntpt = None 2023-12-16T21:37:41.240 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2023-12-16T21:37:41.240 INFO:tasks.cephfs.mount:Setting the 'None' netns for '/home/ubuntu/cephtest/mnt.1' 2023-12-16T21:37:41.240 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-12-16T21:37:41.240 DEBUG:teuthology.orchestra.run.smithi169:> ip addr 2023-12-16T21:37:41.259 INFO:teuthology.orchestra.run.smithi169.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2023-12-16T21:37:41.259 INFO:teuthology.orchestra.run.smithi169.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2023-12-16T21:37:41.259 INFO:teuthology.orchestra.run.smithi169.stdout: inet 127.0.0.1/8 scope host lo 2023-12-16T21:37:41.259 INFO:teuthology.orchestra.run.smithi169.stdout: valid_lft forever preferred_lft forever 2023-12-16T21:37:41.260 INFO:teuthology.orchestra.run.smithi169.stdout: inet6 ::1/128 scope host 2023-12-16T21:37:41.260 INFO:teuthology.orchestra.run.smithi169.stdout: valid_lft forever preferred_lft forever 2023-12-16T21:37:41.260 INFO:teuthology.orchestra.run.smithi169.stdout:2: eno1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2023-12-16T21:37:41.260 INFO:teuthology.orchestra.run.smithi169.stdout: link/ether 0c:c4:7a:d9:3b:b4 brd ff:ff:ff:ff:ff:ff 2023-12-16T21:37:41.260 INFO:teuthology.orchestra.run.smithi169.stdout: altname enp7s0f0 2023-12-16T21:37:41.260 INFO:teuthology.orchestra.run.smithi169.stdout:3: eno2: mtu 1500 qdisc mq state DOWN group default qlen 1000 2023-12-16T21:37:41.260 INFO:teuthology.orchestra.run.smithi169.stdout: link/ether 0c:c4:7a:d9:3b:b5 brd ff:ff:ff:ff:ff:ff 2023-12-16T21:37:41.261 INFO:teuthology.orchestra.run.smithi169.stdout: altname enp7s0f1 2023-12-16T21:37:41.261 INFO:teuthology.orchestra.run.smithi169.stdout:4: enp3s0f0: mtu 1500 qdisc mq state DOWN group default qlen 1000 2023-12-16T21:37:41.261 INFO:teuthology.orchestra.run.smithi169.stdout: link/ether 0c:c4:7a:88:7f:4c brd ff:ff:ff:ff:ff:ff 2023-12-16T21:37:41.261 INFO:teuthology.orchestra.run.smithi169.stdout:5: enp3s0f1: mtu 1500 qdisc mq state UP group default qlen 1000 2023-12-16T21:37:41.261 INFO:teuthology.orchestra.run.smithi169.stdout: link/ether 0c:c4:7a:88:7f:4d brd ff:ff:ff:ff:ff:ff 2023-12-16T21:37:41.261 INFO:teuthology.orchestra.run.smithi169.stdout: inet 172.21.15.169/20 brd 172.21.15.255 scope global dynamic noprefixroute enp3s0f1 2023-12-16T21:37:41.262 INFO:teuthology.orchestra.run.smithi169.stdout: valid_lft 42327sec preferred_lft 42327sec 2023-12-16T21:37:41.262 INFO:teuthology.orchestra.run.smithi169.stdout: inet6 fe80::ec4:7aff:fe88:7f4d/64 scope link 2023-12-16T21:37:41.262 INFO:teuthology.orchestra.run.smithi169.stdout: valid_lft forever preferred_lft forever 2023-12-16T21:37:41.262 INFO:tasks.cephfs.mount:Setuping the 'ceph-brx' with 192.168.255.254/16 2023-12-16T21:37:41.262 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-12-16T21:37:41.262 DEBUG:teuthology.orchestra.run.smithi169:> (cd / && exec bash -c ' 2023-12-16T21:37:41.263 DEBUG:teuthology.orchestra.run.smithi169:> set -e 2023-12-16T21:37:41.263 DEBUG:teuthology.orchestra.run.smithi169:> sudo ip link add name ceph-brx type bridge 2023-12-16T21:37:41.263 DEBUG:teuthology.orchestra.run.smithi169:> sudo ip addr flush dev ceph-brx 2023-12-16T21:37:41.263 DEBUG:teuthology.orchestra.run.smithi169:> sudo ip link set ceph-brx up 2023-12-16T21:37:41.263 DEBUG:teuthology.orchestra.run.smithi169:> sudo ip addr add 192.168.255.254/16 brd 192.168.255.255 dev ceph-brx 2023-12-16T21:37:41.263 DEBUG:teuthology.orchestra.run.smithi169:> ') 2023-12-16T21:37:41.438 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-12-16T21:37:41.438 DEBUG:teuthology.orchestra.run.smithi169:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2023-12-16T21:37:41.465 INFO:teuthology.orchestra.run.smithi169.stdout:1 2023-12-16T21:37:41.467 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-12-16T21:37:41.467 DEBUG:teuthology.orchestra.run.smithi169:> route 2023-12-16T21:37:41.534 INFO:teuthology.orchestra.run.smithi169.stdout:Kernel IP routing table 2023-12-16T21:37:41.534 INFO:teuthology.orchestra.run.smithi169.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2023-12-16T21:37:41.534 INFO:teuthology.orchestra.run.smithi169.stdout:default _gateway 0.0.0.0 UG 100 0 0 enp3s0f1 2023-12-16T21:37:41.534 INFO:teuthology.orchestra.run.smithi169.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 enp3s0f1 2023-12-16T21:37:41.534 INFO:teuthology.orchestra.run.smithi169.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2023-12-16T21:37:41.535 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-12-16T21:37:41.535 DEBUG:teuthology.orchestra.run.smithi169:> (cd / && exec bash -c ' 2023-12-16T21:37:41.535 DEBUG:teuthology.orchestra.run.smithi169:> set -e 2023-12-16T21:37:41.535 DEBUG:teuthology.orchestra.run.smithi169:> sudo iptables -A FORWARD -o enp3s0f1 -i ceph-brx -j ACCEPT 2023-12-16T21:37:41.535 DEBUG:teuthology.orchestra.run.smithi169:> sudo iptables -A FORWARD -i enp3s0f1 -o ceph-brx -j ACCEPT 2023-12-16T21:37:41.535 DEBUG:teuthology.orchestra.run.smithi169:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o enp3s0f1 -j MASQUERADE 2023-12-16T21:37:41.535 DEBUG:teuthology.orchestra.run.smithi169:> ') 2023-12-16T21:37:41.819 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-12-16T21:37:41.819 DEBUG:teuthology.orchestra.run.smithi169:> ip netns list 2023-12-16T21:37:41.834 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-12-16T21:37:41.834 DEBUG:teuthology.orchestra.run.smithi169:> ip netns list-id 2023-12-16T21:37:41.883 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Dec 16 21:37:41 smithi169 ceph-mon[88350]: pgmap v127: 65 pgs: 65 active+clean; 579 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 4.9 KiB/s rd, 0 B/s wr, 5 op/s 2023-12-16T21:37:41.883 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Dec 16 21:37:41 smithi169 ceph-mon[88350]: from='client.? 172.21.15.45:0/823991253' entity='client.admin' cmd=[{"prefix": "fs ls"}]: dispatch 2023-12-16T21:37:41.889 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-12-16T21:37:41.889 DEBUG:teuthology.orchestra.run.smithi169:> (cd / && exec bash -c ' 2023-12-16T21:37:41.889 DEBUG:teuthology.orchestra.run.smithi169:> set -e 2023-12-16T21:37:41.889 DEBUG:teuthology.orchestra.run.smithi169:> sudo ip netns add ceph-ns--home-ubuntu-cephtest-mnt.1 2023-12-16T21:37:41.889 DEBUG:teuthology.orchestra.run.smithi169:> sudo ip netns set ceph-ns--home-ubuntu-cephtest-mnt.1 0 2023-12-16T21:37:41.889 DEBUG:teuthology.orchestra.run.smithi169:> ') 2023-12-16T21:37:41.972 INFO:tasks.cephfs.mount:Setuping the netns 'ceph-ns--home-ubuntu-cephtest-mnt.1' with 192.168.0.1/16 2023-12-16T21:37:41.972 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-12-16T21:37:41.972 DEBUG:teuthology.orchestra.run.smithi169:> (cd / && exec bash -c ' 2023-12-16T21:37:41.972 DEBUG:teuthology.orchestra.run.smithi169:> set -e 2023-12-16T21:37:41.973 DEBUG:teuthology.orchestra.run.smithi169:> sudo ip link add veth0 netns ceph-ns--home-ubuntu-cephtest-mnt.1 type veth peer name brx.0 2023-12-16T21:37:41.973 DEBUG:teuthology.orchestra.run.smithi169:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.1 ip addr add 192.168.0.1/16 brd 192.168.255.255 dev veth0 2023-12-16T21:37:41.973 DEBUG:teuthology.orchestra.run.smithi169:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.1 ip link set veth0 up 2023-12-16T21:37:41.973 DEBUG:teuthology.orchestra.run.smithi169:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.1 ip link set lo up 2023-12-16T21:37:41.973 DEBUG:teuthology.orchestra.run.smithi169:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.1 ip route add default via 192.168.255.254 2023-12-16T21:37:41.973 DEBUG:teuthology.orchestra.run.smithi169:> ') 2023-12-16T21:37:42.067 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Dec 16 21:37:41 smithi045 ceph-mon[80763]: pgmap v127: 65 pgs: 65 active+clean; 579 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 4.9 KiB/s rd, 0 B/s wr, 5 op/s 2023-12-16T21:37:42.067 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Dec 16 21:37:41 smithi045 ceph-mon[80763]: from='client.? 172.21.15.45:0/823991253' entity='client.admin' cmd=[{"prefix": "fs ls"}]: dispatch 2023-12-16T21:37:42.119 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-12-16T21:37:42.119 DEBUG:teuthology.orchestra.run.smithi169:> (cd / && exec bash -c ' 2023-12-16T21:37:42.120 DEBUG:teuthology.orchestra.run.smithi169:> set -e 2023-12-16T21:37:42.120 DEBUG:teuthology.orchestra.run.smithi169:> sudo ip link set brx.0 up 2023-12-16T21:37:42.120 DEBUG:teuthology.orchestra.run.smithi169:> sudo ip link set dev brx.0 master ceph-brx 2023-12-16T21:37:42.120 DEBUG:teuthology.orchestra.run.smithi169:> ') 2023-12-16T21:37:42.161 INFO:teuthology.orchestra.run:Running command with timeout 60 2023-12-16T21:37:42.161 DEBUG:teuthology.orchestra.run.smithi169:> mkdir -p -v /home/ubuntu/cephtest/mnt.1 2023-12-16T21:37:42.216 INFO:teuthology.orchestra.run.smithi169.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.1' 2023-12-16T21:37:42.217 INFO:teuthology.orchestra.run:Running command with timeout 60 2023-12-16T21:37:42.217 DEBUG:teuthology.orchestra.run.smithi169:> chmod 0000 /home/ubuntu/cephtest/mnt.1 2023-12-16T21:37:42.271 INFO:tasks.cephfs.kernel_mount:mounting using device: :/ 2023-12-16T21:37:42.272 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-12-16T21:37:42.272 DEBUG:teuthology.orchestra.run.smithi169:> sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage /bin/mount -t ceph :/ /home/ubuntu/cephtest/mnt.1 -v -o norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=1,mds_namespace=cephfs 2023-12-16T21:37:42.371 INFO:teuthology.orchestra.run.smithi169.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=1,mds_namespace=cephfs 2023-12-16T21:37:42.371 INFO:teuthology.orchestra.run.smithi169.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=1,mds_namespace=cephfs". 2023-12-16T21:37:42.371 INFO:teuthology.orchestra.run.smithi169.stdout:invalid new device string format 2023-12-16T21:37:42.484 INFO:teuthology.orchestra.run.smithi169.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=1,mds_namespace=cephfs 2023-12-16T21:37:42.484 INFO:teuthology.orchestra.run.smithi169.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=1,mds_namespace=cephfs". 2023-12-16T21:37:42.484 INFO:teuthology.orchestra.run.smithi169.stdout:invalid new device string format 2023-12-16T21:37:42.484 INFO:teuthology.orchestra.run.smithi169.stdout:mount.ceph: resolved to: "172.21.15.45:3300,172.21.15.169:3300" 2023-12-16T21:37:42.485 INFO:teuthology.orchestra.run.smithi169.stdout:mount.ceph: switching to using v1 address with old syntax 2023-12-16T21:37:42.488 INFO:teuthology.orchestra.run.smithi169.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=1,mds_namespace=cephfs 2023-12-16T21:37:42.488 INFO:teuthology.orchestra.run.smithi169.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=1,mds_namespace=cephfs". 2023-12-16T21:37:42.489 INFO:teuthology.orchestra.run.smithi169.stdout:invalid new device string format 2023-12-16T21:37:42.489 INFO:teuthology.orchestra.run.smithi169.stdout:mount.ceph: resolved to: "172.21.15.45:3300,172.21.15.169:3300" 2023-12-16T21:37:42.489 INFO:teuthology.orchestra.run.smithi169.stdout:mount.ceph: switching to using v1 address with old syntax 2023-12-16T21:37:42.489 INFO:teuthology.orchestra.run.smithi169.stdout:mount.ceph: resolved to: "172.21.15.45:6789,172.21.15.169:6789" 2023-12-16T21:37:42.489 INFO:teuthology.orchestra.run.smithi169.stdout:mount.ceph: trying mount with old device syntax: 172.21.15.45:6789,172.21.15.169:6789:/ 2023-12-16T21:37:42.489 INFO:teuthology.orchestra.run.smithi169.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=1,mds_namespace=cephfs,key=1,fsid=88ff363c-9c5a-11ee-95a5-87774f69a715" will pass to kernel 2023-12-16T21:37:42.489 INFO:teuthology.orchestra.run.smithi169.stdout:mount: /home/ubuntu/cephtest/mnt.1 does not contain SELinux labels. 2023-12-16T21:37:42.489 INFO:teuthology.orchestra.run.smithi169.stdout: You just mounted a file system that supports labels which does not 2023-12-16T21:37:42.489 INFO:teuthology.orchestra.run.smithi169.stdout: contain labels, onto an SELinux box. It is likely that confined 2023-12-16T21:37:42.490 INFO:teuthology.orchestra.run.smithi169.stdout: applications will generate AVC messages and not be allowed access to 2023-12-16T21:37:42.490 INFO:teuthology.orchestra.run.smithi169.stdout: this file system. For more details see restorecon(8) and mount(8). 2023-12-16T21:37:42.491 INFO:tasks.cephfs.kernel_mount:mount command passed 2023-12-16T21:37:42.491 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-12-16T21:37:42.491 DEBUG:teuthology.orchestra.run.smithi169:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.1 2023-12-16T21:37:42.518 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-12-16T21:37:42.518 DEBUG:teuthology.orchestra.run.smithi169:> (cd /home/ubuntu/cephtest/mnt.1 && exec bash -c 'getfattr --only-values -n ceph.client_id .') 2023-12-16T21:37:42.574 INFO:teuthology.orchestra.run.smithi169.stdout:client14536 2023-12-16T21:37:42.574 DEBUG:teuthology.orchestra.run.smithi169:> set -ex 2023-12-16T21:37:42.574 DEBUG:teuthology.orchestra.run.smithi169:> dd if=/proc/self/mounts of=/dev/stdout 2023-12-16T21:37:42.630 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-12-16T21:37:42.630 DEBUG:teuthology.orchestra.run.smithi169:> (cd /home/ubuntu/cephtest/mnt.1 && exec bash -c 'getfattr --only-values -n ceph.client_id .') 2023-12-16T21:37:42.688 INFO:teuthology.orchestra.run.smithi169.stdout:client14536 2023-12-16T21:37:42.688 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-12-16T21:37:42.688 DEBUG:teuthology.orchestra.run.smithi169:> (cd /home/ubuntu/cephtest/mnt.1 && exec bash -c 'getfattr --only-values -n ceph.client_id .') 2023-12-16T21:37:42.746 INFO:teuthology.orchestra.run.smithi169.stdout:client14536 2023-12-16T21:37:42.746 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-12-16T21:37:42.746 DEBUG:teuthology.orchestra.run.smithi169:> (cd /home/ubuntu/cephtest/mnt.1 && exec bash -c 'sudo dd if=/sys/kernel/debug/ceph/88ff363c-9c5a-11ee-95a5-87774f69a715.client14536/status') 2023-12-16T21:37:42.813 INFO:teuthology.orchestra.run.smithi169.stdout:instance: client.14536 (3)172.21.15.169:0/3963500143 2023-12-16T21:37:42.813 INFO:teuthology.orchestra.run.smithi169.stdout:blocklisted: false 2023-12-16T21:37:42.814 INFO:teuthology.orchestra.run.smithi169.stderr:0+1 records in 2023-12-16T21:37:42.814 INFO:teuthology.orchestra.run.smithi169.stderr:0+1 records out 2023-12-16T21:37:42.814 INFO:teuthology.orchestra.run.smithi169.stderr:72 bytes copied, 2.0667e-05 s, 3.5 MB/s 2023-12-16T21:37:42.815 INFO:teuthology.run_tasks:Running task print... 2023-12-16T21:37:42.833 INFO:teuthology.task.print:**** done client 2023-12-16T21:37:42.833 INFO:teuthology.run_tasks:Running task parallel... 2023-12-16T21:37:42.849 INFO:teuthology.task.parallel:starting parallel... 2023-12-16T21:37:42.849 INFO:teuthology.task.parallel:In parallel, running task sequential... 2023-12-16T21:37:42.850 INFO:teuthology.task.sequential:In sequential, running task cephadm.shell... 2023-12-16T21:37:42.850 INFO:tasks.cephadm:Running commands on role host.a host ubuntu@smithi045.front.sepia.ceph.com 2023-12-16T21:37:42.851 DEBUG:teuthology.orchestra.run.smithi045:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 88ff363c-9c5a-11ee-95a5-87774f69a715 -e sha1=c7586e95519b7be76f3b6e3a4b333888baf46bc9 -- bash -c 'ceph config set mon mon_warn_on_insecure_global_id_reclaim false --force' 2023-12-16T21:37:42.851 INFO:teuthology.task.parallel:In parallel, running task sequential... 2023-12-16T21:37:42.851 INFO:teuthology.task.sequential:In sequential, running task workunit... 2023-12-16T21:37:42.854 INFO:tasks.workunit:Pulling workunits from ref c7586e95519b7be76f3b6e3a4b333888baf46bc9 2023-12-16T21:37:42.855 INFO:tasks.workunit:Making a separate scratch dir for every client... 2023-12-16T21:37:42.855 INFO:tasks.workunit:timeout=3h 2023-12-16T21:37:42.855 INFO:tasks.workunit:cleanup=True 2023-12-16T21:37:42.855 DEBUG:teuthology.orchestra.run.smithi045:> stat -- /home/ubuntu/cephtest/mnt.0 2023-12-16T21:37:42.877 INFO:teuthology.orchestra.run.smithi045.stdout: File: /home/ubuntu/cephtest/mnt.0 2023-12-16T21:37:42.877 INFO:teuthology.orchestra.run.smithi045.stdout: Size: 0 Blocks: 0 IO Block: 65536 directory 2023-12-16T21:37:42.877 INFO:teuthology.orchestra.run.smithi045.stdout:Device: 94h/148d Inode: 1 Links: 2 2023-12-16T21:37:42.877 INFO:teuthology.orchestra.run.smithi045.stdout:Access: (1777/drwxrwxrwt) Uid: ( 0/ root) Gid: ( 0/ root) 2023-12-16T21:37:42.877 INFO:teuthology.orchestra.run.smithi045.stdout:Context: system_u:object_r:unlabeled_t:s0 2023-12-16T21:37:42.877 INFO:teuthology.orchestra.run.smithi045.stdout:Access: 1970-01-01 00:00:00.000000000 +0000 2023-12-16T21:37:42.877 INFO:teuthology.orchestra.run.smithi045.stdout:Modify: 2023-12-16 21:37:22.305198517 +0000 2023-12-16T21:37:42.877 INFO:teuthology.orchestra.run.smithi045.stdout:Change: 2023-12-16 21:37:39.340313960 +0000 2023-12-16T21:37:42.877 INFO:teuthology.orchestra.run.smithi045.stdout: Birth: 2023-12-16 21:37:22.305198517 +0000 2023-12-16T21:37:42.878 INFO:tasks.workunit:Did not need to create dir /home/ubuntu/cephtest/mnt.0 2023-12-16T21:37:42.878 DEBUG:teuthology.orchestra.run.smithi045:> cd -- /home/ubuntu/cephtest/mnt.0 && sudo install -d -m 0755 --owner=ubuntu -- client.0 2023-12-16T21:37:42.953 DEBUG:teuthology.orchestra.run.smithi169:> stat -- /home/ubuntu/cephtest/mnt.1 2023-12-16T21:37:42.969 INFO:teuthology.orchestra.run.smithi169.stdout: File: /home/ubuntu/cephtest/mnt.1 2023-12-16T21:37:42.970 INFO:teuthology.orchestra.run.smithi169.stdout: Size: 1 Blocks: 0 IO Block: 65536 directory 2023-12-16T21:37:42.970 INFO:teuthology.orchestra.run.smithi169.stdout:Device: 73h/115d Inode: 1 Links: 3 2023-12-16T21:37:42.970 INFO:teuthology.orchestra.run.smithi169.stdout:Access: (1777/drwxrwxrwt) Uid: ( 0/ root) Gid: ( 0/ root) 2023-12-16T21:37:42.970 INFO:teuthology.orchestra.run.smithi169.stdout:Context: system_u:object_r:unlabeled_t:s0 2023-12-16T21:37:42.970 INFO:teuthology.orchestra.run.smithi169.stdout:Access: 1970-01-01 00:00:00.000000000 +0000 2023-12-16T21:37:42.970 INFO:teuthology.orchestra.run.smithi169.stdout:Modify: 2023-12-16 21:37:42.946257425 +0000 2023-12-16T21:37:42.970 INFO:teuthology.orchestra.run.smithi169.stdout:Change: 2023-12-16 21:37:42.946257425 +0000 2023-12-16T21:37:42.970 INFO:teuthology.orchestra.run.smithi169.stdout: Birth: 2023-12-16 21:37:22.305198517 +0000 2023-12-16T21:37:42.970 INFO:tasks.workunit:Did not need to create dir /home/ubuntu/cephtest/mnt.1 2023-12-16T21:37:42.970 DEBUG:teuthology.orchestra.run.smithi169:> cd -- /home/ubuntu/cephtest/mnt.1 && sudo install -d -m 0755 --owner=ubuntu -- client.1 2023-12-16T21:37:43.041 DEBUG:teuthology.orchestra.run.smithi045:> rm -rf /home/ubuntu/cephtest/clone.client.0 && git clone https://github.com/ceph/ceph-ci.git /home/ubuntu/cephtest/clone.client.0 && cd /home/ubuntu/cephtest/clone.client.0 && git checkout c7586e95519b7be76f3b6e3a4b333888baf46bc9 2023-12-16T21:37:43.042 DEBUG:teuthology.orchestra.run.smithi169:> rm -rf /home/ubuntu/cephtest/clone.client.1 && git clone https://github.com/ceph/ceph-ci.git /home/ubuntu/cephtest/clone.client.1 && cd /home/ubuntu/cephtest/clone.client.1 && git checkout c7586e95519b7be76f3b6e3a4b333888baf46bc9 2023-12-16T21:37:43.058 INFO:tasks.workunit.client.0.smithi045.stderr:Cloning into '/home/ubuntu/cephtest/clone.client.0'... 2023-12-16T21:37:43.097 INFO:tasks.workunit.client.1.smithi169.stderr:Cloning into '/home/ubuntu/cephtest/clone.client.1'... 2023-12-16T21:37:43.138 INFO:teuthology.orchestra.run.smithi045.stderr:Inferring config /var/lib/ceph/88ff363c-9c5a-11ee-95a5-87774f69a715/mon.smithi045/config 2023-12-16T21:37:43.921 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:37:43.917+0000 7fa555ca5700 1 -- 172.21.15.45:0/917463073 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fa5500fcd00 msgr2=0x7fa5500fd150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:37:43.921 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:37:43.917+0000 7fa555ca5700 1 --2- 172.21.15.45:0/917463073 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fa5500fcd00 0x7fa5500fd150 secure :-1 s=READY pgs=278 cs=0 l=1 rev1=1 crypto rx=0x7fa544004940 tx=0x7fa544009e30 comp rx=0 tx=0).stop 2023-12-16T21:37:43.921 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Dec 16 21:37:43 smithi045 ceph-mon[80763]: pgmap v128: 65 pgs: 65 active+clean; 579 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 4.9 KiB/s rd, 0 B/s wr, 5 op/s 2023-12-16T21:37:43.921 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Dec 16 21:37:43 smithi045 ceph-mon[80763]: from='mgr.14214 172.21.15.45:0/1227909437' entity='mgr.smithi045.nhtmwc' 2023-12-16T21:37:43.921 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Dec 16 21:37:43 smithi045 ceph-mon[80763]: from='mgr.14214 172.21.15.45:0/1227909437' entity='mgr.smithi045.nhtmwc' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2023-12-16T21:37:43.923 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:37:43.917+0000 7fa555ca5700 1 -- 172.21.15.45:0/917463073 shutdown_connections 2023-12-16T21:37:43.923 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:37:43.917+0000 7fa555ca5700 1 --2- 172.21.15.45:0/917463073 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fa5500fcd00 0x7fa5500fd150 unknown :-1 s=CLOSED pgs=278 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:37:43.923 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:37:43.917+0000 7fa555ca5700 1 --2- 172.21.15.45:0/917463073 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fa5500fbaf0 0x7fa5500fbf00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:37:43.923 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:37:43.918+0000 7fa555ca5700 1 -- 172.21.15.45:0/917463073 >> 172.21.15.45:0/917463073 conn(0x7fa5500f71d0 msgr2=0x7fa5500f95e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-16T21:37:43.923 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:37:43.918+0000 7fa555ca5700 1 -- 172.21.15.45:0/917463073 shutdown_connections 2023-12-16T21:37:43.924 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:37:43.918+0000 7fa555ca5700 1 -- 172.21.15.45:0/917463073 wait complete. 2023-12-16T21:37:43.924 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:37:43.918+0000 7fa555ca5700 1 Processor -- start 2023-12-16T21:37:43.924 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:37:43.918+0000 7fa555ca5700 1 -- start start 2023-12-16T21:37:43.924 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:37:43.919+0000 7fa555ca5700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fa5500fbaf0 0x7fa550110200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:37:43.924 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:37:43.919+0000 7fa555ca5700 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fa5500fcd00 0x7fa550112750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:37:43.924 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:37:43.919+0000 7fa555ca5700 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa550066a90 con 0x7fa5500fbaf0 2023-12-16T21:37:43.924 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:37:43.919+0000 7fa555ca5700 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa550066bd0 con 0x7fa5500fcd00 2023-12-16T21:37:43.924 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:37:43.919+0000 7fa54f7fe700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fa5500fbaf0 0x7fa550110200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:37:43.924 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:37:43.919+0000 7fa54f7fe700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fa5500fbaf0 0x7fa550110200 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:52108/0 (socket says 172.21.15.45:52108) 2023-12-16T21:37:43.924 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:37:43.919+0000 7fa54f7fe700 1 -- 172.21.15.45:0/75128020 learned_addr learned my addr 172.21.15.45:0/75128020 (peer_addr_for_me v2:172.21.15.45:0/0) 2023-12-16T21:37:43.924 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:37:43.919+0000 7fa54effd700 1 --2- 172.21.15.45:0/75128020 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fa5500fcd00 0x7fa550112750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:37:43.924 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:37:43.919+0000 7fa54f7fe700 1 -- 172.21.15.45:0/75128020 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fa5500fcd00 msgr2=0x7fa550112750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:37:43.924 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:37:43.919+0000 7fa54f7fe700 1 --2- 172.21.15.45:0/75128020 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fa5500fcd00 0x7fa550112750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:37:43.925 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:37:43.919+0000 7fa54f7fe700 1 -- 172.21.15.45:0/75128020 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa544005040 con 0x7fa5500fbaf0 2023-12-16T21:37:43.925 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:37:43.919+0000 7fa54f7fe700 1 --2- 172.21.15.45:0/75128020 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fa5500fbaf0 0x7fa550110200 secure :-1 s=READY pgs=279 cs=0 l=1 rev1=1 crypto rx=0x7fa54000b790 tx=0x7fa540009220 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:37:43.925 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:37:43.920+0000 7fa537fff700 1 -- 172.21.15.45:0/75128020 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fa54000f6f0 con 0x7fa5500fbaf0 2023-12-16T21:37:43.925 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:37:43.920+0000 7fa555ca5700 1 -- 172.21.15.45:0/75128020 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa550112e30 con 0x7fa5500fbaf0 2023-12-16T21:37:43.925 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:37:43.920+0000 7fa537fff700 1 -- 172.21.15.45:0/75128020 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fa54000fd00 con 0x7fa5500fbaf0 2023-12-16T21:37:43.925 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:37:43.920+0000 7fa537fff700 1 -- 172.21.15.45:0/75128020 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fa540011e90 con 0x7fa5500fbaf0 2023-12-16T21:37:43.925 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:37:43.920+0000 7fa555ca5700 1 -- 172.21.15.45:0/75128020 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa5501132e0 con 0x7fa5500fbaf0 2023-12-16T21:37:43.925 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:37:43.921+0000 7fa555ca5700 1 -- 172.21.15.45:0/75128020 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa55004bb80 con 0x7fa5500fbaf0 2023-12-16T21:37:43.926 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:37:43.921+0000 7fa537fff700 1 -- 172.21.15.45:0/75128020 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 19) v1 ==== 86447+0+0 (secure 0 0 0) 0x7fa54000f850 con 0x7fa5500fbaf0 2023-12-16T21:37:43.926 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:37:43.922+0000 7fa537fff700 1 --2- 172.21.15.45:0/75128020 >> [v2:172.21.15.45:6800/2012105536,v1:172.21.15.45:6801/2012105536] conn(0x7fa538068f60 0x7fa53806b410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:37:43.926 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:37:43.922+0000 7fa537fff700 1 -- 172.21.15.45:0/75128020 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(43..43 src has 1..43) v4 ==== 5550+0+0 (secure 0 0 0) 0x7fa540086bb0 con 0x7fa5500fbaf0 2023-12-16T21:37:43.926 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:37:43.922+0000 7fa54effd700 1 --2- 172.21.15.45:0/75128020 >> [v2:172.21.15.45:6800/2012105536,v1:172.21.15.45:6801/2012105536] conn(0x7fa538068f60 0x7fa53806b410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:37:43.926 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:37:43.922+0000 7fa54effd700 1 --2- 172.21.15.45:0/75128020 >> [v2:172.21.15.45:6800/2012105536,v1:172.21.15.45:6801/2012105536] conn(0x7fa538068f60 0x7fa53806b410 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7fa54400b3b0 tx=0x7fa544009e30 comp rx=0 tx=0).ready entity=mgr.14214 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:37:43.929 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:37:43.926+0000 7fa537fff700 1 -- 172.21.15.45:0/75128020 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+174570 (secure 0 0 0) 0x7fa54008d080 con 0x7fa5500fbaf0 2023-12-16T21:37:44.123 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:37:44.120+0000 7fa555ca5700 1 -- 172.21.15.45:0/75128020 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command([{prefix=config set, name=mon_warn_on_insecure_global_id_reclaim}] v 0) v1 -- 0x7fa550104530 con 0x7fa5500fbaf0 2023-12-16T21:37:44.123 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:37:44.121+0000 7fa537fff700 1 -- 172.21.15.45:0/75128020 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mon_warn_on_insecure_global_id_reclaim}]=0 v21)=0 v21) v1 ==== 155+0+0 (secure 0 0 0) 0x7fa5400530e0 con 0x7fa5500fbaf0 2023-12-16T21:37:44.125 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:37:44.123+0000 7fa555ca5700 1 -- 172.21.15.45:0/75128020 >> [v2:172.21.15.45:6800/2012105536,v1:172.21.15.45:6801/2012105536] conn(0x7fa538068f60 msgr2=0x7fa53806b410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:37:44.126 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:37:44.123+0000 7fa555ca5700 1 --2- 172.21.15.45:0/75128020 >> [v2:172.21.15.45:6800/2012105536,v1:172.21.15.45:6801/2012105536] conn(0x7fa538068f60 0x7fa53806b410 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7fa54400b3b0 tx=0x7fa544009e30 comp rx=0 tx=0).stop 2023-12-16T21:37:44.126 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:37:44.123+0000 7fa555ca5700 1 -- 172.21.15.45:0/75128020 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fa5500fbaf0 msgr2=0x7fa550110200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:37:44.126 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:37:44.123+0000 7fa555ca5700 1 --2- 172.21.15.45:0/75128020 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fa5500fbaf0 0x7fa550110200 secure :-1 s=READY pgs=279 cs=0 l=1 rev1=1 crypto rx=0x7fa54000b790 tx=0x7fa540009220 comp rx=0 tx=0).stop 2023-12-16T21:37:44.126 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:37:44.123+0000 7fa555ca5700 1 -- 172.21.15.45:0/75128020 shutdown_connections 2023-12-16T21:37:44.126 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:37:44.123+0000 7fa555ca5700 1 --2- 172.21.15.45:0/75128020 >> [v2:172.21.15.45:6800/2012105536,v1:172.21.15.45:6801/2012105536] conn(0x7fa538068f60 0x7fa53806b410 unknown :-1 s=CLOSED pgs=109 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:37:44.126 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:37:44.123+0000 7fa555ca5700 1 --2- 172.21.15.45:0/75128020 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fa5500fbaf0 0x7fa550110200 unknown :-1 s=CLOSED pgs=279 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:37:44.126 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:37:44.123+0000 7fa555ca5700 1 --2- 172.21.15.45:0/75128020 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fa5500fcd00 0x7fa550112750 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:37:44.126 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:37:44.123+0000 7fa555ca5700 1 -- 172.21.15.45:0/75128020 >> 172.21.15.45:0/75128020 conn(0x7fa5500f71d0 msgr2=0x7fa5500fff30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-16T21:37:44.126 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:37:44.123+0000 7fa555ca5700 1 -- 172.21.15.45:0/75128020 shutdown_connections 2023-12-16T21:37:44.126 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:37:44.123+0000 7fa555ca5700 1 -- 172.21.15.45:0/75128020 wait complete. 2023-12-16T21:37:44.133 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Dec 16 21:37:43 smithi169 ceph-mon[88350]: pgmap v128: 65 pgs: 65 active+clean; 579 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 4.9 KiB/s rd, 0 B/s wr, 5 op/s 2023-12-16T21:37:44.133 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Dec 16 21:37:43 smithi169 ceph-mon[88350]: from='mgr.14214 172.21.15.45:0/1227909437' entity='mgr.smithi045.nhtmwc' 2023-12-16T21:37:44.133 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Dec 16 21:37:43 smithi169 ceph-mon[88350]: from='mgr.14214 172.21.15.45:0/1227909437' entity='mgr.smithi045.nhtmwc' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2023-12-16T21:37:44.417 DEBUG:teuthology.orchestra.run.smithi045:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 88ff363c-9c5a-11ee-95a5-87774f69a715 -e sha1=c7586e95519b7be76f3b6e3a4b333888baf46bc9 -- bash -c 'ceph config set mon mon_warn_on_insecure_global_id_reclaim_allowed false --force' 2023-12-16T21:37:44.727 INFO:teuthology.orchestra.run.smithi045.stderr:Inferring config /var/lib/ceph/88ff363c-9c5a-11ee-95a5-87774f69a715/mon.smithi045/config 2023-12-16T21:37:45.479 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:37:45.476+0000 7f498f3a7700 1 -- 172.21.15.45:0/1076768933 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f49880fcd00 msgr2=0x7f49880fd150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:37:45.480 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:37:45.476+0000 7f498f3a7700 1 --2- 172.21.15.45:0/1076768933 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f49880fcd00 0x7f49880fd150 secure :-1 s=READY pgs=280 cs=0 l=1 rev1=1 crypto rx=0x7f4978004940 tx=0x7f4978009e30 comp rx=0 tx=0).stop 2023-12-16T21:37:45.480 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:37:45.476+0000 7f498f3a7700 1 -- 172.21.15.45:0/1076768933 shutdown_connections 2023-12-16T21:37:45.480 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:37:45.476+0000 7f498f3a7700 1 --2- 172.21.15.45:0/1076768933 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f49880fcd00 0x7f49880fd150 unknown :-1 s=CLOSED pgs=280 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:37:45.480 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:37:45.476+0000 7f498f3a7700 1 --2- 172.21.15.45:0/1076768933 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f49880fbaf0 0x7f49880fbf00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:37:45.480 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:37:45.476+0000 7f498f3a7700 1 -- 172.21.15.45:0/1076768933 >> 172.21.15.45:0/1076768933 conn(0x7f49880f7190 msgr2=0x7f49880f95e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-16T21:37:45.480 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:37:45.476+0000 7f498f3a7700 1 -- 172.21.15.45:0/1076768933 shutdown_connections 2023-12-16T21:37:45.480 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:37:45.476+0000 7f498f3a7700 1 -- 172.21.15.45:0/1076768933 wait complete. 2023-12-16T21:37:45.480 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:37:45.477+0000 7f498f3a7700 1 Processor -- start 2023-12-16T21:37:45.480 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:37:45.477+0000 7f498f3a7700 1 -- start start 2023-12-16T21:37:45.480 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:37:45.477+0000 7f498f3a7700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f49880fbaf0 0x7f498810d590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:37:45.480 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:37:45.477+0000 7f498f3a7700 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f49880fcd00 0x7f498810dad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:37:45.480 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:37:45.477+0000 7f498f3a7700 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f498810e0d0 con 0x7f49880fbaf0 2023-12-16T21:37:45.480 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:37:45.477+0000 7f498f3a7700 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f498810e970 con 0x7f49880fcd00 2023-12-16T21:37:45.480 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:37:45.477+0000 7f498d143700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f49880fbaf0 0x7f498810d590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:37:45.481 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:37:45.477+0000 7f498d143700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f49880fbaf0 0x7f498810d590 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:52120/0 (socket says 172.21.15.45:52120) 2023-12-16T21:37:45.481 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:37:45.477+0000 7f498d143700 1 -- 172.21.15.45:0/3950693556 learned_addr learned my addr 172.21.15.45:0/3950693556 (peer_addr_for_me v2:172.21.15.45:0/0) 2023-12-16T21:37:45.481 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:37:45.477+0000 7f498c942700 1 --2- 172.21.15.45:0/3950693556 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f49880fcd00 0x7f498810dad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:37:45.481 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:37:45.477+0000 7f498d143700 1 -- 172.21.15.45:0/3950693556 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f49880fcd00 msgr2=0x7f498810dad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:37:45.481 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:37:45.477+0000 7f498d143700 1 --2- 172.21.15.45:0/3950693556 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f49880fcd00 0x7f498810dad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:37:45.481 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:37:45.477+0000 7f498d143700 1 -- 172.21.15.45:0/3950693556 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4978005040 con 0x7f49880fbaf0 2023-12-16T21:37:45.482 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:37:45.477+0000 7f498d143700 1 --2- 172.21.15.45:0/3950693556 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f49880fbaf0 0x7f498810d590 secure :-1 s=READY pgs=281 cs=0 l=1 rev1=1 crypto rx=0x7f498400c800 tx=0x7f4984004470 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:37:45.482 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:37:45.478+0000 7f497dffb700 1 -- 172.21.15.45:0/3950693556 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f49840106f0 con 0x7f49880fbaf0 2023-12-16T21:37:45.483 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:37:45.478+0000 7f497dffb700 1 -- 172.21.15.45:0/3950693556 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f4984010d00 con 0x7f49880fbaf0 2023-12-16T21:37:45.483 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:37:45.478+0000 7f498f3a7700 1 -- 172.21.15.45:0/3950693556 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f498810ec50 con 0x7f49880fbaf0 2023-12-16T21:37:45.483 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:37:45.478+0000 7f497dffb700 1 -- 172.21.15.45:0/3950693556 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f49840113a0 con 0x7f49880fbaf0 2023-12-16T21:37:45.483 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:37:45.478+0000 7f498f3a7700 1 -- 172.21.15.45:0/3950693556 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f498810f190 con 0x7f49880fbaf0 2023-12-16T21:37:45.483 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:37:45.479+0000 7f498f3a7700 1 -- 172.21.15.45:0/3950693556 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f498804bb80 con 0x7f49880fbaf0 2023-12-16T21:37:45.483 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:37:45.479+0000 7f497dffb700 1 -- 172.21.15.45:0/3950693556 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 19) v1 ==== 86447+0+0 (secure 0 0 0) 0x7f4984012400 con 0x7f49880fbaf0 2023-12-16T21:37:45.483 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:37:45.480+0000 7f497dffb700 1 --2- 172.21.15.45:0/3950693556 >> [v2:172.21.15.45:6800/2012105536,v1:172.21.15.45:6801/2012105536] conn(0x7f4974068f70 0x7f497406b420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:37:45.483 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:37:45.480+0000 7f497dffb700 1 -- 172.21.15.45:0/3950693556 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(43..43 src has 1..43) v4 ==== 5550+0+0 (secure 0 0 0) 0x7f4984086d80 con 0x7f49880fbaf0 2023-12-16T21:37:45.483 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:37:45.480+0000 7f498c942700 1 --2- 172.21.15.45:0/3950693556 >> [v2:172.21.15.45:6800/2012105536,v1:172.21.15.45:6801/2012105536] conn(0x7f4974068f70 0x7f497406b420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:37:45.483 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:37:45.480+0000 7f498c942700 1 --2- 172.21.15.45:0/3950693556 >> [v2:172.21.15.45:6800/2012105536,v1:172.21.15.45:6801/2012105536] conn(0x7f4974068f70 0x7f497406b420 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7f497800b3b0 tx=0x7f4978009e30 comp rx=0 tx=0).ready entity=mgr.14214 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:37:45.486 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:37:45.484+0000 7f497dffb700 1 -- 172.21.15.45:0/3950693556 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+174570 (secure 0 0 0) 0x7f4984056c90 con 0x7f49880fbaf0 2023-12-16T21:37:45.683 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:37:45.681+0000 7f498f3a7700 1 -- 172.21.15.45:0/3950693556 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command([{prefix=config set, name=mon_warn_on_insecure_global_id_reclaim_allowed}] v 0) v1 -- 0x7f49881044b0 con 0x7f49880fbaf0 2023-12-16T21:37:45.684 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:37:45.681+0000 7f497dffb700 1 -- 172.21.15.45:0/3950693556 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mon_warn_on_insecure_global_id_reclaim_allowed}]=0 v21)=0 v21) v1 ==== 163+0+0 (secure 0 0 0) 0x7f4984056820 con 0x7f49880fbaf0 2023-12-16T21:37:45.686 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:37:45.683+0000 7f498f3a7700 1 -- 172.21.15.45:0/3950693556 >> [v2:172.21.15.45:6800/2012105536,v1:172.21.15.45:6801/2012105536] conn(0x7f4974068f70 msgr2=0x7f497406b420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:37:45.686 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:37:45.683+0000 7f498f3a7700 1 --2- 172.21.15.45:0/3950693556 >> [v2:172.21.15.45:6800/2012105536,v1:172.21.15.45:6801/2012105536] conn(0x7f4974068f70 0x7f497406b420 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7f497800b3b0 tx=0x7f4978009e30 comp rx=0 tx=0).stop 2023-12-16T21:37:45.686 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:37:45.683+0000 7f498f3a7700 1 -- 172.21.15.45:0/3950693556 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f49880fbaf0 msgr2=0x7f498810d590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:37:45.686 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:37:45.683+0000 7f498f3a7700 1 --2- 172.21.15.45:0/3950693556 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f49880fbaf0 0x7f498810d590 secure :-1 s=READY pgs=281 cs=0 l=1 rev1=1 crypto rx=0x7f498400c800 tx=0x7f4984004470 comp rx=0 tx=0).stop 2023-12-16T21:37:45.686 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:37:45.683+0000 7f498f3a7700 1 -- 172.21.15.45:0/3950693556 shutdown_connections 2023-12-16T21:37:45.686 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:37:45.683+0000 7f498f3a7700 1 --2- 172.21.15.45:0/3950693556 >> [v2:172.21.15.45:6800/2012105536,v1:172.21.15.45:6801/2012105536] conn(0x7f4974068f70 0x7f497406b420 unknown :-1 s=CLOSED pgs=110 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:37:45.686 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:37:45.683+0000 7f498f3a7700 1 --2- 172.21.15.45:0/3950693556 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f49880fbaf0 0x7f498810d590 unknown :-1 s=CLOSED pgs=281 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:37:45.686 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:37:45.683+0000 7f498f3a7700 1 --2- 172.21.15.45:0/3950693556 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f49880fcd00 0x7f498810dad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:37:45.686 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:37:45.683+0000 7f498f3a7700 1 -- 172.21.15.45:0/3950693556 >> 172.21.15.45:0/3950693556 conn(0x7f49880f7190 msgr2=0x7f4988065370 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-16T21:37:45.686 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:37:45.683+0000 7f498f3a7700 1 -- 172.21.15.45:0/3950693556 shutdown_connections 2023-12-16T21:37:45.686 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:37:45.683+0000 7f498f3a7700 1 -- 172.21.15.45:0/3950693556 wait complete. 2023-12-16T21:37:45.936 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Dec 16 21:37:45 smithi169 ceph-mon[88350]: pgmap v129: 65 pgs: 65 active+clean; 580 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 3.2 KiB/s rd, 341 B/s wr, 4 op/s 2023-12-16T21:37:46.066 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Dec 16 21:37:45 smithi045 ceph-mon[80763]: pgmap v129: 65 pgs: 65 active+clean; 580 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 3.2 KiB/s rd, 341 B/s wr, 4 op/s 2023-12-16T21:37:46.138 DEBUG:teuthology.orchestra.run.smithi045:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 88ff363c-9c5a-11ee-95a5-87774f69a715 -e sha1=c7586e95519b7be76f3b6e3a4b333888baf46bc9 -- bash -c 'ceph config set global log_to_journald false --force' 2023-12-16T21:37:46.406 INFO:teuthology.orchestra.run.smithi045.stderr:Inferring config /var/lib/ceph/88ff363c-9c5a-11ee-95a5-87774f69a715/mon.smithi045/config 2023-12-16T21:37:47.219 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:37:47.214+0000 7f7c38257700 1 -- 172.21.15.45:0/1507519299 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f7c300fcbf0 msgr2=0x7f7c300fd000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:37:47.219 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:37:47.214+0000 7f7c38257700 1 --2- 172.21.15.45:0/1507519299 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f7c300fcbf0 0x7f7c300fd000 secure :-1 s=READY pgs=282 cs=0 l=1 rev1=1 crypto rx=0x7f7c2c004940 tx=0x7f7c2c009e30 comp rx=0 tx=0).stop 2023-12-16T21:37:47.219 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:37:47.214+0000 7f7c38257700 1 -- 172.21.15.45:0/1507519299 shutdown_connections 2023-12-16T21:37:47.219 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:37:47.214+0000 7f7c38257700 1 --2- 172.21.15.45:0/1507519299 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f7c300fde00 0x7f7c300fe250 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:37:47.219 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:37:47.214+0000 7f7c38257700 1 --2- 172.21.15.45:0/1507519299 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f7c300fcbf0 0x7f7c300fd000 unknown :-1 s=CLOSED pgs=282 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:37:47.219 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:37:47.214+0000 7f7c38257700 1 -- 172.21.15.45:0/1507519299 >> 172.21.15.45:0/1507519299 conn(0x7f7c300f82d0 msgr2=0x7f7c300fa6e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-16T21:37:47.219 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:37:47.215+0000 7f7c38257700 1 -- 172.21.15.45:0/1507519299 shutdown_connections 2023-12-16T21:37:47.219 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:37:47.215+0000 7f7c38257700 1 -- 172.21.15.45:0/1507519299 wait complete. 2023-12-16T21:37:47.219 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:37:47.215+0000 7f7c38257700 1 Processor -- start 2023-12-16T21:37:47.219 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:37:47.215+0000 7f7c38257700 1 -- start start 2023-12-16T21:37:47.219 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:37:47.215+0000 7f7c38257700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f7c300fcbf0 0x7f7c30071b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:37:47.220 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:37:47.215+0000 7f7c38257700 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f7c300fde00 0x7f7c3006ff90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:37:47.220 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:37:47.215+0000 7f7c38257700 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7c30070590 con 0x7f7c300fcbf0 2023-12-16T21:37:47.220 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:37:47.215+0000 7f7c38257700 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7c30070700 con 0x7f7c300fde00 2023-12-16T21:37:47.220 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:37:47.215+0000 7f7c35ff3700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f7c300fcbf0 0x7f7c30071b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:37:47.220 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:37:47.215+0000 7f7c35ff3700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f7c300fcbf0 0x7f7c30071b10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:52138/0 (socket says 172.21.15.45:52138) 2023-12-16T21:37:47.220 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:37:47.215+0000 7f7c35ff3700 1 -- 172.21.15.45:0/171954197 learned_addr learned my addr 172.21.15.45:0/171954197 (peer_addr_for_me v2:172.21.15.45:0/0) 2023-12-16T21:37:47.220 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:37:47.216+0000 7f7c357f2700 1 --2- 172.21.15.45:0/171954197 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f7c300fde00 0x7f7c3006ff90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:37:47.220 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:37:47.216+0000 7f7c35ff3700 1 -- 172.21.15.45:0/171954197 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f7c300fde00 msgr2=0x7f7c3006ff90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:37:47.220 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:37:47.216+0000 7f7c35ff3700 1 --2- 172.21.15.45:0/171954197 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f7c300fde00 0x7f7c3006ff90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:37:47.220 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:37:47.216+0000 7f7c35ff3700 1 -- 172.21.15.45:0/171954197 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7c2c005040 con 0x7f7c300fcbf0 2023-12-16T21:37:47.220 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:37:47.216+0000 7f7c35ff3700 1 --2- 172.21.15.45:0/171954197 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f7c300fcbf0 0x7f7c30071b10 secure :-1 s=READY pgs=283 cs=0 l=1 rev1=1 crypto rx=0x7f7c2c004f90 tx=0x7f7c2c009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:37:47.220 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:37:47.216+0000 7f7c267fc700 1 -- 172.21.15.45:0/171954197 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f7c2c00e040 con 0x7f7c300fcbf0 2023-12-16T21:37:47.220 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:37:47.216+0000 7f7c267fc700 1 -- 172.21.15.45:0/171954197 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f7c2c00bc50 con 0x7f7c300fcbf0 2023-12-16T21:37:47.220 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:37:47.216+0000 7f7c38257700 1 -- 172.21.15.45:0/171954197 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7c30070980 con 0x7f7c300fcbf0 2023-12-16T21:37:47.220 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:37:47.216+0000 7f7c267fc700 1 -- 172.21.15.45:0/171954197 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f7c2c01c630 con 0x7f7c300fcbf0 2023-12-16T21:37:47.221 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:37:47.216+0000 7f7c38257700 1 -- 172.21.15.45:0/171954197 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7c30115590 con 0x7f7c300fcbf0 2023-12-16T21:37:47.223 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:37:47.217+0000 7f7c38257700 1 -- 172.21.15.45:0/171954197 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7c3004bb80 con 0x7f7c300fcbf0 2023-12-16T21:37:47.223 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:37:47.218+0000 7f7c267fc700 1 -- 172.21.15.45:0/171954197 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 19) v1 ==== 86447+0+0 (secure 0 0 0) 0x7f7c2c013420 con 0x7f7c300fcbf0 2023-12-16T21:37:47.223 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:37:47.218+0000 7f7c267fc700 1 --2- 172.21.15.45:0/171954197 >> [v2:172.21.15.45:6800/2012105536,v1:172.21.15.45:6801/2012105536] conn(0x7f7c1c068f60 0x7f7c1c06b410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:37:47.223 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:37:47.218+0000 7f7c267fc700 1 -- 172.21.15.45:0/171954197 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(43..43 src has 1..43) v4 ==== 5550+0+0 (secure 0 0 0) 0x7f7c2c0881a0 con 0x7f7c300fcbf0 2023-12-16T21:37:47.223 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:37:47.218+0000 7f7c357f2700 1 --2- 172.21.15.45:0/171954197 >> [v2:172.21.15.45:6800/2012105536,v1:172.21.15.45:6801/2012105536] conn(0x7f7c1c068f60 0x7f7c1c06b410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:37:47.223 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:37:47.219+0000 7f7c357f2700 1 --2- 172.21.15.45:0/171954197 >> [v2:172.21.15.45:6800/2012105536,v1:172.21.15.45:6801/2012105536] conn(0x7f7c1c068f60 0x7f7c1c06b410 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7f7c20004f40 tx=0x7f7c2000b040 comp rx=0 tx=0).ready entity=mgr.14214 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:37:47.225 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:37:47.222+0000 7f7c267fc700 1 -- 172.21.15.45:0/171954197 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+174570 (secure 0 0 0) 0x7f7c2c0580b0 con 0x7f7c300fcbf0 2023-12-16T21:37:47.383 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Dec 16 21:37:47 smithi169 ceph-mon[88350]: pgmap v130: 65 pgs: 65 active+clean; 580 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 938 B/s rd, 426 B/s wr, 2 op/s 2023-12-16T21:37:47.422 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:37:47.420+0000 7f7c38257700 1 -- 172.21.15.45:0/171954197 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command([{prefix=config set, name=log_to_journald}] v 0) v1 -- 0x7f7c30105590 con 0x7f7c300fcbf0 2023-12-16T21:37:47.422 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Dec 16 21:37:47 smithi045 ceph-mon[80763]: pgmap v130: 65 pgs: 65 active+clean; 580 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 938 B/s rd, 426 B/s wr, 2 op/s 2023-12-16T21:37:47.423 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:37:47.420+0000 7f7c267fc700 1 -- 172.21.15.45:0/171954197 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mon_command_ack([{prefix=config set, name=log_to_journald}]=0 v21)=0 v21) v1 ==== 135+0+0 (secure 0 0 0) 0x7f7c2c057c40 con 0x7f7c300fcbf0 2023-12-16T21:37:47.425 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:37:47.422+0000 7f7c38257700 1 -- 172.21.15.45:0/171954197 >> [v2:172.21.15.45:6800/2012105536,v1:172.21.15.45:6801/2012105536] conn(0x7f7c1c068f60 msgr2=0x7f7c1c06b410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:37:47.425 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:37:47.422+0000 7f7c38257700 1 --2- 172.21.15.45:0/171954197 >> [v2:172.21.15.45:6800/2012105536,v1:172.21.15.45:6801/2012105536] conn(0x7f7c1c068f60 0x7f7c1c06b410 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7f7c20004f40 tx=0x7f7c2000b040 comp rx=0 tx=0).stop 2023-12-16T21:37:47.425 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:37:47.422+0000 7f7c38257700 1 -- 172.21.15.45:0/171954197 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f7c300fcbf0 msgr2=0x7f7c30071b10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:37:47.425 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:37:47.422+0000 7f7c38257700 1 --2- 172.21.15.45:0/171954197 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f7c300fcbf0 0x7f7c30071b10 secure :-1 s=READY pgs=283 cs=0 l=1 rev1=1 crypto rx=0x7f7c2c004f90 tx=0x7f7c2c009e30 comp rx=0 tx=0).stop 2023-12-16T21:37:47.425 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:37:47.422+0000 7f7c38257700 1 -- 172.21.15.45:0/171954197 shutdown_connections 2023-12-16T21:37:47.425 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:37:47.422+0000 7f7c38257700 1 --2- 172.21.15.45:0/171954197 >> [v2:172.21.15.45:6800/2012105536,v1:172.21.15.45:6801/2012105536] conn(0x7f7c1c068f60 0x7f7c1c06b410 unknown :-1 s=CLOSED pgs=111 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:37:47.425 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:37:47.422+0000 7f7c38257700 1 --2- 172.21.15.45:0/171954197 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f7c300fcbf0 0x7f7c30071b10 unknown :-1 s=CLOSED pgs=283 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:37:47.425 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:37:47.422+0000 7f7c38257700 1 --2- 172.21.15.45:0/171954197 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f7c300fde00 0x7f7c3006ff90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:37:47.425 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:37:47.422+0000 7f7c38257700 1 -- 172.21.15.45:0/171954197 >> 172.21.15.45:0/171954197 conn(0x7f7c300f82d0 msgr2=0x7f7c30101030 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-16T21:37:47.425 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:37:47.422+0000 7f7c38257700 1 -- 172.21.15.45:0/171954197 shutdown_connections 2023-12-16T21:37:47.425 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:37:47.422+0000 7f7c38257700 1 -- 172.21.15.45:0/171954197 wait complete. 2023-12-16T21:37:47.697 DEBUG:teuthology.orchestra.run.smithi045:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 88ff363c-9c5a-11ee-95a5-87774f69a715 -e sha1=c7586e95519b7be76f3b6e3a4b333888baf46bc9 -- bash -c 'ceph orch upgrade start --image quay.ceph.io/ceph-ci/ceph:$sha1 --daemon-types mgr' 2023-12-16T21:37:47.964 INFO:teuthology.orchestra.run.smithi045.stderr:Inferring config /var/lib/ceph/88ff363c-9c5a-11ee-95a5-87774f69a715/mon.smithi045/config 2023-12-16T21:37:48.748 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:37:48.744+0000 7fdf85739700 1 -- 172.21.15.45:0/758877145 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fdf800ffd50 msgr2=0x7fdf801021d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:37:48.748 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:37:48.744+0000 7fdf85739700 1 --2- 172.21.15.45:0/758877145 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fdf800ffd50 0x7fdf801021d0 secure :-1 s=READY pgs=284 cs=0 l=1 rev1=1 crypto rx=0x7fdf74004940 tx=0x7fdf74009e30 comp rx=0 tx=0).stop 2023-12-16T21:37:48.748 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:37:48.744+0000 7fdf85739700 1 -- 172.21.15.45:0/758877145 shutdown_connections 2023-12-16T21:37:48.748 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:37:48.744+0000 7fdf85739700 1 --2- 172.21.15.45:0/758877145 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fdf800ffd50 0x7fdf801021d0 unknown :-1 s=CLOSED pgs=284 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:37:48.748 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:37:48.744+0000 7fdf85739700 1 --2- 172.21.15.45:0/758877145 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fdf800660b0 0x7fdf800ff780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:37:48.748 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:37:48.744+0000 7fdf85739700 1 -- 172.21.15.45:0/758877145 >> 172.21.15.45:0/758877145 conn(0x7fdf800f7190 msgr2=0x7fdf800f95e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-16T21:37:48.748 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:37:48.745+0000 7fdf85739700 1 -- 172.21.15.45:0/758877145 shutdown_connections 2023-12-16T21:37:48.749 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:37:48.745+0000 7fdf85739700 1 -- 172.21.15.45:0/758877145 wait complete. 2023-12-16T21:37:48.749 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:37:48.745+0000 7fdf85739700 1 Processor -- start 2023-12-16T21:37:48.749 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:37:48.745+0000 7fdf85739700 1 -- start start 2023-12-16T21:37:48.749 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:37:48.745+0000 7fdf85739700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fdf800660b0 0x7fdf8010d510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:37:48.749 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:37:48.745+0000 7fdf85739700 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fdf800ffd50 0x7fdf8010da50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:37:48.749 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:37:48.745+0000 7fdf85739700 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdf8010e050 con 0x7fdf800660b0 2023-12-16T21:37:48.749 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:37:48.745+0000 7fdf85739700 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdf8010e8f0 con 0x7fdf800ffd50 2023-12-16T21:37:48.749 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:37:48.746+0000 7fdf7effd700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fdf800660b0 0x7fdf8010d510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:37:48.749 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:37:48.746+0000 7fdf7effd700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fdf800660b0 0x7fdf8010d510 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:52152/0 (socket says 172.21.15.45:52152) 2023-12-16T21:37:48.749 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:37:48.746+0000 7fdf7effd700 1 -- 172.21.15.45:0/4028991436 learned_addr learned my addr 172.21.15.45:0/4028991436 (peer_addr_for_me v2:172.21.15.45:0/0) 2023-12-16T21:37:48.749 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:37:48.746+0000 7fdf7e7fc700 1 --2- 172.21.15.45:0/4028991436 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fdf800ffd50 0x7fdf8010da50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:37:48.750 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:37:48.746+0000 7fdf7effd700 1 -- 172.21.15.45:0/4028991436 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fdf800ffd50 msgr2=0x7fdf8010da50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:37:48.750 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:37:48.746+0000 7fdf7effd700 1 --2- 172.21.15.45:0/4028991436 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fdf800ffd50 0x7fdf8010da50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:37:48.750 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:37:48.746+0000 7fdf7effd700 1 -- 172.21.15.45:0/4028991436 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdf74005040 con 0x7fdf800660b0 2023-12-16T21:37:48.750 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:37:48.746+0000 7fdf7effd700 1 --2- 172.21.15.45:0/4028991436 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fdf800660b0 0x7fdf8010d510 secure :-1 s=READY pgs=285 cs=0 l=1 rev1=1 crypto rx=0x7fdf70002a50 tx=0x7fdf700046b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:37:48.751 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:37:48.746+0000 7fdf677fe700 1 -- 172.21.15.45:0/4028991436 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fdf70007cc0 con 0x7fdf800660b0 2023-12-16T21:37:48.751 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:37:48.746+0000 7fdf85739700 1 -- 172.21.15.45:0/4028991436 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdf8010ebd0 con 0x7fdf800660b0 2023-12-16T21:37:48.751 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:37:48.746+0000 7fdf677fe700 1 -- 172.21.15.45:0/4028991436 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fdf7000f040 con 0x7fdf800660b0 2023-12-16T21:37:48.751 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:37:48.746+0000 7fdf677fe700 1 -- 172.21.15.45:0/4028991436 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fdf7000d9c0 con 0x7fdf800660b0 2023-12-16T21:37:48.751 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:37:48.747+0000 7fdf85739700 1 -- 172.21.15.45:0/4028991436 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdf8010f110 con 0x7fdf800660b0 2023-12-16T21:37:48.751 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:37:48.747+0000 7fdf85739700 1 -- 172.21.15.45:0/4028991436 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdf801075c0 con 0x7fdf800660b0 2023-12-16T21:37:48.751 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:37:48.748+0000 7fdf677fe700 1 -- 172.21.15.45:0/4028991436 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 19) v1 ==== 86447+0+0 (secure 0 0 0) 0x7fdf7000db20 con 0x7fdf800660b0 2023-12-16T21:37:48.752 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:37:48.748+0000 7fdf677fe700 1 --2- 172.21.15.45:0/4028991436 >> [v2:172.21.15.45:6800/2012105536,v1:172.21.15.45:6801/2012105536] conn(0x7fdf68068f60 0x7fdf6806b410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:37:48.752 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:37:48.749+0000 7fdf677fe700 1 -- 172.21.15.45:0/4028991436 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(43..43 src has 1..43) v4 ==== 5550+0+0 (secure 0 0 0) 0x7fdf70086a30 con 0x7fdf800660b0 2023-12-16T21:37:48.752 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:37:48.749+0000 7fdf7e7fc700 1 --2- 172.21.15.45:0/4028991436 >> [v2:172.21.15.45:6800/2012105536,v1:172.21.15.45:6801/2012105536] conn(0x7fdf68068f60 0x7fdf6806b410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:37:48.752 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:37:48.749+0000 7fdf7e7fc700 1 --2- 172.21.15.45:0/4028991436 >> [v2:172.21.15.45:6800/2012105536,v1:172.21.15.45:6801/2012105536] conn(0x7fdf68068f60 0x7fdf6806b410 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7fdf7400b3b0 tx=0x7fdf74009e30 comp rx=0 tx=0).ready entity=mgr.14214 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:37:48.755 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:37:48.753+0000 7fdf677fe700 1 -- 172.21.15.45:0/4028991436 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+174570 (secure 0 0 0) 0x7fdf70056940 con 0x7fdf800660b0 2023-12-16T21:37:48.976 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:37:48.973+0000 7fdf85739700 1 -- 172.21.15.45:0/4028991436 --> [v2:172.21.15.45:6800/2012105536,v1:172.21.15.45:6801/2012105536] -- mgr_command(tid 0: {"prefix": "orch upgrade start", "image": "quay.ceph.io/ceph-ci/ceph:c7586e95519b7be76f3b6e3a4b333888baf46bc9", "daemon_types": "mgr", "target": ["mon-mgr", ""]}) v1 -- 0x7fdf8010f4f0 con 0x7fdf68068f60 2023-12-16T21:37:50.066 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Dec 16 21:37:49 smithi045 ceph-mon[80763]: pgmap v131: 65 pgs: 65 active+clean; 584 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 1.8 KiB/s rd, 767 B/s wr, 3 op/s 2023-12-16T21:37:50.133 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Dec 16 21:37:49 smithi169 ceph-mon[88350]: pgmap v131: 65 pgs: 65 active+clean; 584 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 1.8 KiB/s rd, 767 B/s wr, 3 op/s 2023-12-16T21:37:51.066 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Dec 16 21:37:50 smithi045 ceph-mon[80763]: from='client.14552 -' entity='client.admin' cmd=[{"prefix": "orch upgrade start", "image": "quay.ceph.io/ceph-ci/ceph:c7586e95519b7be76f3b6e3a4b333888baf46bc9", "daemon_types": "mgr", "target": ["mon-mgr", ""]}]: dispatch 2023-12-16T21:37:51.133 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Dec 16 21:37:50 smithi169 ceph-mon[88350]: from='client.14552 -' entity='client.admin' cmd=[{"prefix": "orch upgrade start", "image": "quay.ceph.io/ceph-ci/ceph:c7586e95519b7be76f3b6e3a4b333888baf46bc9", "daemon_types": "mgr", "target": ["mon-mgr", ""]}]: dispatch 2023-12-16T21:37:51.883 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Dec 16 21:37:51 smithi169 ceph-mon[88350]: pgmap v132: 65 pgs: 65 active+clean; 584 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 1.4 KiB/s rd, 767 B/s wr, 2 op/s 2023-12-16T21:37:52.066 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Dec 16 21:37:51 smithi045 ceph-mon[80763]: pgmap v132: 65 pgs: 65 active+clean; 584 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 1.4 KiB/s rd, 767 B/s wr, 2 op/s 2023-12-16T21:37:54.066 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Dec 16 21:37:53 smithi045 ceph-mon[80763]: pgmap v133: 65 pgs: 65 active+clean; 584 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 1.4 KiB/s rd, 767 B/s wr, 2 op/s 2023-12-16T21:37:54.133 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Dec 16 21:37:53 smithi169 ceph-mon[88350]: pgmap v133: 65 pgs: 65 active+clean; 584 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 1.4 KiB/s rd, 767 B/s wr, 2 op/s 2023-12-16T21:37:56.066 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Dec 16 21:37:55 smithi045 ceph-mon[80763]: pgmap v134: 65 pgs: 65 active+clean; 584 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 1.8 KiB/s rd, 767 B/s wr, 3 op/s 2023-12-16T21:37:56.133 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Dec 16 21:37:55 smithi169 ceph-mon[88350]: pgmap v134: 65 pgs: 65 active+clean; 584 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 1.8 KiB/s rd, 767 B/s wr, 3 op/s 2023-12-16T21:37:57.566 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Dec 16 21:37:57 smithi045 ceph-mon[80763]: pgmap v135: 65 pgs: 65 active+clean; 584 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 1.3 KiB/s rd, 426 B/s wr, 2 op/s 2023-12-16T21:37:57.633 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Dec 16 21:37:57 smithi169 ceph-mon[88350]: pgmap v135: 65 pgs: 65 active+clean; 584 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 1.3 KiB/s rd, 426 B/s wr, 2 op/s 2023-12-16T21:37:59.066 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Dec 16 21:37:58 smithi045 ceph-mon[80763]: from='mgr.14214 172.21.15.45:0/1227909437' entity='mgr.smithi045.nhtmwc' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2023-12-16T21:37:59.133 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Dec 16 21:37:58 smithi169 ceph-mon[88350]: from='mgr.14214 172.21.15.45:0/1227909437' entity='mgr.smithi045.nhtmwc' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2023-12-16T21:38:00.066 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Dec 16 21:37:59 smithi045 ceph-mon[80763]: pgmap v136: 65 pgs: 65 active+clean; 584 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 341 B/s wr, 2 op/s 2023-12-16T21:38:00.133 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Dec 16 21:37:59 smithi169 ceph-mon[88350]: pgmap v136: 65 pgs: 65 active+clean; 584 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 341 B/s wr, 2 op/s 2023-12-16T21:38:02.066 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Dec 16 21:38:01 smithi045 ceph-mon[80763]: pgmap v137: 65 pgs: 65 active+clean; 584 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 853 B/s rd, 1 op/s 2023-12-16T21:38:02.133 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Dec 16 21:38:01 smithi169 ceph-mon[88350]: pgmap v137: 65 pgs: 65 active+clean; 584 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 853 B/s rd, 1 op/s 2023-12-16T21:38:04.069 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Dec 16 21:38:03 smithi045 ceph-mon[80763]: pgmap v138: 65 pgs: 65 active+clean; 584 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 853 B/s rd, 1 op/s 2023-12-16T21:38:04.133 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Dec 16 21:38:03 smithi169 ceph-mon[88350]: pgmap v138: 65 pgs: 65 active+clean; 584 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 853 B/s rd, 1 op/s 2023-12-16T21:38:06.066 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Dec 16 21:38:05 smithi045 ceph-mon[80763]: pgmap v139: 65 pgs: 65 active+clean; 584 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 1.2 KiB/s rd, 2 op/s 2023-12-16T21:38:06.133 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Dec 16 21:38:05 smithi169 ceph-mon[88350]: pgmap v139: 65 pgs: 65 active+clean; 584 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 1.2 KiB/s rd, 2 op/s 2023-12-16T21:38:07.566 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Dec 16 21:38:07 smithi045 ceph-mon[80763]: pgmap v140: 65 pgs: 65 active+clean; 584 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 853 B/s rd, 1 op/s 2023-12-16T21:38:07.633 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Dec 16 21:38:07 smithi169 ceph-mon[88350]: pgmap v140: 65 pgs: 65 active+clean; 584 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 853 B/s rd, 1 op/s 2023-12-16T21:38:10.066 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Dec 16 21:38:09 smithi045 ceph-mon[80763]: pgmap v141: 65 pgs: 65 active+clean; 584 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 1.2 KiB/s rd, 2 op/s 2023-12-16T21:38:10.133 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Dec 16 21:38:09 smithi169 ceph-mon[88350]: pgmap v141: 65 pgs: 65 active+clean; 584 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 1.2 KiB/s rd, 2 op/s 2023-12-16T21:38:12.066 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Dec 16 21:38:11 smithi045 ceph-mon[80763]: pgmap v142: 65 pgs: 65 active+clean; 584 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 853 B/s rd, 1 op/s 2023-12-16T21:38:12.133 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Dec 16 21:38:11 smithi169 ceph-mon[88350]: pgmap v142: 65 pgs: 65 active+clean; 584 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 853 B/s rd, 1 op/s 2023-12-16T21:38:14.066 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Dec 16 21:38:13 smithi045 ceph-mon[80763]: pgmap v143: 65 pgs: 65 active+clean; 584 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 853 B/s rd, 1 op/s 2023-12-16T21:38:14.066 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Dec 16 21:38:13 smithi045 ceph-mon[80763]: from='mgr.14214 172.21.15.45:0/1227909437' entity='mgr.smithi045.nhtmwc' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2023-12-16T21:38:14.133 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Dec 16 21:38:13 smithi169 ceph-mon[88350]: pgmap v143: 65 pgs: 65 active+clean; 584 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 853 B/s rd, 1 op/s 2023-12-16T21:38:14.133 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Dec 16 21:38:13 smithi169 ceph-mon[88350]: from='mgr.14214 172.21.15.45:0/1227909437' entity='mgr.smithi045.nhtmwc' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2023-12-16T21:38:15.940 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Dec 16 21:38:15 smithi169 ceph-mon[88350]: pgmap v144: 65 pgs: 65 active+clean; 584 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 1.2 KiB/s rd, 2 op/s 2023-12-16T21:38:16.066 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Dec 16 21:38:15 smithi045 ceph-mon[80763]: pgmap v144: 65 pgs: 65 active+clean; 584 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 1.2 KiB/s rd, 2 op/s 2023-12-16T21:38:17.341 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:38:17.338+0000 7fdf677fe700 1 -- 172.21.15.45:0/4028991436 <== mgr.14214 v2:172.21.15.45:6800/2012105536 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+89 (secure 0 0 0) 0x7fdf8010f4f0 con 0x7fdf68068f60 2023-12-16T21:38:17.341 INFO:teuthology.orchestra.run.smithi045.stdout:Initiating upgrade to quay.ceph.io/ceph-ci/ceph:c7586e95519b7be76f3b6e3a4b333888baf46bc9 2023-12-16T21:38:17.343 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:38:17.340+0000 7fdf85739700 1 -- 172.21.15.45:0/4028991436 >> [v2:172.21.15.45:6800/2012105536,v1:172.21.15.45:6801/2012105536] conn(0x7fdf68068f60 msgr2=0x7fdf6806b410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:38:17.343 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:38:17.340+0000 7fdf85739700 1 --2- 172.21.15.45:0/4028991436 >> [v2:172.21.15.45:6800/2012105536,v1:172.21.15.45:6801/2012105536] conn(0x7fdf68068f60 0x7fdf6806b410 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7fdf7400b3b0 tx=0x7fdf74009e30 comp rx=0 tx=0).stop 2023-12-16T21:38:17.343 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:38:17.340+0000 7fdf85739700 1 -- 172.21.15.45:0/4028991436 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fdf800660b0 msgr2=0x7fdf8010d510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:38:17.343 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:38:17.340+0000 7fdf85739700 1 --2- 172.21.15.45:0/4028991436 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fdf800660b0 0x7fdf8010d510 secure :-1 s=READY pgs=285 cs=0 l=1 rev1=1 crypto rx=0x7fdf70002a50 tx=0x7fdf700046b0 comp rx=0 tx=0).stop 2023-12-16T21:38:17.343 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:38:17.341+0000 7fdf85739700 1 -- 172.21.15.45:0/4028991436 shutdown_connections 2023-12-16T21:38:17.343 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:38:17.341+0000 7fdf85739700 1 --2- 172.21.15.45:0/4028991436 >> [v2:172.21.15.45:6800/2012105536,v1:172.21.15.45:6801/2012105536] conn(0x7fdf68068f60 0x7fdf6806b410 unknown :-1 s=CLOSED pgs=112 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:38:17.344 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:38:17.341+0000 7fdf85739700 1 --2- 172.21.15.45:0/4028991436 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fdf800660b0 0x7fdf8010d510 unknown :-1 s=CLOSED pgs=285 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:38:17.344 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:38:17.341+0000 7fdf85739700 1 --2- 172.21.15.45:0/4028991436 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fdf800ffd50 0x7fdf8010da50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:38:17.344 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:38:17.341+0000 7fdf85739700 1 -- 172.21.15.45:0/4028991436 >> 172.21.15.45:0/4028991436 conn(0x7fdf800f7190 msgr2=0x7fdf800f95e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-16T21:38:17.344 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:38:17.341+0000 7fdf85739700 1 -- 172.21.15.45:0/4028991436 shutdown_connections 2023-12-16T21:38:17.344 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:38:17.341+0000 7fdf85739700 1 -- 172.21.15.45:0/4028991436 wait complete. 2023-12-16T21:38:17.566 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Dec 16 21:38:17 smithi045 ceph-mon[80763]: pgmap v145: 65 pgs: 65 active+clean; 584 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 853 B/s rd, 1 op/s 2023-12-16T21:38:17.633 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Dec 16 21:38:17 smithi169 ceph-mon[88350]: pgmap v145: 65 pgs: 65 active+clean; 584 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 853 B/s rd, 1 op/s 2023-12-16T21:38:17.658 DEBUG:teuthology.orchestra.run.smithi045:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 88ff363c-9c5a-11ee-95a5-87774f69a715 -e sha1=c7586e95519b7be76f3b6e3a4b333888baf46bc9 -- bash -c 'while ceph orch upgrade status | jq '"'"'.in_progress'"'"' | grep true && ! ceph orch upgrade status | jq '"'"'.message'"'"' | grep Error ; do ceph orch ps ; ceph versions ; ceph orch upgrade status ; sleep 30 ; done' 2023-12-16T21:38:17.997 INFO:teuthology.orchestra.run.smithi045.stderr:Inferring config /var/lib/ceph/88ff363c-9c5a-11ee-95a5-87774f69a715/mon.smithi045/config 2023-12-16T21:38:18.492 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Dec 16 21:38:18 smithi045 ceph-mon[80763]: Upgrade: Started with target quay.ceph.io/ceph-ci/ceph:c7586e95519b7be76f3b6e3a4b333888baf46bc9 2023-12-16T21:38:18.493 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Dec 16 21:38:18 smithi045 ceph-mon[80763]: from='mgr.14214 172.21.15.45:0/1227909437' entity='mgr.smithi045.nhtmwc' 2023-12-16T21:38:18.493 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Dec 16 21:38:18 smithi045 ceph-mon[80763]: from='mgr.14214 172.21.15.45:0/1227909437' entity='mgr.smithi045.nhtmwc' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-12-16T21:38:18.493 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Dec 16 21:38:18 smithi045 ceph-mon[80763]: from='mgr.14214 172.21.15.45:0/1227909437' entity='mgr.smithi045.nhtmwc' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-12-16T21:38:18.493 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Dec 16 21:38:18 smithi045 ceph-mon[80763]: from='mgr.14214 172.21.15.45:0/1227909437' entity='mgr.smithi045.nhtmwc' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-12-16T21:38:18.493 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Dec 16 21:38:18 smithi045 ceph-mon[80763]: from='mgr.14214 172.21.15.45:0/1227909437' entity='mgr.smithi045.nhtmwc' 2023-12-16T21:38:18.633 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Dec 16 21:38:18 smithi169 ceph-mon[88350]: Upgrade: Started with target quay.ceph.io/ceph-ci/ceph:c7586e95519b7be76f3b6e3a4b333888baf46bc9 2023-12-16T21:38:18.633 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Dec 16 21:38:18 smithi169 ceph-mon[88350]: from='mgr.14214 172.21.15.45:0/1227909437' entity='mgr.smithi045.nhtmwc' 2023-12-16T21:38:18.634 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Dec 16 21:38:18 smithi169 ceph-mon[88350]: from='mgr.14214 172.21.15.45:0/1227909437' entity='mgr.smithi045.nhtmwc' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-12-16T21:38:18.634 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Dec 16 21:38:18 smithi169 ceph-mon[88350]: from='mgr.14214 172.21.15.45:0/1227909437' entity='mgr.smithi045.nhtmwc' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-12-16T21:38:18.634 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Dec 16 21:38:18 smithi169 ceph-mon[88350]: from='mgr.14214 172.21.15.45:0/1227909437' entity='mgr.smithi045.nhtmwc' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-12-16T21:38:18.634 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Dec 16 21:38:18 smithi169 ceph-mon[88350]: from='mgr.14214 172.21.15.45:0/1227909437' entity='mgr.smithi045.nhtmwc' 2023-12-16T21:38:18.866 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:38:18.861+0000 7fbe7c92e700 1 -- 172.21.15.45:0/3055203455 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fbe78100050 msgr2=0x7fbe78102430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:38:18.866 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:38:18.861+0000 7fbe7c92e700 1 --2- 172.21.15.45:0/3055203455 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fbe78100050 0x7fbe78102430 secure :-1 s=READY pgs=286 cs=0 l=1 rev1=1 crypto rx=0x7fbe6c004940 tx=0x7fbe6c009e30 comp rx=0 tx=0).stop 2023-12-16T21:38:18.866 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:38:18.861+0000 7fbe7c92e700 1 -- 172.21.15.45:0/3055203455 shutdown_connections 2023-12-16T21:38:18.866 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:38:18.861+0000 7fbe7c92e700 1 --2- 172.21.15.45:0/3055203455 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fbe78100050 0x7fbe78102430 unknown :-1 s=CLOSED pgs=286 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:38:18.866 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:38:18.861+0000 7fbe7c92e700 1 --2- 172.21.15.45:0/3055203455 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fbe780fd730 0x7fbe780ffb10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:38:18.867 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:38:18.861+0000 7fbe7c92e700 1 -- 172.21.15.45:0/3055203455 >> 172.21.15.45:0/3055203455 conn(0x7fbe780f7190 msgr2=0x7fbe780f95e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-16T21:38:18.867 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:38:18.861+0000 7fbe7c92e700 1 -- 172.21.15.45:0/3055203455 shutdown_connections 2023-12-16T21:38:18.867 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:38:18.861+0000 7fbe7c92e700 1 -- 172.21.15.45:0/3055203455 wait complete. 2023-12-16T21:38:18.867 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:38:18.862+0000 7fbe7c92e700 1 Processor -- start 2023-12-16T21:38:18.867 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:38:18.862+0000 7fbe7c92e700 1 -- start start 2023-12-16T21:38:18.867 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:38:18.862+0000 7fbe7c92e700 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fbe780fd730 0x7fbe7810d4a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:38:18.867 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:38:18.862+0000 7fbe7c92e700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fbe78100050 0x7fbe7810d9e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:38:18.868 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:38:18.863+0000 7fbe75d9b700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fbe78100050 0x7fbe7810d9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:38:18.868 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:38:18.863+0000 7fbe7659c700 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fbe780fd730 0x7fbe7810d4a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:38:18.868 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:38:18.863+0000 7fbe7c92e700 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbe7810dfe0 con 0x7fbe78100050 2023-12-16T21:38:18.868 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:38:18.863+0000 7fbe7c92e700 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbe7810e880 con 0x7fbe780fd730 2023-12-16T21:38:18.868 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:38:18.863+0000 7fbe7659c700 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fbe780fd730 0x7fbe7810d4a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.169:3300/0 says I am v2:172.21.15.45:59482/0 (socket says 172.21.15.45:59482) 2023-12-16T21:38:18.868 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:38:18.863+0000 7fbe75d9b700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fbe78100050 0x7fbe7810d9e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:47980/0 (socket says 172.21.15.45:47980) 2023-12-16T21:38:18.869 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:38:18.863+0000 7fbe7659c700 1 -- 172.21.15.45:0/3645174068 learned_addr learned my addr 172.21.15.45:0/3645174068 (peer_addr_for_me v2:172.21.15.45:0/0) 2023-12-16T21:38:18.869 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:38:18.863+0000 7fbe75d9b700 1 -- 172.21.15.45:0/3645174068 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fbe780fd730 msgr2=0x7fbe7810d4a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:38:18.869 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:38:18.863+0000 7fbe75d9b700 1 --2- 172.21.15.45:0/3645174068 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fbe780fd730 0x7fbe7810d4a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:38:18.869 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:38:18.863+0000 7fbe75d9b700 1 -- 172.21.15.45:0/3645174068 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbe6c005040 con 0x7fbe78100050 2023-12-16T21:38:18.869 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:38:18.863+0000 7fbe7659c700 1 --2- 172.21.15.45:0/3645174068 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fbe780fd730 0x7fbe7810d4a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-12-16T21:38:18.869 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:38:18.863+0000 7fbe75d9b700 1 --2- 172.21.15.45:0/3645174068 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fbe78100050 0x7fbe7810d9e0 secure :-1 s=READY pgs=287 cs=0 l=1 rev1=1 crypto rx=0x7fbe6c000c00 tx=0x7fbe6c009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:38:18.871 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:38:18.864+0000 7fbe66ffd700 1 -- 172.21.15.45:0/3645174068 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fbe6c00e030 con 0x7fbe78100050 2023-12-16T21:38:18.872 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:38:18.864+0000 7fbe66ffd700 1 -- 172.21.15.45:0/3645174068 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fbe6c01a570 con 0x7fbe78100050 2023-12-16T21:38:18.872 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:38:18.864+0000 7fbe7c92e700 1 -- 172.21.15.45:0/3645174068 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbe7810eb00 con 0x7fbe78100050 2023-12-16T21:38:18.872 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:38:18.864+0000 7fbe7c92e700 1 -- 172.21.15.45:0/3645174068 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbe7810ef80 con 0x7fbe78100050 2023-12-16T21:38:18.872 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:38:18.864+0000 7fbe66ffd700 1 -- 172.21.15.45:0/3645174068 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fbe6c01f070 con 0x7fbe78100050 2023-12-16T21:38:18.872 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:38:18.865+0000 7fbe7c92e700 1 -- 172.21.15.45:0/3645174068 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbe7804bb80 con 0x7fbe78100050 2023-12-16T21:38:18.872 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:38:18.866+0000 7fbe66ffd700 1 -- 172.21.15.45:0/3645174068 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 19) v1 ==== 86447+0+0 (secure 0 0 0) 0x7fbe6c0256b0 con 0x7fbe78100050 2023-12-16T21:38:18.872 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:38:18.867+0000 7fbe66ffd700 1 --2- 172.21.15.45:0/3645174068 >> [v2:172.21.15.45:6800/2012105536,v1:172.21.15.45:6801/2012105536] conn(0x7fbe60069020 0x7fbe6006b4d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:38:18.872 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:38:18.867+0000 7fbe66ffd700 1 -- 172.21.15.45:0/3645174068 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(43..43 src has 1..43) v4 ==== 5550+0+0 (secure 0 0 0) 0x7fbe6c08c360 con 0x7fbe78100050 2023-12-16T21:38:18.873 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:38:18.867+0000 7fbe7659c700 1 --2- 172.21.15.45:0/3645174068 >> [v2:172.21.15.45:6800/2012105536,v1:172.21.15.45:6801/2012105536] conn(0x7fbe60069020 0x7fbe6006b4d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:38:18.873 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:38:18.867+0000 7fbe7659c700 1 --2- 172.21.15.45:0/3645174068 >> [v2:172.21.15.45:6800/2012105536,v1:172.21.15.45:6801/2012105536] conn(0x7fbe60069020 0x7fbe6006b4d0 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7fbe680041d0 tx=0x7fbe6800b040 comp rx=0 tx=0).ready entity=mgr.14214 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:38:18.874 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:38:18.871+0000 7fbe66ffd700 1 -- 172.21.15.45:0/3645174068 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+174570 (secure 0 0 0) 0x7fbe6c05c2c0 con 0x7fbe78100050 2023-12-16T21:38:19.197 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:38:19.194+0000 7fbe7c92e700 1 -- 172.21.15.45:0/3645174068 --> [v2:172.21.15.45:6800/2012105536,v1:172.21.15.45:6801/2012105536] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7fbe7810f470 con 0x7fbe60069020 2023-12-16T21:38:19.198 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:38:19.196+0000 7fbe66ffd700 1 -- 172.21.15.45:0/3645174068 <== mgr.14214 v2:172.21.15.45:6800/2012105536 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+351 (secure 0 0 0) 0x7fbe7810f470 con 0x7fbe60069020 2023-12-16T21:38:19.201 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:38:19.198+0000 7fbe7c92e700 1 -- 172.21.15.45:0/3645174068 >> [v2:172.21.15.45:6800/2012105536,v1:172.21.15.45:6801/2012105536] conn(0x7fbe60069020 msgr2=0x7fbe6006b4d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:38:19.201 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:38:19.198+0000 7fbe7c92e700 1 --2- 172.21.15.45:0/3645174068 >> [v2:172.21.15.45:6800/2012105536,v1:172.21.15.45:6801/2012105536] conn(0x7fbe60069020 0x7fbe6006b4d0 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7fbe680041d0 tx=0x7fbe6800b040 comp rx=0 tx=0).stop 2023-12-16T21:38:19.201 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:38:19.198+0000 7fbe7c92e700 1 -- 172.21.15.45:0/3645174068 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fbe78100050 msgr2=0x7fbe7810d9e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:38:19.201 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:38:19.198+0000 7fbe7c92e700 1 --2- 172.21.15.45:0/3645174068 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fbe78100050 0x7fbe7810d9e0 secure :-1 s=READY pgs=287 cs=0 l=1 rev1=1 crypto rx=0x7fbe6c000c00 tx=0x7fbe6c009e30 comp rx=0 tx=0).stop 2023-12-16T21:38:19.201 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:38:19.199+0000 7fbe7c92e700 1 -- 172.21.15.45:0/3645174068 shutdown_connections 2023-12-16T21:38:19.201 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:38:19.199+0000 7fbe7c92e700 1 --2- 172.21.15.45:0/3645174068 >> [v2:172.21.15.45:6800/2012105536,v1:172.21.15.45:6801/2012105536] conn(0x7fbe60069020 0x7fbe6006b4d0 unknown :-1 s=CLOSED pgs=113 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:38:19.201 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:38:19.199+0000 7fbe7c92e700 1 --2- 172.21.15.45:0/3645174068 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fbe780fd730 0x7fbe7810d4a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:38:19.202 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:38:19.199+0000 7fbe7c92e700 1 --2- 172.21.15.45:0/3645174068 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fbe78100050 0x7fbe7810d9e0 unknown :-1 s=CLOSED pgs=287 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:38:19.202 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:38:19.199+0000 7fbe7c92e700 1 -- 172.21.15.45:0/3645174068 >> 172.21.15.45:0/3645174068 conn(0x7fbe780f7190 msgr2=0x7fbe780f95e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-16T21:38:19.202 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:38:19.199+0000 7fbe7c92e700 1 -- 172.21.15.45:0/3645174068 shutdown_connections 2023-12-16T21:38:19.202 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:38:19.199+0000 7fbe7c92e700 1 -- 172.21.15.45:0/3645174068 wait complete. 2023-12-16T21:38:19.215 INFO:teuthology.orchestra.run.smithi045.stdout:true 2023-12-16T21:38:19.314 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:38:19.311+0000 7fa575cbb700 1 -- 172.21.15.45:0/3188675496 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fa5700fcd00 msgr2=0x7fa5700fd150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:38:19.314 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:38:19.311+0000 7fa575cbb700 1 --2- 172.21.15.45:0/3188675496 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fa5700fcd00 0x7fa5700fd150 secure :-1 s=READY pgs=288 cs=0 l=1 rev1=1 crypto rx=0x7fa564004940 tx=0x7fa564009e30 comp rx=0 tx=0).stop 2023-12-16T21:38:19.314 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:38:19.311+0000 7fa575cbb700 1 -- 172.21.15.45:0/3188675496 shutdown_connections 2023-12-16T21:38:19.314 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:38:19.311+0000 7fa575cbb700 1 --2- 172.21.15.45:0/3188675496 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fa5700fcd00 0x7fa5700fd150 unknown :-1 s=CLOSED pgs=288 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:38:19.315 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:38:19.311+0000 7fa575cbb700 1 --2- 172.21.15.45:0/3188675496 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fa5700fbaf0 0x7fa5700fbf00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:38:19.315 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:38:19.311+0000 7fa575cbb700 1 -- 172.21.15.45:0/3188675496 >> 172.21.15.45:0/3188675496 conn(0x7fa5700f71b0 msgr2=0x7fa5700f95e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-16T21:38:19.315 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:38:19.312+0000 7fa575cbb700 1 -- 172.21.15.45:0/3188675496 shutdown_connections 2023-12-16T21:38:19.316 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:38:19.312+0000 7fa575cbb700 1 -- 172.21.15.45:0/3188675496 wait complete. 2023-12-16T21:38:19.316 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:38:19.312+0000 7fa575cbb700 1 Processor -- start 2023-12-16T21:38:19.316 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:38:19.313+0000 7fa575cbb700 1 -- start start 2023-12-16T21:38:19.316 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:38:19.313+0000 7fa575cbb700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fa5700fbaf0 0x7fa5701101a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:38:19.316 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:38:19.313+0000 7fa575cbb700 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fa5700fcd00 0x7fa5701126f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:38:19.316 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:38:19.313+0000 7fa575cbb700 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa5700669a0 con 0x7fa5700fbaf0 2023-12-16T21:38:19.316 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:38:19.313+0000 7fa575cbb700 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa570066ae0 con 0x7fa5700fcd00 2023-12-16T21:38:19.316 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:38:19.313+0000 7fa56f7fe700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fa5700fbaf0 0x7fa5701101a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:38:19.316 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:38:19.313+0000 7fa56f7fe700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fa5700fbaf0 0x7fa5701101a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:48010/0 (socket says 172.21.15.45:48010) 2023-12-16T21:38:19.316 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:38:19.313+0000 7fa56f7fe700 1 -- 172.21.15.45:0/1091063639 learned_addr learned my addr 172.21.15.45:0/1091063639 (peer_addr_for_me v2:172.21.15.45:0/0) 2023-12-16T21:38:19.316 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:38:19.313+0000 7fa56effd700 1 --2- 172.21.15.45:0/1091063639 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fa5700fcd00 0x7fa5701126f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:38:19.317 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:38:19.314+0000 7fa56f7fe700 1 -- 172.21.15.45:0/1091063639 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fa5700fcd00 msgr2=0x7fa5701126f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:38:19.317 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:38:19.314+0000 7fa56f7fe700 1 --2- 172.21.15.45:0/1091063639 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fa5700fcd00 0x7fa5701126f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:38:19.317 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:38:19.314+0000 7fa56f7fe700 1 -- 172.21.15.45:0/1091063639 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa564005040 con 0x7fa5700fbaf0 2023-12-16T21:38:19.320 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:38:19.317+0000 7fa56f7fe700 1 --2- 172.21.15.45:0/1091063639 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fa5700fbaf0 0x7fa5701101a0 secure :-1 s=READY pgs=289 cs=0 l=1 rev1=1 crypto rx=0x7fa56000eb70 tx=0x7fa56000b040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:38:19.320 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:38:19.318+0000 7fa557fff700 1 -- 172.21.15.45:0/1091063639 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fa560007cc0 con 0x7fa5700fbaf0 2023-12-16T21:38:19.320 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:38:19.318+0000 7fa575cbb700 1 -- 172.21.15.45:0/1091063639 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa570112cf0 con 0x7fa5700fbaf0 2023-12-16T21:38:19.320 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:38:19.318+0000 7fa557fff700 1 -- 172.21.15.45:0/1091063639 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fa56000f040 con 0x7fa5700fbaf0 2023-12-16T21:38:19.321 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:38:19.318+0000 7fa557fff700 1 -- 172.21.15.45:0/1091063639 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fa560007cc0 con 0x7fa5700fbaf0 2023-12-16T21:38:19.322 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:38:19.318+0000 7fa575cbb700 1 -- 172.21.15.45:0/1091063639 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa570113180 con 0x7fa5700fbaf0 2023-12-16T21:38:19.322 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:38:19.319+0000 7fa575cbb700 1 -- 172.21.15.45:0/1091063639 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa57004bb80 con 0x7fa5700fbaf0 2023-12-16T21:38:19.323 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:38:19.320+0000 7fa557fff700 1 -- 172.21.15.45:0/1091063639 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 19) v1 ==== 86447+0+0 (secure 0 0 0) 0x7fa5600090d0 con 0x7fa5700fbaf0 2023-12-16T21:38:19.323 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:38:19.321+0000 7fa557fff700 1 --2- 172.21.15.45:0/1091063639 >> [v2:172.21.15.45:6800/2012105536,v1:172.21.15.45:6801/2012105536] conn(0x7fa558068fc0 0x7fa55806b470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:38:19.324 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:38:19.321+0000 7fa56effd700 1 --2- 172.21.15.45:0/1091063639 >> [v2:172.21.15.45:6800/2012105536,v1:172.21.15.45:6801/2012105536] conn(0x7fa558068fc0 0x7fa55806b470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:38:19.324 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:38:19.321+0000 7fa557fff700 1 -- 172.21.15.45:0/1091063639 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(43..43 src has 1..43) v4 ==== 5550+0+0 (secure 0 0 0) 0x7fa560007e20 con 0x7fa5700fbaf0 2023-12-16T21:38:19.324 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:38:19.322+0000 7fa56effd700 1 --2- 172.21.15.45:0/1091063639 >> [v2:172.21.15.45:6800/2012105536,v1:172.21.15.45:6801/2012105536] conn(0x7fa558068fc0 0x7fa55806b470 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7fa564013680 tx=0x7fa564009e30 comp rx=0 tx=0).ready entity=mgr.14214 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:38:19.333 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:38:19.331+0000 7fa557fff700 1 -- 172.21.15.45:0/1091063639 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+174570 (secure 0 0 0) 0x7fa560053090 con 0x7fa5700fbaf0 2023-12-16T21:38:19.552 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:38:19.549+0000 7fa575cbb700 1 -- 172.21.15.45:0/1091063639 --> [v2:172.21.15.45:6800/2012105536,v1:172.21.15.45:6801/2012105536] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7fa570101650 con 0x7fa558068fc0 2023-12-16T21:38:19.553 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:38:19.550+0000 7fa557fff700 1 -- 172.21.15.45:0/1091063639 <== mgr.14214 v2:172.21.15.45:6800/2012105536 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+351 (secure 0 0 0) 0x7fa570101650 con 0x7fa558068fc0 2023-12-16T21:38:19.555 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:38:19.553+0000 7fa575cbb700 1 -- 172.21.15.45:0/1091063639 >> [v2:172.21.15.45:6800/2012105536,v1:172.21.15.45:6801/2012105536] conn(0x7fa558068fc0 msgr2=0x7fa55806b470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:38:19.555 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:38:19.553+0000 7fa575cbb700 1 --2- 172.21.15.45:0/1091063639 >> [v2:172.21.15.45:6800/2012105536,v1:172.21.15.45:6801/2012105536] conn(0x7fa558068fc0 0x7fa55806b470 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7fa564013680 tx=0x7fa564009e30 comp rx=0 tx=0).stop 2023-12-16T21:38:19.556 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:38:19.553+0000 7fa575cbb700 1 -- 172.21.15.45:0/1091063639 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fa5700fbaf0 msgr2=0x7fa5701101a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:38:19.556 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:38:19.553+0000 7fa575cbb700 1 --2- 172.21.15.45:0/1091063639 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fa5700fbaf0 0x7fa5701101a0 secure :-1 s=READY pgs=289 cs=0 l=1 rev1=1 crypto rx=0x7fa56000eb70 tx=0x7fa56000b040 comp rx=0 tx=0).stop 2023-12-16T21:38:19.556 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:38:19.553+0000 7fa575cbb700 1 -- 172.21.15.45:0/1091063639 shutdown_connections 2023-12-16T21:38:19.556 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:38:19.553+0000 7fa575cbb700 1 --2- 172.21.15.45:0/1091063639 >> [v2:172.21.15.45:6800/2012105536,v1:172.21.15.45:6801/2012105536] conn(0x7fa558068fc0 0x7fa55806b470 unknown :-1 s=CLOSED pgs=114 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:38:19.556 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:38:19.553+0000 7fa575cbb700 1 --2- 172.21.15.45:0/1091063639 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fa5700fbaf0 0x7fa5701101a0 unknown :-1 s=CLOSED pgs=289 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:38:19.556 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:38:19.553+0000 7fa575cbb700 1 --2- 172.21.15.45:0/1091063639 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fa5700fcd00 0x7fa5701126f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:38:19.556 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:38:19.553+0000 7fa575cbb700 1 -- 172.21.15.45:0/1091063639 >> 172.21.15.45:0/1091063639 conn(0x7fa5700f71b0 msgr2=0x7fa5700fff30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-16T21:38:19.556 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:38:19.553+0000 7fa575cbb700 1 -- 172.21.15.45:0/1091063639 shutdown_connections 2023-12-16T21:38:19.556 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:38:19.553+0000 7fa575cbb700 1 -- 172.21.15.45:0/1091063639 wait complete. 2023-12-16T21:38:19.633 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Dec 16 21:38:19 smithi169 ceph-mon[88350]: Upgrade: First pull of quay.ceph.io/ceph-ci/ceph:c7586e95519b7be76f3b6e3a4b333888baf46bc9 2023-12-16T21:38:19.633 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Dec 16 21:38:19 smithi169 ceph-mon[88350]: pgmap v146: 65 pgs: 65 active+clean; 586 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 1.4 KiB/s rd, 255 B/s wr, 2 op/s 2023-12-16T21:38:19.665 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:38:19.663+0000 7f5e12f62700 1 -- 172.21.15.45:0/2177747707 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f5e0c066920 msgr2=0x7f5e0c0ff7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:38:19.666 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:38:19.663+0000 7f5e12f62700 1 --2- 172.21.15.45:0/2177747707 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f5e0c066920 0x7f5e0c0ff7a0 secure :-1 s=READY pgs=290 cs=0 l=1 rev1=1 crypto rx=0x7f5e00004940 tx=0x7f5e00009e30 comp rx=0 tx=0).stop 2023-12-16T21:38:19.666 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Dec 16 21:38:19 smithi045 ceph-mon[80763]: Upgrade: First pull of quay.ceph.io/ceph-ci/ceph:c7586e95519b7be76f3b6e3a4b333888baf46bc9 2023-12-16T21:38:19.666 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Dec 16 21:38:19 smithi045 ceph-mon[80763]: pgmap v146: 65 pgs: 65 active+clean; 586 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 1.4 KiB/s rd, 255 B/s wr, 2 op/s 2023-12-16T21:38:19.667 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:38:19.665+0000 7f5e12f62700 1 -- 172.21.15.45:0/2177747707 shutdown_connections 2023-12-16T21:38:19.667 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:38:19.665+0000 7f5e12f62700 1 --2- 172.21.15.45:0/2177747707 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f5e0c0ffd70 0x7f5e0c1021d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:38:19.667 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:38:19.665+0000 7f5e12f62700 1 --2- 172.21.15.45:0/2177747707 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f5e0c066920 0x7f5e0c0ff7a0 unknown :-1 s=CLOSED pgs=290 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:38:19.667 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:38:19.665+0000 7f5e12f62700 1 -- 172.21.15.45:0/2177747707 >> 172.21.15.45:0/2177747707 conn(0x7f5e0c0f71d0 msgr2=0x7f5e0c0f95e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-16T21:38:19.668 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:38:19.665+0000 7f5e12f62700 1 -- 172.21.15.45:0/2177747707 shutdown_connections 2023-12-16T21:38:19.668 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:38:19.665+0000 7f5e12f62700 1 -- 172.21.15.45:0/2177747707 wait complete. 2023-12-16T21:38:19.668 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:38:19.666+0000 7f5e12f62700 1 Processor -- start 2023-12-16T21:38:19.668 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:38:19.666+0000 7f5e12f62700 1 -- start start 2023-12-16T21:38:19.669 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:38:19.666+0000 7f5e12f62700 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f5e0c066920 0x7f5e0c10d4b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:38:19.669 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:38:19.666+0000 7f5e12f62700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f5e0c0ffd70 0x7f5e0c10d9f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:38:19.669 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:38:19.666+0000 7f5e12f62700 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5e0c10dff0 con 0x7f5e0c0ffd70 2023-12-16T21:38:19.669 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:38:19.666+0000 7f5e12f62700 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5e0c10e890 con 0x7f5e0c066920 2023-12-16T21:38:19.669 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:38:19.666+0000 7f5e10cfe700 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f5e0c066920 0x7f5e0c10d4b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:38:19.669 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:38:19.666+0000 7f5e10cfe700 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f5e0c066920 0x7f5e0c10d4b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.169:3300/0 says I am v2:172.21.15.45:59510/0 (socket says 172.21.15.45:59510) 2023-12-16T21:38:19.669 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:38:19.666+0000 7f5e10cfe700 1 -- 172.21.15.45:0/1064231219 learned_addr learned my addr 172.21.15.45:0/1064231219 (peer_addr_for_me v2:172.21.15.45:0/0) 2023-12-16T21:38:19.669 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:38:19.666+0000 7f5e0bfff700 1 --2- 172.21.15.45:0/1064231219 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f5e0c0ffd70 0x7f5e0c10d9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:38:19.670 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:38:19.667+0000 7f5e10cfe700 1 -- 172.21.15.45:0/1064231219 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f5e0c0ffd70 msgr2=0x7f5e0c10d9f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:38:19.670 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:38:19.667+0000 7f5e10cfe700 1 --2- 172.21.15.45:0/1064231219 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f5e0c0ffd70 0x7f5e0c10d9f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:38:19.670 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:38:19.667+0000 7f5e10cfe700 1 -- 172.21.15.45:0/1064231219 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5df8005040 con 0x7f5e0c066920 2023-12-16T21:38:19.670 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:38:19.667+0000 7f5e10cfe700 1 --2- 172.21.15.45:0/1064231219 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f5e0c066920 0x7f5e0c10d4b0 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7f5e00009fd0 tx=0x7f5e00009e30 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:38:19.670 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:38:19.667+0000 7f5e097fa700 1 -- 172.21.15.45:0/1064231219 <== mon.1 v2:172.21.15.169:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f5e00021070 con 0x7f5e0c066920 2023-12-16T21:38:19.670 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:38:19.667+0000 7f5e097fa700 1 -- 172.21.15.45:0/1064231219 <== mon.1 v2:172.21.15.169:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f5e0001c400 con 0x7f5e0c066920 2023-12-16T21:38:19.670 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:38:19.667+0000 7f5e097fa700 1 -- 172.21.15.45:0/1064231219 <== mon.1 v2:172.21.15.169:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f5e0000e070 con 0x7f5e0c066920 2023-12-16T21:38:19.670 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:38:19.667+0000 7f5e12f62700 1 -- 172.21.15.45:0/1064231219 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5e00005040 con 0x7f5e0c066920 2023-12-16T21:38:19.671 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:38:19.668+0000 7f5e12f62700 1 -- 172.21.15.45:0/1064231219 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5e0c10ee70 con 0x7f5e0c066920 2023-12-16T21:38:19.671 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:38:19.669+0000 7f5e12f62700 1 -- 172.21.15.45:0/1064231219 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5e0c04bb80 con 0x7f5e0c066920 2023-12-16T21:38:19.672 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:38:19.670+0000 7f5e097fa700 1 -- 172.21.15.45:0/1064231219 <== mon.1 v2:172.21.15.169:3300/0 4 ==== mgrmap(e 19) v1 ==== 86447+0+0 (secure 0 0 0) 0x7f5e0001ca00 con 0x7f5e0c066920 2023-12-16T21:38:19.672 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:38:19.670+0000 7f5e097fa700 1 --2- 172.21.15.45:0/1064231219 >> [v2:172.21.15.45:6800/2012105536,v1:172.21.15.45:6801/2012105536] conn(0x7f5dfc068fc0 0x7f5dfc06b470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:38:19.673 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:38:19.670+0000 7f5e097fa700 1 -- 172.21.15.45:0/1064231219 <== mon.1 v2:172.21.15.169:3300/0 5 ==== osd_map(43..43 src has 1..43) v4 ==== 5550+0+0 (secure 0 0 0) 0x7f5e00088070 con 0x7f5e0c066920 2023-12-16T21:38:19.673 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:38:19.671+0000 7f5e0bfff700 1 --2- 172.21.15.45:0/1064231219 >> [v2:172.21.15.45:6800/2012105536,v1:172.21.15.45:6801/2012105536] conn(0x7f5dfc068fc0 0x7f5dfc06b470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:38:19.673 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:38:19.671+0000 7f5e0bfff700 1 --2- 172.21.15.45:0/1064231219 >> [v2:172.21.15.45:6800/2012105536,v1:172.21.15.45:6801/2012105536] conn(0x7f5dfc068fc0 0x7f5dfc06b470 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7f5e0c066660 tx=0x7f5df800b040 comp rx=0 tx=0).ready entity=mgr.14214 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:38:19.680 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:38:19.678+0000 7f5e097fa700 1 -- 172.21.15.45:0/1064231219 <== mon.1 v2:172.21.15.169:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+174570 (secure 0 0 0) 0x7f5e00057f00 con 0x7f5e0c066920 2023-12-16T21:38:20.013 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:38:20.010+0000 7f5e12f62700 1 -- 172.21.15.45:0/1064231219 --> [v2:172.21.15.45:6800/2012105536,v1:172.21.15.45:6801/2012105536] -- mgr_command(tid 0: {"prefix": "orch ps", "target": ["mon-mgr", ""]}) v1 -- 0x7f5e0c10f530 con 0x7f5dfc068fc0 2023-12-16T21:38:20.030 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:38:20.027+0000 7f5e097fa700 1 -- 172.21.15.45:0/1064231219 <== mgr.14214 v2:172.21.15.45:6800/2012105536 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+3410 (secure 0 0 0) 0x7f5e0c10f530 con 0x7f5dfc068fc0 2023-12-16T21:38:20.030 INFO:teuthology.orchestra.run.smithi045.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2023-12-16T21:38:20.030 INFO:teuthology.orchestra.run.smithi045.stdout:alertmanager.smithi045 smithi045 *:9093,9094 running (3m) 46s ago 4m 26.1M - 0.25.0 c8568f914cd2 d9982231d4a2 2023-12-16T21:38:20.031 INFO:teuthology.orchestra.run.smithi045.stdout:crash.smithi045 smithi045 running (4m) 46s ago 4m 6660k - 17.2.7-125-g1f2f5510 60cc83a9c8f1 fc997f70d946 2023-12-16T21:38:20.031 INFO:teuthology.orchestra.run.smithi045.stdout:crash.smithi169 smithi169 running (3m) 48s ago 3m 6665k - 17.2.7-125-g1f2f5510 60cc83a9c8f1 f29fac8b7321 2023-12-16T21:38:20.031 INFO:teuthology.orchestra.run.smithi045.stdout:grafana.smithi045 smithi045 *:3000 running (3m) 46s ago 4m 84.9M - 9.4.7 954c08fa6188 dddae79633f0 2023-12-16T21:38:20.031 INFO:teuthology.orchestra.run.smithi045.stdout:mds.cephfs.smithi045.fgsbgq smithi045 running (58s) 46s ago 58s 10.4M - 17.2.7-125-g1f2f5510 60cc83a9c8f1 9c41632d4c6e 2023-12-16T21:38:20.031 INFO:teuthology.orchestra.run.smithi045.stdout:mds.cephfs.smithi045.hkjgzw smithi045 running (55s) 46s ago 55s 11.9M - 17.2.7-125-g1f2f5510 60cc83a9c8f1 140efc1dc83a 2023-12-16T21:38:20.031 INFO:teuthology.orchestra.run.smithi045.stdout:mds.cephfs.smithi169.mokxju smithi169 running (57s) 48s ago 56s 13.1M - 17.2.7-125-g1f2f5510 60cc83a9c8f1 2c6edcc7b959 2023-12-16T21:38:20.031 INFO:teuthology.orchestra.run.smithi045.stdout:mds.cephfs.smithi169.pihlib smithi169 running (53s) 48s ago 53s 13.4M - 17.2.7-125-g1f2f5510 60cc83a9c8f1 ca40c392fd88 2023-12-16T21:38:20.031 INFO:teuthology.orchestra.run.smithi045.stdout:mgr.smithi045.nhtmwc smithi045 *:9283 running (5m) 46s ago 5m 477M - 17.2.7-125-g1f2f5510 60cc83a9c8f1 a4ec7bd78bc7 2023-12-16T21:38:20.031 INFO:teuthology.orchestra.run.smithi045.stdout:mgr.smithi169.cncbgh smithi169 *:8443,9283 running (3m) 48s ago 3m 432M - 17.2.7-125-g1f2f5510 60cc83a9c8f1 034b205acb79 2023-12-16T21:38:20.031 INFO:teuthology.orchestra.run.smithi045.stdout:mon.smithi045 smithi045 running (5m) 46s ago 5m 50.0M 2048M 17.2.7-125-g1f2f5510 60cc83a9c8f1 1cddbf5059f8 2023-12-16T21:38:20.031 INFO:teuthology.orchestra.run.smithi045.stdout:mon.smithi169 smithi169 running (3m) 48s ago 3m 43.2M 2048M 17.2.7-125-g1f2f5510 60cc83a9c8f1 f964ea9d81bd 2023-12-16T21:38:20.031 INFO:teuthology.orchestra.run.smithi045.stdout:node-exporter.smithi045 smithi045 *:9100 running (4m) 46s ago 4m 16.0M - 1.5.0 0da6a335fe13 c535836aef3c 2023-12-16T21:38:20.031 INFO:teuthology.orchestra.run.smithi045.stdout:node-exporter.smithi169 smithi169 *:9100 running (3m) 48s ago 3m 15.9M - 1.5.0 0da6a335fe13 f6778c39f7f1 2023-12-16T21:38:20.031 INFO:teuthology.orchestra.run.smithi045.stdout:osd.0 smithi045 running (3m) 46s ago 3m 44.5M 4289M 17.2.7-125-g1f2f5510 60cc83a9c8f1 eeea01051a62 2023-12-16T21:38:20.031 INFO:teuthology.orchestra.run.smithi045.stdout:osd.1 smithi045 running (2m) 46s ago 2m 45.4M 4289M 17.2.7-125-g1f2f5510 60cc83a9c8f1 94ded6ce4ce1 2023-12-16T21:38:20.032 INFO:teuthology.orchestra.run.smithi045.stdout:osd.2 smithi045 running (2m) 46s ago 2m 43.9M 4289M 17.2.7-125-g1f2f5510 60cc83a9c8f1 5fc744d14d11 2023-12-16T21:38:20.032 INFO:teuthology.orchestra.run.smithi045.stdout:osd.3 smithi169 running (2m) 48s ago 2m 44.1M 5312M 17.2.7-125-g1f2f5510 60cc83a9c8f1 f5ee44f8b3e2 2023-12-16T21:38:20.032 INFO:teuthology.orchestra.run.smithi045.stdout:osd.4 smithi169 running (2m) 48s ago 2m 41.1M 5312M 17.2.7-125-g1f2f5510 60cc83a9c8f1 8a74d1fa8b75 2023-12-16T21:38:20.032 INFO:teuthology.orchestra.run.smithi045.stdout:osd.5 smithi169 running (108s) 48s ago 108s 40.6M 5312M 17.2.7-125-g1f2f5510 60cc83a9c8f1 9c0a8a497615 2023-12-16T21:38:20.032 INFO:teuthology.orchestra.run.smithi045.stdout:prometheus.smithi045 smithi045 *:9095 running (3m) 46s ago 3m 44.4M - 2.43.0 a07b618ecd1d 1e1226889abe 2023-12-16T21:38:20.032 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:38:20.030+0000 7f5e12f62700 1 -- 172.21.15.45:0/1064231219 >> [v2:172.21.15.45:6800/2012105536,v1:172.21.15.45:6801/2012105536] conn(0x7f5dfc068fc0 msgr2=0x7f5dfc06b470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:38:20.032 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:38:20.030+0000 7f5e12f62700 1 --2- 172.21.15.45:0/1064231219 >> [v2:172.21.15.45:6800/2012105536,v1:172.21.15.45:6801/2012105536] conn(0x7f5dfc068fc0 0x7f5dfc06b470 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7f5e0c066660 tx=0x7f5df800b040 comp rx=0 tx=0).stop 2023-12-16T21:38:20.032 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:38:20.030+0000 7f5e12f62700 1 -- 172.21.15.45:0/1064231219 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f5e0c066920 msgr2=0x7f5e0c10d4b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:38:20.032 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:38:20.030+0000 7f5e12f62700 1 --2- 172.21.15.45:0/1064231219 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f5e0c066920 0x7f5e0c10d4b0 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7f5e00009fd0 tx=0x7f5e00009e30 comp rx=0 tx=0).stop 2023-12-16T21:38:20.033 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:38:20.030+0000 7f5e12f62700 1 -- 172.21.15.45:0/1064231219 shutdown_connections 2023-12-16T21:38:20.033 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:38:20.030+0000 7f5e12f62700 1 --2- 172.21.15.45:0/1064231219 >> [v2:172.21.15.45:6800/2012105536,v1:172.21.15.45:6801/2012105536] conn(0x7f5dfc068fc0 0x7f5dfc06b470 unknown :-1 s=CLOSED pgs=115 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:38:20.033 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:38:20.030+0000 7f5e12f62700 1 --2- 172.21.15.45:0/1064231219 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f5e0c066920 0x7f5e0c10d4b0 unknown :-1 s=CLOSED pgs=38 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:38:20.033 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:38:20.030+0000 7f5e12f62700 1 --2- 172.21.15.45:0/1064231219 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f5e0c0ffd70 0x7f5e0c10d9f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:38:20.033 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:38:20.030+0000 7f5e12f62700 1 -- 172.21.15.45:0/1064231219 >> 172.21.15.45:0/1064231219 conn(0x7f5e0c0f71d0 msgr2=0x7f5e0c0f95b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-16T21:38:20.033 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:38:20.030+0000 7f5e12f62700 1 -- 172.21.15.45:0/1064231219 shutdown_connections 2023-12-16T21:38:20.033 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:38:20.030+0000 7f5e12f62700 1 -- 172.21.15.45:0/1064231219 wait complete. 2023-12-16T21:38:20.214 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:38:20.211+0000 7f0f09fee700 1 -- 172.21.15.45:0/1143567193 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f0f040fad40 msgr2=0x7f0f040fb150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:38:20.215 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:38:20.211+0000 7f0f09fee700 1 --2- 172.21.15.45:0/1143567193 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f0f040fad40 0x7f0f040fb150 secure :-1 s=READY pgs=291 cs=0 l=1 rev1=1 crypto rx=0x7f0ef8004940 tx=0x7f0ef8009e30 comp rx=0 tx=0).stop 2023-12-16T21:38:20.215 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:38:20.212+0000 7f0f09fee700 1 -- 172.21.15.45:0/1143567193 shutdown_connections 2023-12-16T21:38:20.215 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:38:20.212+0000 7f0f09fee700 1 --2- 172.21.15.45:0/1143567193 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f0f040fb720 0x7f0f040fdfd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:38:20.215 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:38:20.212+0000 7f0f09fee700 1 --2- 172.21.15.45:0/1143567193 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f0f040fad40 0x7f0f040fb150 unknown :-1 s=CLOSED pgs=291 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:38:20.215 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:38:20.212+0000 7f0f09fee700 1 -- 172.21.15.45:0/1143567193 >> 172.21.15.45:0/1143567193 conn(0x7f0f040f6720 msgr2=0x7f0f040f8b50 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-16T21:38:20.217 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:38:20.212+0000 7f0f09fee700 1 -- 172.21.15.45:0/1143567193 shutdown_connections 2023-12-16T21:38:20.217 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:38:20.213+0000 7f0f09fee700 1 -- 172.21.15.45:0/1143567193 wait complete. 2023-12-16T21:38:20.217 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:38:20.213+0000 7f0f09fee700 1 Processor -- start 2023-12-16T21:38:20.217 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:38:20.213+0000 7f0f09fee700 1 -- start start 2023-12-16T21:38:20.217 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:38:20.214+0000 7f0f09fee700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f0f040fad40 0x7f0f0410b230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:38:20.218 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:38:20.214+0000 7f0f09fee700 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f0f0410b770 0x7f0f0410c660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:38:20.218 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:38:20.214+0000 7f0f09fee700 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0f0410bc30 con 0x7f0f040fad40 2023-12-16T21:38:20.218 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:38:20.214+0000 7f0f09fee700 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0f0410bda0 con 0x7f0f0410b770 2023-12-16T21:38:20.218 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:38:20.214+0000 7f0f08fec700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f0f040fad40 0x7f0f0410b230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:38:20.218 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:38:20.214+0000 7f0f08fec700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f0f040fad40 0x7f0f0410b230 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:48046/0 (socket says 172.21.15.45:48046) 2023-12-16T21:38:20.219 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:38:20.214+0000 7f0f08fec700 1 -- 172.21.15.45:0/3880984217 learned_addr learned my addr 172.21.15.45:0/3880984217 (peer_addr_for_me v2:172.21.15.45:0/0) 2023-12-16T21:38:20.219 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:38:20.214+0000 7f0f03fff700 1 --2- 172.21.15.45:0/3880984217 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f0f0410b770 0x7f0f0410c660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:38:20.220 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:38:20.215+0000 7f0f03fff700 1 -- 172.21.15.45:0/3880984217 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f0f040fad40 msgr2=0x7f0f0410b230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:38:20.220 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:38:20.215+0000 7f0f03fff700 1 --2- 172.21.15.45:0/3880984217 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f0f040fad40 0x7f0f0410b230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:38:20.220 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:38:20.215+0000 7f0f03fff700 1 -- 172.21.15.45:0/3880984217 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0ef8005040 con 0x7f0f0410b770 2023-12-16T21:38:20.220 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:38:20.215+0000 7f0f08fec700 1 --2- 172.21.15.45:0/3880984217 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f0f040fad40 0x7f0f0410b230 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-12-16T21:38:20.220 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:38:20.215+0000 7f0f03fff700 1 --2- 172.21.15.45:0/3880984217 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f0f0410b770 0x7f0f0410c660 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7f0ef0009fd0 tx=0x7f0ef000b040 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:38:20.220 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:38:20.216+0000 7f0f017fa700 1 -- 172.21.15.45:0/3880984217 <== mon.1 v2:172.21.15.169:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f0ef0004e90 con 0x7f0f0410b770 2023-12-16T21:38:20.221 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:38:20.216+0000 7f0f017fa700 1 -- 172.21.15.45:0/3880984217 <== mon.1 v2:172.21.15.169:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f0ef0004e90 con 0x7f0f0410b770 2023-12-16T21:38:20.221 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:38:20.217+0000 7f0f017fa700 1 -- 172.21.15.45:0/3880984217 <== mon.1 v2:172.21.15.169:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f0ef000c690 con 0x7f0f0410b770 2023-12-16T21:38:20.221 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:38:20.217+0000 7f0f09fee700 1 -- 172.21.15.45:0/3880984217 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0f0410cc60 con 0x7f0f0410b770 2023-12-16T21:38:20.221 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:38:20.217+0000 7f0f09fee700 1 -- 172.21.15.45:0/3880984217 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0f0410d150 con 0x7f0f0410b770 2023-12-16T21:38:20.221 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:38:20.218+0000 7f0f09fee700 1 -- 172.21.15.45:0/3880984217 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0f0404bb80 con 0x7f0f0410b770 2023-12-16T21:38:20.222 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:38:20.220+0000 7f0f017fa700 1 -- 172.21.15.45:0/3880984217 <== mon.1 v2:172.21.15.169:3300/0 4 ==== mgrmap(e 19) v1 ==== 86447+0+0 (secure 0 0 0) 0x7f0ef0007cc0 con 0x7f0f0410b770 2023-12-16T21:38:20.223 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:38:20.221+0000 7f0f017fa700 1 --2- 172.21.15.45:0/3880984217 >> [v2:172.21.15.45:6800/2012105536,v1:172.21.15.45:6801/2012105536] conn(0x7f0ef40690b0 0x7f0ef406b560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:38:20.224 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:38:20.221+0000 7f0f017fa700 1 -- 172.21.15.45:0/3880984217 <== mon.1 v2:172.21.15.169:3300/0 5 ==== osd_map(43..43 src has 1..43) v4 ==== 5550+0+0 (secure 0 0 0) 0x7f0ef0086bd0 con 0x7f0f0410b770 2023-12-16T21:38:20.224 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:38:20.221+0000 7f0f08fec700 1 --2- 172.21.15.45:0/3880984217 >> [v2:172.21.15.45:6800/2012105536,v1:172.21.15.45:6801/2012105536] conn(0x7f0ef40690b0 0x7f0ef406b560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:38:20.225 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:38:20.222+0000 7f0f08fec700 1 --2- 172.21.15.45:0/3880984217 >> [v2:172.21.15.45:6800/2012105536,v1:172.21.15.45:6801/2012105536] conn(0x7f0ef40690b0 0x7f0ef406b560 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7f0ef8004890 tx=0x7f0ef8009e30 comp rx=0 tx=0).ready entity=mgr.14214 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:38:20.230 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:38:20.224+0000 7f0f017fa700 1 -- 172.21.15.45:0/3880984217 <== mon.1 v2:172.21.15.169:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+174570 (secure 0 0 0) 0x7f0ef0052fb0 con 0x7f0f0410b770 2023-12-16T21:38:20.494 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Dec 16 21:38:20 smithi045 ceph-mon[80763]: from='client.14556 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2023-12-16T21:38:20.494 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Dec 16 21:38:20 smithi045 ceph-mon[80763]: from='client.14560 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2023-12-16T21:38:20.559 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:38:20.556+0000 7f0f09fee700 1 -- 172.21.15.45:0/3880984217 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_command({"prefix": "versions"} v 0) v1 -- 0x7f0f0410d430 con 0x7f0f0410b770 2023-12-16T21:38:20.559 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:38:20.557+0000 7f0f017fa700 1 -- 172.21.15.45:0/3880984217 <== mon.1 v2:172.21.15.169:3300/0 7 ==== mon_command_ack([{"prefix": "versions"}]=0 v0) v1 ==== 56+0+638 (secure 0 0 0) 0x7f0ef00565d0 con 0x7f0f0410b770 2023-12-16T21:38:20.560 INFO:teuthology.orchestra.run.smithi045.stdout:{ 2023-12-16T21:38:20.560 INFO:teuthology.orchestra.run.smithi045.stdout: "mon": { 2023-12-16T21:38:20.560 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 17.2.7-125-g1f2f5510 (1f2f55109dc1ed2dc2dda9de1db3c70923ba0dfd) quincy (stable)": 2 2023-12-16T21:38:20.560 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2023-12-16T21:38:20.560 INFO:teuthology.orchestra.run.smithi045.stdout: "mgr": { 2023-12-16T21:38:20.560 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 17.2.7-125-g1f2f5510 (1f2f55109dc1ed2dc2dda9de1db3c70923ba0dfd) quincy (stable)": 2 2023-12-16T21:38:20.560 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2023-12-16T21:38:20.560 INFO:teuthology.orchestra.run.smithi045.stdout: "osd": { 2023-12-16T21:38:20.560 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 17.2.7-125-g1f2f5510 (1f2f55109dc1ed2dc2dda9de1db3c70923ba0dfd) quincy (stable)": 6 2023-12-16T21:38:20.560 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2023-12-16T21:38:20.560 INFO:teuthology.orchestra.run.smithi045.stdout: "mds": { 2023-12-16T21:38:20.561 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 17.2.7-125-g1f2f5510 (1f2f55109dc1ed2dc2dda9de1db3c70923ba0dfd) quincy (stable)": 4 2023-12-16T21:38:20.561 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2023-12-16T21:38:20.561 INFO:teuthology.orchestra.run.smithi045.stdout: "overall": { 2023-12-16T21:38:20.561 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 17.2.7-125-g1f2f5510 (1f2f55109dc1ed2dc2dda9de1db3c70923ba0dfd) quincy (stable)": 14 2023-12-16T21:38:20.561 INFO:teuthology.orchestra.run.smithi045.stdout: } 2023-12-16T21:38:20.561 INFO:teuthology.orchestra.run.smithi045.stdout:} 2023-12-16T21:38:20.562 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:38:20.559+0000 7f0f09fee700 1 -- 172.21.15.45:0/3880984217 >> [v2:172.21.15.45:6800/2012105536,v1:172.21.15.45:6801/2012105536] conn(0x7f0ef40690b0 msgr2=0x7f0ef406b560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:38:20.562 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:38:20.559+0000 7f0f09fee700 1 --2- 172.21.15.45:0/3880984217 >> [v2:172.21.15.45:6800/2012105536,v1:172.21.15.45:6801/2012105536] conn(0x7f0ef40690b0 0x7f0ef406b560 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7f0ef8004890 tx=0x7f0ef8009e30 comp rx=0 tx=0).stop 2023-12-16T21:38:20.562 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:38:20.559+0000 7f0f09fee700 1 -- 172.21.15.45:0/3880984217 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f0f0410b770 msgr2=0x7f0f0410c660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:38:20.563 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:38:20.559+0000 7f0f09fee700 1 --2- 172.21.15.45:0/3880984217 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f0f0410b770 0x7f0f0410c660 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7f0ef0009fd0 tx=0x7f0ef000b040 comp rx=0 tx=0).stop 2023-12-16T21:38:20.563 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:38:20.560+0000 7f0f09fee700 1 -- 172.21.15.45:0/3880984217 shutdown_connections 2023-12-16T21:38:20.563 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:38:20.560+0000 7f0f09fee700 1 --2- 172.21.15.45:0/3880984217 >> [v2:172.21.15.45:6800/2012105536,v1:172.21.15.45:6801/2012105536] conn(0x7f0ef40690b0 0x7f0ef406b560 unknown :-1 s=CLOSED pgs=116 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:38:20.563 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:38:20.560+0000 7f0f09fee700 1 --2- 172.21.15.45:0/3880984217 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f0f040fad40 0x7f0f0410b230 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:38:20.563 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:38:20.560+0000 7f0f09fee700 1 --2- 172.21.15.45:0/3880984217 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f0f0410b770 0x7f0f0410c660 unknown :-1 s=CLOSED pgs=39 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:38:20.563 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:38:20.560+0000 7f0f09fee700 1 -- 172.21.15.45:0/3880984217 >> 172.21.15.45:0/3880984217 conn(0x7f0f040f6720 msgr2=0x7f0f040ff1b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-16T21:38:20.563 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:38:20.560+0000 7f0f09fee700 1 -- 172.21.15.45:0/3880984217 shutdown_connections 2023-12-16T21:38:20.563 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:38:20.560+0000 7f0f09fee700 1 -- 172.21.15.45:0/3880984217 wait complete. 2023-12-16T21:38:20.633 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Dec 16 21:38:20 smithi169 ceph-mon[88350]: from='client.14556 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2023-12-16T21:38:20.633 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Dec 16 21:38:20 smithi169 ceph-mon[88350]: from='client.14560 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2023-12-16T21:38:20.734 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:38:20.730+0000 7f571e8a1700 1 -- 172.21.15.45:0/949070549 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f5718100050 msgr2=0x7f5718102430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:38:20.734 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:38:20.730+0000 7f571e8a1700 1 --2- 172.21.15.45:0/949070549 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f5718100050 0x7f5718102430 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7f5708004880 tx=0x7f5708009e30 comp rx=0 tx=0).stop 2023-12-16T21:38:20.735 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:38:20.730+0000 7f571e8a1700 1 -- 172.21.15.45:0/949070549 shutdown_connections 2023-12-16T21:38:20.735 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:38:20.730+0000 7f571e8a1700 1 --2- 172.21.15.45:0/949070549 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f5718100050 0x7f5718102430 secure :-1 s=CLOSED pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7f5708004880 tx=0x7f5708009e30 comp rx=0 tx=0).stop 2023-12-16T21:38:20.735 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:38:20.730+0000 7f571e8a1700 1 --2- 172.21.15.45:0/949070549 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f57180fd730 0x7f57180ffb10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:38:20.735 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:38:20.730+0000 7f571e8a1700 1 -- 172.21.15.45:0/949070549 >> 172.21.15.45:0/949070549 conn(0x7f57180f71b0 msgr2=0x7f57180f95e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-16T21:38:20.735 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:38:20.731+0000 7f571e8a1700 1 -- 172.21.15.45:0/949070549 shutdown_connections 2023-12-16T21:38:20.735 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:38:20.731+0000 7f571e8a1700 1 -- 172.21.15.45:0/949070549 wait complete. 2023-12-16T21:38:20.735 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:38:20.731+0000 7f571e8a1700 1 Processor -- start 2023-12-16T21:38:20.735 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:38:20.732+0000 7f571e8a1700 1 -- start start 2023-12-16T21:38:20.735 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:38:20.732+0000 7f571e8a1700 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f57180fd730 0x7f57180fcdd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:38:20.735 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:38:20.732+0000 7f571e8a1700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f57180fd310 0x7f57180fb900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:38:20.736 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:38:20.732+0000 7f571e8a1700 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f571811af30 con 0x7f57180fd310 2023-12-16T21:38:20.736 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:38:20.732+0000 7f571e8a1700 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f571811b0a0 con 0x7f57180fd730 2023-12-16T21:38:20.736 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:38:20.732+0000 7f5717fff700 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f57180fd730 0x7f57180fcdd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:38:20.736 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:38:20.732+0000 7f5717fff700 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f57180fd730 0x7f57180fcdd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.169:3300/0 says I am v2:172.21.15.45:59558/0 (socket says 172.21.15.45:59558) 2023-12-16T21:38:20.736 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:38:20.732+0000 7f5717fff700 1 -- 172.21.15.45:0/3497997974 learned_addr learned my addr 172.21.15.45:0/3497997974 (peer_addr_for_me v2:172.21.15.45:0/0) 2023-12-16T21:38:20.736 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:38:20.732+0000 7f57177fe700 1 --2- 172.21.15.45:0/3497997974 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f57180fd310 0x7f57180fb900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:38:20.737 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:38:20.733+0000 7f57177fe700 1 -- 172.21.15.45:0/3497997974 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f57180fd730 msgr2=0x7f57180fcdd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:38:20.737 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:38:20.733+0000 7f57177fe700 1 --2- 172.21.15.45:0/3497997974 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f57180fd730 0x7f57180fcdd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:38:20.737 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:38:20.733+0000 7f57177fe700 1 -- 172.21.15.45:0/3497997974 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5708005040 con 0x7f57180fd310 2023-12-16T21:38:20.737 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:38:20.734+0000 7f57177fe700 1 --2- 172.21.15.45:0/3497997974 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f57180fd310 0x7f57180fb900 secure :-1 s=READY pgs=292 cs=0 l=1 rev1=1 crypto rx=0x7f5708008000 tx=0x7f5708009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:38:20.737 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:38:20.735+0000 7f5714ff9700 1 -- 172.21.15.45:0/3497997974 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f570800e030 con 0x7f57180fd310 2023-12-16T21:38:20.737 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:38:20.735+0000 7f5714ff9700 1 -- 172.21.15.45:0/3497997974 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f570801c400 con 0x7f57180fd310 2023-12-16T21:38:20.738 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:38:20.735+0000 7f5714ff9700 1 -- 172.21.15.45:0/3497997974 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f5708012600 con 0x7f57180fd310 2023-12-16T21:38:20.738 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:38:20.735+0000 7f571e8a1700 1 -- 172.21.15.45:0/3497997974 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f571811b320 con 0x7f57180fd310 2023-12-16T21:38:20.738 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:38:20.735+0000 7f571e8a1700 1 -- 172.21.15.45:0/3497997974 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f571811b830 con 0x7f57180fd310 2023-12-16T21:38:20.739 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:38:20.737+0000 7f571e8a1700 1 -- 172.21.15.45:0/3497997974 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f571804bb80 con 0x7f57180fd310 2023-12-16T21:38:20.740 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:38:20.738+0000 7f5714ff9700 1 -- 172.21.15.45:0/3497997974 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 19) v1 ==== 86447+0+0 (secure 0 0 0) 0x7f570801c970 con 0x7f57180fd310 2023-12-16T21:38:20.741 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:38:20.739+0000 7f5714ff9700 1 --2- 172.21.15.45:0/3497997974 >> [v2:172.21.15.45:6800/2012105536,v1:172.21.15.45:6801/2012105536] conn(0x7f5704069040 0x7f570406b4f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:38:20.741 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:38:20.739+0000 7f5714ff9700 1 -- 172.21.15.45:0/3497997974 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(43..43 src has 1..43) v4 ==== 5550+0+0 (secure 0 0 0) 0x7f5708087c20 con 0x7f57180fd310 2023-12-16T21:38:20.741 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:38:20.739+0000 7f5717fff700 1 --2- 172.21.15.45:0/3497997974 >> [v2:172.21.15.45:6800/2012105536,v1:172.21.15.45:6801/2012105536] conn(0x7f5704069040 0x7f570406b4f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:38:20.742 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:38:20.740+0000 7f5717fff700 1 --2- 172.21.15.45:0/3497997974 >> [v2:172.21.15.45:6800/2012105536,v1:172.21.15.45:6801/2012105536] conn(0x7f5704069040 0x7f570406b4f0 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7f571806fc60 tx=0x7f570000b040 comp rx=0 tx=0).ready entity=mgr.14214 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:38:20.751 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:38:20.748+0000 7f5714ff9700 1 -- 172.21.15.45:0/3497997974 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+174570 (secure 0 0 0) 0x7f5708054080 con 0x7f57180fd310 2023-12-16T21:38:21.200 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:38:21.197+0000 7f571e8a1700 1 -- 172.21.15.45:0/3497997974 --> [v2:172.21.15.45:6800/2012105536,v1:172.21.15.45:6801/2012105536] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f57180fbe40 con 0x7f5704069040 2023-12-16T21:38:21.201 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:38:21.199+0000 7f5714ff9700 1 -- 172.21.15.45:0/3497997974 <== mgr.14214 v2:172.21.15.45:6800/2012105536 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+439 (secure 0 0 0) 0x7f57180fbe40 con 0x7f5704069040 2023-12-16T21:38:21.201 INFO:teuthology.orchestra.run.smithi045.stdout:{ 2023-12-16T21:38:21.201 INFO:teuthology.orchestra.run.smithi045.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:45cdc87ca300414c8899f73cd28f3b099dc14234a3689c0162531020ce800bc5", 2023-12-16T21:38:21.201 INFO:teuthology.orchestra.run.smithi045.stdout: "in_progress": true, 2023-12-16T21:38:21.202 INFO:teuthology.orchestra.run.smithi045.stdout: "which": "Upgrading daemons of type(s) mgr", 2023-12-16T21:38:21.202 INFO:teuthology.orchestra.run.smithi045.stdout: "services_complete": [], 2023-12-16T21:38:21.202 INFO:teuthology.orchestra.run.smithi045.stdout: "progress": "0/2 daemons upgraded", 2023-12-16T21:38:21.202 INFO:teuthology.orchestra.run.smithi045.stdout: "message": "Pulling quay.ceph.io/ceph-ci/ceph@sha256:45cdc87ca300414c8899f73cd28f3b099dc14234a3689c0162531020ce800bc5 image on host smithi169", 2023-12-16T21:38:21.202 INFO:teuthology.orchestra.run.smithi045.stdout: "is_paused": false 2023-12-16T21:38:21.202 INFO:teuthology.orchestra.run.smithi045.stdout:} 2023-12-16T21:38:21.205 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:38:21.201+0000 7f571e8a1700 1 -- 172.21.15.45:0/3497997974 >> [v2:172.21.15.45:6800/2012105536,v1:172.21.15.45:6801/2012105536] conn(0x7f5704069040 msgr2=0x7f570406b4f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:38:21.205 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:38:21.201+0000 7f571e8a1700 1 --2- 172.21.15.45:0/3497997974 >> [v2:172.21.15.45:6800/2012105536,v1:172.21.15.45:6801/2012105536] conn(0x7f5704069040 0x7f570406b4f0 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7f571806fc60 tx=0x7f570000b040 comp rx=0 tx=0).stop 2023-12-16T21:38:21.205 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:38:21.201+0000 7f571e8a1700 1 -- 172.21.15.45:0/3497997974 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f57180fd310 msgr2=0x7f57180fb900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:38:21.205 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:38:21.201+0000 7f571e8a1700 1 --2- 172.21.15.45:0/3497997974 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f57180fd310 0x7f57180fb900 secure :-1 s=READY pgs=292 cs=0 l=1 rev1=1 crypto rx=0x7f5708008000 tx=0x7f5708009e30 comp rx=0 tx=0).stop 2023-12-16T21:38:21.205 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:38:21.202+0000 7f571e8a1700 1 -- 172.21.15.45:0/3497997974 shutdown_connections 2023-12-16T21:38:21.205 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:38:21.202+0000 7f571e8a1700 1 --2- 172.21.15.45:0/3497997974 >> [v2:172.21.15.45:6800/2012105536,v1:172.21.15.45:6801/2012105536] conn(0x7f5704069040 0x7f570406b4f0 unknown :-1 s=CLOSED pgs=117 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:38:21.206 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:38:21.202+0000 7f571e8a1700 1 --2- 172.21.15.45:0/3497997974 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f57180fd730 0x7f57180fcdd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:38:21.206 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:38:21.202+0000 7f571e8a1700 1 --2- 172.21.15.45:0/3497997974 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f57180fd310 0x7f57180fb900 unknown :-1 s=CLOSED pgs=292 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:38:21.206 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:38:21.202+0000 7f571e8a1700 1 -- 172.21.15.45:0/3497997974 >> 172.21.15.45:0/3497997974 conn(0x7f57180f71b0 msgr2=0x7f57180f95e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-16T21:38:21.206 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:38:21.202+0000 7f571e8a1700 1 -- 172.21.15.45:0/3497997974 shutdown_connections 2023-12-16T21:38:21.206 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:38:21.202+0000 7f571e8a1700 1 -- 172.21.15.45:0/3497997974 wait complete. 2023-12-16T21:38:21.566 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Dec 16 21:38:21 smithi045 ceph-mon[80763]: from='client.24357 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2023-12-16T21:38:21.566 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Dec 16 21:38:21 smithi045 ceph-mon[80763]: from='client.? 172.21.15.45:0/3880984217' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2023-12-16T21:38:21.566 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Dec 16 21:38:21 smithi045 ceph-mon[80763]: pgmap v147: 65 pgs: 65 active+clean; 586 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 1023 B/s rd, 255 B/s wr, 1 op/s 2023-12-16T21:38:21.566 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Dec 16 21:38:21 smithi045 ceph-mon[80763]: from='mgr.14214 172.21.15.45:0/1227909437' entity='mgr.smithi045.nhtmwc' 2023-12-16T21:38:21.567 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Dec 16 21:38:21 smithi045 ceph-mon[80763]: Upgrade: Target is version 19.0.0-202-gc7586e95 (unknown) 2023-12-16T21:38:21.567 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Dec 16 21:38:21 smithi045 ceph-mon[80763]: Upgrade: Target container is quay.ceph.io/ceph-ci/ceph@sha256:45cdc87ca300414c8899f73cd28f3b099dc14234a3689c0162531020ce800bc5, digests ['quay.ceph.io/ceph-ci/ceph@sha256:45cdc87ca300414c8899f73cd28f3b099dc14234a3689c0162531020ce800bc5'] 2023-12-16T21:38:21.567 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Dec 16 21:38:21 smithi045 ceph-mon[80763]: from='mgr.14214 172.21.15.45:0/1227909437' entity='mgr.smithi045.nhtmwc' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-12-16T21:38:21.567 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Dec 16 21:38:21 smithi045 ceph-mon[80763]: from='mgr.14214 172.21.15.45:0/1227909437' entity='mgr.smithi045.nhtmwc' 2023-12-16T21:38:21.567 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Dec 16 21:38:21 smithi045 ceph-mon[80763]: Upgrade: Need to upgrade myself (mgr.smithi045.nhtmwc) 2023-12-16T21:38:21.633 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Dec 16 21:38:21 smithi169 ceph-mon[88350]: from='client.24357 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2023-12-16T21:38:21.634 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Dec 16 21:38:21 smithi169 ceph-mon[88350]: from='client.? 172.21.15.45:0/3880984217' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2023-12-16T21:38:21.634 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Dec 16 21:38:21 smithi169 ceph-mon[88350]: pgmap v147: 65 pgs: 65 active+clean; 586 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 1023 B/s rd, 255 B/s wr, 1 op/s 2023-12-16T21:38:21.634 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Dec 16 21:38:21 smithi169 ceph-mon[88350]: from='mgr.14214 172.21.15.45:0/1227909437' entity='mgr.smithi045.nhtmwc' 2023-12-16T21:38:21.634 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Dec 16 21:38:21 smithi169 ceph-mon[88350]: Upgrade: Target is version 19.0.0-202-gc7586e95 (unknown) 2023-12-16T21:38:21.634 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Dec 16 21:38:21 smithi169 ceph-mon[88350]: Upgrade: Target container is quay.ceph.io/ceph-ci/ceph@sha256:45cdc87ca300414c8899f73cd28f3b099dc14234a3689c0162531020ce800bc5, digests ['quay.ceph.io/ceph-ci/ceph@sha256:45cdc87ca300414c8899f73cd28f3b099dc14234a3689c0162531020ce800bc5'] 2023-12-16T21:38:21.634 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Dec 16 21:38:21 smithi169 ceph-mon[88350]: from='mgr.14214 172.21.15.45:0/1227909437' entity='mgr.smithi045.nhtmwc' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-12-16T21:38:21.634 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Dec 16 21:38:21 smithi169 ceph-mon[88350]: from='mgr.14214 172.21.15.45:0/1227909437' entity='mgr.smithi045.nhtmwc' 2023-12-16T21:38:21.634 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Dec 16 21:38:21 smithi169 ceph-mon[88350]: Upgrade: Need to upgrade myself (mgr.smithi045.nhtmwc) 2023-12-16T21:38:22.633 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Dec 16 21:38:22 smithi169 ceph-mon[88350]: Upgrade: Pulling quay.ceph.io/ceph-ci/ceph@sha256:45cdc87ca300414c8899f73cd28f3b099dc14234a3689c0162531020ce800bc5 on smithi169 2023-12-16T21:38:22.633 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Dec 16 21:38:22 smithi169 ceph-mon[88350]: from='client.14572 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2023-12-16T21:38:22.816 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Dec 16 21:38:22 smithi045 ceph-mon[80763]: Upgrade: Pulling quay.ceph.io/ceph-ci/ceph@sha256:45cdc87ca300414c8899f73cd28f3b099dc14234a3689c0162531020ce800bc5 on smithi169 2023-12-16T21:38:22.816 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Dec 16 21:38:22 smithi045 ceph-mon[80763]: from='client.14572 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2023-12-16T21:38:23.633 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Dec 16 21:38:23 smithi169 ceph-mon[88350]: pgmap v148: 65 pgs: 65 active+clean; 586 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 1023 B/s rd, 341 B/s wr, 2 op/s 2023-12-16T21:38:23.816 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Dec 16 21:38:23 smithi045 ceph-mon[80763]: pgmap v148: 65 pgs: 65 active+clean; 586 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 1023 B/s rd, 341 B/s wr, 2 op/s 2023-12-16T21:38:26.066 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Dec 16 21:38:25 smithi045 ceph-mon[80763]: pgmap v149: 65 pgs: 65 active+clean; 587 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 1.6 KiB/s rd, 511 B/s wr, 3 op/s 2023-12-16T21:38:26.133 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Dec 16 21:38:25 smithi169 ceph-mon[88350]: pgmap v149: 65 pgs: 65 active+clean; 587 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 1.6 KiB/s rd, 511 B/s wr, 3 op/s 2023-12-16T21:38:27.633 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Dec 16 21:38:27 smithi169 ceph-mon[88350]: pgmap v150: 65 pgs: 65 active+clean; 587 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 1.2 KiB/s rd, 597 B/s wr, 2 op/s 2023-12-16T21:38:27.816 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Dec 16 21:38:27 smithi045 ceph-mon[80763]: pgmap v150: 65 pgs: 65 active+clean; 587 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 1.2 KiB/s rd, 597 B/s wr, 2 op/s 2023-12-16T21:38:29.066 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Dec 16 21:38:28 smithi045 ceph-mon[80763]: from='mgr.14214 172.21.15.45:0/1227909437' entity='mgr.smithi045.nhtmwc' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2023-12-16T21:38:29.133 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Dec 16 21:38:28 smithi169 ceph-mon[88350]: from='mgr.14214 172.21.15.45:0/1227909437' entity='mgr.smithi045.nhtmwc' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2023-12-16T21:38:30.066 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Dec 16 21:38:29 smithi045 ceph-mon[80763]: pgmap v151: 65 pgs: 65 active+clean; 587 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 1.6 KiB/s rd, 597 B/s wr, 3 op/s 2023-12-16T21:38:30.133 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Dec 16 21:38:29 smithi169 ceph-mon[88350]: pgmap v151: 65 pgs: 65 active+clean; 587 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 1.6 KiB/s rd, 597 B/s wr, 3 op/s 2023-12-16T21:38:31.974 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Dec 16 21:38:31 smithi045 ceph-mon[80763]: pgmap v152: 65 pgs: 65 active+clean; 587 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 1023 B/s rd, 341 B/s wr, 2 op/s 2023-12-16T21:38:32.133 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Dec 16 21:38:31 smithi169 ceph-mon[88350]: pgmap v152: 65 pgs: 65 active+clean; 587 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 1023 B/s rd, 341 B/s wr, 2 op/s 2023-12-16T21:38:34.067 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Dec 16 21:38:33 smithi045 ceph-mon[80763]: pgmap v153: 65 pgs: 65 active+clean; 587 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 1023 B/s rd, 426 B/s wr, 2 op/s 2023-12-16T21:38:34.133 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Dec 16 21:38:33 smithi169 ceph-mon[88350]: pgmap v153: 65 pgs: 65 active+clean; 587 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 1023 B/s rd, 426 B/s wr, 2 op/s 2023-12-16T21:38:34.247 INFO:tasks.workunit.client.1.smithi169.stderr:Updating files: 94% (11030/11620) Updating files: 95% (11039/11620) Updating files: 96% (11156/11620) Updating files: 97% (11272/11620) Updating files: 98% (11388/11620) Updating files: 99% (11504/11620) Updating files: 100% (11620/11620) Updating files: 100% (11620/11620), done. 2023-12-16T21:38:34.943 INFO:tasks.workunit.client.1.smithi169.stderr:Note: switching to 'c7586e95519b7be76f3b6e3a4b333888baf46bc9'. 2023-12-16T21:38:34.943 INFO:tasks.workunit.client.1.smithi169.stderr: 2023-12-16T21:38:34.943 INFO:tasks.workunit.client.1.smithi169.stderr:You are in 'detached HEAD' state. You can look around, make experimental 2023-12-16T21:38:34.943 INFO:tasks.workunit.client.1.smithi169.stderr:changes and commit them, and you can discard any commits you make in this 2023-12-16T21:38:34.943 INFO:tasks.workunit.client.1.smithi169.stderr:state without impacting any branches by switching back to a branch. 2023-12-16T21:38:34.943 INFO:tasks.workunit.client.1.smithi169.stderr: 2023-12-16T21:38:34.943 INFO:tasks.workunit.client.1.smithi169.stderr:If you want to create a new branch to retain commits you create, you may 2023-12-16T21:38:34.943 INFO:tasks.workunit.client.1.smithi169.stderr:do so (now or later) by using -c with the switch command. Example: 2023-12-16T21:38:34.944 INFO:tasks.workunit.client.1.smithi169.stderr: 2023-12-16T21:38:34.944 INFO:tasks.workunit.client.1.smithi169.stderr: git switch -c 2023-12-16T21:38:34.944 INFO:tasks.workunit.client.1.smithi169.stderr: 2023-12-16T21:38:34.944 INFO:tasks.workunit.client.1.smithi169.stderr:Or undo this operation with: 2023-12-16T21:38:34.944 INFO:tasks.workunit.client.1.smithi169.stderr: 2023-12-16T21:38:34.944 INFO:tasks.workunit.client.1.smithi169.stderr: git switch - 2023-12-16T21:38:34.944 INFO:tasks.workunit.client.1.smithi169.stderr: 2023-12-16T21:38:34.944 INFO:tasks.workunit.client.1.smithi169.stderr:Turn off this advice by setting config variable advice.detachedHead to false 2023-12-16T21:38:34.944 INFO:tasks.workunit.client.1.smithi169.stderr: 2023-12-16T21:38:34.944 INFO:tasks.workunit.client.1.smithi169.stderr:HEAD is now at c7586e95519 Merge branch 'wip-qa-container-distros-s' of https://github.com/cbodley/ceph into wip-yuri8-testing-2023-12-15-0911 2023-12-16T21:38:34.948 DEBUG:teuthology.orchestra.run.smithi169:> cd -- /home/ubuntu/cephtest/clone.client.1/qa/workunits && if test -e Makefile ; then make ; fi && find -executable -type f -printf '%P\0' >/home/ubuntu/cephtest/workunits.list.client.1 2023-12-16T21:38:35.014 INFO:tasks.workunit.client.1.smithi169.stdout:for d in direct_io fs ; do ( cd $d ; make all ) ; done 2023-12-16T21:38:35.015 INFO:tasks.workunit.client.1.smithi169.stdout:make[1]: Entering directory '/home/ubuntu/cephtest/clone.client.1/qa/workunits/direct_io' 2023-12-16T21:38:35.015 INFO:tasks.workunit.client.1.smithi169.stdout:cc -Wall -Wextra -D_GNU_SOURCE direct_io_test.c -o direct_io_test 2023-12-16T21:38:35.798 INFO:tasks.workunit.client.1.smithi169.stdout:cc -Wall -Wextra -D_GNU_SOURCE test_sync_io.c -o test_sync_io 2023-12-16T21:38:35.852 INFO:tasks.workunit.client.1.smithi169.stdout:cc -Wall -Wextra -D_GNU_SOURCE test_short_dio_read.c -o test_short_dio_read 2023-12-16T21:38:35.892 INFO:tasks.workunit.client.1.smithi169.stdout:make[1]: Leaving directory '/home/ubuntu/cephtest/clone.client.1/qa/workunits/direct_io' 2023-12-16T21:38:35.894 INFO:tasks.workunit.client.1.smithi169.stdout:make[1]: Entering directory '/home/ubuntu/cephtest/clone.client.1/qa/workunits/fs' 2023-12-16T21:38:35.894 INFO:tasks.workunit.client.1.smithi169.stdout:cc -Wall -Wextra -D_GNU_SOURCE test_o_trunc.c -o test_o_trunc 2023-12-16T21:38:35.937 INFO:tasks.workunit.client.1.smithi169.stdout:make[1]: Leaving directory '/home/ubuntu/cephtest/clone.client.1/qa/workunits/fs' 2023-12-16T21:38:35.940 DEBUG:teuthology.orchestra.run.smithi169:> set -ex 2023-12-16T21:38:35.940 DEBUG:teuthology.orchestra.run.smithi169:> dd if=/home/ubuntu/cephtest/workunits.list.client.1 of=/dev/stdout 2023-12-16T21:38:35.996 INFO:tasks.workunit:Running workunits matching suites/fsstress.sh on client.1... 2023-12-16T21:38:35.997 INFO:tasks.workunit:Running workunit suites/fsstress.sh... 2023-12-16T21:38:35.997 DEBUG:teuthology.orchestra.run.smithi169:workunit test suites/fsstress.sh> mkdir -p -- /home/ubuntu/cephtest/mnt.1/client.1/tmp && cd -- /home/ubuntu/cephtest/mnt.1/client.1/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=c7586e95519b7be76f3b6e3a4b333888baf46bc9 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="1" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.1 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.1 CEPH_MNT=/home/ubuntu/cephtest/mnt.1 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.1/qa/workunits/suites/fsstress.sh 2023-12-16T21:38:36.056 INFO:tasks.workunit.client.1.smithi169.stderr:+ mkdir -p fsstress 2023-12-16T21:38:36.057 INFO:tasks.workunit.client.1.smithi169.stderr:+ pushd fsstress 2023-12-16T21:38:36.058 INFO:tasks.workunit.client.1.smithi169.stdout:~/cephtest/mnt.1/client.1/tmp/fsstress ~/cephtest/mnt.1/client.1/tmp 2023-12-16T21:38:36.058 INFO:tasks.workunit.client.1.smithi169.stderr:+ wget -q -O ltp-full.tgz http://download.ceph.com/qa/ltp-full-20091231.tgz 2023-12-16T21:38:36.066 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Dec 16 21:38:35 smithi045 ceph-mon[80763]: pgmap v154: 65 pgs: 65 active+clean; 587 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 1.4 KiB/s rd, 341 B/s wr, 3 op/s 2023-12-16T21:38:36.133 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Dec 16 21:38:35 smithi169 ceph-mon[88350]: pgmap v154: 65 pgs: 65 active+clean; 587 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 1.4 KiB/s rd, 341 B/s wr, 3 op/s 2023-12-16T21:38:36.428 INFO:tasks.workunit.client.0.smithi045.stderr:Note: switching to 'c7586e95519b7be76f3b6e3a4b333888baf46bc9'. 2023-12-16T21:38:36.428 INFO:tasks.workunit.client.0.smithi045.stderr: 2023-12-16T21:38:36.428 INFO:tasks.workunit.client.0.smithi045.stderr:You are in 'detached HEAD' state. You can look around, make experimental 2023-12-16T21:38:36.428 INFO:tasks.workunit.client.0.smithi045.stderr:changes and commit them, and you can discard any commits you make in this 2023-12-16T21:38:36.428 INFO:tasks.workunit.client.0.smithi045.stderr:state without impacting any branches by switching back to a branch. 2023-12-16T21:38:36.428 INFO:tasks.workunit.client.0.smithi045.stderr: 2023-12-16T21:38:36.428 INFO:tasks.workunit.client.0.smithi045.stderr:If you want to create a new branch to retain commits you create, you may 2023-12-16T21:38:36.428 INFO:tasks.workunit.client.0.smithi045.stderr:do so (now or later) by using -c with the switch command. Example: 2023-12-16T21:38:36.429 INFO:tasks.workunit.client.0.smithi045.stderr: 2023-12-16T21:38:36.429 INFO:tasks.workunit.client.0.smithi045.stderr: git switch -c 2023-12-16T21:38:36.429 INFO:tasks.workunit.client.0.smithi045.stderr: 2023-12-16T21:38:36.429 INFO:tasks.workunit.client.0.smithi045.stderr:Or undo this operation with: 2023-12-16T21:38:36.429 INFO:tasks.workunit.client.0.smithi045.stderr: 2023-12-16T21:38:36.429 INFO:tasks.workunit.client.0.smithi045.stderr: git switch - 2023-12-16T21:38:36.429 INFO:tasks.workunit.client.0.smithi045.stderr: 2023-12-16T21:38:36.429 INFO:tasks.workunit.client.0.smithi045.stderr:Turn off this advice by setting config variable advice.detachedHead to false 2023-12-16T21:38:36.429 INFO:tasks.workunit.client.0.smithi045.stderr: 2023-12-16T21:38:36.429 INFO:tasks.workunit.client.0.smithi045.stderr:HEAD is now at c7586e95519 Merge branch 'wip-qa-container-distros-s' of https://github.com/cbodley/ceph into wip-yuri8-testing-2023-12-15-0911 2023-12-16T21:38:36.433 DEBUG:teuthology.orchestra.run.smithi045:> cd -- /home/ubuntu/cephtest/clone.client.0/qa/workunits && if test -e Makefile ; then make ; fi && find -executable -type f -printf '%P\0' >/home/ubuntu/cephtest/workunits.list.client.0 2023-12-16T21:38:36.461 INFO:tasks.workunit.client.0.smithi045.stdout:for d in direct_io fs ; do ( cd $d ; make all ) ; done 2023-12-16T21:38:36.463 INFO:tasks.workunit.client.0.smithi045.stdout:make[1]: Entering directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/direct_io' 2023-12-16T21:38:36.464 INFO:tasks.workunit.client.0.smithi045.stdout:cc -Wall -Wextra -D_GNU_SOURCE direct_io_test.c -o direct_io_test 2023-12-16T21:38:36.835 INFO:tasks.workunit.client.1.smithi169.stderr:+ tar xzf ltp-full.tgz 2023-12-16T21:38:37.228 INFO:tasks.workunit.client.0.smithi045.stdout:cc -Wall -Wextra -D_GNU_SOURCE test_sync_io.c -o test_sync_io 2023-12-16T21:38:37.292 INFO:tasks.workunit.client.0.smithi045.stdout:cc -Wall -Wextra -D_GNU_SOURCE test_short_dio_read.c -o test_short_dio_read 2023-12-16T21:38:37.336 INFO:tasks.workunit.client.0.smithi045.stdout:make[1]: Leaving directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/direct_io' 2023-12-16T21:38:37.337 INFO:tasks.workunit.client.0.smithi045.stdout:make[1]: Entering directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/fs' 2023-12-16T21:38:37.337 INFO:tasks.workunit.client.0.smithi045.stdout:cc -Wall -Wextra -D_GNU_SOURCE test_o_trunc.c -o test_o_trunc 2023-12-16T21:38:37.378 INFO:tasks.workunit.client.0.smithi045.stdout:make[1]: Leaving directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/fs' 2023-12-16T21:38:37.381 DEBUG:teuthology.orchestra.run.smithi045:> set -ex 2023-12-16T21:38:37.381 DEBUG:teuthology.orchestra.run.smithi045:> dd if=/home/ubuntu/cephtest/workunits.list.client.0 of=/dev/stdout 2023-12-16T21:38:37.438 INFO:tasks.workunit:Running workunits matching suites/fsstress.sh on client.0... 2023-12-16T21:38:37.440 INFO:tasks.workunit:Running workunit suites/fsstress.sh... 2023-12-16T21:38:37.440 DEBUG:teuthology.orchestra.run.smithi045:workunit test suites/fsstress.sh> mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=c7586e95519b7be76f3b6e3a4b333888baf46bc9 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/suites/fsstress.sh 2023-12-16T21:38:37.500 INFO:tasks.workunit.client.0.smithi045.stderr:+ mkdir -p fsstress 2023-12-16T21:38:37.502 INFO:tasks.workunit.client.0.smithi045.stderr:+ pushd fsstress 2023-12-16T21:38:37.502 INFO:tasks.workunit.client.0.smithi045.stdout:~/cephtest/mnt.0/client.0/tmp/fsstress ~/cephtest/mnt.0/client.0/tmp 2023-12-16T21:38:37.502 INFO:tasks.workunit.client.0.smithi045.stderr:+ wget -q -O ltp-full.tgz http://download.ceph.com/qa/ltp-full-20091231.tgz 2023-12-16T21:38:37.633 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Dec 16 21:38:37 smithi169 ceph-mon[88350]: pgmap v155: 65 pgs: 65 active+clean; 587 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 853 B/s rd, 170 B/s wr, 2 op/s 2023-12-16T21:38:37.816 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Dec 16 21:38:37 smithi045 ceph-mon[80763]: pgmap v155: 65 pgs: 65 active+clean; 587 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 853 B/s rd, 170 B/s wr, 2 op/s 2023-12-16T21:38:38.280 INFO:tasks.workunit.client.0.smithi045.stderr:+ tar xzf ltp-full.tgz 2023-12-16T21:38:40.066 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Dec 16 21:38:39 smithi045 ceph-mon[80763]: pgmap v156: 65 pgs: 65 active+clean; 1.9 MiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 112 KiB/s rd, 111 KiB/s wr, 3 op/s 2023-12-16T21:38:40.133 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Dec 16 21:38:39 smithi169 ceph-mon[88350]: pgmap v156: 65 pgs: 65 active+clean; 1.9 MiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 112 KiB/s rd, 111 KiB/s wr, 3 op/s 2023-12-16T21:38:42.066 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Dec 16 21:38:41 smithi045 ceph-mon[80763]: pgmap v157: 65 pgs: 65 active+clean; 1.9 MiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 111 KiB/s rd, 111 KiB/s wr, 2 op/s 2023-12-16T21:38:42.133 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Dec 16 21:38:41 smithi169 ceph-mon[88350]: pgmap v157: 65 pgs: 65 active+clean; 1.9 MiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 111 KiB/s rd, 111 KiB/s wr, 2 op/s 2023-12-16T21:38:44.066 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Dec 16 21:38:43 smithi045 ceph-mon[80763]: pgmap v158: 65 pgs: 65 active+clean; 8.6 MiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 453 KiB/s rd, 727 KiB/s wr, 64 op/s 2023-12-16T21:38:44.066 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Dec 16 21:38:43 smithi045 ceph-mon[80763]: from='mgr.14214 172.21.15.45:0/1227909437' entity='mgr.smithi045.nhtmwc' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2023-12-16T21:38:44.133 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Dec 16 21:38:43 smithi169 ceph-mon[88350]: pgmap v158: 65 pgs: 65 active+clean; 8.6 MiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 453 KiB/s rd, 727 KiB/s wr, 64 op/s 2023-12-16T21:38:44.133 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Dec 16 21:38:43 smithi169 ceph-mon[88350]: from='mgr.14214 172.21.15.45:0/1227909437' entity='mgr.smithi045.nhtmwc' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2023-12-16T21:38:46.069 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Dec 16 21:38:45 smithi045 ceph-mon[80763]: pgmap v159: 65 pgs: 65 active+clean; 40 MiB data, 1.8 GiB used, 535 GiB / 536 GiB avail; 886 KiB/s rd, 3.5 MiB/s wr, 305 op/s 2023-12-16T21:38:46.133 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Dec 16 21:38:45 smithi169 ceph-mon[88350]: pgmap v159: 65 pgs: 65 active+clean; 40 MiB data, 1.8 GiB used, 535 GiB / 536 GiB avail; 886 KiB/s rd, 3.5 MiB/s wr, 305 op/s 2023-12-16T21:38:47.633 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Dec 16 21:38:47 smithi169 ceph-mon[88350]: pgmap v160: 65 pgs: 65 active+clean; 50 MiB data, 1.9 GiB used, 535 GiB / 536 GiB avail; 885 KiB/s rd, 4.3 MiB/s wr, 374 op/s 2023-12-16T21:38:47.816 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Dec 16 21:38:47 smithi045 ceph-mon[80763]: pgmap v160: 65 pgs: 65 active+clean; 50 MiB data, 1.9 GiB used, 535 GiB / 536 GiB avail; 885 KiB/s rd, 4.3 MiB/s wr, 374 op/s 2023-12-16T21:38:50.066 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Dec 16 21:38:49 smithi045 ceph-mon[80763]: pgmap v161: 65 pgs: 65 active+clean; 68 MiB data, 1.9 GiB used, 534 GiB / 536 GiB avail; 2.0 MiB/s rd, 5.9 MiB/s wr, 444 op/s 2023-12-16T21:38:50.133 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Dec 16 21:38:49 smithi169 ceph-mon[88350]: pgmap v161: 65 pgs: 65 active+clean; 68 MiB data, 1.9 GiB used, 534 GiB / 536 GiB avail; 2.0 MiB/s rd, 5.9 MiB/s wr, 444 op/s 2023-12-16T21:38:51.326 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:38:51.322+0000 7fc2a5506700 1 -- 172.21.15.45:0/329889565 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fc2a0100050 msgr2=0x7fc2a0102430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:38:51.327 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:38:51.322+0000 7fc2a5506700 1 --2- 172.21.15.45:0/329889565 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fc2a0100050 0x7fc2a0102430 secure :-1 s=READY pgs=293 cs=0 l=1 rev1=1 crypto rx=0x7fc288004940 tx=0x7fc288009e30 comp rx=0 tx=0).stop 2023-12-16T21:38:51.327 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:38:51.322+0000 7fc2a5506700 1 -- 172.21.15.45:0/329889565 shutdown_connections 2023-12-16T21:38:51.327 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:38:51.322+0000 7fc2a5506700 1 --2- 172.21.15.45:0/329889565 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fc2a0100050 0x7fc2a0102430 unknown :-1 s=CLOSED pgs=293 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:38:51.327 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:38:51.322+0000 7fc2a5506700 1 --2- 172.21.15.45:0/329889565 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fc2a00fd730 0x7fc2a00ffb10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:38:51.327 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:38:51.322+0000 7fc2a5506700 1 -- 172.21.15.45:0/329889565 >> 172.21.15.45:0/329889565 conn(0x7fc2a00f7190 msgr2=0x7fc2a00f95e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-16T21:38:51.327 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:38:51.322+0000 7fc2a5506700 1 -- 172.21.15.45:0/329889565 shutdown_connections 2023-12-16T21:38:51.327 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:38:51.322+0000 7fc2a5506700 1 -- 172.21.15.45:0/329889565 wait complete. 2023-12-16T21:38:51.327 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:38:51.323+0000 7fc2a5506700 1 Processor -- start 2023-12-16T21:38:51.327 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:38:51.323+0000 7fc2a5506700 1 -- start start 2023-12-16T21:38:51.327 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:38:51.323+0000 7fc2a5506700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fc2a00fd730 0x7fc2a010d4c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:38:51.327 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:38:51.323+0000 7fc2a5506700 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fc2a0100050 0x7fc2a010da00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:38:51.327 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:38:51.323+0000 7fc2a5506700 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc2a010e000 con 0x7fc2a00fd730 2023-12-16T21:38:51.327 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:38:51.323+0000 7fc2a5506700 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc2a010e8a0 con 0x7fc2a0100050 2023-12-16T21:38:51.327 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:38:51.324+0000 7fc29effd700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fc2a00fd730 0x7fc2a010d4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:38:51.328 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:38:51.324+0000 7fc29effd700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fc2a00fd730 0x7fc2a010d4c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:44240/0 (socket says 172.21.15.45:44240) 2023-12-16T21:38:51.328 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:38:51.324+0000 7fc29effd700 1 -- 172.21.15.45:0/1899448318 learned_addr learned my addr 172.21.15.45:0/1899448318 (peer_addr_for_me v2:172.21.15.45:0/0) 2023-12-16T21:38:51.328 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:38:51.324+0000 7fc29e7fc700 1 --2- 172.21.15.45:0/1899448318 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fc2a0100050 0x7fc2a010da00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:38:51.328 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:38:51.324+0000 7fc29effd700 1 -- 172.21.15.45:0/1899448318 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fc2a0100050 msgr2=0x7fc2a010da00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:38:51.329 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:38:51.324+0000 7fc29effd700 1 --2- 172.21.15.45:0/1899448318 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fc2a0100050 0x7fc2a010da00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:38:51.329 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:38:51.324+0000 7fc29effd700 1 -- 172.21.15.45:0/1899448318 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc288005040 con 0x7fc2a00fd730 2023-12-16T21:38:51.329 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:38:51.325+0000 7fc29effd700 1 --2- 172.21.15.45:0/1899448318 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fc2a00fd730 0x7fc2a010d4c0 secure :-1 s=READY pgs=294 cs=0 l=1 rev1=1 crypto rx=0x7fc29000ec30 tx=0x7fc29000b040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:38:51.329 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:38:51.325+0000 7fc2977fe700 1 -- 172.21.15.45:0/1899448318 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fc290007b30 con 0x7fc2a00fd730 2023-12-16T21:38:51.329 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:38:51.325+0000 7fc2977fe700 1 -- 172.21.15.45:0/1899448318 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fc29000f040 con 0x7fc2a00fd730 2023-12-16T21:38:51.329 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:38:51.325+0000 7fc2a5506700 1 -- 172.21.15.45:0/1899448318 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc2a010eb80 con 0x7fc2a00fd730 2023-12-16T21:38:51.329 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:38:51.325+0000 7fc2977fe700 1 -- 172.21.15.45:0/1899448318 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fc29000cbd0 con 0x7fc2a00fd730 2023-12-16T21:38:51.329 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:38:51.325+0000 7fc2a5506700 1 -- 172.21.15.45:0/1899448318 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc2a010f0c0 con 0x7fc2a00fd730 2023-12-16T21:38:51.329 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:38:51.326+0000 7fc2a5506700 1 -- 172.21.15.45:0/1899448318 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc2800052e0 con 0x7fc2a00fd730 2023-12-16T21:38:51.330 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:38:51.327+0000 7fc2977fe700 1 -- 172.21.15.45:0/1899448318 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 19) v1 ==== 86447+0+0 (secure 0 0 0) 0x7fc290014480 con 0x7fc2a00fd730 2023-12-16T21:38:51.330 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:38:51.328+0000 7fc2977fe700 1 --2- 172.21.15.45:0/1899448318 >> [v2:172.21.15.45:6800/2012105536,v1:172.21.15.45:6801/2012105536] conn(0x7fc28c068f60 0x7fc28c06b410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:38:51.330 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:38:51.328+0000 7fc2977fe700 1 -- 172.21.15.45:0/1899448318 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(43..43 src has 1..43) v4 ==== 5550+0+0 (secure 0 0 0) 0x7fc290087290 con 0x7fc2a00fd730 2023-12-16T21:38:51.331 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:38:51.328+0000 7fc29e7fc700 1 --2- 172.21.15.45:0/1899448318 >> [v2:172.21.15.45:6800/2012105536,v1:172.21.15.45:6801/2012105536] conn(0x7fc28c068f60 0x7fc28c06b410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:38:51.331 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:38:51.329+0000 7fc29e7fc700 1 --2- 172.21.15.45:0/1899448318 >> [v2:172.21.15.45:6800/2012105536,v1:172.21.15.45:6801/2012105536] conn(0x7fc28c068f60 0x7fc28c06b410 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7fc288000c00 tx=0x7fc288009e30 comp rx=0 tx=0).ready entity=mgr.14214 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:38:51.338 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:38:51.336+0000 7fc2977fe700 1 -- 172.21.15.45:0/1899448318 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+174570 (secure 0 0 0) 0x7fc290057190 con 0x7fc2a00fd730 2023-12-16T21:38:51.573 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:38:51.571+0000 7fc2a5506700 1 -- 172.21.15.45:0/1899448318 --> [v2:172.21.15.45:6800/2012105536,v1:172.21.15.45:6801/2012105536] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7fc280000bf0 con 0x7fc28c068f60 2023-12-16T21:38:51.574 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:38:51.572+0000 7fc2977fe700 1 -- 172.21.15.45:0/1899448318 <== mgr.14214 v2:172.21.15.45:6800/2012105536 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+439 (secure 0 0 0) 0x7fc280000bf0 con 0x7fc28c068f60 2023-12-16T21:38:51.576 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:38:51.574+0000 7fc2a5506700 1 -- 172.21.15.45:0/1899448318 >> [v2:172.21.15.45:6800/2012105536,v1:172.21.15.45:6801/2012105536] conn(0x7fc28c068f60 msgr2=0x7fc28c06b410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:38:51.576 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:38:51.574+0000 7fc2a5506700 1 --2- 172.21.15.45:0/1899448318 >> [v2:172.21.15.45:6800/2012105536,v1:172.21.15.45:6801/2012105536] conn(0x7fc28c068f60 0x7fc28c06b410 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7fc288000c00 tx=0x7fc288009e30 comp rx=0 tx=0).stop 2023-12-16T21:38:51.576 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:38:51.574+0000 7fc2a5506700 1 -- 172.21.15.45:0/1899448318 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fc2a00fd730 msgr2=0x7fc2a010d4c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:38:51.576 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:38:51.574+0000 7fc2a5506700 1 --2- 172.21.15.45:0/1899448318 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fc2a00fd730 0x7fc2a010d4c0 secure :-1 s=READY pgs=294 cs=0 l=1 rev1=1 crypto rx=0x7fc29000ec30 tx=0x7fc29000b040 comp rx=0 tx=0).stop 2023-12-16T21:38:51.577 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:38:51.574+0000 7fc2a5506700 1 -- 172.21.15.45:0/1899448318 shutdown_connections 2023-12-16T21:38:51.577 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:38:51.574+0000 7fc2a5506700 1 --2- 172.21.15.45:0/1899448318 >> [v2:172.21.15.45:6800/2012105536,v1:172.21.15.45:6801/2012105536] conn(0x7fc28c068f60 0x7fc28c06b410 unknown :-1 s=CLOSED pgs=118 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:38:51.577 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:38:51.574+0000 7fc2a5506700 1 --2- 172.21.15.45:0/1899448318 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fc2a00fd730 0x7fc2a010d4c0 unknown :-1 s=CLOSED pgs=294 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:38:51.577 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:38:51.574+0000 7fc2a5506700 1 --2- 172.21.15.45:0/1899448318 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fc2a0100050 0x7fc2a010da00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:38:51.577 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:38:51.574+0000 7fc2a5506700 1 -- 172.21.15.45:0/1899448318 >> 172.21.15.45:0/1899448318 conn(0x7fc2a00f7190 msgr2=0x7fc2a00f95e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-16T21:38:51.577 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:38:51.574+0000 7fc2a5506700 1 -- 172.21.15.45:0/1899448318 shutdown_connections 2023-12-16T21:38:51.577 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:38:51.574+0000 7fc2a5506700 1 -- 172.21.15.45:0/1899448318 wait complete. 2023-12-16T21:38:51.589 INFO:teuthology.orchestra.run.smithi045.stdout:true 2023-12-16T21:38:51.685 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:38:51.682+0000 7fe16bc6a700 1 -- 172.21.15.45:0/845256396 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fe1640fde00 msgr2=0x7fe1640fe250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:38:51.685 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:38:51.682+0000 7fe16bc6a700 1 --2- 172.21.15.45:0/845256396 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fe1640fde00 0x7fe1640fe250 secure :-1 s=READY pgs=295 cs=0 l=1 rev1=1 crypto rx=0x7fe154004940 tx=0x7fe154009e30 comp rx=0 tx=0).stop 2023-12-16T21:38:51.685 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:38:51.682+0000 7fe16bc6a700 1 -- 172.21.15.45:0/845256396 shutdown_connections 2023-12-16T21:38:51.685 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:38:51.682+0000 7fe16bc6a700 1 --2- 172.21.15.45:0/845256396 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fe1640fde00 0x7fe1640fe250 unknown :-1 s=CLOSED pgs=295 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:38:51.686 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:38:51.682+0000 7fe16bc6a700 1 --2- 172.21.15.45:0/845256396 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fe1640fcbf0 0x7fe1640fd000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:38:51.686 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:38:51.682+0000 7fe16bc6a700 1 -- 172.21.15.45:0/845256396 >> 172.21.15.45:0/845256396 conn(0x7fe1640f82d0 msgr2=0x7fe1640fa6e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-16T21:38:51.686 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:38:51.683+0000 7fe16bc6a700 1 -- 172.21.15.45:0/845256396 shutdown_connections 2023-12-16T21:38:51.686 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:38:51.683+0000 7fe16bc6a700 1 -- 172.21.15.45:0/845256396 wait complete. 2023-12-16T21:38:51.687 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:38:51.683+0000 7fe16bc6a700 1 Processor -- start 2023-12-16T21:38:51.687 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:38:51.683+0000 7fe16bc6a700 1 -- start start 2023-12-16T21:38:51.687 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:38:51.683+0000 7fe16bc6a700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fe1640fcbf0 0x7fe16410f6b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:38:51.687 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:38:51.683+0000 7fe16bc6a700 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fe1640fde00 0x7fe16410fbf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:38:51.688 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:38:51.683+0000 7fe16bc6a700 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe1641101f0 con 0x7fe1640fcbf0 2023-12-16T21:38:51.688 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:38:51.683+0000 7fe16bc6a700 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe164110a90 con 0x7fe1640fde00 2023-12-16T21:38:51.688 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:38:51.684+0000 7fe169a06700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fe1640fcbf0 0x7fe16410f6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:38:51.688 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:38:51.684+0000 7fe169a06700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fe1640fcbf0 0x7fe16410f6b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:44260/0 (socket says 172.21.15.45:44260) 2023-12-16T21:38:51.688 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:38:51.684+0000 7fe169a06700 1 -- 172.21.15.45:0/1891802903 learned_addr learned my addr 172.21.15.45:0/1891802903 (peer_addr_for_me v2:172.21.15.45:0/0) 2023-12-16T21:38:51.688 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:38:51.684+0000 7fe169205700 1 --2- 172.21.15.45:0/1891802903 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fe1640fde00 0x7fe16410fbf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:38:51.688 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:38:51.684+0000 7fe169a06700 1 -- 172.21.15.45:0/1891802903 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fe1640fde00 msgr2=0x7fe16410fbf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:38:51.688 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:38:51.684+0000 7fe169a06700 1 --2- 172.21.15.45:0/1891802903 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fe1640fde00 0x7fe16410fbf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:38:51.688 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:38:51.684+0000 7fe169a06700 1 -- 172.21.15.45:0/1891802903 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe154005040 con 0x7fe1640fcbf0 2023-12-16T21:38:51.688 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:38:51.684+0000 7fe169205700 1 --2- 172.21.15.45:0/1891802903 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fe1640fde00 0x7fe16410fbf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-12-16T21:38:51.688 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:38:51.684+0000 7fe169a06700 1 --2- 172.21.15.45:0/1891802903 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fe1640fcbf0 0x7fe16410f6b0 secure :-1 s=READY pgs=296 cs=0 l=1 rev1=1 crypto rx=0x7fe16000eb70 tx=0x7fe16000b040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:38:51.688 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:38:51.684+0000 7fe15a7fc700 1 -- 172.21.15.45:0/1891802903 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fe160007cc0 con 0x7fe1640fcbf0 2023-12-16T21:38:51.688 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:38:51.684+0000 7fe15a7fc700 1 -- 172.21.15.45:0/1891802903 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fe16000f040 con 0x7fe1640fcbf0 2023-12-16T21:38:51.688 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:38:51.684+0000 7fe16bc6a700 1 -- 172.21.15.45:0/1891802903 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe164110d10 con 0x7fe1640fcbf0 2023-12-16T21:38:51.688 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:38:51.684+0000 7fe15a7fc700 1 -- 172.21.15.45:0/1891802903 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fe16000c9d0 con 0x7fe1640fcbf0 2023-12-16T21:38:51.689 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:38:51.684+0000 7fe16bc6a700 1 -- 172.21.15.45:0/1891802903 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe164111250 con 0x7fe1640fcbf0 2023-12-16T21:38:51.690 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:38:51.685+0000 7fe16bc6a700 1 -- 172.21.15.45:0/1891802903 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe164101ea0 con 0x7fe1640fcbf0 2023-12-16T21:38:51.690 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:38:51.686+0000 7fe15a7fc700 1 -- 172.21.15.45:0/1891802903 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 19) v1 ==== 86447+0+0 (secure 0 0 0) 0x7fe16000cb30 con 0x7fe1640fcbf0 2023-12-16T21:38:51.690 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:38:51.686+0000 7fe15a7fc700 1 --2- 172.21.15.45:0/1891802903 >> [v2:172.21.15.45:6800/2012105536,v1:172.21.15.45:6801/2012105536] conn(0x7fe150068f60 0x7fe15006b410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:38:51.690 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:38:51.687+0000 7fe15a7fc700 1 -- 172.21.15.45:0/1891802903 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(43..43 src has 1..43) v4 ==== 5550+0+0 (secure 0 0 0) 0x7fe160086800 con 0x7fe1640fcbf0 2023-12-16T21:38:51.690 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:38:51.687+0000 7fe169205700 1 --2- 172.21.15.45:0/1891802903 >> [v2:172.21.15.45:6800/2012105536,v1:172.21.15.45:6801/2012105536] conn(0x7fe150068f60 0x7fe15006b410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:38:51.690 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:38:51.687+0000 7fe169205700 1 --2- 172.21.15.45:0/1891802903 >> [v2:172.21.15.45:6800/2012105536,v1:172.21.15.45:6801/2012105536] conn(0x7fe150068f60 0x7fe15006b410 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7fe154000c00 tx=0x7fe154009e30 comp rx=0 tx=0).ready entity=mgr.14214 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:38:51.696 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:38:51.693+0000 7fe15a7fc700 1 -- 172.21.15.45:0/1891802903 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+174570 (secure 0 0 0) 0x7fe16004da00 con 0x7fe1640fcbf0 2023-12-16T21:38:51.955 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:38:51.951+0000 7fe16bc6a700 1 -- 172.21.15.45:0/1891802903 --> [v2:172.21.15.45:6800/2012105536,v1:172.21.15.45:6801/2012105536] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7fe16402b150 con 0x7fe150068f60 2023-12-16T21:38:51.955 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Dec 16 21:38:51 smithi045 ceph-mon[80763]: pgmap v162: 65 pgs: 65 active+clean; 68 MiB data, 1.9 GiB used, 534 GiB / 536 GiB avail; 1.9 MiB/s rd, 5.8 MiB/s wr, 443 op/s 2023-12-16T21:38:51.959 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:38:51.957+0000 7fe15a7fc700 1 -- 172.21.15.45:0/1891802903 <== mgr.14214 v2:172.21.15.45:6800/2012105536 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+439 (secure 0 0 0) 0x7fe16402b150 con 0x7fe150068f60 2023-12-16T21:38:51.963 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:38:51.960+0000 7fe16bc6a700 1 -- 172.21.15.45:0/1891802903 >> [v2:172.21.15.45:6800/2012105536,v1:172.21.15.45:6801/2012105536] conn(0x7fe150068f60 msgr2=0x7fe15006b410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:38:51.963 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:38:51.960+0000 7fe16bc6a700 1 --2- 172.21.15.45:0/1891802903 >> [v2:172.21.15.45:6800/2012105536,v1:172.21.15.45:6801/2012105536] conn(0x7fe150068f60 0x7fe15006b410 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7fe154000c00 tx=0x7fe154009e30 comp rx=0 tx=0).stop 2023-12-16T21:38:51.964 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:38:51.960+0000 7fe16bc6a700 1 -- 172.21.15.45:0/1891802903 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fe1640fcbf0 msgr2=0x7fe16410f6b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:38:51.964 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:38:51.960+0000 7fe16bc6a700 1 --2- 172.21.15.45:0/1891802903 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fe1640fcbf0 0x7fe16410f6b0 secure :-1 s=READY pgs=296 cs=0 l=1 rev1=1 crypto rx=0x7fe16000eb70 tx=0x7fe16000b040 comp rx=0 tx=0).stop 2023-12-16T21:38:51.965 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:38:51.962+0000 7fe16bc6a700 1 -- 172.21.15.45:0/1891802903 shutdown_connections 2023-12-16T21:38:51.965 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:38:51.962+0000 7fe16bc6a700 1 --2- 172.21.15.45:0/1891802903 >> [v2:172.21.15.45:6800/2012105536,v1:172.21.15.45:6801/2012105536] conn(0x7fe150068f60 0x7fe15006b410 unknown :-1 s=CLOSED pgs=119 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:38:51.965 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:38:51.962+0000 7fe16bc6a700 1 --2- 172.21.15.45:0/1891802903 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fe1640fcbf0 0x7fe16410f6b0 unknown :-1 s=CLOSED pgs=296 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:38:51.965 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:38:51.962+0000 7fe16bc6a700 1 --2- 172.21.15.45:0/1891802903 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fe1640fde00 0x7fe16410fbf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:38:51.965 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:38:51.962+0000 7fe16bc6a700 1 -- 172.21.15.45:0/1891802903 >> 172.21.15.45:0/1891802903 conn(0x7fe1640f82d0 msgr2=0x7fe1640ff020 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-16T21:38:51.965 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:38:51.963+0000 7fe16bc6a700 1 -- 172.21.15.45:0/1891802903 shutdown_connections 2023-12-16T21:38:51.966 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:38:51.963+0000 7fe16bc6a700 1 -- 172.21.15.45:0/1891802903 wait complete. 2023-12-16T21:38:52.133 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Dec 16 21:38:51 smithi169 ceph-mon[88350]: pgmap v162: 65 pgs: 65 active+clean; 68 MiB data, 1.9 GiB used, 534 GiB / 536 GiB avail; 1.9 MiB/s rd, 5.8 MiB/s wr, 443 op/s 2023-12-16T21:38:52.162 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:38:52.158+0000 7f14e6fc4700 1 -- 172.21.15.45:0/2016547576 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f14e0104410 msgr2=0x7f14e0104860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:38:52.162 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:38:52.158+0000 7f14e6fc4700 1 --2- 172.21.15.45:0/2016547576 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f14e0104410 0x7f14e0104860 secure :-1 s=READY pgs=297 cs=0 l=1 rev1=1 crypto rx=0x7f14d8009ce0 tx=0x7f14d8004d10 comp rx=0 tx=0).stop 2023-12-16T21:38:52.162 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:38:52.158+0000 7f14e6fc4700 1 -- 172.21.15.45:0/2016547576 shutdown_connections 2023-12-16T21:38:52.162 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:38:52.158+0000 7f14e6fc4700 1 --2- 172.21.15.45:0/2016547576 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f14e0104410 0x7f14e0104860 unknown :-1 s=CLOSED pgs=297 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:38:52.162 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:38:52.158+0000 7f14e6fc4700 1 --2- 172.21.15.45:0/2016547576 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f14e0103250 0x7f14e0103660 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:38:52.162 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:38:52.158+0000 7f14e6fc4700 1 -- 172.21.15.45:0/2016547576 >> 172.21.15.45:0/2016547576 conn(0x7f14e00665a0 msgr2=0x7f14e00669c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-16T21:38:52.162 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:38:52.159+0000 7f14e6fc4700 1 -- 172.21.15.45:0/2016547576 shutdown_connections 2023-12-16T21:38:52.164 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:38:52.160+0000 7f14e6fc4700 1 -- 172.21.15.45:0/2016547576 wait complete. 2023-12-16T21:38:52.164 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:38:52.161+0000 7f14e6fc4700 1 Processor -- start 2023-12-16T21:38:52.164 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:38:52.161+0000 7f14e6fc4700 1 -- start start 2023-12-16T21:38:52.164 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:38:52.161+0000 7f14e6fc4700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f14e0103250 0x7f14e006ee70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:38:52.165 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:38:52.161+0000 7f14e6fc4700 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f14e0104410 0x7f14e006f3f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:38:52.165 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:38:52.161+0000 7f14e6fc4700 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f14e006fa70 con 0x7f14e0103250 2023-12-16T21:38:52.165 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:38:52.161+0000 7f14e6fc4700 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f14e011d800 con 0x7f14e0104410 2023-12-16T21:38:52.165 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:38:52.162+0000 7f14e4d60700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f14e0103250 0x7f14e006ee70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:38:52.166 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:38:52.162+0000 7f14e4d60700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f14e0103250 0x7f14e006ee70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:44274/0 (socket says 172.21.15.45:44274) 2023-12-16T21:38:52.166 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:38:52.162+0000 7f14e4d60700 1 -- 172.21.15.45:0/1231441671 learned_addr learned my addr 172.21.15.45:0/1231441671 (peer_addr_for_me v2:172.21.15.45:0/0) 2023-12-16T21:38:52.166 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:38:52.162+0000 7f14dffff700 1 --2- 172.21.15.45:0/1231441671 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f14e0104410 0x7f14e006f3f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:38:52.167 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:38:52.164+0000 7f14dffff700 1 -- 172.21.15.45:0/1231441671 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f14e0103250 msgr2=0x7f14e006ee70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:38:52.167 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:38:52.164+0000 7f14dffff700 1 --2- 172.21.15.45:0/1231441671 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f14e0103250 0x7f14e006ee70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:38:52.167 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:38:52.164+0000 7f14dffff700 1 -- 172.21.15.45:0/1231441671 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f14d8009940 con 0x7f14e0104410 2023-12-16T21:38:52.167 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:38:52.165+0000 7f14dffff700 1 --2- 172.21.15.45:0/1231441671 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f14e0104410 0x7f14e006f3f0 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7f14d8000f80 tx=0x7f14d8004d10 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:38:52.169 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:38:52.165+0000 7f14dd7fa700 1 -- 172.21.15.45:0/1231441671 <== mon.1 v2:172.21.15.169:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f14d800e070 con 0x7f14e0104410 2023-12-16T21:38:52.169 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:38:52.165+0000 7f14e6fc4700 1 -- 172.21.15.45:0/1231441671 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f14e011da80 con 0x7f14e0104410 2023-12-16T21:38:52.170 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:38:52.165+0000 7f14dd7fa700 1 -- 172.21.15.45:0/1231441671 <== mon.1 v2:172.21.15.169:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f14d80128c0 con 0x7f14e0104410 2023-12-16T21:38:52.170 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:38:52.165+0000 7f14dd7fa700 1 -- 172.21.15.45:0/1231441671 <== mon.1 v2:172.21.15.169:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f14d800e070 con 0x7f14e0104410 2023-12-16T21:38:52.170 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:38:52.165+0000 7f14e6fc4700 1 -- 172.21.15.45:0/1231441671 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f14e011dfc0 con 0x7f14e0104410 2023-12-16T21:38:52.170 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:38:52.167+0000 7f14e6fc4700 1 -- 172.21.15.45:0/1231441671 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f14e0068d90 con 0x7f14e0104410 2023-12-16T21:38:52.171 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:38:52.168+0000 7f14dd7fa700 1 -- 172.21.15.45:0/1231441671 <== mon.1 v2:172.21.15.169:3300/0 4 ==== mgrmap(e 19) v1 ==== 86447+0+0 (secure 0 0 0) 0x7f14d8019040 con 0x7f14e0104410 2023-12-16T21:38:52.171 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:38:52.168+0000 7f14dd7fa700 1 --2- 172.21.15.45:0/1231441671 >> [v2:172.21.15.45:6800/2012105536,v1:172.21.15.45:6801/2012105536] conn(0x7f14cc069070 0x7f14cc06b520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:38:52.171 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:38:52.169+0000 7f14dd7fa700 1 -- 172.21.15.45:0/1231441671 <== mon.1 v2:172.21.15.169:3300/0 5 ==== osd_map(43..43 src has 1..43) v4 ==== 5550+0+0 (secure 0 0 0) 0x7f14d8088290 con 0x7f14e0104410 2023-12-16T21:38:52.171 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:38:52.169+0000 7f14e4d60700 1 --2- 172.21.15.45:0/1231441671 >> [v2:172.21.15.45:6800/2012105536,v1:172.21.15.45:6801/2012105536] conn(0x7f14cc069070 0x7f14cc06b520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:38:52.172 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:38:52.170+0000 7f14e4d60700 1 --2- 172.21.15.45:0/1231441671 >> [v2:172.21.15.45:6800/2012105536,v1:172.21.15.45:6801/2012105536] conn(0x7f14cc069070 0x7f14cc06b520 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7f14e006ec40 tx=0x7f14c800b040 comp rx=0 tx=0).ready entity=mgr.14214 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:38:52.176 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:38:52.173+0000 7f14dd7fa700 1 -- 172.21.15.45:0/1231441671 <== mon.1 v2:172.21.15.169:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+174570 (secure 0 0 0) 0x7f14d80546b0 con 0x7f14e0104410 2023-12-16T21:38:52.496 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:38:52.493+0000 7f14e6fc4700 1 -- 172.21.15.45:0/1231441671 --> [v2:172.21.15.45:6800/2012105536,v1:172.21.15.45:6801/2012105536] -- mgr_command(tid 0: {"prefix": "orch ps", "target": ["mon-mgr", ""]}) v1 -- 0x7f14e011e2a0 con 0x7f14cc069070 2023-12-16T21:38:52.506 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:38:52.503+0000 7f14dd7fa700 1 -- 172.21.15.45:0/1231441671 <== mgr.14214 v2:172.21.15.45:6800/2012105536 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+3366 (secure 0 0 0) 0x7f14e011e2a0 con 0x7f14cc069070 2023-12-16T21:38:52.506 INFO:teuthology.orchestra.run.smithi045.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2023-12-16T21:38:52.506 INFO:teuthology.orchestra.run.smithi045.stdout:alertmanager.smithi045 smithi045 *:9093,9094 running (3m) 78s ago 4m 26.1M - 0.25.0 c8568f914cd2 d9982231d4a2 2023-12-16T21:38:52.506 INFO:teuthology.orchestra.run.smithi045.stdout:crash.smithi045 smithi045 running (5m) 78s ago 5m 6660k - 17.2.7-125-g1f2f5510 60cc83a9c8f1 fc997f70d946 2023-12-16T21:38:52.506 INFO:teuthology.orchestra.run.smithi045.stdout:crash.smithi169 smithi169 running (4m) 80s ago 4m 6665k - 17.2.7-125-g1f2f5510 60cc83a9c8f1 f29fac8b7321 2023-12-16T21:38:52.506 INFO:teuthology.orchestra.run.smithi045.stdout:grafana.smithi045 smithi045 *:3000 running (3m) 78s ago 4m 84.9M - 9.4.7 954c08fa6188 dddae79633f0 2023-12-16T21:38:52.506 INFO:teuthology.orchestra.run.smithi045.stdout:mds.cephfs.smithi045.fgsbgq smithi045 running (91s) 78s ago 90s 10.4M - 17.2.7-125-g1f2f5510 60cc83a9c8f1 9c41632d4c6e 2023-12-16T21:38:52.506 INFO:teuthology.orchestra.run.smithi045.stdout:mds.cephfs.smithi045.hkjgzw smithi045 running (87s) 78s ago 87s 11.9M - 17.2.7-125-g1f2f5510 60cc83a9c8f1 140efc1dc83a 2023-12-16T21:38:52.507 INFO:teuthology.orchestra.run.smithi045.stdout:mds.cephfs.smithi169.mokxju smithi169 running (89s) 80s ago 89s 13.1M - 17.2.7-125-g1f2f5510 60cc83a9c8f1 2c6edcc7b959 2023-12-16T21:38:52.507 INFO:teuthology.orchestra.run.smithi045.stdout:mds.cephfs.smithi169.pihlib smithi169 running (86s) 80s ago 86s 13.4M - 17.2.7-125-g1f2f5510 60cc83a9c8f1 ca40c392fd88 2023-12-16T21:38:52.507 INFO:teuthology.orchestra.run.smithi045.stdout:mgr.smithi045.nhtmwc smithi045 *:9283 running (5m) 78s ago 5m 477M - 17.2.7-125-g1f2f5510 60cc83a9c8f1 a4ec7bd78bc7 2023-12-16T21:38:52.507 INFO:teuthology.orchestra.run.smithi045.stdout:mgr.smithi169.cncbgh smithi169 *:8443,9283 running (4m) 80s ago 4m 432M - 17.2.7-125-g1f2f5510 60cc83a9c8f1 034b205acb79 2023-12-16T21:38:52.507 INFO:teuthology.orchestra.run.smithi045.stdout:mon.smithi045 smithi045 running (6m) 78s ago 6m 50.0M 2048M 17.2.7-125-g1f2f5510 60cc83a9c8f1 1cddbf5059f8 2023-12-16T21:38:52.507 INFO:teuthology.orchestra.run.smithi045.stdout:mon.smithi169 smithi169 running (4m) 80s ago 4m 43.2M 2048M 17.2.7-125-g1f2f5510 60cc83a9c8f1 f964ea9d81bd 2023-12-16T21:38:52.507 INFO:teuthology.orchestra.run.smithi045.stdout:node-exporter.smithi045 smithi045 *:9100 running (5m) 78s ago 5m 16.0M - 1.5.0 0da6a335fe13 c535836aef3c 2023-12-16T21:38:52.507 INFO:teuthology.orchestra.run.smithi045.stdout:node-exporter.smithi169 smithi169 *:9100 running (4m) 80s ago 4m 15.9M - 1.5.0 0da6a335fe13 f6778c39f7f1 2023-12-16T21:38:52.507 INFO:teuthology.orchestra.run.smithi045.stdout:osd.0 smithi045 running (3m) 78s ago 3m 44.5M 4289M 17.2.7-125-g1f2f5510 60cc83a9c8f1 eeea01051a62 2023-12-16T21:38:52.507 INFO:teuthology.orchestra.run.smithi045.stdout:osd.1 smithi045 running (3m) 78s ago 3m 45.4M 4289M 17.2.7-125-g1f2f5510 60cc83a9c8f1 94ded6ce4ce1 2023-12-16T21:38:52.507 INFO:teuthology.orchestra.run.smithi045.stdout:osd.2 smithi045 running (3m) 78s ago 3m 43.9M 4289M 17.2.7-125-g1f2f5510 60cc83a9c8f1 5fc744d14d11 2023-12-16T21:38:52.507 INFO:teuthology.orchestra.run.smithi045.stdout:osd.3 smithi169 running (2m) 80s ago 2m 44.1M 5312M 17.2.7-125-g1f2f5510 60cc83a9c8f1 f5ee44f8b3e2 2023-12-16T21:38:52.507 INFO:teuthology.orchestra.run.smithi045.stdout:osd.4 smithi169 running (2m) 80s ago 2m 41.1M 5312M 17.2.7-125-g1f2f5510 60cc83a9c8f1 8a74d1fa8b75 2023-12-16T21:38:52.507 INFO:teuthology.orchestra.run.smithi045.stdout:osd.5 smithi169 running (2m) 80s ago 2m 40.6M 5312M 17.2.7-125-g1f2f5510 60cc83a9c8f1 9c0a8a497615 2023-12-16T21:38:52.508 INFO:teuthology.orchestra.run.smithi045.stdout:prometheus.smithi045 smithi045 *:9095 running (3m) 78s ago 4m 44.4M - 2.43.0 a07b618ecd1d 1e1226889abe 2023-12-16T21:38:52.508 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:38:52.505+0000 7f14e6fc4700 1 -- 172.21.15.45:0/1231441671 >> [v2:172.21.15.45:6800/2012105536,v1:172.21.15.45:6801/2012105536] conn(0x7f14cc069070 msgr2=0x7f14cc06b520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:38:52.508 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:38:52.505+0000 7f14e6fc4700 1 --2- 172.21.15.45:0/1231441671 >> [v2:172.21.15.45:6800/2012105536,v1:172.21.15.45:6801/2012105536] conn(0x7f14cc069070 0x7f14cc06b520 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7f14e006ec40 tx=0x7f14c800b040 comp rx=0 tx=0).stop 2023-12-16T21:38:52.508 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:38:52.505+0000 7f14e6fc4700 1 -- 172.21.15.45:0/1231441671 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f14e0104410 msgr2=0x7f14e006f3f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:38:52.508 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:38:52.505+0000 7f14e6fc4700 1 --2- 172.21.15.45:0/1231441671 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f14e0104410 0x7f14e006f3f0 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7f14d8000f80 tx=0x7f14d8004d10 comp rx=0 tx=0).stop 2023-12-16T21:38:52.508 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:38:52.505+0000 7f14e6fc4700 1 -- 172.21.15.45:0/1231441671 shutdown_connections 2023-12-16T21:38:52.509 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:38:52.505+0000 7f14e6fc4700 1 --2- 172.21.15.45:0/1231441671 >> [v2:172.21.15.45:6800/2012105536,v1:172.21.15.45:6801/2012105536] conn(0x7f14cc069070 0x7f14cc06b520 unknown :-1 s=CLOSED pgs=120 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:38:52.509 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:38:52.505+0000 7f14e6fc4700 1 --2- 172.21.15.45:0/1231441671 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f14e0103250 0x7f14e006ee70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:38:52.509 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:38:52.505+0000 7f14e6fc4700 1 --2- 172.21.15.45:0/1231441671 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f14e0104410 0x7f14e006f3f0 unknown :-1 s=CLOSED pgs=41 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:38:52.509 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:38:52.505+0000 7f14e6fc4700 1 -- 172.21.15.45:0/1231441671 >> 172.21.15.45:0/1231441671 conn(0x7f14e00665a0 msgr2=0x7f14e0075ce0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-16T21:38:52.509 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:38:52.506+0000 7f14e6fc4700 1 -- 172.21.15.45:0/1231441671 shutdown_connections 2023-12-16T21:38:52.509 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:38:52.506+0000 7f14e6fc4700 1 -- 172.21.15.45:0/1231441671 wait complete. 2023-12-16T21:38:52.633 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:38:52.628+0000 7f10f8bc2700 1 -- 172.21.15.45:0/1126810290 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f10f4066090 msgr2=0x7f10f40ff7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:38:52.633 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:38:52.628+0000 7f10f8bc2700 1 --2- 172.21.15.45:0/1126810290 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f10f4066090 0x7f10f40ff7a0 secure :-1 s=READY pgs=298 cs=0 l=1 rev1=1 crypto rx=0x7f10e4004940 tx=0x7f10e4009e30 comp rx=0 tx=0).stop 2023-12-16T21:38:52.633 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:38:52.628+0000 7f10f8bc2700 1 -- 172.21.15.45:0/1126810290 shutdown_connections 2023-12-16T21:38:52.633 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:38:52.628+0000 7f10f8bc2700 1 --2- 172.21.15.45:0/1126810290 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f10f40ffd70 0x7f10f41021d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:38:52.633 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:38:52.628+0000 7f10f8bc2700 1 --2- 172.21.15.45:0/1126810290 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f10f4066090 0x7f10f40ff7a0 unknown :-1 s=CLOSED pgs=298 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:38:52.633 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:38:52.628+0000 7f10f8bc2700 1 -- 172.21.15.45:0/1126810290 >> 172.21.15.45:0/1126810290 conn(0x7f10f40f7190 msgr2=0x7f10f40f95e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-16T21:38:52.633 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:38:52.628+0000 7f10f8bc2700 1 -- 172.21.15.45:0/1126810290 shutdown_connections 2023-12-16T21:38:52.633 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:38:52.628+0000 7f10f8bc2700 1 -- 172.21.15.45:0/1126810290 wait complete. 2023-12-16T21:38:52.633 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:38:52.629+0000 7f10f8bc2700 1 Processor -- start 2023-12-16T21:38:52.633 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:38:52.629+0000 7f10f8bc2700 1 -- start start 2023-12-16T21:38:52.633 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:38:52.629+0000 7f10f8bc2700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f10f4066090 0x7f10f40fd220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:38:52.634 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:38:52.629+0000 7f10f8bc2700 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f10f40ffd70 0x7f10f40fb6a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:38:52.634 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:38:52.629+0000 7f10f8bc2700 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f10f40fbca0 con 0x7f10f4066090 2023-12-16T21:38:52.634 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:38:52.629+0000 7f10f8bc2700 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f10f40fbe10 con 0x7f10f40ffd70 2023-12-16T21:38:52.634 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:38:52.629+0000 7f10f259c700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f10f4066090 0x7f10f40fd220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:38:52.634 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:38:52.629+0000 7f10f259c700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f10f4066090 0x7f10f40fd220 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:45468/0 (socket says 172.21.15.45:45468) 2023-12-16T21:38:52.634 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:38:52.629+0000 7f10f259c700 1 -- 172.21.15.45:0/1443271374 learned_addr learned my addr 172.21.15.45:0/1443271374 (peer_addr_for_me v2:172.21.15.45:0/0) 2023-12-16T21:38:52.634 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:38:52.629+0000 7f10f1d9b700 1 --2- 172.21.15.45:0/1443271374 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f10f40ffd70 0x7f10f40fb6a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:38:52.634 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:38:52.629+0000 7f10f1d9b700 1 -- 172.21.15.45:0/1443271374 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f10f4066090 msgr2=0x7f10f40fd220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:38:52.634 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:38:52.629+0000 7f10f1d9b700 1 --2- 172.21.15.45:0/1443271374 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f10f4066090 0x7f10f40fd220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:38:52.634 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:38:52.629+0000 7f10f1d9b700 1 -- 172.21.15.45:0/1443271374 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f10e8005040 con 0x7f10f40ffd70 2023-12-16T21:38:52.634 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:38:52.629+0000 7f10f259c700 1 --2- 172.21.15.45:0/1443271374 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f10f4066090 0x7f10f40fd220 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-12-16T21:38:52.634 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:38:52.630+0000 7f10f1d9b700 1 --2- 172.21.15.45:0/1443271374 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f10f40ffd70 0x7f10f40fb6a0 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7f10e8009fd0 tx=0x7f10e8009e30 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:38:52.634 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:38:52.630+0000 7f10e2ffd700 1 -- 172.21.15.45:0/1443271374 <== mon.1 v2:172.21.15.169:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f10e8012570 con 0x7f10f40ffd70 2023-12-16T21:38:52.634 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:38:52.630+0000 7f10f8bc2700 1 -- 172.21.15.45:0/1443271374 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f10e4005040 con 0x7f10f40ffd70 2023-12-16T21:38:52.635 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:38:52.630+0000 7f10e2ffd700 1 -- 172.21.15.45:0/1443271374 <== mon.1 v2:172.21.15.169:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f10e800f040 con 0x7f10f40ffd70 2023-12-16T21:38:52.635 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:38:52.630+0000 7f10e2ffd700 1 -- 172.21.15.45:0/1443271374 <== mon.1 v2:172.21.15.169:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f10e8013bd0 con 0x7f10f40ffd70 2023-12-16T21:38:52.635 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:38:52.630+0000 7f10f8bc2700 1 -- 172.21.15.45:0/1443271374 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f10f40fc4a0 con 0x7f10f40ffd70 2023-12-16T21:38:52.635 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:38:52.631+0000 7f10f8bc2700 1 -- 172.21.15.45:0/1443271374 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f10f404bb80 con 0x7f10f40ffd70 2023-12-16T21:38:52.636 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:38:52.632+0000 7f10e2ffd700 1 -- 172.21.15.45:0/1443271374 <== mon.1 v2:172.21.15.169:3300/0 4 ==== mgrmap(e 19) v1 ==== 86447+0+0 (secure 0 0 0) 0x7f10e80126d0 con 0x7f10f40ffd70 2023-12-16T21:38:52.636 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:38:52.632+0000 7f10e2ffd700 1 --2- 172.21.15.45:0/1443271374 >> [v2:172.21.15.45:6800/2012105536,v1:172.21.15.45:6801/2012105536] conn(0x7f10dc068f60 0x7f10dc06b410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:38:52.636 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:38:52.632+0000 7f10e2ffd700 1 -- 172.21.15.45:0/1443271374 <== mon.1 v2:172.21.15.169:3300/0 5 ==== osd_map(43..43 src has 1..43) v4 ==== 5550+0+0 (secure 0 0 0) 0x7f10e8013e40 con 0x7f10f40ffd70 2023-12-16T21:38:52.636 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:38:52.632+0000 7f10f259c700 1 --2- 172.21.15.45:0/1443271374 >> [v2:172.21.15.45:6800/2012105536,v1:172.21.15.45:6801/2012105536] conn(0x7f10dc068f60 0x7f10dc06b410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:38:52.636 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:38:52.633+0000 7f10f259c700 1 --2- 172.21.15.45:0/1443271374 >> [v2:172.21.15.45:6800/2012105536,v1:172.21.15.45:6801/2012105536] conn(0x7f10dc068f60 0x7f10dc06b410 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7f10e4004580 tx=0x7f10e4009e30 comp rx=0 tx=0).ready entity=mgr.14214 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:38:52.643 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:38:52.640+0000 7f10e2ffd700 1 -- 172.21.15.45:0/1443271374 <== mon.1 v2:172.21.15.169:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+174570 (secure 0 0 0) 0x7f10e8056bc0 con 0x7f10f40ffd70 2023-12-16T21:38:52.816 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Dec 16 21:38:52 smithi045 ceph-mon[80763]: from='client.14576 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2023-12-16T21:38:52.927 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:38:52.925+0000 7f10f8bc2700 1 -- 172.21.15.45:0/1443271374 --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_command({"prefix": "versions"} v 0) v1 -- 0x7f10f41055a0 con 0x7f10f40ffd70 2023-12-16T21:38:52.928 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:38:52.925+0000 7f10e2ffd700 1 -- 172.21.15.45:0/1443271374 <== mon.1 v2:172.21.15.169:3300/0 7 ==== mon_command_ack([{"prefix": "versions"}]=0 v0) v1 ==== 56+0+638 (secure 0 0 0) 0x7f10e8056750 con 0x7f10f40ffd70 2023-12-16T21:38:52.928 INFO:teuthology.orchestra.run.smithi045.stdout:{ 2023-12-16T21:38:52.928 INFO:teuthology.orchestra.run.smithi045.stdout: "mon": { 2023-12-16T21:38:52.928 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 17.2.7-125-g1f2f5510 (1f2f55109dc1ed2dc2dda9de1db3c70923ba0dfd) quincy (stable)": 2 2023-12-16T21:38:52.928 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2023-12-16T21:38:52.928 INFO:teuthology.orchestra.run.smithi045.stdout: "mgr": { 2023-12-16T21:38:52.928 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 17.2.7-125-g1f2f5510 (1f2f55109dc1ed2dc2dda9de1db3c70923ba0dfd) quincy (stable)": 2 2023-12-16T21:38:52.928 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2023-12-16T21:38:52.928 INFO:teuthology.orchestra.run.smithi045.stdout: "osd": { 2023-12-16T21:38:52.928 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 17.2.7-125-g1f2f5510 (1f2f55109dc1ed2dc2dda9de1db3c70923ba0dfd) quincy (stable)": 6 2023-12-16T21:38:52.928 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2023-12-16T21:38:52.929 INFO:teuthology.orchestra.run.smithi045.stdout: "mds": { 2023-12-16T21:38:52.929 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 17.2.7-125-g1f2f5510 (1f2f55109dc1ed2dc2dda9de1db3c70923ba0dfd) quincy (stable)": 4 2023-12-16T21:38:52.929 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2023-12-16T21:38:52.929 INFO:teuthology.orchestra.run.smithi045.stdout: "overall": { 2023-12-16T21:38:52.929 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph version 17.2.7-125-g1f2f5510 (1f2f55109dc1ed2dc2dda9de1db3c70923ba0dfd) quincy (stable)": 14 2023-12-16T21:38:52.929 INFO:teuthology.orchestra.run.smithi045.stdout: } 2023-12-16T21:38:52.929 INFO:teuthology.orchestra.run.smithi045.stdout:} 2023-12-16T21:38:52.930 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:38:52.927+0000 7f10f8bc2700 1 -- 172.21.15.45:0/1443271374 >> [v2:172.21.15.45:6800/2012105536,v1:172.21.15.45:6801/2012105536] conn(0x7f10dc068f60 msgr2=0x7f10dc06b410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:38:52.930 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:38:52.927+0000 7f10f8bc2700 1 --2- 172.21.15.45:0/1443271374 >> [v2:172.21.15.45:6800/2012105536,v1:172.21.15.45:6801/2012105536] conn(0x7f10dc068f60 0x7f10dc06b410 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7f10e4004580 tx=0x7f10e4009e30 comp rx=0 tx=0).stop 2023-12-16T21:38:52.930 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:38:52.927+0000 7f10f8bc2700 1 -- 172.21.15.45:0/1443271374 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f10f40ffd70 msgr2=0x7f10f40fb6a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:38:52.930 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:38:52.927+0000 7f10f8bc2700 1 --2- 172.21.15.45:0/1443271374 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f10f40ffd70 0x7f10f40fb6a0 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7f10e8009fd0 tx=0x7f10e8009e30 comp rx=0 tx=0).stop 2023-12-16T21:38:52.930 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:38:52.927+0000 7f10f8bc2700 1 -- 172.21.15.45:0/1443271374 shutdown_connections 2023-12-16T21:38:52.930 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:38:52.927+0000 7f10f8bc2700 1 --2- 172.21.15.45:0/1443271374 >> [v2:172.21.15.45:6800/2012105536,v1:172.21.15.45:6801/2012105536] conn(0x7f10dc068f60 0x7f10dc06b410 unknown :-1 s=CLOSED pgs=121 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:38:52.930 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:38:52.927+0000 7f10f8bc2700 1 --2- 172.21.15.45:0/1443271374 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f10f4066090 0x7f10f40fd220 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:38:52.931 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:38:52.927+0000 7f10f8bc2700 1 --2- 172.21.15.45:0/1443271374 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7f10f40ffd70 0x7f10f40fb6a0 unknown :-1 s=CLOSED pgs=42 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:38:52.931 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:38:52.927+0000 7f10f8bc2700 1 -- 172.21.15.45:0/1443271374 >> 172.21.15.45:0/1443271374 conn(0x7f10f40f7190 msgr2=0x7f10f40f95e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-16T21:38:52.931 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:38:52.928+0000 7f10f8bc2700 1 -- 172.21.15.45:0/1443271374 shutdown_connections 2023-12-16T21:38:52.931 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:38:52.928+0000 7f10f8bc2700 1 -- 172.21.15.45:0/1443271374 wait complete. 2023-12-16T21:38:53.036 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:38:53.033+0000 7fd7ed6f8700 1 -- 172.21.15.45:0/2679090691 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fd7e80fcbf0 msgr2=0x7fd7e80fd000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:38:53.037 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:38:53.034+0000 7fd7ed6f8700 1 --2- 172.21.15.45:0/2679090691 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fd7e80fcbf0 0x7fd7e80fd000 secure :-1 s=READY pgs=299 cs=0 l=1 rev1=1 crypto rx=0x7fd7d8004940 tx=0x7fd7d8009e30 comp rx=0 tx=0).stop 2023-12-16T21:38:53.037 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:38:53.034+0000 7fd7ed6f8700 1 -- 172.21.15.45:0/2679090691 shutdown_connections 2023-12-16T21:38:53.037 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:38:53.034+0000 7fd7ed6f8700 1 --2- 172.21.15.45:0/2679090691 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fd7e80fdea0 0x7fd7e80fe310 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:38:53.037 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:38:53.034+0000 7fd7ed6f8700 1 --2- 172.21.15.45:0/2679090691 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fd7e80fcbf0 0x7fd7e80fd000 unknown :-1 s=CLOSED pgs=299 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:38:53.037 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:38:53.034+0000 7fd7ed6f8700 1 -- 172.21.15.45:0/2679090691 >> 172.21.15.45:0/2679090691 conn(0x7fd7e80f8330 msgr2=0x7fd7e80fa780 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-16T21:38:53.037 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:38:53.034+0000 7fd7ed6f8700 1 -- 172.21.15.45:0/2679090691 shutdown_connections 2023-12-16T21:38:53.037 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:38:53.034+0000 7fd7ed6f8700 1 -- 172.21.15.45:0/2679090691 wait complete. 2023-12-16T21:38:53.038 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:38:53.034+0000 7fd7ed6f8700 1 Processor -- start 2023-12-16T21:38:53.039 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:38:53.035+0000 7fd7ed6f8700 1 -- start start 2023-12-16T21:38:53.039 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:38:53.035+0000 7fd7ed6f8700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fd7e80fcbf0 0x7fd7e810d500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:38:53.039 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:38:53.035+0000 7fd7ed6f8700 1 --2- >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fd7e80fdea0 0x7fd7e810da40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:38:53.039 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:38:53.035+0000 7fd7ed6f8700 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd7e810e040 con 0x7fd7e80fcbf0 2023-12-16T21:38:53.039 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:38:53.035+0000 7fd7ed6f8700 1 -- --> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd7e810e8e0 con 0x7fd7e80fdea0 2023-12-16T21:38:53.039 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:38:53.035+0000 7fd7e6ffd700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fd7e80fcbf0 0x7fd7e810d500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:38:53.039 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:38:53.035+0000 7fd7e6ffd700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fd7e80fcbf0 0x7fd7e810d500 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:45486/0 (socket says 172.21.15.45:45486) 2023-12-16T21:38:53.039 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:38:53.035+0000 7fd7e6ffd700 1 -- 172.21.15.45:0/3212535120 learned_addr learned my addr 172.21.15.45:0/3212535120 (peer_addr_for_me v2:172.21.15.45:0/0) 2023-12-16T21:38:53.039 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:38:53.035+0000 7fd7e67fc700 1 --2- 172.21.15.45:0/3212535120 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fd7e80fdea0 0x7fd7e810da40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:38:53.039 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:38:53.035+0000 7fd7e6ffd700 1 -- 172.21.15.45:0/3212535120 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fd7e80fdea0 msgr2=0x7fd7e810da40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:38:53.039 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:38:53.035+0000 7fd7e6ffd700 1 --2- 172.21.15.45:0/3212535120 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fd7e80fdea0 0x7fd7e810da40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:38:53.039 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:38:53.035+0000 7fd7e6ffd700 1 -- 172.21.15.45:0/3212535120 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd7d8005040 con 0x7fd7e80fcbf0 2023-12-16T21:38:53.039 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:38:53.035+0000 7fd7e6ffd700 1 --2- 172.21.15.45:0/3212535120 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fd7e80fcbf0 0x7fd7e810d500 secure :-1 s=READY pgs=300 cs=0 l=1 rev1=1 crypto rx=0x7fd7d8000c00 tx=0x7fd7d8009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:38:53.039 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:38:53.036+0000 7fd7cf7fe700 1 -- 172.21.15.45:0/3212535120 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fd7d8021070 con 0x7fd7e80fcbf0 2023-12-16T21:38:53.039 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:38:53.036+0000 7fd7ed6f8700 1 -- 172.21.15.45:0/3212535120 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd7e810eb60 con 0x7fd7e80fcbf0 2023-12-16T21:38:53.040 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:38:53.036+0000 7fd7cf7fe700 1 -- 172.21.15.45:0/3212535120 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fd7d801c400 con 0x7fd7e80fcbf0 2023-12-16T21:38:53.040 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:38:53.036+0000 7fd7cf7fe700 1 -- 172.21.15.45:0/3212535120 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fd7d800e070 con 0x7fd7e80fcbf0 2023-12-16T21:38:53.040 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:38:53.036+0000 7fd7ed6f8700 1 -- 172.21.15.45:0/3212535120 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd7e810f040 con 0x7fd7e80fcbf0 2023-12-16T21:38:53.040 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:38:53.036+0000 7fd7ed6f8700 1 -- 172.21.15.45:0/3212535120 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd7e804bb80 con 0x7fd7e80fcbf0 2023-12-16T21:38:53.041 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:38:53.037+0000 7fd7cf7fe700 1 -- 172.21.15.45:0/3212535120 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 19) v1 ==== 86447+0+0 (secure 0 0 0) 0x7fd7d801c9c0 con 0x7fd7e80fcbf0 2023-12-16T21:38:53.041 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:38:53.037+0000 7fd7cf7fe700 1 --2- 172.21.15.45:0/3212535120 >> [v2:172.21.15.45:6800/2012105536,v1:172.21.15.45:6801/2012105536] conn(0x7fd7d0068f60 0x7fd7d006b410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-16T21:38:53.041 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:38:53.038+0000 7fd7cf7fe700 1 -- 172.21.15.45:0/3212535120 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(43..43 src has 1..43) v4 ==== 5550+0+0 (secure 0 0 0) 0x7fd7d8088040 con 0x7fd7e80fcbf0 2023-12-16T21:38:53.041 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:38:53.038+0000 7fd7e67fc700 1 --2- 172.21.15.45:0/3212535120 >> [v2:172.21.15.45:6800/2012105536,v1:172.21.15.45:6801/2012105536] conn(0x7fd7d0068f60 0x7fd7d006b410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-16T21:38:53.041 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:38:53.038+0000 7fd7e67fc700 1 --2- 172.21.15.45:0/3212535120 >> [v2:172.21.15.45:6800/2012105536,v1:172.21.15.45:6801/2012105536] conn(0x7fd7d0068f60 0x7fd7d006b410 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7fd7dc009b80 tx=0x7fd7dc00b040 comp rx=0 tx=0).ready entity=mgr.14214 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-16T21:38:53.044 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:38:53.042+0000 7fd7cf7fe700 1 -- 172.21.15.45:0/3212535120 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+174570 (secure 0 0 0) 0x7fd7d8057f50 con 0x7fd7e80fcbf0 2023-12-16T21:38:53.133 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Dec 16 21:38:52 smithi169 ceph-mon[88350]: from='client.14576 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2023-12-16T21:38:53.279 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:38:53.277+0000 7fd7ed6f8700 1 -- 172.21.15.45:0/3212535120 --> [v2:172.21.15.45:6800/2012105536,v1:172.21.15.45:6801/2012105536] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7fd7e81027f0 con 0x7fd7d0068f60 2023-12-16T21:38:53.280 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:38:53.278+0000 7fd7cf7fe700 1 -- 172.21.15.45:0/3212535120 <== mgr.14214 v2:172.21.15.45:6800/2012105536 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+439 (secure 0 0 0) 0x7fd7e81027f0 con 0x7fd7d0068f60 2023-12-16T21:38:53.280 INFO:teuthology.orchestra.run.smithi045.stdout:{ 2023-12-16T21:38:53.281 INFO:teuthology.orchestra.run.smithi045.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:45cdc87ca300414c8899f73cd28f3b099dc14234a3689c0162531020ce800bc5", 2023-12-16T21:38:53.281 INFO:teuthology.orchestra.run.smithi045.stdout: "in_progress": true, 2023-12-16T21:38:53.281 INFO:teuthology.orchestra.run.smithi045.stdout: "which": "Upgrading daemons of type(s) mgr", 2023-12-16T21:38:53.281 INFO:teuthology.orchestra.run.smithi045.stdout: "services_complete": [], 2023-12-16T21:38:53.281 INFO:teuthology.orchestra.run.smithi045.stdout: "progress": "0/2 daemons upgraded", 2023-12-16T21:38:53.281 INFO:teuthology.orchestra.run.smithi045.stdout: "message": "Pulling quay.ceph.io/ceph-ci/ceph@sha256:45cdc87ca300414c8899f73cd28f3b099dc14234a3689c0162531020ce800bc5 image on host smithi169", 2023-12-16T21:38:53.281 INFO:teuthology.orchestra.run.smithi045.stdout: "is_paused": false 2023-12-16T21:38:53.281 INFO:teuthology.orchestra.run.smithi045.stdout:} 2023-12-16T21:38:53.283 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:38:53.280+0000 7fd7ed6f8700 1 -- 172.21.15.45:0/3212535120 >> [v2:172.21.15.45:6800/2012105536,v1:172.21.15.45:6801/2012105536] conn(0x7fd7d0068f60 msgr2=0x7fd7d006b410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:38:53.283 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:38:53.280+0000 7fd7ed6f8700 1 --2- 172.21.15.45:0/3212535120 >> [v2:172.21.15.45:6800/2012105536,v1:172.21.15.45:6801/2012105536] conn(0x7fd7d0068f60 0x7fd7d006b410 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7fd7dc009b80 tx=0x7fd7dc00b040 comp rx=0 tx=0).stop 2023-12-16T21:38:53.283 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:38:53.280+0000 7fd7ed6f8700 1 -- 172.21.15.45:0/3212535120 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fd7e80fcbf0 msgr2=0x7fd7e810d500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-16T21:38:53.283 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:38:53.280+0000 7fd7ed6f8700 1 --2- 172.21.15.45:0/3212535120 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fd7e80fcbf0 0x7fd7e810d500 secure :-1 s=READY pgs=300 cs=0 l=1 rev1=1 crypto rx=0x7fd7d8000c00 tx=0x7fd7d8009e30 comp rx=0 tx=0).stop 2023-12-16T21:38:53.283 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:38:53.280+0000 7fd7ed6f8700 1 -- 172.21.15.45:0/3212535120 shutdown_connections 2023-12-16T21:38:53.284 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:38:53.281+0000 7fd7ed6f8700 1 --2- 172.21.15.45:0/3212535120 >> [v2:172.21.15.45:6800/2012105536,v1:172.21.15.45:6801/2012105536] conn(0x7fd7d0068f60 0x7fd7d006b410 unknown :-1 s=CLOSED pgs=122 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:38:53.284 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:38:53.281+0000 7fd7ed6f8700 1 --2- 172.21.15.45:0/3212535120 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fd7e80fcbf0 0x7fd7e810d500 unknown :-1 s=CLOSED pgs=300 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:38:53.284 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:38:53.281+0000 7fd7ed6f8700 1 --2- 172.21.15.45:0/3212535120 >> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] conn(0x7fd7e80fdea0 0x7fd7e810da40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-16T21:38:53.284 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:38:53.281+0000 7fd7ed6f8700 1 -- 172.21.15.45:0/3212535120 >> 172.21.15.45:0/3212535120 conn(0x7fd7e80f8330 msgr2=0x7fd7e81010d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-16T21:38:53.284 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:38:53.281+0000 7fd7ed6f8700 1 -- 172.21.15.45:0/3212535120 shutdown_connections 2023-12-16T21:38:53.284 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:38:53.281+0000 7fd7ed6f8700 1 -- 172.21.15.45:0/3212535120 wait complete. 2023-12-16T21:38:53.947 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Dec 16 21:38:53 smithi169 ceph-mon[88350]: from='client.14580 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2023-12-16T21:38:53.947 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Dec 16 21:38:53 smithi169 ceph-mon[88350]: from='client.24377 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2023-12-16T21:38:53.947 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Dec 16 21:38:53 smithi169 ceph-mon[88350]: pgmap v163: 65 pgs: 65 active+clean; 70 MiB data, 2.0 GiB used, 534 GiB / 536 GiB avail; 1.9 MiB/s rd, 6.0 MiB/s wr, 496 op/s 2023-12-16T21:38:53.947 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Dec 16 21:38:53 smithi169 ceph-mon[88350]: from='client.? 172.21.15.45:0/1443271374' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2023-12-16T21:38:54.066 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Dec 16 21:38:53 smithi045 ceph-mon[80763]: from='client.14580 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2023-12-16T21:38:54.066 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Dec 16 21:38:53 smithi045 ceph-mon[80763]: from='client.24377 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2023-12-16T21:38:54.066 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Dec 16 21:38:53 smithi045 ceph-mon[80763]: pgmap v163: 65 pgs: 65 active+clean; 70 MiB data, 2.0 GiB used, 534 GiB / 536 GiB avail; 1.9 MiB/s rd, 6.0 MiB/s wr, 496 op/s 2023-12-16T21:38:54.066 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Dec 16 21:38:53 smithi045 ceph-mon[80763]: from='client.? 172.21.15.45:0/1443271374' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2023-12-16T21:38:54.846 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Dec 16 21:38:54 smithi169 ceph-mon[88350]: from='client.14592 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2023-12-16T21:38:55.066 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Dec 16 21:38:54 smithi045 ceph-mon[80763]: from='client.14592 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2023-12-16T21:38:56.068 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Dec 16 21:38:55 smithi045 ceph-mon[80763]: pgmap v164: 65 pgs: 65 active+clean; 92 MiB data, 2.0 GiB used, 534 GiB / 536 GiB avail; 2.6 MiB/s rd, 7.3 MiB/s wr, 599 op/s 2023-12-16T21:38:56.133 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Dec 16 21:38:55 smithi169 ceph-mon[88350]: pgmap v164: 65 pgs: 65 active+clean; 92 MiB data, 2.0 GiB used, 534 GiB / 536 GiB avail; 2.6 MiB/s rd, 7.3 MiB/s wr, 599 op/s 2023-12-16T21:38:57.633 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Dec 16 21:38:57 smithi169 ceph-mon[88350]: pgmap v165: 65 pgs: 65 active+clean; 97 MiB data, 2.1 GiB used, 534 GiB / 536 GiB avail; 2.1 MiB/s rd, 5.0 MiB/s wr, 449 op/s 2023-12-16T21:38:57.816 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Dec 16 21:38:57 smithi045 ceph-mon[80763]: pgmap v165: 65 pgs: 65 active+clean; 97 MiB data, 2.1 GiB used, 534 GiB / 536 GiB avail; 2.1 MiB/s rd, 5.0 MiB/s wr, 449 op/s 2023-12-16T21:38:59.728 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Dec 16 21:38:59 smithi169 ceph-mon[88350]: Upgrade: Updating mgr.smithi169.cncbgh 2023-12-16T21:38:59.729 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Dec 16 21:38:59 smithi169 ceph-mon[88350]: from='mgr.14214 172.21.15.45:0/1227909437' entity='mgr.smithi045.nhtmwc' 2023-12-16T21:38:59.729 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Dec 16 21:38:59 smithi169 ceph-mon[88350]: from='mgr.14214 172.21.15.45:0/1227909437' entity='mgr.smithi045.nhtmwc' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.smithi169.cncbgh", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2023-12-16T21:38:59.729 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Dec 16 21:38:59 smithi169 ceph-mon[88350]: from='mgr.14214 172.21.15.45:0/1227909437' entity='mgr.smithi045.nhtmwc' cmd=[{"prefix": "mgr services"}]: dispatch 2023-12-16T21:38:59.729 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Dec 16 21:38:59 smithi169 ceph-mon[88350]: from='mgr.14214 172.21.15.45:0/1227909437' entity='mgr.smithi045.nhtmwc' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-12-16T21:38:59.729 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Dec 16 21:38:59 smithi169 ceph-mon[88350]: Deploying daemon mgr.smithi169.cncbgh on smithi169 2023-12-16T21:38:59.729 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Dec 16 21:38:59 smithi169 ceph-mon[88350]: pgmap v166: 65 pgs: 65 active+clean; 117 MiB data, 2.2 GiB used, 534 GiB / 536 GiB avail; 2.8 MiB/s rd, 5.9 MiB/s wr, 594 op/s 2023-12-16T21:38:59.730 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Dec 16 21:38:59 smithi169 ceph-mon[88350]: from='mgr.14214 172.21.15.45:0/1227909437' entity='mgr.smithi045.nhtmwc' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2023-12-16T21:38:59.794 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Dec 16 21:38:59 smithi045 ceph-mon[80763]: Upgrade: Updating mgr.smithi169.cncbgh 2023-12-16T21:38:59.794 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Dec 16 21:38:59 smithi045 ceph-mon[80763]: from='mgr.14214 172.21.15.45:0/1227909437' entity='mgr.smithi045.nhtmwc' 2023-12-16T21:38:59.795 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Dec 16 21:38:59 smithi045 ceph-mon[80763]: from='mgr.14214 172.21.15.45:0/1227909437' entity='mgr.smithi045.nhtmwc' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.smithi169.cncbgh", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2023-12-16T21:38:59.795 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Dec 16 21:38:59 smithi045 ceph-mon[80763]: from='mgr.14214 172.21.15.45:0/1227909437' entity='mgr.smithi045.nhtmwc' cmd=[{"prefix": "mgr services"}]: dispatch 2023-12-16T21:38:59.795 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Dec 16 21:38:59 smithi045 ceph-mon[80763]: from='mgr.14214 172.21.15.45:0/1227909437' entity='mgr.smithi045.nhtmwc' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-12-16T21:38:59.795 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Dec 16 21:38:59 smithi045 ceph-mon[80763]: Deploying daemon mgr.smithi169.cncbgh on smithi169 2023-12-16T21:38:59.795 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Dec 16 21:38:59 smithi045 ceph-mon[80763]: pgmap v166: 65 pgs: 65 active+clean; 117 MiB data, 2.2 GiB used, 534 GiB / 536 GiB avail; 2.8 MiB/s rd, 5.9 MiB/s wr, 594 op/s 2023-12-16T21:38:59.795 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Dec 16 21:38:59 smithi045 ceph-mon[80763]: from='mgr.14214 172.21.15.45:0/1227909437' entity='mgr.smithi045.nhtmwc' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2023-12-16T21:39:02.066 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Dec 16 21:39:01 smithi045 ceph-mon[80763]: pgmap v167: 65 pgs: 65 active+clean; 127 MiB data, 2.2 GiB used, 534 GiB / 536 GiB avail; 2.0 MiB/s rd, 5.3 MiB/s wr, 616 op/s 2023-12-16T21:39:02.133 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Dec 16 21:39:01 smithi169 ceph-mon[88350]: pgmap v167: 65 pgs: 65 active+clean; 127 MiB data, 2.2 GiB used, 534 GiB / 536 GiB avail; 2.0 MiB/s rd, 5.3 MiB/s wr, 616 op/s 2023-12-16T21:39:03.883 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Dec 16 21:39:03 smithi169 ceph-mon[88350]: pgmap v168: 65 pgs: 65 active+clean; 130 MiB data, 2.2 GiB used, 534 GiB / 536 GiB avail; 2.0 MiB/s rd, 5.5 MiB/s wr, 664 op/s 2023-12-16T21:39:04.069 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Dec 16 21:39:03 smithi045 ceph-mon[80763]: pgmap v168: 65 pgs: 65 active+clean; 130 MiB data, 2.2 GiB used, 534 GiB / 536 GiB avail; 2.0 MiB/s rd, 5.5 MiB/s wr, 664 op/s 2023-12-16T21:39:06.042 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Dec 16 21:39:05 smithi169 ceph-mon[88350]: pgmap v169: 65 pgs: 65 active+clean; 147 MiB data, 2.2 GiB used, 534 GiB / 536 GiB avail; 3.1 MiB/s rd, 6.8 MiB/s wr, 763 op/s 2023-12-16T21:39:06.066 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Dec 16 21:39:05 smithi045 ceph-mon[80763]: pgmap v169: 65 pgs: 65 active+clean; 147 MiB data, 2.2 GiB used, 534 GiB / 536 GiB avail; 3.1 MiB/s rd, 6.8 MiB/s wr, 763 op/s 2023-12-16T21:39:07.575 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Dec 16 21:39:07 smithi169 ceph-mon[88350]: pgmap v170: 65 pgs: 65 active+clean; 154 MiB data, 2.3 GiB used, 534 GiB / 536 GiB avail; 2.5 MiB/s rd, 5.6 MiB/s wr, 636 op/s 2023-12-16T21:39:07.816 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Dec 16 21:39:07 smithi045 ceph-mon[80763]: pgmap v170: 65 pgs: 65 active+clean; 154 MiB data, 2.3 GiB used, 534 GiB / 536 GiB avail; 2.5 MiB/s rd, 5.6 MiB/s wr, 636 op/s 2023-12-16T21:39:08.500 INFO:tasks.workunit.client.1.smithi169.stderr:+ pushd ltp-full-20091231/testcases/kernel/fs/fsstress 2023-12-16T21:39:08.523 INFO:tasks.workunit.client.1.smithi169.stdout:~/cephtest/mnt.1/client.1/tmp/fsstress/ltp-full-20091231/testcases/kernel/fs/fsstress ~/cephtest/mnt.1/client.1/tmp/fsstress ~/cephtest/mnt.1/client.1/tmp 2023-12-16T21:39:08.523 INFO:tasks.workunit.client.1.smithi169.stderr:+ make 2023-12-16T21:39:09.120 INFO:tasks.workunit.client.1.smithi169.stdout:cc -DNO_XFS -I/home/ubuntu/cephtest/mnt.1/client.1/tmp/fsstress/ltp-full-20091231/testcases/kernel/fs/fsstress -D_LARGEFILE64_SOURCE -D_GNU_SOURCE -I../../../../include -I../../../../include -L../../../../lib fsstress.c -o fsstress 2023-12-16T21:39:10.066 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Dec 16 21:39:09 smithi045 ceph-mon[80763]: pgmap v171: 65 pgs: 65 active+clean; 156 MiB data, 2.3 GiB used, 534 GiB / 536 GiB avail; 2.6 MiB/s rd, 5.2 MiB/s wr, 547 op/s 2023-12-16T21:39:10.066 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Dec 16 21:39:09 smithi045 ceph-mon[80763]: from='mgr.14214 172.21.15.45:0/1227909437' entity='mgr.smithi045.nhtmwc' 2023-12-16T21:39:10.066 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Dec 16 21:39:09 smithi045 ceph-mon[80763]: from='mgr.14214 172.21.15.45:0/1227909437' entity='mgr.smithi045.nhtmwc' 2023-12-16T21:39:10.066 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Dec 16 21:39:09 smithi045 ceph-mon[80763]: from='mgr.14214 172.21.15.45:0/1227909437' entity='mgr.smithi045.nhtmwc' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-12-16T21:39:10.133 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Dec 16 21:39:09 smithi169 ceph-mon[88350]: pgmap v171: 65 pgs: 65 active+clean; 156 MiB data, 2.3 GiB used, 534 GiB / 536 GiB avail; 2.6 MiB/s rd, 5.2 MiB/s wr, 547 op/s 2023-12-16T21:39:10.133 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Dec 16 21:39:09 smithi169 ceph-mon[88350]: from='mgr.14214 172.21.15.45:0/1227909437' entity='mgr.smithi045.nhtmwc' 2023-12-16T21:39:10.133 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Dec 16 21:39:09 smithi169 ceph-mon[88350]: from='mgr.14214 172.21.15.45:0/1227909437' entity='mgr.smithi045.nhtmwc' 2023-12-16T21:39:10.134 INFO:journalctl@ceph.mon.smithi169.smithi169.stdout:Dec 16 21:39:09 smithi169 ceph-mon[88350]: from='mgr.14214 172.21.15.45:0/1227909437' entity='mgr.smithi045.nhtmwc' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-12-16T21:39:31.316 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:39:30 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:39:30.932+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:05.622397+0000 front 2023-12-16T21:39:05.622556+0000 (oldest deadline 2023-12-16T21:39:30.322546+0000) 2023-12-16T21:39:31.316 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:39:30 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:39:30.932+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:05.622506+0000 front 2023-12-16T21:39:05.622528+0000 (oldest deadline 2023-12-16T21:39:30.322546+0000) 2023-12-16T21:39:31.316 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:39:30 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:39:30.932+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:05.622479+0000 front 2023-12-16T21:39:05.622580+0000 (oldest deadline 2023-12-16T21:39:30.322546+0000) 2023-12-16T21:39:32.317 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:39:31 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:39:31.892+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:05.622397+0000 front 2023-12-16T21:39:05.622556+0000 (oldest deadline 2023-12-16T21:39:30.322546+0000) 2023-12-16T21:39:32.317 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:39:31 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:39:31.892+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:05.622506+0000 front 2023-12-16T21:39:05.622528+0000 (oldest deadline 2023-12-16T21:39:30.322546+0000) 2023-12-16T21:39:32.317 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:39:31 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:39:31.892+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:05.622479+0000 front 2023-12-16T21:39:05.622580+0000 (oldest deadline 2023-12-16T21:39:30.322546+0000) 2023-12-16T21:39:33.066 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:39:32 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:39:32.891+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:05.622397+0000 front 2023-12-16T21:39:05.622556+0000 (oldest deadline 2023-12-16T21:39:30.322546+0000) 2023-12-16T21:39:33.066 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:39:32 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:39:32.891+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:05.622506+0000 front 2023-12-16T21:39:05.622528+0000 (oldest deadline 2023-12-16T21:39:30.322546+0000) 2023-12-16T21:39:33.066 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:39:32 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:39:32.891+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:05.622479+0000 front 2023-12-16T21:39:05.622580+0000 (oldest deadline 2023-12-16T21:39:30.322546+0000) 2023-12-16T21:39:33.067 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:39:32 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:39:32.743+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:09.828911+0000 front 2023-12-16T21:39:09.828958+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:39:33.067 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:39:32 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:39:32.743+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:09.828868+0000 front 2023-12-16T21:39:09.828878+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:39:33.067 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:39:32 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:39:32.743+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:09.828947+0000 front 2023-12-16T21:39:09.828993+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:39:33.767 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:39:33 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:39:33.637+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:09.306583+0000 front 2023-12-16T21:39:09.306633+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:39:33.767 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:39:33 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:39:33.637+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:09.306551+0000 front 2023-12-16T21:39:09.306652+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:39:33.767 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:39:33 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:39:33.637+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:09.306605+0000 front 2023-12-16T21:39:09.306613+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:39:34.066 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:39:33 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:39:33.764+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:09.828911+0000 front 2023-12-16T21:39:09.828958+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:39:34.066 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:39:33 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:39:33.764+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:09.828868+0000 front 2023-12-16T21:39:09.828878+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:39:34.066 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:39:33 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:39:33.764+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:09.828947+0000 front 2023-12-16T21:39:09.828993+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:39:34.066 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:39:33 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:39:33.855+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:05.622397+0000 front 2023-12-16T21:39:05.622556+0000 (oldest deadline 2023-12-16T21:39:30.322546+0000) 2023-12-16T21:39:34.066 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:39:33 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:39:33.855+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:05.622506+0000 front 2023-12-16T21:39:05.622528+0000 (oldest deadline 2023-12-16T21:39:30.322546+0000) 2023-12-16T21:39:34.067 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:39:33 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:39:33.855+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:05.622479+0000 front 2023-12-16T21:39:05.622580+0000 (oldest deadline 2023-12-16T21:39:30.322546+0000) 2023-12-16T21:39:35.066 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:39:34 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:39:34.809+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:09.828911+0000 front 2023-12-16T21:39:09.828958+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:39:35.066 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:39:34 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:39:34.809+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:09.828868+0000 front 2023-12-16T21:39:09.828878+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:39:35.066 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:39:34 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:39:34.809+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:09.828947+0000 front 2023-12-16T21:39:09.828993+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:39:35.066 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:39:34 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:39:34.635+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:09.306583+0000 front 2023-12-16T21:39:09.306633+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:39:35.067 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:39:34 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:39:34.635+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:09.306551+0000 front 2023-12-16T21:39:09.306652+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:39:35.067 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:39:34 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:39:34.635+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:09.306605+0000 front 2023-12-16T21:39:09.306613+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:39:35.067 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:39:34 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:39:34.821+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:05.622397+0000 front 2023-12-16T21:39:05.622556+0000 (oldest deadline 2023-12-16T21:39:30.322546+0000) 2023-12-16T21:39:35.067 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:39:34 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:39:34.821+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:05.622506+0000 front 2023-12-16T21:39:05.622528+0000 (oldest deadline 2023-12-16T21:39:30.322546+0000) 2023-12-16T21:39:35.067 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:39:34 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:39:34.821+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:05.622479+0000 front 2023-12-16T21:39:05.622580+0000 (oldest deadline 2023-12-16T21:39:30.322546+0000) 2023-12-16T21:39:36.066 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:39:35 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:39:35.772+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:09.828911+0000 front 2023-12-16T21:39:09.828958+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:39:36.066 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:39:35 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:39:35.772+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:09.828868+0000 front 2023-12-16T21:39:09.828878+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:39:36.066 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:39:35 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:39:35.772+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:09.828947+0000 front 2023-12-16T21:39:09.828993+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:39:36.067 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:39:35 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:39:35.614+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:09.306583+0000 front 2023-12-16T21:39:09.306633+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:39:36.067 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:39:35 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:39:35.614+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:09.306551+0000 front 2023-12-16T21:39:09.306652+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:39:36.067 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:39:35 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:39:35.614+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:09.306605+0000 front 2023-12-16T21:39:09.306613+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:39:36.067 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:39:35 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:39:35.775+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:05.622397+0000 front 2023-12-16T21:39:05.622556+0000 (oldest deadline 2023-12-16T21:39:30.322546+0000) 2023-12-16T21:39:36.067 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:39:35 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:39:35.775+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:05.622506+0000 front 2023-12-16T21:39:05.622528+0000 (oldest deadline 2023-12-16T21:39:30.322546+0000) 2023-12-16T21:39:36.067 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:39:35 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:39:35.775+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:05.622479+0000 front 2023-12-16T21:39:05.622580+0000 (oldest deadline 2023-12-16T21:39:30.322546+0000) 2023-12-16T21:39:37.066 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:39:36 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:39:36.795+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:09.828911+0000 front 2023-12-16T21:39:09.828958+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:39:37.067 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:39:36 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:39:36.795+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:09.828868+0000 front 2023-12-16T21:39:09.828878+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:39:37.067 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:39:36 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:39:36.795+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:09.828947+0000 front 2023-12-16T21:39:09.828993+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:39:37.067 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:39:36 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:39:36.640+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:09.306583+0000 front 2023-12-16T21:39:09.306633+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:39:37.067 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:39:36 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:39:36.640+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:09.306551+0000 front 2023-12-16T21:39:09.306652+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:39:37.067 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:39:36 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:39:36.640+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:09.306605+0000 front 2023-12-16T21:39:09.306613+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:39:37.067 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:39:36 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:39:36.808+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:05.622397+0000 front 2023-12-16T21:39:05.622556+0000 (oldest deadline 2023-12-16T21:39:30.322546+0000) 2023-12-16T21:39:37.067 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:39:36 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:39:36.808+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:05.622506+0000 front 2023-12-16T21:39:05.622528+0000 (oldest deadline 2023-12-16T21:39:30.322546+0000) 2023-12-16T21:39:37.068 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:39:36 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:39:36.808+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:05.622479+0000 front 2023-12-16T21:39:05.622580+0000 (oldest deadline 2023-12-16T21:39:30.322546+0000) 2023-12-16T21:39:37.811 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:39:37 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:39:37.613+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:09.306583+0000 front 2023-12-16T21:39:09.306633+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:39:37.811 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:39:37 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:39:37.613+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:09.306551+0000 front 2023-12-16T21:39:09.306652+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:39:37.811 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:39:37 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:39:37.613+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:09.306605+0000 front 2023-12-16T21:39:09.306613+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:39:38.066 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:39:37 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:39:37.808+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:09.828911+0000 front 2023-12-16T21:39:09.828958+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:39:38.066 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:39:37 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:39:37.808+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:09.828868+0000 front 2023-12-16T21:39:09.828878+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:39:38.066 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:39:37 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:39:37.808+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:09.828947+0000 front 2023-12-16T21:39:09.828993+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:39:38.067 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:39:37 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:39:37.811+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:05.622397+0000 front 2023-12-16T21:39:05.622556+0000 (oldest deadline 2023-12-16T21:39:30.322546+0000) 2023-12-16T21:39:38.067 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:39:37 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:39:37.811+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:05.622506+0000 front 2023-12-16T21:39:05.622528+0000 (oldest deadline 2023-12-16T21:39:30.322546+0000) 2023-12-16T21:39:38.067 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:39:37 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:39:37.811+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:05.622479+0000 front 2023-12-16T21:39:05.622580+0000 (oldest deadline 2023-12-16T21:39:30.322546+0000) 2023-12-16T21:39:39.066 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:39:38 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:39:38.773+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:05.622397+0000 front 2023-12-16T21:39:05.622556+0000 (oldest deadline 2023-12-16T21:39:30.322546+0000) 2023-12-16T21:39:39.066 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:39:38 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:39:38.773+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:05.622506+0000 front 2023-12-16T21:39:05.622528+0000 (oldest deadline 2023-12-16T21:39:30.322546+0000) 2023-12-16T21:39:39.066 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:39:38 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:39:38.773+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:05.622479+0000 front 2023-12-16T21:39:05.622580+0000 (oldest deadline 2023-12-16T21:39:30.322546+0000) 2023-12-16T21:39:39.067 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:39:38 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:39:38.760+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:09.828911+0000 front 2023-12-16T21:39:09.828958+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:39:39.067 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:39:38 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:39:38.760+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:09.828868+0000 front 2023-12-16T21:39:09.828878+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:39:39.067 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:39:38 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:39:38.760+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:09.828947+0000 front 2023-12-16T21:39:09.828993+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:39:39.067 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:39:38 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:39:38.589+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:09.306583+0000 front 2023-12-16T21:39:09.306633+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:39:39.067 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:39:38 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:39:38.589+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:09.306551+0000 front 2023-12-16T21:39:09.306652+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:39:39.067 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:39:38 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:39:38.589+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:09.306605+0000 front 2023-12-16T21:39:09.306613+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:39:40.066 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:39:39 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:39:39.728+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:05.622397+0000 front 2023-12-16T21:39:05.622556+0000 (oldest deadline 2023-12-16T21:39:30.322546+0000) 2023-12-16T21:39:40.066 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:39:39 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:39:39.728+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:05.622506+0000 front 2023-12-16T21:39:05.622528+0000 (oldest deadline 2023-12-16T21:39:30.322546+0000) 2023-12-16T21:39:40.066 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:39:39 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:39:39.728+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:05.622479+0000 front 2023-12-16T21:39:05.622580+0000 (oldest deadline 2023-12-16T21:39:30.322546+0000) 2023-12-16T21:39:40.067 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:39:39 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:39:39.614+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:09.306583+0000 front 2023-12-16T21:39:09.306633+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:39:40.067 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:39:39 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:39:39.614+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:09.306551+0000 front 2023-12-16T21:39:09.306652+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:39:40.067 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:39:39 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:39:39.614+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:09.306605+0000 front 2023-12-16T21:39:09.306613+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:39:40.067 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:39:39 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:39:39.711+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:09.828911+0000 front 2023-12-16T21:39:09.828958+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:39:40.067 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:39:39 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:39:39.711+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:09.828868+0000 front 2023-12-16T21:39:09.828878+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:39:40.067 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:39:39 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:39:39.711+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:09.828947+0000 front 2023-12-16T21:39:09.828993+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:39:41.067 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:39:40 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:39:40.752+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:09.828911+0000 front 2023-12-16T21:39:09.828958+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:39:41.067 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:39:40 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:39:40.752+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:09.828868+0000 front 2023-12-16T21:39:09.828878+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:39:41.067 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:39:40 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:39:40.752+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:09.828947+0000 front 2023-12-16T21:39:09.828993+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:39:41.067 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:39:40 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:39:40.679+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:05.622397+0000 front 2023-12-16T21:39:05.622556+0000 (oldest deadline 2023-12-16T21:39:30.322546+0000) 2023-12-16T21:39:41.067 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:39:40 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:39:40.679+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:05.622506+0000 front 2023-12-16T21:39:05.622528+0000 (oldest deadline 2023-12-16T21:39:30.322546+0000) 2023-12-16T21:39:41.067 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:39:40 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:39:40.679+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:05.622479+0000 front 2023-12-16T21:39:05.622580+0000 (oldest deadline 2023-12-16T21:39:30.322546+0000) 2023-12-16T21:39:41.068 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:39:40 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:39:40.649+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:09.306583+0000 front 2023-12-16T21:39:09.306633+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:39:41.068 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:39:40 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:39:40.649+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:09.306551+0000 front 2023-12-16T21:39:09.306652+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:39:41.068 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:39:40 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:39:40.649+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:09.306605+0000 front 2023-12-16T21:39:09.306613+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:39:42.066 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:39:41 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:39:41.630+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:05.622397+0000 front 2023-12-16T21:39:05.622556+0000 (oldest deadline 2023-12-16T21:39:30.322546+0000) 2023-12-16T21:39:42.066 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:39:41 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:39:41.630+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:05.622506+0000 front 2023-12-16T21:39:05.622528+0000 (oldest deadline 2023-12-16T21:39:30.322546+0000) 2023-12-16T21:39:42.066 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:39:41 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:39:41.630+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:05.622479+0000 front 2023-12-16T21:39:05.622580+0000 (oldest deadline 2023-12-16T21:39:30.322546+0000) 2023-12-16T21:39:42.067 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:39:41 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:39:41.772+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:09.828911+0000 front 2023-12-16T21:39:09.828958+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:39:42.067 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:39:41 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:39:41.772+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:09.828868+0000 front 2023-12-16T21:39:09.828878+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:39:42.067 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:39:41 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:39:41.772+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:09.828947+0000 front 2023-12-16T21:39:09.828993+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:39:42.067 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:39:41 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:39:41.601+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:09.306583+0000 front 2023-12-16T21:39:09.306633+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:39:42.067 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:39:41 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:39:41.601+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:09.306551+0000 front 2023-12-16T21:39:09.306652+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:39:42.067 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:39:41 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:39:41.601+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:09.306605+0000 front 2023-12-16T21:39:09.306613+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:39:43.066 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:39:42 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:39:42.628+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:09.306583+0000 front 2023-12-16T21:39:09.306633+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:39:43.066 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:39:42 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:39:42.628+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:09.306551+0000 front 2023-12-16T21:39:09.306652+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:39:43.066 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:39:42 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:39:42.628+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:09.306605+0000 front 2023-12-16T21:39:09.306613+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:39:43.067 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:39:42 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:39:42.665+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:05.622397+0000 front 2023-12-16T21:39:05.622556+0000 (oldest deadline 2023-12-16T21:39:30.322546+0000) 2023-12-16T21:39:43.067 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:39:42 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:39:42.665+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:05.622506+0000 front 2023-12-16T21:39:05.622528+0000 (oldest deadline 2023-12-16T21:39:30.322546+0000) 2023-12-16T21:39:43.067 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:39:42 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:39:42.665+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:05.622479+0000 front 2023-12-16T21:39:05.622580+0000 (oldest deadline 2023-12-16T21:39:30.322546+0000) 2023-12-16T21:39:43.067 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:39:42 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:39:42.742+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:09.828911+0000 front 2023-12-16T21:39:09.828958+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:39:43.067 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:39:42 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:39:42.742+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:09.828868+0000 front 2023-12-16T21:39:09.828878+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:39:43.067 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:39:42 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:39:42.742+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:09.828947+0000 front 2023-12-16T21:39:09.828993+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:39:43.816 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:39:43 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:39:43.677+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:09.306583+0000 front 2023-12-16T21:39:09.306633+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:39:43.816 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:39:43 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:39:43.677+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:09.306551+0000 front 2023-12-16T21:39:09.306652+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:39:43.816 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:39:43 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:39:43.677+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:09.306605+0000 front 2023-12-16T21:39:09.306613+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:39:43.817 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:39:43 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:39:43.702+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:05.622397+0000 front 2023-12-16T21:39:05.622556+0000 (oldest deadline 2023-12-16T21:39:30.322546+0000) 2023-12-16T21:39:43.817 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:39:43 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:39:43.702+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:05.622506+0000 front 2023-12-16T21:39:05.622528+0000 (oldest deadline 2023-12-16T21:39:30.322546+0000) 2023-12-16T21:39:43.817 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:39:43 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:39:43.702+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:05.622479+0000 front 2023-12-16T21:39:05.622580+0000 (oldest deadline 2023-12-16T21:39:30.322546+0000) 2023-12-16T21:39:43.817 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:39:43 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:39:43.724+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:09.828911+0000 front 2023-12-16T21:39:09.828958+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:39:43.817 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:39:43 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:39:43.724+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:09.828868+0000 front 2023-12-16T21:39:09.828878+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:39:43.817 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:39:43 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:39:43.724+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:09.828947+0000 front 2023-12-16T21:39:09.828993+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:39:45.066 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:39:44 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:39:44.757+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:09.828911+0000 front 2023-12-16T21:39:09.828958+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:39:45.067 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:39:44 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:39:44.757+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:09.828868+0000 front 2023-12-16T21:39:09.828878+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:39:45.067 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:39:44 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:39:44.757+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:09.828947+0000 front 2023-12-16T21:39:09.828993+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:39:45.067 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:39:44 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:39:44.637+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:09.306583+0000 front 2023-12-16T21:39:09.306633+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:39:45.067 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:39:44 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:39:44.637+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:09.306551+0000 front 2023-12-16T21:39:09.306652+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:39:45.067 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:39:44 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:39:44.637+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:09.306605+0000 front 2023-12-16T21:39:09.306613+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:39:45.068 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:39:44 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:39:44.729+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:05.622397+0000 front 2023-12-16T21:39:05.622556+0000 (oldest deadline 2023-12-16T21:39:30.322546+0000) 2023-12-16T21:39:45.068 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:39:44 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:39:44.729+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:05.622506+0000 front 2023-12-16T21:39:05.622528+0000 (oldest deadline 2023-12-16T21:39:30.322546+0000) 2023-12-16T21:39:45.068 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:39:44 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:39:44.729+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:05.622479+0000 front 2023-12-16T21:39:05.622580+0000 (oldest deadline 2023-12-16T21:39:30.322546+0000) 2023-12-16T21:39:46.066 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:39:45 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:39:45.635+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:09.306583+0000 front 2023-12-16T21:39:09.306633+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:39:46.067 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:39:45 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:39:45.635+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:09.306551+0000 front 2023-12-16T21:39:09.306652+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:39:46.067 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:39:45 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:39:45.635+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:09.306605+0000 front 2023-12-16T21:39:09.306613+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:39:46.067 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:39:45 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:39:45.764+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:09.828911+0000 front 2023-12-16T21:39:09.828958+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:39:46.067 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:39:45 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:39:45.764+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:09.828868+0000 front 2023-12-16T21:39:09.828878+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:39:46.067 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:39:45 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:39:45.764+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:09.828947+0000 front 2023-12-16T21:39:09.828993+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:39:46.067 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:39:45 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:39:45.713+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:05.622397+0000 front 2023-12-16T21:39:05.622556+0000 (oldest deadline 2023-12-16T21:39:30.322546+0000) 2023-12-16T21:39:46.067 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:39:45 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:39:45.713+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:05.622506+0000 front 2023-12-16T21:39:05.622528+0000 (oldest deadline 2023-12-16T21:39:30.322546+0000) 2023-12-16T21:39:46.068 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:39:45 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:39:45.713+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:05.622479+0000 front 2023-12-16T21:39:05.622580+0000 (oldest deadline 2023-12-16T21:39:30.322546+0000) 2023-12-16T21:39:47.066 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:39:46 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:39:46.610+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:09.306583+0000 front 2023-12-16T21:39:09.306633+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:39:47.066 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:39:46 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:39:46.610+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:09.306551+0000 front 2023-12-16T21:39:09.306652+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:39:47.066 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:39:46 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:39:46.610+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:09.306605+0000 front 2023-12-16T21:39:09.306613+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:39:47.067 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:39:46 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:39:46.750+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:09.828911+0000 front 2023-12-16T21:39:09.828958+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:39:47.067 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:39:46 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:39:46.750+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:09.828868+0000 front 2023-12-16T21:39:09.828878+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:39:47.067 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:39:46 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:39:46.750+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:09.828947+0000 front 2023-12-16T21:39:09.828993+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:39:47.067 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:39:46 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:39:46.713+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:05.622397+0000 front 2023-12-16T21:39:05.622556+0000 (oldest deadline 2023-12-16T21:39:30.322546+0000) 2023-12-16T21:39:47.067 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:39:46 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:39:46.713+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:05.622506+0000 front 2023-12-16T21:39:05.622528+0000 (oldest deadline 2023-12-16T21:39:30.322546+0000) 2023-12-16T21:39:47.067 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:39:46 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:39:46.713+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:05.622479+0000 front 2023-12-16T21:39:05.622580+0000 (oldest deadline 2023-12-16T21:39:30.322546+0000) 2023-12-16T21:39:48.067 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:39:47 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:39:47.647+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:09.306583+0000 front 2023-12-16T21:39:09.306633+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:39:48.067 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:39:47 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:39:47.647+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:09.306551+0000 front 2023-12-16T21:39:09.306652+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:39:48.067 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:39:47 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:39:47.647+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:09.306605+0000 front 2023-12-16T21:39:09.306613+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:39:48.068 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:39:47 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:39:47.712+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:09.828911+0000 front 2023-12-16T21:39:09.828958+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:39:48.068 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:39:47 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:39:47.712+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:09.828868+0000 front 2023-12-16T21:39:09.828878+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:39:48.068 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:39:47 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:39:47.712+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:09.828947+0000 front 2023-12-16T21:39:09.828993+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:39:48.069 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:39:47 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:39:47.720+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:05.622397+0000 front 2023-12-16T21:39:05.622556+0000 (oldest deadline 2023-12-16T21:39:30.322546+0000) 2023-12-16T21:39:48.069 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:39:47 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:39:47.720+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:05.622506+0000 front 2023-12-16T21:39:05.622528+0000 (oldest deadline 2023-12-16T21:39:30.322546+0000) 2023-12-16T21:39:48.070 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:39:47 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:39:47.720+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:05.622479+0000 front 2023-12-16T21:39:05.622580+0000 (oldest deadline 2023-12-16T21:39:30.322546+0000) 2023-12-16T21:39:49.066 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:39:48 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:39:48.717+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:05.622397+0000 front 2023-12-16T21:39:05.622556+0000 (oldest deadline 2023-12-16T21:39:30.322546+0000) 2023-12-16T21:39:49.066 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:39:48 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:39:48.717+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:05.622506+0000 front 2023-12-16T21:39:05.622528+0000 (oldest deadline 2023-12-16T21:39:30.322546+0000) 2023-12-16T21:39:49.066 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:39:48 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:39:48.717+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:05.622479+0000 front 2023-12-16T21:39:05.622580+0000 (oldest deadline 2023-12-16T21:39:30.322546+0000) 2023-12-16T21:39:49.067 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:39:48 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:39:48.681+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:09.306583+0000 front 2023-12-16T21:39:09.306633+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:39:49.067 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:39:48 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:39:48.681+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:09.306551+0000 front 2023-12-16T21:39:09.306652+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:39:49.067 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:39:48 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:39:48.681+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:09.306605+0000 front 2023-12-16T21:39:09.306613+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:39:49.067 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:39:48 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:39:48.732+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:09.828911+0000 front 2023-12-16T21:39:09.828958+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:39:49.067 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:39:48 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:39:48.732+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:09.828868+0000 front 2023-12-16T21:39:09.828878+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:39:49.067 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:39:48 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:39:48.732+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:09.828947+0000 front 2023-12-16T21:39:09.828993+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:39:49.967 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:39:49 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:39:49.702+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:05.622397+0000 front 2023-12-16T21:39:05.622556+0000 (oldest deadline 2023-12-16T21:39:30.322546+0000) 2023-12-16T21:39:49.968 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:39:49 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:39:49.702+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:05.622506+0000 front 2023-12-16T21:39:05.622528+0000 (oldest deadline 2023-12-16T21:39:30.322546+0000) 2023-12-16T21:39:49.968 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:39:49 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:39:49.702+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:05.622479+0000 front 2023-12-16T21:39:05.622580+0000 (oldest deadline 2023-12-16T21:39:30.322546+0000) 2023-12-16T21:39:49.968 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:39:49 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:39:49.744+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:09.828911+0000 front 2023-12-16T21:39:09.828958+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:39:49.968 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:39:49 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:39:49.744+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:09.828868+0000 front 2023-12-16T21:39:09.828878+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:39:49.968 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:39:49 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:39:49.744+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:09.828947+0000 front 2023-12-16T21:39:09.828993+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:39:49.969 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:39:49 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:39:49.651+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:09.306583+0000 front 2023-12-16T21:39:09.306633+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:39:49.969 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:39:49 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:39:49.651+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:09.306551+0000 front 2023-12-16T21:39:09.306652+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:39:49.969 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:39:49 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:39:49.651+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:09.306605+0000 front 2023-12-16T21:39:09.306613+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:39:51.066 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:39:50 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:39:50.710+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:05.622397+0000 front 2023-12-16T21:39:05.622556+0000 (oldest deadline 2023-12-16T21:39:30.322546+0000) 2023-12-16T21:39:51.066 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:39:50 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:39:50.710+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:05.622506+0000 front 2023-12-16T21:39:05.622528+0000 (oldest deadline 2023-12-16T21:39:30.322546+0000) 2023-12-16T21:39:51.066 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:39:50 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:39:50.710+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:05.622479+0000 front 2023-12-16T21:39:05.622580+0000 (oldest deadline 2023-12-16T21:39:30.322546+0000) 2023-12-16T21:39:51.067 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:39:50 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:39:50.664+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:09.306583+0000 front 2023-12-16T21:39:09.306633+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:39:51.067 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:39:50 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:39:50.664+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:09.306551+0000 front 2023-12-16T21:39:09.306652+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:39:51.067 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:39:50 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:39:50.664+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:09.306605+0000 front 2023-12-16T21:39:09.306613+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:39:51.067 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:39:50 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:39:50.778+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:09.828911+0000 front 2023-12-16T21:39:09.828958+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:39:51.067 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:39:50 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:39:50.778+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:09.828868+0000 front 2023-12-16T21:39:09.828878+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:39:51.067 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:39:50 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:39:50.778+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:09.828947+0000 front 2023-12-16T21:39:09.828993+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:39:52.066 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:39:51 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:39:51.669+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:05.622397+0000 front 2023-12-16T21:39:05.622556+0000 (oldest deadline 2023-12-16T21:39:30.322546+0000) 2023-12-16T21:39:52.066 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:39:51 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:39:51.669+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:05.622506+0000 front 2023-12-16T21:39:05.622528+0000 (oldest deadline 2023-12-16T21:39:30.322546+0000) 2023-12-16T21:39:52.066 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:39:51 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:39:51.669+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:05.622479+0000 front 2023-12-16T21:39:05.622580+0000 (oldest deadline 2023-12-16T21:39:30.322546+0000) 2023-12-16T21:39:52.066 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:39:51 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:39:51.730+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:09.828911+0000 front 2023-12-16T21:39:09.828958+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:39:52.067 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:39:51 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:39:51.730+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:09.828868+0000 front 2023-12-16T21:39:09.828878+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:39:52.067 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:39:51 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:39:51.730+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:09.828947+0000 front 2023-12-16T21:39:09.828993+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:39:52.067 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:39:51 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:39:51.689+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:09.306583+0000 front 2023-12-16T21:39:09.306633+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:39:52.067 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:39:51 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:39:51.689+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:09.306551+0000 front 2023-12-16T21:39:09.306652+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:39:52.067 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:39:51 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:39:51.689+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:09.306605+0000 front 2023-12-16T21:39:09.306613+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:39:53.066 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:39:52 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:39:52.725+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:09.306583+0000 front 2023-12-16T21:39:09.306633+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:39:53.066 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:39:52 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:39:52.725+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:09.306551+0000 front 2023-12-16T21:39:09.306652+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:39:53.066 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:39:52 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:39:52.725+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:09.306605+0000 front 2023-12-16T21:39:09.306613+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:39:53.067 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:39:52 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:39:52.742+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:09.828911+0000 front 2023-12-16T21:39:09.828958+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:39:53.067 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:39:52 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:39:52.742+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:09.828868+0000 front 2023-12-16T21:39:09.828878+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:39:53.067 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:39:52 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:39:52.742+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:09.828947+0000 front 2023-12-16T21:39:09.828993+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:39:53.067 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:39:52 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:39:52.627+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:05.622397+0000 front 2023-12-16T21:39:05.622556+0000 (oldest deadline 2023-12-16T21:39:30.322546+0000) 2023-12-16T21:39:53.067 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:39:52 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:39:52.627+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:05.622506+0000 front 2023-12-16T21:39:05.622528+0000 (oldest deadline 2023-12-16T21:39:30.322546+0000) 2023-12-16T21:39:53.067 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:39:52 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:39:52.627+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:05.622479+0000 front 2023-12-16T21:39:05.622580+0000 (oldest deadline 2023-12-16T21:39:30.322546+0000) 2023-12-16T21:39:53.816 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:39:53 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:39:53.742+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:09.828911+0000 front 2023-12-16T21:39:09.828958+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:39:53.816 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:39:53 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:39:53.742+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:09.828868+0000 front 2023-12-16T21:39:09.828878+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:39:53.816 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:39:53 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:39:53.742+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:09.828947+0000 front 2023-12-16T21:39:09.828993+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:39:53.817 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:39:53 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:39:53.578+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:05.622397+0000 front 2023-12-16T21:39:05.622556+0000 (oldest deadline 2023-12-16T21:39:30.322546+0000) 2023-12-16T21:39:53.817 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:39:53 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:39:53.578+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:05.622506+0000 front 2023-12-16T21:39:05.622528+0000 (oldest deadline 2023-12-16T21:39:30.322546+0000) 2023-12-16T21:39:53.817 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:39:53 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:39:53.578+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:05.622479+0000 front 2023-12-16T21:39:05.622580+0000 (oldest deadline 2023-12-16T21:39:30.322546+0000) 2023-12-16T21:39:53.817 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:39:53 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:39:53.697+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:09.306583+0000 front 2023-12-16T21:39:09.306633+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:39:53.817 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:39:53 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:39:53.697+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:09.306551+0000 front 2023-12-16T21:39:09.306652+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:39:53.817 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:39:53 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:39:53.697+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:09.306605+0000 front 2023-12-16T21:39:09.306613+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:39:54.816 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:39:54 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:39:54.685+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:09.306583+0000 front 2023-12-16T21:39:09.306633+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:39:54.816 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:39:54 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:39:54.685+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:09.306551+0000 front 2023-12-16T21:39:09.306652+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:39:54.816 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:39:54 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:39:54.685+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:09.306605+0000 front 2023-12-16T21:39:09.306613+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:39:54.817 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:39:54 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:39:54.532+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:05.622397+0000 front 2023-12-16T21:39:05.622556+0000 (oldest deadline 2023-12-16T21:39:30.322546+0000) 2023-12-16T21:39:54.817 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:39:54 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:39:54.532+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:05.622506+0000 front 2023-12-16T21:39:05.622528+0000 (oldest deadline 2023-12-16T21:39:30.322546+0000) 2023-12-16T21:39:54.817 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:39:54 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:39:54.532+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:05.622479+0000 front 2023-12-16T21:39:05.622580+0000 (oldest deadline 2023-12-16T21:39:30.322546+0000) 2023-12-16T21:39:54.817 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:39:54 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:39:54.772+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:09.828911+0000 front 2023-12-16T21:39:09.828958+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:39:54.817 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:39:54 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:39:54.772+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:09.828868+0000 front 2023-12-16T21:39:09.828878+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:39:54.817 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:39:54 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:39:54.772+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:09.828947+0000 front 2023-12-16T21:39:09.828993+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:39:55.817 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:39:55 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:39:55.683+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:09.306583+0000 front 2023-12-16T21:39:09.306633+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:39:55.818 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:39:55 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:39:55.683+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:09.306551+0000 front 2023-12-16T21:39:09.306652+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:39:55.818 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:39:55 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:39:55.683+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:09.306605+0000 front 2023-12-16T21:39:09.306613+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:39:55.818 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:39:55 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:39:55.805+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:09.828911+0000 front 2023-12-16T21:39:09.828958+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:39:55.818 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:39:55 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:39:55.805+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:09.828868+0000 front 2023-12-16T21:39:09.828878+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:39:55.818 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:39:55 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:39:55.805+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:09.828947+0000 front 2023-12-16T21:39:09.828993+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:39:55.818 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:39:55 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:39:55.559+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:05.622397+0000 front 2023-12-16T21:39:05.622556+0000 (oldest deadline 2023-12-16T21:39:30.322546+0000) 2023-12-16T21:39:55.819 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:39:55 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:39:55.559+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:05.622506+0000 front 2023-12-16T21:39:05.622528+0000 (oldest deadline 2023-12-16T21:39:30.322546+0000) 2023-12-16T21:39:55.819 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:39:55 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:39:55.559+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:05.622479+0000 front 2023-12-16T21:39:05.622580+0000 (oldest deadline 2023-12-16T21:39:30.322546+0000) 2023-12-16T21:39:56.694 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:39:56 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:39:56.559+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:05.622397+0000 front 2023-12-16T21:39:05.622556+0000 (oldest deadline 2023-12-16T21:39:30.322546+0000) 2023-12-16T21:39:56.694 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:39:56 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:39:56.559+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:05.622506+0000 front 2023-12-16T21:39:05.622528+0000 (oldest deadline 2023-12-16T21:39:30.322546+0000) 2023-12-16T21:39:56.694 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:39:56 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:39:56.559+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:05.622479+0000 front 2023-12-16T21:39:05.622580+0000 (oldest deadline 2023-12-16T21:39:30.322546+0000) 2023-12-16T21:39:57.066 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:39:56 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:39:56.690+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:09.306583+0000 front 2023-12-16T21:39:09.306633+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:39:57.066 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:39:56 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:39:56.690+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:09.306551+0000 front 2023-12-16T21:39:09.306652+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:39:57.067 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:39:56 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:39:56.690+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:09.306605+0000 front 2023-12-16T21:39:09.306613+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:39:57.067 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:39:56 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:39:56.780+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:09.828911+0000 front 2023-12-16T21:39:09.828958+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:39:57.067 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:39:56 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:39:56.780+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:09.828868+0000 front 2023-12-16T21:39:09.828878+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:39:57.067 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:39:56 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:39:56.780+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:09.828947+0000 front 2023-12-16T21:39:09.828993+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:39:57.802 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:39:57 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:39:57.732+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:09.306583+0000 front 2023-12-16T21:39:09.306633+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:39:57.802 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:39:57 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:39:57.732+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:09.306551+0000 front 2023-12-16T21:39:09.306652+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:39:57.802 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:39:57 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:39:57.732+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:09.306605+0000 front 2023-12-16T21:39:09.306613+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:39:57.802 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:39:57 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:39:57.510+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:05.622397+0000 front 2023-12-16T21:39:05.622556+0000 (oldest deadline 2023-12-16T21:39:30.322546+0000) 2023-12-16T21:39:57.802 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:39:57 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:39:57.510+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:05.622506+0000 front 2023-12-16T21:39:05.622528+0000 (oldest deadline 2023-12-16T21:39:30.322546+0000) 2023-12-16T21:39:57.802 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:39:57 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:39:57.510+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:05.622479+0000 front 2023-12-16T21:39:05.622580+0000 (oldest deadline 2023-12-16T21:39:30.322546+0000) 2023-12-16T21:39:58.066 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:39:57 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:39:57.799+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:09.828911+0000 front 2023-12-16T21:39:09.828958+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:39:58.066 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:39:57 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:39:57.799+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:09.828868+0000 front 2023-12-16T21:39:09.828878+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:39:58.066 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:39:57 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:39:57.799+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:09.828947+0000 front 2023-12-16T21:39:09.828993+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:39:58.816 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:39:58 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:39:58.484+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:05.622397+0000 front 2023-12-16T21:39:05.622556+0000 (oldest deadline 2023-12-16T21:39:30.322546+0000) 2023-12-16T21:39:58.816 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:39:58 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:39:58.484+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:05.622506+0000 front 2023-12-16T21:39:05.622528+0000 (oldest deadline 2023-12-16T21:39:30.322546+0000) 2023-12-16T21:39:58.816 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:39:58 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:39:58.484+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:05.622479+0000 front 2023-12-16T21:39:05.622580+0000 (oldest deadline 2023-12-16T21:39:30.322546+0000) 2023-12-16T21:39:58.817 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:39:58 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:39:58.723+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:09.306583+0000 front 2023-12-16T21:39:09.306633+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:39:58.817 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:39:58 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:39:58.723+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:09.306551+0000 front 2023-12-16T21:39:09.306652+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:39:58.817 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:39:58 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:39:58.723+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:09.306605+0000 front 2023-12-16T21:39:09.306613+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:39:59.316 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:39:58 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:39:58.826+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:09.828911+0000 front 2023-12-16T21:39:09.828958+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:39:59.316 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:39:58 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:39:58.826+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:09.828868+0000 front 2023-12-16T21:39:09.828878+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:39:59.316 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:39:58 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:39:58.826+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:09.828947+0000 front 2023-12-16T21:39:09.828993+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:39:59.747 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:39:59 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:39:59.447+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:05.622397+0000 front 2023-12-16T21:39:05.622556+0000 (oldest deadline 2023-12-16T21:39:30.322546+0000) 2023-12-16T21:39:59.748 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:39:59 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:39:59.447+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:05.622506+0000 front 2023-12-16T21:39:05.622528+0000 (oldest deadline 2023-12-16T21:39:30.322546+0000) 2023-12-16T21:39:59.748 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:39:59 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:39:59.447+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:05.622479+0000 front 2023-12-16T21:39:05.622580+0000 (oldest deadline 2023-12-16T21:39:30.322546+0000) 2023-12-16T21:40:00.066 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:39:59 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:39:59.745+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:09.306583+0000 front 2023-12-16T21:39:09.306633+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:40:00.066 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:39:59 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:39:59.745+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:09.306551+0000 front 2023-12-16T21:39:09.306652+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:40:00.066 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:39:59 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:39:59.745+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:09.306605+0000 front 2023-12-16T21:39:09.306613+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:40:00.066 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:39:59 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:39:59.818+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:09.828911+0000 front 2023-12-16T21:39:09.828958+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:40:00.067 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:39:59 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:39:59.818+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:09.828868+0000 front 2023-12-16T21:39:09.828878+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:40:00.067 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:39:59 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:39:59.818+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:09.828947+0000 front 2023-12-16T21:39:09.828993+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:40:00.743 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:40:00 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:40:00.435+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:05.622397+0000 front 2023-12-16T21:39:05.622556+0000 (oldest deadline 2023-12-16T21:39:30.322546+0000) 2023-12-16T21:40:00.744 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:40:00 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:40:00.435+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:05.622506+0000 front 2023-12-16T21:39:05.622528+0000 (oldest deadline 2023-12-16T21:39:30.322546+0000) 2023-12-16T21:40:00.744 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:40:00 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:40:00.435+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:05.622479+0000 front 2023-12-16T21:39:05.622580+0000 (oldest deadline 2023-12-16T21:39:30.322546+0000) 2023-12-16T21:40:01.066 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:40:00 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:40:00.740+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:09.306583+0000 front 2023-12-16T21:39:09.306633+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:40:01.066 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:40:00 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:40:00.740+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:09.306551+0000 front 2023-12-16T21:39:09.306652+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:40:01.066 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:40:00 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:40:00.741+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:09.306605+0000 front 2023-12-16T21:39:09.306613+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:40:01.067 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:40:00 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:40:00.801+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:09.828911+0000 front 2023-12-16T21:39:09.828958+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:40:01.067 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:40:00 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:40:00.801+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:09.828868+0000 front 2023-12-16T21:39:09.828878+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:40:01.067 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:40:00 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:40:00.801+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:09.828947+0000 front 2023-12-16T21:39:09.828993+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:40:01.754 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:40:01 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:40:01.425+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:05.622397+0000 front 2023-12-16T21:39:05.622556+0000 (oldest deadline 2023-12-16T21:39:30.322546+0000) 2023-12-16T21:40:01.754 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:40:01 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:40:01.425+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:05.622506+0000 front 2023-12-16T21:39:05.622528+0000 (oldest deadline 2023-12-16T21:39:30.322546+0000) 2023-12-16T21:40:01.754 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:40:01 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:40:01.425+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:05.622479+0000 front 2023-12-16T21:39:05.622580+0000 (oldest deadline 2023-12-16T21:39:30.322546+0000) 2023-12-16T21:40:02.066 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:40:01 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:40:01.798+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:09.828911+0000 front 2023-12-16T21:39:09.828958+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:40:02.066 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:40:01 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:40:01.798+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:09.828868+0000 front 2023-12-16T21:39:09.828878+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:40:02.066 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:40:01 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:40:01.798+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:09.828947+0000 front 2023-12-16T21:39:09.828993+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:40:02.067 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:40:01 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:40:01.751+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:09.306583+0000 front 2023-12-16T21:39:09.306633+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:40:02.067 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:40:01 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:40:01.751+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:09.306551+0000 front 2023-12-16T21:39:09.306652+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:40:02.067 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:40:01 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:40:01.751+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:09.306605+0000 front 2023-12-16T21:39:09.306613+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:40:02.798 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:40:02 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:40:02.432+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:05.622397+0000 front 2023-12-16T21:39:05.622556+0000 (oldest deadline 2023-12-16T21:39:30.322546+0000) 2023-12-16T21:40:02.799 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:40:02 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:40:02.432+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:05.622506+0000 front 2023-12-16T21:39:05.622528+0000 (oldest deadline 2023-12-16T21:39:30.322546+0000) 2023-12-16T21:40:02.799 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:40:02 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:40:02.432+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:05.622479+0000 front 2023-12-16T21:39:05.622580+0000 (oldest deadline 2023-12-16T21:39:30.322546+0000) 2023-12-16T21:40:03.066 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:40:02 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:40:02.796+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:09.306583+0000 front 2023-12-16T21:39:09.306633+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:40:03.066 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:40:02 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:40:02.796+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:09.306551+0000 front 2023-12-16T21:39:09.306652+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:40:03.066 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:40:02 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:40:02.796+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:09.306605+0000 front 2023-12-16T21:39:09.306613+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:40:03.067 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:40:02 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:40:02.839+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:09.828911+0000 front 2023-12-16T21:39:09.828958+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:40:03.067 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:40:02 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:40:02.839+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:09.828868+0000 front 2023-12-16T21:39:09.828878+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:40:03.067 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:40:02 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:40:02.839+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:09.828947+0000 front 2023-12-16T21:39:09.828993+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:40:03.779 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:40:03 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:40:03.432+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:05.622397+0000 front 2023-12-16T21:39:05.622556+0000 (oldest deadline 2023-12-16T21:39:30.322546+0000) 2023-12-16T21:40:03.779 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:40:03 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:40:03.432+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:05.622506+0000 front 2023-12-16T21:39:05.622528+0000 (oldest deadline 2023-12-16T21:39:30.322546+0000) 2023-12-16T21:40:03.779 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:40:03 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:40:03.432+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:05.622479+0000 front 2023-12-16T21:39:05.622580+0000 (oldest deadline 2023-12-16T21:39:30.322546+0000) 2023-12-16T21:40:04.066 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:40:03 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:40:03.775+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:09.306583+0000 front 2023-12-16T21:39:09.306633+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:40:04.067 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:40:03 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:40:03.775+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:09.306551+0000 front 2023-12-16T21:39:09.306652+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:40:04.067 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:40:03 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:40:03.775+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:09.306605+0000 front 2023-12-16T21:39:09.306613+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:40:04.067 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:40:03 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:40:03.856+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:09.828911+0000 front 2023-12-16T21:39:09.828958+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:40:04.067 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:40:03 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:40:03.856+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:09.828868+0000 front 2023-12-16T21:39:09.828878+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:40:04.067 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:40:03 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:40:03.856+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:09.828947+0000 front 2023-12-16T21:39:09.828993+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:40:04.757 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:40:04 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:40:04.478+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:05.622397+0000 front 2023-12-16T21:39:05.622556+0000 (oldest deadline 2023-12-16T21:39:30.322546+0000) 2023-12-16T21:40:04.758 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:40:04 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:40:04.478+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:05.622506+0000 front 2023-12-16T21:39:05.622528+0000 (oldest deadline 2023-12-16T21:39:30.322546+0000) 2023-12-16T21:40:04.758 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:40:04 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:40:04.478+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:05.622479+0000 front 2023-12-16T21:39:05.622580+0000 (oldest deadline 2023-12-16T21:39:30.322546+0000) 2023-12-16T21:40:05.066 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:40:04 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:40:04.755+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:09.306583+0000 front 2023-12-16T21:39:09.306633+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:40:05.066 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:40:04 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:40:04.755+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:09.306551+0000 front 2023-12-16T21:39:09.306652+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:40:05.066 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:40:04 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:40:04.755+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:09.306605+0000 front 2023-12-16T21:39:09.306613+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:40:05.066 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:40:04 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:40:04.846+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:09.828911+0000 front 2023-12-16T21:39:09.828958+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:40:05.067 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:40:04 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:40:04.846+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:09.828868+0000 front 2023-12-16T21:39:09.828878+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:40:05.067 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:40:04 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:40:04.846+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:09.828947+0000 front 2023-12-16T21:39:09.828993+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:40:05.757 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:40:05 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:40:05.454+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:05.622397+0000 front 2023-12-16T21:39:05.622556+0000 (oldest deadline 2023-12-16T21:39:30.322546+0000) 2023-12-16T21:40:05.757 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:40:05 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:40:05.454+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:05.622506+0000 front 2023-12-16T21:39:05.622528+0000 (oldest deadline 2023-12-16T21:39:30.322546+0000) 2023-12-16T21:40:05.757 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:40:05 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:40:05.454+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:05.622479+0000 front 2023-12-16T21:39:05.622580+0000 (oldest deadline 2023-12-16T21:39:30.322546+0000) 2023-12-16T21:40:06.066 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:40:05 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:40:05.819+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:09.828911+0000 front 2023-12-16T21:39:09.828958+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:40:06.066 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:40:05 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:40:05.819+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:09.828868+0000 front 2023-12-16T21:39:09.828878+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:40:06.067 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:40:05 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:40:05.819+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:09.828947+0000 front 2023-12-16T21:39:09.828993+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:40:06.067 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:40:05 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:40:05.754+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:09.306583+0000 front 2023-12-16T21:39:09.306633+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:40:06.067 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:40:05 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:40:05.754+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:09.306551+0000 front 2023-12-16T21:39:09.306652+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:40:06.067 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:40:05 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:40:05.754+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:09.306605+0000 front 2023-12-16T21:39:09.306613+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:40:06.729 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:40:06 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:40:06.430+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:05.622397+0000 front 2023-12-16T21:39:05.622556+0000 (oldest deadline 2023-12-16T21:39:30.322546+0000) 2023-12-16T21:40:06.729 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:40:06 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:40:06.430+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:05.622506+0000 front 2023-12-16T21:39:05.622528+0000 (oldest deadline 2023-12-16T21:39:30.322546+0000) 2023-12-16T21:40:06.729 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:40:06 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:40:06.430+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:05.622479+0000 front 2023-12-16T21:39:05.622580+0000 (oldest deadline 2023-12-16T21:39:30.322546+0000) 2023-12-16T21:40:07.066 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:40:06 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:40:06.726+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:09.306583+0000 front 2023-12-16T21:39:09.306633+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:40:07.066 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:40:06 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:40:06.727+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:09.306551+0000 front 2023-12-16T21:39:09.306652+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:40:07.066 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:40:06 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:40:06.727+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:09.306605+0000 front 2023-12-16T21:39:09.306613+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:40:07.067 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:40:06 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:40:06.869+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:09.828911+0000 front 2023-12-16T21:39:09.828958+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:40:07.067 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:40:06 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:40:06.869+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:09.828868+0000 front 2023-12-16T21:39:09.828878+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:40:07.067 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:40:06 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:40:06.869+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:09.828947+0000 front 2023-12-16T21:39:09.828993+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:40:07.727 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:40:07 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:40:07.456+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:05.622397+0000 front 2023-12-16T21:39:05.622556+0000 (oldest deadline 2023-12-16T21:39:30.322546+0000) 2023-12-16T21:40:07.727 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:40:07 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:40:07.456+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:05.622506+0000 front 2023-12-16T21:39:05.622528+0000 (oldest deadline 2023-12-16T21:39:30.322546+0000) 2023-12-16T21:40:07.727 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:40:07 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:40:07.456+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:05.622479+0000 front 2023-12-16T21:39:05.622580+0000 (oldest deadline 2023-12-16T21:39:30.322546+0000) 2023-12-16T21:40:08.066 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:40:07 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:40:07.723+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:09.306583+0000 front 2023-12-16T21:39:09.306633+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:40:08.066 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:40:07 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:40:07.723+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:09.306551+0000 front 2023-12-16T21:39:09.306652+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:40:08.066 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:40:07 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:40:07.723+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:09.306605+0000 front 2023-12-16T21:39:09.306613+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:40:08.066 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:40:07 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:40:07.905+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:09.828911+0000 front 2023-12-16T21:39:09.828958+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:40:08.066 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:40:07 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:40:07.905+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:09.828868+0000 front 2023-12-16T21:39:09.828878+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:40:08.067 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:40:07 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:40:07.905+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:09.828947+0000 front 2023-12-16T21:39:09.828993+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:40:08.816 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:40:08 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:40:08.684+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:09.306583+0000 front 2023-12-16T21:39:09.306633+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:40:08.817 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:40:08 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:40:08.684+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:09.306551+0000 front 2023-12-16T21:39:09.306652+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:40:08.817 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:40:08 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:40:08.684+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:09.306605+0000 front 2023-12-16T21:39:09.306613+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:40:08.817 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:40:08 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:40:08.472+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:05.622397+0000 front 2023-12-16T21:39:05.622556+0000 (oldest deadline 2023-12-16T21:39:30.322546+0000) 2023-12-16T21:40:08.817 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:40:08 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:40:08.472+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:05.622506+0000 front 2023-12-16T21:39:05.622528+0000 (oldest deadline 2023-12-16T21:39:30.322546+0000) 2023-12-16T21:40:08.817 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:40:08 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:40:08.472+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:05.622479+0000 front 2023-12-16T21:39:05.622580+0000 (oldest deadline 2023-12-16T21:39:30.322546+0000) 2023-12-16T21:40:09.316 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:40:08 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:40:08.904+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:09.828911+0000 front 2023-12-16T21:39:09.828958+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:40:09.316 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:40:08 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:40:08.904+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:09.828868+0000 front 2023-12-16T21:39:09.828878+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:40:09.316 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:40:08 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:40:08.904+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:09.828947+0000 front 2023-12-16T21:39:09.828993+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:40:09.730 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:40:09 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:40:09.447+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:05.622397+0000 front 2023-12-16T21:39:05.622556+0000 (oldest deadline 2023-12-16T21:39:30.322546+0000) 2023-12-16T21:40:09.731 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:40:09 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:40:09.447+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:05.622506+0000 front 2023-12-16T21:39:05.622528+0000 (oldest deadline 2023-12-16T21:39:30.322546+0000) 2023-12-16T21:40:09.731 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:40:09 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:40:09.447+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:05.622479+0000 front 2023-12-16T21:39:05.622580+0000 (oldest deadline 2023-12-16T21:39:30.322546+0000) 2023-12-16T21:40:10.066 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:40:09 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:40:09.952+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:09.828911+0000 front 2023-12-16T21:39:09.828958+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:40:10.066 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:40:09 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:40:09.952+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:09.828868+0000 front 2023-12-16T21:39:09.828878+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:40:10.066 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:40:09 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:40:09.952+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:09.828947+0000 front 2023-12-16T21:39:09.828993+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:40:10.066 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:40:09 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:40:09.727+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:09.306583+0000 front 2023-12-16T21:39:09.306633+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:40:10.066 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:40:09 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:40:09.727+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:09.306551+0000 front 2023-12-16T21:39:09.306652+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:40:10.067 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:40:09 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:40:09.727+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:09.306605+0000 front 2023-12-16T21:39:09.306613+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:40:10.753 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:40:10 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:40:10.463+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:05.622397+0000 front 2023-12-16T21:39:05.622556+0000 (oldest deadline 2023-12-16T21:39:30.322546+0000) 2023-12-16T21:40:10.753 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:40:10 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:40:10.463+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:05.622506+0000 front 2023-12-16T21:39:05.622528+0000 (oldest deadline 2023-12-16T21:39:30.322546+0000) 2023-12-16T21:40:10.754 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:40:10 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:40:10.463+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:05.622479+0000 front 2023-12-16T21:39:05.622580+0000 (oldest deadline 2023-12-16T21:39:30.322546+0000) 2023-12-16T21:40:11.066 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:40:10 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:40:10.957+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:09.828911+0000 front 2023-12-16T21:39:09.828958+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:40:11.066 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:40:10 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:40:10.957+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:09.828868+0000 front 2023-12-16T21:39:09.828878+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:40:11.066 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:40:10 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:40:10.957+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:09.828947+0000 front 2023-12-16T21:39:09.828993+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:40:11.066 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:40:10 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:40:10.750+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:09.306583+0000 front 2023-12-16T21:39:09.306633+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:40:11.066 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:40:10 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:40:10.750+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:09.306551+0000 front 2023-12-16T21:39:09.306652+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:40:11.067 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:40:10 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:40:10.750+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:09.306605+0000 front 2023-12-16T21:39:09.306613+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:40:11.786 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:40:11 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:40:11.453+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:05.622397+0000 front 2023-12-16T21:39:05.622556+0000 (oldest deadline 2023-12-16T21:39:30.322546+0000) 2023-12-16T21:40:11.787 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:40:11 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:40:11.453+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:05.622506+0000 front 2023-12-16T21:39:05.622528+0000 (oldest deadline 2023-12-16T21:39:30.322546+0000) 2023-12-16T21:40:11.787 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:40:11 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:40:11.453+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:05.622479+0000 front 2023-12-16T21:39:05.622580+0000 (oldest deadline 2023-12-16T21:39:30.322546+0000) 2023-12-16T21:40:12.066 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:40:11 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:40:11.783+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:09.306583+0000 front 2023-12-16T21:39:09.306633+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:40:12.066 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:40:11 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:40:11.783+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:09.306551+0000 front 2023-12-16T21:39:09.306652+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:40:12.066 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:40:11 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:40:11.783+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:09.306605+0000 front 2023-12-16T21:39:09.306613+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:40:12.067 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:40:11 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:40:11.948+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:09.828911+0000 front 2023-12-16T21:39:09.828958+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:40:12.067 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:40:11 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:40:11.948+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:09.828868+0000 front 2023-12-16T21:39:09.828878+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:40:12.067 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:40:11 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:40:11.948+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:09.828947+0000 front 2023-12-16T21:39:09.828993+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:40:12.816 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:40:12 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:40:12.417+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:05.622397+0000 front 2023-12-16T21:39:05.622556+0000 (oldest deadline 2023-12-16T21:39:30.322546+0000) 2023-12-16T21:40:12.817 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:40:12 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:40:12.417+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:05.622506+0000 front 2023-12-16T21:39:05.622528+0000 (oldest deadline 2023-12-16T21:39:30.322546+0000) 2023-12-16T21:40:12.817 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:40:12 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:40:12.417+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:05.622479+0000 front 2023-12-16T21:39:05.622580+0000 (oldest deadline 2023-12-16T21:39:30.322546+0000) 2023-12-16T21:40:13.316 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:40:12 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:40:12.981+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:09.828911+0000 front 2023-12-16T21:39:09.828958+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:40:13.316 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:40:12 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:40:12.981+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:09.828868+0000 front 2023-12-16T21:39:09.828878+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:40:13.316 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:40:12 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:40:12.981+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:09.828947+0000 front 2023-12-16T21:39:09.828993+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:40:13.317 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:40:12 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:40:12.817+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:09.306583+0000 front 2023-12-16T21:39:09.306633+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:40:13.317 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:40:12 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:40:12.817+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:09.306551+0000 front 2023-12-16T21:39:09.306652+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:40:13.317 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:40:12 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:40:12.817+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:09.306605+0000 front 2023-12-16T21:39:09.306613+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:40:13.713 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:40:13 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:40:13.393+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:05.622397+0000 front 2023-12-16T21:39:05.622556+0000 (oldest deadline 2023-12-16T21:39:30.322546+0000) 2023-12-16T21:40:13.713 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:40:13 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:40:13.393+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:05.622506+0000 front 2023-12-16T21:39:05.622528+0000 (oldest deadline 2023-12-16T21:39:30.322546+0000) 2023-12-16T21:40:13.713 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:40:13 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:40:13.393+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:05.622479+0000 front 2023-12-16T21:39:05.622580+0000 (oldest deadline 2023-12-16T21:39:30.322546+0000) 2023-12-16T21:40:13.984 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:40:13 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:40:13.815+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:09.306583+0000 front 2023-12-16T21:39:09.306633+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:40:13.985 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:40:13 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:40:13.815+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:09.306551+0000 front 2023-12-16T21:39:09.306652+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:40:13.985 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:40:13 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:40:13.815+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:09.306605+0000 front 2023-12-16T21:39:09.306613+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:40:14.316 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:40:13 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:40:13.981+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:09.828911+0000 front 2023-12-16T21:39:09.828958+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:40:14.316 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:40:13 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:40:13.981+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:09.828868+0000 front 2023-12-16T21:39:09.828878+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:40:14.316 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:40:13 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:40:13.981+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:09.828947+0000 front 2023-12-16T21:39:09.828993+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:40:14.816 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:40:14 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:40:14.441+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:05.622397+0000 front 2023-12-16T21:39:05.622556+0000 (oldest deadline 2023-12-16T21:39:30.322546+0000) 2023-12-16T21:40:14.816 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:40:14 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:40:14.441+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:05.622506+0000 front 2023-12-16T21:39:05.622528+0000 (oldest deadline 2023-12-16T21:39:30.322546+0000) 2023-12-16T21:40:14.816 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:40:14 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:40:14.441+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:05.622479+0000 front 2023-12-16T21:39:05.622580+0000 (oldest deadline 2023-12-16T21:39:30.322546+0000) 2023-12-16T21:40:15.316 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:40:14 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:40:14.851+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:09.306583+0000 front 2023-12-16T21:39:09.306633+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:40:15.316 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:40:14 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:40:14.851+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:09.306551+0000 front 2023-12-16T21:39:09.306652+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:40:15.317 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:40:14 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:40:14.851+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:09.306605+0000 front 2023-12-16T21:39:09.306613+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:40:15.317 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:40:14 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:40:14.949+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:09.828911+0000 front 2023-12-16T21:39:09.828958+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:40:15.317 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:40:14 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:40:14.949+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:09.828868+0000 front 2023-12-16T21:39:09.828878+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:40:15.317 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:40:14 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:40:14.949+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:09.828947+0000 front 2023-12-16T21:39:09.828993+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:40:15.816 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:40:15 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:40:15.459+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:05.622397+0000 front 2023-12-16T21:39:05.622556+0000 (oldest deadline 2023-12-16T21:39:30.322546+0000) 2023-12-16T21:40:15.817 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:40:15 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:40:15.459+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:05.622506+0000 front 2023-12-16T21:39:05.622528+0000 (oldest deadline 2023-12-16T21:39:30.322546+0000) 2023-12-16T21:40:15.817 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:40:15 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:40:15.459+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:05.622479+0000 front 2023-12-16T21:39:05.622580+0000 (oldest deadline 2023-12-16T21:39:30.322546+0000) 2023-12-16T21:40:16.316 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:40:15 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:40:15.871+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:09.306583+0000 front 2023-12-16T21:39:09.306633+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:40:16.316 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:40:15 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:40:15.871+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:09.306551+0000 front 2023-12-16T21:39:09.306652+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:40:16.316 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:40:15 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:40:15.871+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:09.306605+0000 front 2023-12-16T21:39:09.306613+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:40:16.316 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:40:15 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:40:15.911+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:09.828911+0000 front 2023-12-16T21:39:09.828958+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:40:16.317 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:40:15 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:40:15.911+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:09.828868+0000 front 2023-12-16T21:39:09.828878+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:40:16.317 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:40:15 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:40:15.911+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:09.828947+0000 front 2023-12-16T21:39:09.828993+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:40:16.816 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:40:16 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:40:16.500+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:05.622397+0000 front 2023-12-16T21:39:05.622556+0000 (oldest deadline 2023-12-16T21:39:30.322546+0000) 2023-12-16T21:40:16.816 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:40:16 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:40:16.500+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:05.622506+0000 front 2023-12-16T21:39:05.622528+0000 (oldest deadline 2023-12-16T21:39:30.322546+0000) 2023-12-16T21:40:16.816 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:40:16 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:40:16.500+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:05.622479+0000 front 2023-12-16T21:39:05.622580+0000 (oldest deadline 2023-12-16T21:39:30.322546+0000) 2023-12-16T21:40:17.316 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:40:16 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:40:16.860+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:09.306583+0000 front 2023-12-16T21:39:09.306633+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:40:17.316 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:40:16 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:40:16.860+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:09.306551+0000 front 2023-12-16T21:39:09.306652+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:40:17.316 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:40:16 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:40:16.860+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:09.306605+0000 front 2023-12-16T21:39:09.306613+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:40:17.317 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:40:16 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:40:16.951+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:09.828911+0000 front 2023-12-16T21:39:09.828958+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:40:17.317 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:40:16 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:40:16.951+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:09.828868+0000 front 2023-12-16T21:39:09.828878+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:40:17.317 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:40:16 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:40:16.951+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:09.828947+0000 front 2023-12-16T21:39:09.828993+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:40:17.815 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:40:17 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:40:17.547+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:05.622397+0000 front 2023-12-16T21:39:05.622556+0000 (oldest deadline 2023-12-16T21:39:30.322546+0000) 2023-12-16T21:40:17.815 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:40:17 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:40:17.547+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:05.622506+0000 front 2023-12-16T21:39:05.622528+0000 (oldest deadline 2023-12-16T21:39:30.322546+0000) 2023-12-16T21:40:17.815 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:40:17 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:40:17.547+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:05.622479+0000 front 2023-12-16T21:39:05.622580+0000 (oldest deadline 2023-12-16T21:39:30.322546+0000) 2023-12-16T21:40:18.066 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:40:17 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:40:17.812+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:09.306583+0000 front 2023-12-16T21:39:09.306633+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:40:18.066 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:40:17 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:40:17.812+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:09.306551+0000 front 2023-12-16T21:39:09.306652+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:40:18.067 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:40:17 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:40:17.812+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:09.306605+0000 front 2023-12-16T21:39:09.306613+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:40:18.067 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:40:17 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:40:17.941+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:09.828911+0000 front 2023-12-16T21:39:09.828958+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:40:18.067 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:40:17 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:40:17.941+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:09.828868+0000 front 2023-12-16T21:39:09.828878+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:40:18.067 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:40:17 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:40:17.941+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:09.828947+0000 front 2023-12-16T21:39:09.828993+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:40:18.816 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:40:18 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:40:18.792+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:09.306583+0000 front 2023-12-16T21:39:09.306633+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:40:18.816 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:40:18 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:40:18.793+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:09.306551+0000 front 2023-12-16T21:39:09.306652+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:40:18.817 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:40:18 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:40:18.793+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:09.306605+0000 front 2023-12-16T21:39:09.306613+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:40:18.817 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:40:18 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:40:18.546+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:05.622397+0000 front 2023-12-16T21:39:05.622556+0000 (oldest deadline 2023-12-16T21:39:30.322546+0000) 2023-12-16T21:40:18.817 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:40:18 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:40:18.546+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:05.622506+0000 front 2023-12-16T21:39:05.622528+0000 (oldest deadline 2023-12-16T21:39:30.322546+0000) 2023-12-16T21:40:18.817 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:40:18 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:40:18.546+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:05.622479+0000 front 2023-12-16T21:39:05.622580+0000 (oldest deadline 2023-12-16T21:39:30.322546+0000) 2023-12-16T21:40:19.316 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:40:18 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:40:18.961+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:09.828911+0000 front 2023-12-16T21:39:09.828958+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:40:19.316 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:40:18 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:40:18.961+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:09.828868+0000 front 2023-12-16T21:39:09.828878+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:40:19.316 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:40:18 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:40:18.962+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:09.828947+0000 front 2023-12-16T21:39:09.828993+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:40:20.013 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:40:19 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:40:19.582+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:05.622397+0000 front 2023-12-16T21:39:05.622556+0000 (oldest deadline 2023-12-16T21:39:30.322546+0000) 2023-12-16T21:40:20.013 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:40:19 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:40:19.582+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:05.622506+0000 front 2023-12-16T21:39:05.622528+0000 (oldest deadline 2023-12-16T21:39:30.322546+0000) 2023-12-16T21:40:20.013 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:40:19 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:40:19.582+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:05.622479+0000 front 2023-12-16T21:39:05.622580+0000 (oldest deadline 2023-12-16T21:39:30.322546+0000) 2023-12-16T21:40:20.013 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:40:19 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:40:19.756+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:09.306583+0000 front 2023-12-16T21:39:09.306633+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:40:20.014 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:40:19 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:40:19.756+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:09.306551+0000 front 2023-12-16T21:39:09.306652+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:40:20.014 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:40:19 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:40:19.756+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:09.306605+0000 front 2023-12-16T21:39:09.306613+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:40:20.316 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:40:20 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:40:20.010+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:09.828911+0000 front 2023-12-16T21:39:09.828958+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:40:20.316 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:40:20 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:40:20.010+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:09.828868+0000 front 2023-12-16T21:39:09.828878+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:40:20.316 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:40:20 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:40:20.010+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:09.828947+0000 front 2023-12-16T21:39:09.828993+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:40:20.816 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:40:20 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:40:20.729+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:09.306583+0000 front 2023-12-16T21:39:09.306633+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:40:20.816 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:40:20 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:40:20.729+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:09.306551+0000 front 2023-12-16T21:39:09.306652+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:40:20.816 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:40:20 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:40:20.729+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:09.306605+0000 front 2023-12-16T21:39:09.306613+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:40:20.817 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:40:20 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:40:20.544+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:05.622397+0000 front 2023-12-16T21:39:05.622556+0000 (oldest deadline 2023-12-16T21:39:30.322546+0000) 2023-12-16T21:40:20.817 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:40:20 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:40:20.544+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:05.622506+0000 front 2023-12-16T21:39:05.622528+0000 (oldest deadline 2023-12-16T21:39:30.322546+0000) 2023-12-16T21:40:20.817 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:40:20 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:40:20.544+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:05.622479+0000 front 2023-12-16T21:39:05.622580+0000 (oldest deadline 2023-12-16T21:39:30.322546+0000) 2023-12-16T21:40:21.316 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:40:20 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:40:20.998+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:09.828911+0000 front 2023-12-16T21:39:09.828958+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:40:21.316 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:40:20 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:40:20.998+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:09.828868+0000 front 2023-12-16T21:39:09.828878+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:40:21.316 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:40:20 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:40:20.998+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:09.828947+0000 front 2023-12-16T21:39:09.828993+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:40:21.753 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:40:21 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:40:21.565+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:05.622397+0000 front 2023-12-16T21:39:05.622556+0000 (oldest deadline 2023-12-16T21:39:30.322546+0000) 2023-12-16T21:40:21.754 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:40:21 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:40:21.565+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:05.622506+0000 front 2023-12-16T21:39:05.622528+0000 (oldest deadline 2023-12-16T21:39:30.322546+0000) 2023-12-16T21:40:21.754 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:40:21 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:40:21.565+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:05.622479+0000 front 2023-12-16T21:39:05.622580+0000 (oldest deadline 2023-12-16T21:39:30.322546+0000) 2023-12-16T21:40:22.004 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:40:21 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:40:21.751+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:09.306583+0000 front 2023-12-16T21:39:09.306633+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:40:22.004 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:40:21 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:40:21.751+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:09.306551+0000 front 2023-12-16T21:39:09.306652+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:40:22.004 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:40:21 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:40:21.751+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:09.306605+0000 front 2023-12-16T21:39:09.306613+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:40:22.316 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:40:22 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:40:22.001+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:09.828911+0000 front 2023-12-16T21:39:09.828958+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:40:22.316 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:40:22 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:40:22.001+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:09.828868+0000 front 2023-12-16T21:39:09.828878+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:40:22.316 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:40:22 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:40:22.001+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:09.828947+0000 front 2023-12-16T21:39:09.828993+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:40:22.928 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:40:22 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:40:22.721+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:09.306583+0000 front 2023-12-16T21:39:09.306633+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:40:22.929 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:40:22 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:40:22.721+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:09.306551+0000 front 2023-12-16T21:39:09.306652+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:40:22.929 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:40:22 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:40:22.721+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:09.306605+0000 front 2023-12-16T21:39:09.306613+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:40:22.929 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:40:22 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:40:22.613+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:05.622397+0000 front 2023-12-16T21:39:05.622556+0000 (oldest deadline 2023-12-16T21:39:30.322546+0000) 2023-12-16T21:40:22.929 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:40:22 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:40:22.613+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:05.622506+0000 front 2023-12-16T21:39:05.622528+0000 (oldest deadline 2023-12-16T21:39:30.322546+0000) 2023-12-16T21:40:22.929 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:40:22 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:40:22.613+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:05.622479+0000 front 2023-12-16T21:39:05.622580+0000 (oldest deadline 2023-12-16T21:39:30.322546+0000) 2023-12-16T21:40:23.316 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:40:23 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:40:23.028+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:09.828911+0000 front 2023-12-16T21:39:09.828958+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:40:23.316 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:40:23 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:40:23.028+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:09.828868+0000 front 2023-12-16T21:39:09.828878+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:40:23.316 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:40:23 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:40:23.028+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:09.828947+0000 front 2023-12-16T21:39:09.828993+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:40:23.816 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:40:23 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:40:23.711+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:09.306583+0000 front 2023-12-16T21:39:09.306633+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:40:23.816 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:40:23 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:40:23.711+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:09.306551+0000 front 2023-12-16T21:39:09.306652+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:40:23.816 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:40:23 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:40:23.711+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:09.306605+0000 front 2023-12-16T21:39:09.306613+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:40:23.816 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:40:23 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:40:23.564+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:05.622397+0000 front 2023-12-16T21:39:05.622556+0000 (oldest deadline 2023-12-16T21:39:30.322546+0000) 2023-12-16T21:40:23.816 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:40:23 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:40:23.564+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:05.622506+0000 front 2023-12-16T21:39:05.622528+0000 (oldest deadline 2023-12-16T21:39:30.322546+0000) 2023-12-16T21:40:23.817 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:40:23 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:40:23.564+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:05.622479+0000 front 2023-12-16T21:39:05.622580+0000 (oldest deadline 2023-12-16T21:39:30.322546+0000) 2023-12-16T21:40:24.553 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:40:24 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:40:24.073+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:09.828911+0000 front 2023-12-16T21:39:09.828958+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:40:24.553 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:40:24 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:40:24.073+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:09.828868+0000 front 2023-12-16T21:39:09.828878+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:40:24.553 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:40:24 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:40:24.073+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:09.828947+0000 front 2023-12-16T21:39:09.828993+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:40:24.816 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:40:24 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:40:24.550+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:05.622397+0000 front 2023-12-16T21:39:05.622556+0000 (oldest deadline 2023-12-16T21:39:30.322546+0000) 2023-12-16T21:40:24.816 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:40:24 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:40:24.550+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:05.622506+0000 front 2023-12-16T21:39:05.622528+0000 (oldest deadline 2023-12-16T21:39:30.322546+0000) 2023-12-16T21:40:24.816 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:40:24 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:40:24.550+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:05.622479+0000 front 2023-12-16T21:39:05.622580+0000 (oldest deadline 2023-12-16T21:39:30.322546+0000) 2023-12-16T21:40:24.816 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:40:24 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:40:24.740+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:09.306583+0000 front 2023-12-16T21:39:09.306633+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:40:24.817 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:40:24 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:40:24.740+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:09.306551+0000 front 2023-12-16T21:39:09.306652+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:40:24.817 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:40:24 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:40:24.740+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:09.306605+0000 front 2023-12-16T21:39:09.306613+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:40:25.316 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:40:25 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:40:25.050+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:09.828911+0000 front 2023-12-16T21:39:09.828958+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:40:25.316 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:40:25 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:40:25.050+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:09.828868+0000 front 2023-12-16T21:39:09.828878+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:40:25.316 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:40:25 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:40:25.050+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:09.828947+0000 front 2023-12-16T21:39:09.828993+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:40:26.001 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:40:25 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:40:25.782+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:09.306583+0000 front 2023-12-16T21:39:09.306633+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:40:26.001 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:40:25 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:40:25.782+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:09.306551+0000 front 2023-12-16T21:39:09.306652+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:40:26.001 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:40:25 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:40:25.782+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:09.306605+0000 front 2023-12-16T21:39:09.306613+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:40:26.001 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:40:25 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:40:25.597+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:05.622397+0000 front 2023-12-16T21:39:05.622556+0000 (oldest deadline 2023-12-16T21:39:30.322546+0000) 2023-12-16T21:40:26.001 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:40:25 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:40:25.597+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:05.622506+0000 front 2023-12-16T21:39:05.622528+0000 (oldest deadline 2023-12-16T21:39:30.322546+0000) 2023-12-16T21:40:26.002 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:40:25 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:40:25.597+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:05.622479+0000 front 2023-12-16T21:39:05.622580+0000 (oldest deadline 2023-12-16T21:39:30.322546+0000) 2023-12-16T21:40:26.316 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:40:26 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:40:26.039+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:09.828911+0000 front 2023-12-16T21:39:09.828958+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:40:26.316 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:40:26 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:40:26.039+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:09.828868+0000 front 2023-12-16T21:39:09.828878+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:40:26.316 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:40:26 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:40:26.039+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:09.828947+0000 front 2023-12-16T21:39:09.828993+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:40:27.058 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:40:26 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:40:26.639+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:05.622397+0000 front 2023-12-16T21:39:05.622556+0000 (oldest deadline 2023-12-16T21:39:30.322546+0000) 2023-12-16T21:40:27.058 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:40:26 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:40:26.639+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:05.622506+0000 front 2023-12-16T21:39:05.622528+0000 (oldest deadline 2023-12-16T21:39:30.322546+0000) 2023-12-16T21:40:27.059 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:40:26 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:40:26.639+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:05.622479+0000 front 2023-12-16T21:39:05.622580+0000 (oldest deadline 2023-12-16T21:39:30.322546+0000) 2023-12-16T21:40:27.059 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:40:26 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:40:26.738+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:09.306583+0000 front 2023-12-16T21:39:09.306633+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:40:27.059 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:40:26 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:40:26.738+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:09.306551+0000 front 2023-12-16T21:39:09.306652+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:40:27.059 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:40:26 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:40:26.738+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:09.306605+0000 front 2023-12-16T21:39:09.306613+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:40:27.316 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:40:27 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:40:27.055+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:09.828911+0000 front 2023-12-16T21:39:09.828958+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:40:27.316 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:40:27 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:40:27.055+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:09.828868+0000 front 2023-12-16T21:39:09.828878+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:40:27.317 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:40:27 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:40:27.055+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:09.828947+0000 front 2023-12-16T21:39:09.828993+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:40:28.061 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:40:27 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:40:27.757+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:09.306583+0000 front 2023-12-16T21:39:09.306633+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:40:28.061 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:40:27 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:40:27.757+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:09.306551+0000 front 2023-12-16T21:39:09.306652+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:40:28.061 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:40:27 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:40:27.757+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:09.306605+0000 front 2023-12-16T21:39:09.306613+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:40:28.062 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:40:27 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:40:27.647+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:05.622397+0000 front 2023-12-16T21:39:05.622556+0000 (oldest deadline 2023-12-16T21:39:30.322546+0000) 2023-12-16T21:40:28.062 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:40:27 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:40:27.647+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:05.622506+0000 front 2023-12-16T21:39:05.622528+0000 (oldest deadline 2023-12-16T21:39:30.322546+0000) 2023-12-16T21:40:28.062 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:40:27 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:40:27.647+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:05.622479+0000 front 2023-12-16T21:39:05.622580+0000 (oldest deadline 2023-12-16T21:39:30.322546+0000) 2023-12-16T21:40:28.316 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:40:28 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:40:28.058+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:09.828911+0000 front 2023-12-16T21:39:09.828958+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:40:28.316 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:40:28 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:40:28.058+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:09.828868+0000 front 2023-12-16T21:39:09.828878+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:40:28.316 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:40:28 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:40:28.058+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:09.828947+0000 front 2023-12-16T21:39:09.828993+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:40:29.066 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:40:28 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:40:28.781+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:09.306583+0000 front 2023-12-16T21:39:09.306633+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:40:29.066 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:40:28 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:40:28.781+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:09.306551+0000 front 2023-12-16T21:39:09.306652+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:40:29.066 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:40:28 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:40:28.781+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:09.306605+0000 front 2023-12-16T21:39:09.306613+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:40:29.066 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:40:28 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:40:28.648+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:05.622397+0000 front 2023-12-16T21:39:05.622556+0000 (oldest deadline 2023-12-16T21:39:30.322546+0000) 2023-12-16T21:40:29.066 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:40:28 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:40:28.648+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:05.622506+0000 front 2023-12-16T21:39:05.622528+0000 (oldest deadline 2023-12-16T21:39:30.322546+0000) 2023-12-16T21:40:29.066 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:40:28 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:40:28.648+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:05.622479+0000 front 2023-12-16T21:39:05.622580+0000 (oldest deadline 2023-12-16T21:39:30.322546+0000) 2023-12-16T21:40:29.316 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:40:29 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:40:29.063+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:09.828911+0000 front 2023-12-16T21:39:09.828958+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:40:29.316 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:40:29 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:40:29.063+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:09.828868+0000 front 2023-12-16T21:39:09.828878+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:40:29.316 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:40:29 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:40:29.063+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:09.828947+0000 front 2023-12-16T21:39:09.828993+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:40:30.066 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:40:29 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:40:29.762+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:09.306583+0000 front 2023-12-16T21:39:09.306633+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:40:30.066 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:40:29 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:40:29.762+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:09.306551+0000 front 2023-12-16T21:39:09.306652+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:40:30.066 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:40:29 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:40:29.762+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:09.306605+0000 front 2023-12-16T21:39:09.306613+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:40:30.067 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:40:29 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:40:29.654+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:05.622397+0000 front 2023-12-16T21:39:05.622556+0000 (oldest deadline 2023-12-16T21:39:30.322546+0000) 2023-12-16T21:40:30.067 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:40:29 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:40:29.654+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:05.622506+0000 front 2023-12-16T21:39:05.622528+0000 (oldest deadline 2023-12-16T21:39:30.322546+0000) 2023-12-16T21:40:30.067 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:40:29 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:40:29.654+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:05.622479+0000 front 2023-12-16T21:39:05.622580+0000 (oldest deadline 2023-12-16T21:39:30.322546+0000) 2023-12-16T21:40:30.566 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:40:30 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:40:30.069+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:09.828911+0000 front 2023-12-16T21:39:09.828958+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:40:30.566 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:40:30 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:40:30.069+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:09.828868+0000 front 2023-12-16T21:39:09.828878+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:40:30.566 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:40:30 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:40:30.069+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:09.828947+0000 front 2023-12-16T21:39:09.828993+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:40:31.026 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:40:30 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:40:30.806+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:09.306583+0000 front 2023-12-16T21:39:09.306633+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:40:31.026 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:40:30 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:40:30.806+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:09.306551+0000 front 2023-12-16T21:39:09.306652+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:40:31.026 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:40:30 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:40:30.806+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:09.306605+0000 front 2023-12-16T21:39:09.306613+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:40:31.026 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:40:30 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:40:30.607+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:05.622397+0000 front 2023-12-16T21:39:05.622556+0000 (oldest deadline 2023-12-16T21:39:30.322546+0000) 2023-12-16T21:40:31.026 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:40:30 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:40:30.607+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:05.622506+0000 front 2023-12-16T21:39:05.622528+0000 (oldest deadline 2023-12-16T21:39:30.322546+0000) 2023-12-16T21:40:31.027 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:40:30 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:40:30.607+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:05.622479+0000 front 2023-12-16T21:39:05.622580+0000 (oldest deadline 2023-12-16T21:39:30.322546+0000) 2023-12-16T21:40:31.316 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:40:31 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:40:31.023+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:09.828911+0000 front 2023-12-16T21:39:09.828958+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:40:31.316 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:40:31 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:40:31.023+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:09.828868+0000 front 2023-12-16T21:39:09.828878+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:40:31.316 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:40:31 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:40:31.023+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:09.828947+0000 front 2023-12-16T21:39:09.828993+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:40:31.627 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:40:31 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:40:31.569+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:05.622397+0000 front 2023-12-16T21:39:05.622556+0000 (oldest deadline 2023-12-16T21:39:30.322546+0000) 2023-12-16T21:40:31.627 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:40:31 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:40:31.569+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:05.622506+0000 front 2023-12-16T21:39:05.622528+0000 (oldest deadline 2023-12-16T21:39:30.322546+0000) 2023-12-16T21:40:31.627 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:40:31 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:40:31.569+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:05.622479+0000 front 2023-12-16T21:39:05.622580+0000 (oldest deadline 2023-12-16T21:39:30.322546+0000) 2023-12-16T21:40:32.058 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:40:31 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:40:31.811+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:09.306583+0000 front 2023-12-16T21:39:09.306633+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:40:32.058 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:40:31 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:40:31.811+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:09.306551+0000 front 2023-12-16T21:39:09.306652+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:40:32.058 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:40:31 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:40:31.811+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:09.306605+0000 front 2023-12-16T21:39:09.306613+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:40:32.316 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:40:32 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:40:32.055+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:09.828911+0000 front 2023-12-16T21:39:09.828958+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:40:32.316 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:40:32 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:40:32.055+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:09.828868+0000 front 2023-12-16T21:39:09.828878+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:40:32.316 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:40:32 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:40:32.055+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:09.828947+0000 front 2023-12-16T21:39:09.828993+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:40:32.816 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:40:32 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:40:32.543+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:05.622397+0000 front 2023-12-16T21:39:05.622556+0000 (oldest deadline 2023-12-16T21:39:30.322546+0000) 2023-12-16T21:40:32.816 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:40:32 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:40:32.543+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:05.622506+0000 front 2023-12-16T21:39:05.622528+0000 (oldest deadline 2023-12-16T21:39:30.322546+0000) 2023-12-16T21:40:32.816 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:40:32 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:40:32.543+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:05.622479+0000 front 2023-12-16T21:39:05.622580+0000 (oldest deadline 2023-12-16T21:39:30.322546+0000) 2023-12-16T21:40:33.316 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:40:33 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:40:33.066+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:09.828911+0000 front 2023-12-16T21:39:09.828958+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:40:33.316 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:40:33 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:40:33.066+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:09.828868+0000 front 2023-12-16T21:39:09.828878+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:40:33.316 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:40:33 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:40:33.066+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:09.828947+0000 front 2023-12-16T21:39:09.828993+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:40:33.317 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:40:32 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:40:32.837+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:09.306583+0000 front 2023-12-16T21:39:09.306633+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:40:33.317 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:40:32 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:40:32.837+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:09.306551+0000 front 2023-12-16T21:39:09.306652+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:40:33.317 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:40:32 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:40:32.837+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:09.306605+0000 front 2023-12-16T21:39:09.306613+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:40:33.816 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:40:33 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:40:33.518+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:05.622397+0000 front 2023-12-16T21:39:05.622556+0000 (oldest deadline 2023-12-16T21:39:30.322546+0000) 2023-12-16T21:40:33.816 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:40:33 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:40:33.518+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:05.622506+0000 front 2023-12-16T21:39:05.622528+0000 (oldest deadline 2023-12-16T21:39:30.322546+0000) 2023-12-16T21:40:33.817 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:40:33 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:40:33.518+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:05.622479+0000 front 2023-12-16T21:39:05.622580+0000 (oldest deadline 2023-12-16T21:39:30.322546+0000) 2023-12-16T21:40:34.316 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:40:34 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:40:34.088+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:09.828911+0000 front 2023-12-16T21:39:09.828958+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:40:34.317 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:40:34 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:40:34.088+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:09.828868+0000 front 2023-12-16T21:39:09.828878+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:40:34.317 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:40:34 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:40:34.088+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:09.828947+0000 front 2023-12-16T21:39:09.828993+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:40:34.317 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:40:33 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:40:33.852+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:09.306583+0000 front 2023-12-16T21:39:09.306633+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:40:34.317 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:40:33 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:40:33.852+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:09.306551+0000 front 2023-12-16T21:39:09.306652+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:40:34.317 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:40:33 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:40:33.852+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:09.306605+0000 front 2023-12-16T21:39:09.306613+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:40:34.816 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:40:34 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:40:34.563+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:05.622397+0000 front 2023-12-16T21:39:05.622556+0000 (oldest deadline 2023-12-16T21:39:30.322546+0000) 2023-12-16T21:40:34.816 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:40:34 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:40:34.563+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:05.622506+0000 front 2023-12-16T21:39:05.622528+0000 (oldest deadline 2023-12-16T21:39:30.322546+0000) 2023-12-16T21:40:34.817 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:40:34 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:40:34.563+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:05.622479+0000 front 2023-12-16T21:39:05.622580+0000 (oldest deadline 2023-12-16T21:39:30.322546+0000) 2023-12-16T21:40:35.217 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:40:35 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:40:35.045+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:09.828911+0000 front 2023-12-16T21:39:09.828958+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:40:35.217 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:40:35 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:40:35.045+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:09.828868+0000 front 2023-12-16T21:39:09.828878+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:40:35.217 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:40:35 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:40:35.045+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:09.828947+0000 front 2023-12-16T21:39:09.828993+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:40:35.218 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:40:34 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:40:34.878+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:09.306583+0000 front 2023-12-16T21:39:09.306633+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:40:35.218 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:40:34 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:40:34.878+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:09.306551+0000 front 2023-12-16T21:39:09.306652+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:40:35.218 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:40:34 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:40:34.878+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:09.306605+0000 front 2023-12-16T21:39:09.306613+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:40:35.902 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:40:35 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:40:35.603+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:05.622397+0000 front 2023-12-16T21:39:05.622556+0000 (oldest deadline 2023-12-16T21:39:30.322546+0000) 2023-12-16T21:40:35.902 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:40:35 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:40:35.603+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:05.622506+0000 front 2023-12-16T21:39:05.622528+0000 (oldest deadline 2023-12-16T21:39:30.322546+0000) 2023-12-16T21:40:35.902 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:40:35 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:40:35.603+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:05.622479+0000 front 2023-12-16T21:39:05.622580+0000 (oldest deadline 2023-12-16T21:39:30.322546+0000) 2023-12-16T21:40:36.316 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:40:36 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:40:36.017+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:09.828911+0000 front 2023-12-16T21:39:09.828958+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:40:36.316 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:40:36 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:40:36.017+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:09.828868+0000 front 2023-12-16T21:39:09.828878+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:40:36.317 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:40:36 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:40:36.017+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:09.828947+0000 front 2023-12-16T21:39:09.828993+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:40:36.317 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:40:35 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:40:35.899+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:09.306583+0000 front 2023-12-16T21:39:09.306633+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:40:36.317 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:40:35 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:40:35.899+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:09.306551+0000 front 2023-12-16T21:39:09.306652+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:40:36.317 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:40:35 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:40:35.899+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:09.306605+0000 front 2023-12-16T21:39:09.306613+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:40:36.887 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:40:36 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:40:36.607+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:05.622397+0000 front 2023-12-16T21:39:05.622556+0000 (oldest deadline 2023-12-16T21:39:30.322546+0000) 2023-12-16T21:40:36.888 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:40:36 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:40:36.607+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:05.622506+0000 front 2023-12-16T21:39:05.622528+0000 (oldest deadline 2023-12-16T21:39:30.322546+0000) 2023-12-16T21:40:36.888 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:40:36 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:40:36.607+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:05.622479+0000 front 2023-12-16T21:39:05.622580+0000 (oldest deadline 2023-12-16T21:39:30.322546+0000) 2023-12-16T21:40:37.316 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:40:36 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:40:36.985+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:09.828911+0000 front 2023-12-16T21:39:09.828958+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:40:37.316 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:40:36 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:40:36.985+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:09.828868+0000 front 2023-12-16T21:39:09.828878+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:40:37.316 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:40:36 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:40:36.985+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:09.828947+0000 front 2023-12-16T21:39:09.828993+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:40:37.316 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:40:36 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:40:36.885+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:09.306583+0000 front 2023-12-16T21:39:09.306633+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:40:37.316 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:40:36 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:40:36.885+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:09.306551+0000 front 2023-12-16T21:39:09.306652+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:40:37.317 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:40:36 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:40:36.885+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:09.306605+0000 front 2023-12-16T21:39:09.306613+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:40:37.998 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:40:37 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:40:37.609+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:05.622397+0000 front 2023-12-16T21:39:05.622556+0000 (oldest deadline 2023-12-16T21:39:30.322546+0000) 2023-12-16T21:40:37.999 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:40:37 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:40:37.609+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:05.622506+0000 front 2023-12-16T21:39:05.622528+0000 (oldest deadline 2023-12-16T21:39:30.322546+0000) 2023-12-16T21:40:37.999 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:40:37 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:40:37.609+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:05.622479+0000 front 2023-12-16T21:39:05.622580+0000 (oldest deadline 2023-12-16T21:39:30.322546+0000) 2023-12-16T21:40:37.999 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:40:37 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:40:37.855+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:09.306583+0000 front 2023-12-16T21:39:09.306633+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:40:37.999 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:40:37 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:40:37.855+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:09.306551+0000 front 2023-12-16T21:39:09.306652+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:40:37.999 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:40:37 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:40:37.855+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:09.306605+0000 front 2023-12-16T21:39:09.306613+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:40:38.316 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:40:37 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:40:37.995+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:09.828911+0000 front 2023-12-16T21:39:09.828958+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:40:38.316 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:40:37 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:40:37.995+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:09.828868+0000 front 2023-12-16T21:39:09.828878+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:40:38.316 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:40:37 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:40:37.995+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:09.828947+0000 front 2023-12-16T21:39:09.828993+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:40:39.044 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:40:38 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:40:38.855+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:09.306583+0000 front 2023-12-16T21:39:09.306633+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:40:39.044 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:40:38 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:40:38.855+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:09.306551+0000 front 2023-12-16T21:39:09.306652+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:40:39.045 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:40:38 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:40:38.855+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:09.306605+0000 front 2023-12-16T21:39:09.306613+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:40:39.045 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:40:38 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:40:38.634+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:05.622397+0000 front 2023-12-16T21:39:05.622556+0000 (oldest deadline 2023-12-16T21:39:30.322546+0000) 2023-12-16T21:40:39.045 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:40:38 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:40:38.634+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:05.622506+0000 front 2023-12-16T21:39:05.622528+0000 (oldest deadline 2023-12-16T21:39:30.322546+0000) 2023-12-16T21:40:39.045 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:40:38 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:40:38.634+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:05.622479+0000 front 2023-12-16T21:39:05.622580+0000 (oldest deadline 2023-12-16T21:39:30.322546+0000) 2023-12-16T21:40:39.316 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:40:39 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:40:39.041+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:09.828911+0000 front 2023-12-16T21:39:09.828958+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:40:39.316 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:40:39 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:40:39.041+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:09.828868+0000 front 2023-12-16T21:39:09.828878+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:40:39.316 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:40:39 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:40:39.041+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:09.828947+0000 front 2023-12-16T21:39:09.828993+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:40:40.039 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:40:39 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:40:39.881+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:09.306583+0000 front 2023-12-16T21:39:09.306633+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:40:40.096 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:40:39 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:40:39.881+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:09.306551+0000 front 2023-12-16T21:39:09.306652+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:40:40.096 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:40:39 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:40:39.881+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:09.306605+0000 front 2023-12-16T21:39:09.306613+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:40:40.097 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:40:39 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:40:39.666+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:05.622397+0000 front 2023-12-16T21:39:05.622556+0000 (oldest deadline 2023-12-16T21:39:30.322546+0000) 2023-12-16T21:40:40.097 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:40:39 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:40:39.666+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:05.622506+0000 front 2023-12-16T21:39:05.622528+0000 (oldest deadline 2023-12-16T21:39:30.322546+0000) 2023-12-16T21:40:40.097 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:40:39 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:40:39.666+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:05.622479+0000 front 2023-12-16T21:39:05.622580+0000 (oldest deadline 2023-12-16T21:39:30.322546+0000) 2023-12-16T21:40:40.316 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:40:40 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:40:40.035+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:09.828911+0000 front 2023-12-16T21:39:09.828958+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:40:40.316 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:40:40 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:40:40.035+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:09.828868+0000 front 2023-12-16T21:39:09.828878+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:40:40.316 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:40:40 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:40:40.035+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:09.828947+0000 front 2023-12-16T21:39:09.828993+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:40:40.997 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:40:40 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:40:40.659+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:05.622397+0000 front 2023-12-16T21:39:05.622556+0000 (oldest deadline 2023-12-16T21:39:30.322546+0000) 2023-12-16T21:40:40.997 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:40:40 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:40:40.659+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:05.622506+0000 front 2023-12-16T21:39:05.622528+0000 (oldest deadline 2023-12-16T21:39:30.322546+0000) 2023-12-16T21:40:40.997 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:40:40 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:40:40.659+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:05.622479+0000 front 2023-12-16T21:39:05.622580+0000 (oldest deadline 2023-12-16T21:39:30.322546+0000) 2023-12-16T21:40:40.997 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:40:40 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:40:40.895+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:09.306583+0000 front 2023-12-16T21:39:09.306633+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:40:40.998 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:40:40 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:40:40.895+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:09.306551+0000 front 2023-12-16T21:39:09.306652+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:40:40.998 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:40:40 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:40:40.895+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:09.306605+0000 front 2023-12-16T21:39:09.306613+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:40:41.316 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:40:40 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:40:40.994+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:09.828911+0000 front 2023-12-16T21:39:09.828958+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:40:41.316 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:40:40 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:40:40.994+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:09.828868+0000 front 2023-12-16T21:39:09.828878+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:40:41.316 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:40:40 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:40:40.994+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:09.828947+0000 front 2023-12-16T21:39:09.828993+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:40:41.867 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:40:41 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:40:41.612+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:05.622397+0000 front 2023-12-16T21:39:05.622556+0000 (oldest deadline 2023-12-16T21:39:30.322546+0000) 2023-12-16T21:40:41.867 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:40:41 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:40:41.612+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:05.622506+0000 front 2023-12-16T21:39:05.622528+0000 (oldest deadline 2023-12-16T21:39:30.322546+0000) 2023-12-16T21:40:41.867 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:40:41 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:40:41.612+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:05.622479+0000 front 2023-12-16T21:39:05.622580+0000 (oldest deadline 2023-12-16T21:39:30.322546+0000) 2023-12-16T21:40:42.316 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:40:41 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:40:41.864+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:09.306583+0000 front 2023-12-16T21:39:09.306633+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:40:42.316 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:40:41 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:40:41.864+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:09.306551+0000 front 2023-12-16T21:39:09.306652+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:40:42.316 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:40:41 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:40:41.864+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:09.306605+0000 front 2023-12-16T21:39:09.306613+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:40:42.316 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:40:41 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:40:41.949+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:09.828911+0000 front 2023-12-16T21:39:09.828958+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:40:42.316 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:40:41 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:40:41.949+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:09.828868+0000 front 2023-12-16T21:39:09.828878+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:40:42.317 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:40:41 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:40:41.949+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:09.828947+0000 front 2023-12-16T21:39:09.828993+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:40:42.868 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:40:42 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:40:42.615+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:05.622397+0000 front 2023-12-16T21:39:05.622556+0000 (oldest deadline 2023-12-16T21:39:30.322546+0000) 2023-12-16T21:40:42.868 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:40:42 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:40:42.615+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:05.622506+0000 front 2023-12-16T21:39:05.622528+0000 (oldest deadline 2023-12-16T21:39:30.322546+0000) 2023-12-16T21:40:42.868 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:40:42 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:40:42.615+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:05.622479+0000 front 2023-12-16T21:39:05.622580+0000 (oldest deadline 2023-12-16T21:39:30.322546+0000) 2023-12-16T21:40:43.316 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:40:42 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:40:42.865+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:09.306583+0000 front 2023-12-16T21:39:09.306633+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:40:43.316 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:40:42 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:40:42.865+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:09.306551+0000 front 2023-12-16T21:39:09.306652+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:40:43.316 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:40:42 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:40:42.865+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:09.306605+0000 front 2023-12-16T21:39:09.306613+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:40:43.316 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:40:42 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:40:42.928+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:09.828911+0000 front 2023-12-16T21:39:09.828958+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:40:43.317 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:40:42 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:40:42.928+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:09.828868+0000 front 2023-12-16T21:39:09.828878+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:40:43.317 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:40:42 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:40:42.928+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:09.828947+0000 front 2023-12-16T21:39:09.828993+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:40:43.816 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:40:43 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:40:43.613+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:05.622397+0000 front 2023-12-16T21:39:05.622556+0000 (oldest deadline 2023-12-16T21:39:30.322546+0000) 2023-12-16T21:40:43.816 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:40:43 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:40:43.613+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:05.622506+0000 front 2023-12-16T21:39:05.622528+0000 (oldest deadline 2023-12-16T21:39:30.322546+0000) 2023-12-16T21:40:43.816 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:40:43 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:40:43.613+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:05.622479+0000 front 2023-12-16T21:39:05.622580+0000 (oldest deadline 2023-12-16T21:39:30.322546+0000) 2023-12-16T21:40:44.316 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:40:43 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:40:43.928+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:09.828911+0000 front 2023-12-16T21:39:09.828958+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:40:44.316 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:40:43 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:40:43.928+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:09.828868+0000 front 2023-12-16T21:39:09.828878+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:40:44.316 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:40:43 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:40:43.928+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:09.828947+0000 front 2023-12-16T21:39:09.828993+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:40:44.317 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:40:43 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:40:43.817+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:09.306583+0000 front 2023-12-16T21:39:09.306633+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:40:44.317 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:40:43 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:40:43.817+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:09.306551+0000 front 2023-12-16T21:39:09.306652+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:40:44.317 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:40:43 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:40:43.817+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:09.306605+0000 front 2023-12-16T21:39:09.306613+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:40:44.816 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:40:44 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:40:44.599+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:05.622397+0000 front 2023-12-16T21:39:05.622556+0000 (oldest deadline 2023-12-16T21:39:30.322546+0000) 2023-12-16T21:40:44.816 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:40:44 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:40:44.599+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:05.622506+0000 front 2023-12-16T21:39:05.622528+0000 (oldest deadline 2023-12-16T21:39:30.322546+0000) 2023-12-16T21:40:44.816 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:40:44 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:40:44.599+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:05.622479+0000 front 2023-12-16T21:39:05.622580+0000 (oldest deadline 2023-12-16T21:39:30.322546+0000) 2023-12-16T21:40:45.316 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:40:44 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:40:44.819+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:09.306583+0000 front 2023-12-16T21:39:09.306633+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:40:45.316 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:40:44 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:40:44.819+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:09.306551+0000 front 2023-12-16T21:39:09.306652+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:40:45.317 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:40:44 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:40:44.819+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:09.306605+0000 front 2023-12-16T21:39:09.306613+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:40:45.317 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:40:44 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:40:44.926+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:09.828911+0000 front 2023-12-16T21:39:09.828958+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:40:45.317 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:40:44 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:40:44.926+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:09.828868+0000 front 2023-12-16T21:39:09.828878+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:40:45.317 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:40:44 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:40:44.926+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:09.828947+0000 front 2023-12-16T21:39:09.828993+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:40:45.926 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:40:45 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:40:45.784+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:09.306583+0000 front 2023-12-16T21:39:09.306633+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:40:45.926 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:40:45 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:40:45.784+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:09.306551+0000 front 2023-12-16T21:39:09.306652+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:40:45.926 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:40:45 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:40:45.784+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:09.306605+0000 front 2023-12-16T21:39:09.306613+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:40:45.926 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:40:45 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:40:45.589+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:05.622397+0000 front 2023-12-16T21:39:05.622556+0000 (oldest deadline 2023-12-16T21:39:30.322546+0000) 2023-12-16T21:40:45.926 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:40:45 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:40:45.589+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:05.622506+0000 front 2023-12-16T21:39:05.622528+0000 (oldest deadline 2023-12-16T21:39:30.322546+0000) 2023-12-16T21:40:45.926 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:40:45 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:40:45.589+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:05.622479+0000 front 2023-12-16T21:39:05.622580+0000 (oldest deadline 2023-12-16T21:39:30.322546+0000) 2023-12-16T21:40:46.316 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:40:45 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:40:45.923+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:09.828911+0000 front 2023-12-16T21:39:09.828958+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:40:46.316 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:40:45 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:40:45.923+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:09.828868+0000 front 2023-12-16T21:39:09.828878+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:40:46.316 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:40:45 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:40:45.923+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:09.828947+0000 front 2023-12-16T21:39:09.828993+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:40:46.758 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:40:46 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:40:46.553+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:05.622397+0000 front 2023-12-16T21:39:05.622556+0000 (oldest deadline 2023-12-16T21:39:30.322546+0000) 2023-12-16T21:40:46.758 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:40:46 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:40:46.553+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:05.622506+0000 front 2023-12-16T21:39:05.622528+0000 (oldest deadline 2023-12-16T21:39:30.322546+0000) 2023-12-16T21:40:46.758 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:40:46 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:40:46.553+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:05.622479+0000 front 2023-12-16T21:39:05.622580+0000 (oldest deadline 2023-12-16T21:39:30.322546+0000) 2023-12-16T21:40:47.066 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:40:46 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:40:46.941+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:09.828911+0000 front 2023-12-16T21:39:09.828958+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:40:47.066 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:40:46 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:40:46.941+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:09.828868+0000 front 2023-12-16T21:39:09.828878+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:40:47.066 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:40:46 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:40:46.941+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:09.828947+0000 front 2023-12-16T21:39:09.828993+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:40:47.066 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:40:46 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:40:46.755+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:09.306583+0000 front 2023-12-16T21:39:09.306633+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:40:47.067 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:40:46 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:40:46.755+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:09.306551+0000 front 2023-12-16T21:39:09.306652+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:40:47.067 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:40:46 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:40:46.755+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:09.306605+0000 front 2023-12-16T21:39:09.306613+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:40:47.788 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:40:47 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:40:47.569+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:05.622397+0000 front 2023-12-16T21:39:05.622556+0000 (oldest deadline 2023-12-16T21:39:30.322546+0000) 2023-12-16T21:40:47.788 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:40:47 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:40:47.569+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:05.622506+0000 front 2023-12-16T21:39:05.622528+0000 (oldest deadline 2023-12-16T21:39:30.322546+0000) 2023-12-16T21:40:47.788 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:40:47 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:40:47.569+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:05.622479+0000 front 2023-12-16T21:39:05.622580+0000 (oldest deadline 2023-12-16T21:39:30.322546+0000) 2023-12-16T21:40:48.066 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:40:47 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:40:47.931+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:09.828911+0000 front 2023-12-16T21:39:09.828958+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:40:48.067 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:40:47 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:40:47.931+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:09.828868+0000 front 2023-12-16T21:39:09.828878+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:40:48.067 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:40:47 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:40:47.931+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:09.828947+0000 front 2023-12-16T21:39:09.828993+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:40:48.067 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:40:47 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:40:47.785+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:09.306583+0000 front 2023-12-16T21:39:09.306633+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:40:48.067 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:40:47 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:40:47.785+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:09.306551+0000 front 2023-12-16T21:39:09.306652+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:40:48.067 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:40:47 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:40:47.785+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:09.306605+0000 front 2023-12-16T21:39:09.306613+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:40:48.901 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:40:48 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:40:48.756+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:09.306583+0000 front 2023-12-16T21:39:09.306633+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:40:48.901 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:40:48 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:40:48.756+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:09.306551+0000 front 2023-12-16T21:39:09.306652+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:40:48.902 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:40:48 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:40:48.756+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:09.306605+0000 front 2023-12-16T21:39:09.306613+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:40:48.902 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:40:48 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:40:48.572+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:05.622397+0000 front 2023-12-16T21:39:05.622556+0000 (oldest deadline 2023-12-16T21:39:30.322546+0000) 2023-12-16T21:40:48.902 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:40:48 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:40:48.572+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:05.622506+0000 front 2023-12-16T21:39:05.622528+0000 (oldest deadline 2023-12-16T21:39:30.322546+0000) 2023-12-16T21:40:48.902 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:40:48 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:40:48.572+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:05.622479+0000 front 2023-12-16T21:39:05.622580+0000 (oldest deadline 2023-12-16T21:39:30.322546+0000) 2023-12-16T21:40:49.316 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:40:48 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:40:48.898+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:09.828911+0000 front 2023-12-16T21:39:09.828958+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:40:49.317 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:40:48 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:40:48.898+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:09.828868+0000 front 2023-12-16T21:39:09.828878+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:40:49.317 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:40:48 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:40:48.898+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:09.828947+0000 front 2023-12-16T21:39:09.828993+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:40:49.816 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:40:49 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:40:49.549+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:05.622397+0000 front 2023-12-16T21:39:05.622556+0000 (oldest deadline 2023-12-16T21:39:30.322546+0000) 2023-12-16T21:40:49.816 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:40:49 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:40:49.549+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:05.622506+0000 front 2023-12-16T21:39:05.622528+0000 (oldest deadline 2023-12-16T21:39:30.322546+0000) 2023-12-16T21:40:49.816 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:40:49 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:40:49.549+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:05.622479+0000 front 2023-12-16T21:39:05.622580+0000 (oldest deadline 2023-12-16T21:39:30.322546+0000) 2023-12-16T21:40:49.817 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:40:49 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:40:49.794+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:09.306583+0000 front 2023-12-16T21:39:09.306633+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:40:49.817 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:40:49 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:40:49.794+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:09.306551+0000 front 2023-12-16T21:39:09.306652+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:40:49.817 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:40:49 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:40:49.794+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:09.306605+0000 front 2023-12-16T21:39:09.306613+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:40:50.316 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:40:49 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:40:49.852+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:09.828911+0000 front 2023-12-16T21:39:09.828958+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:40:50.316 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:40:49 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:40:49.852+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:09.828868+0000 front 2023-12-16T21:39:09.828878+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:40:50.316 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:40:49 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:40:49.852+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:09.828947+0000 front 2023-12-16T21:39:09.828993+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:40:51.066 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:40:50 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:40:50.582+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:05.622397+0000 front 2023-12-16T21:39:05.622556+0000 (oldest deadline 2023-12-16T21:39:30.322546+0000) 2023-12-16T21:40:51.066 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:40:50 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:40:50.582+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:05.622506+0000 front 2023-12-16T21:39:05.622528+0000 (oldest deadline 2023-12-16T21:39:30.322546+0000) 2023-12-16T21:40:51.066 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:40:50 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:40:50.582+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:05.622479+0000 front 2023-12-16T21:39:05.622580+0000 (oldest deadline 2023-12-16T21:39:30.322546+0000) 2023-12-16T21:40:51.067 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:40:50 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:40:50.808+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:09.828911+0000 front 2023-12-16T21:39:09.828958+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:40:51.067 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:40:50 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:40:50.808+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:09.828868+0000 front 2023-12-16T21:39:09.828878+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:40:51.067 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:40:50 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:40:50.808+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:09.828947+0000 front 2023-12-16T21:39:09.828993+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:40:51.067 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:40:50 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:40:50.807+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:09.306583+0000 front 2023-12-16T21:39:09.306633+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:40:51.067 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:40:50 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:40:50.807+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:09.306551+0000 front 2023-12-16T21:39:09.306652+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:40:51.067 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:40:50 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:40:50.807+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:09.306605+0000 front 2023-12-16T21:39:09.306613+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:40:51.629 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:40:51 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:40:51.547+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:05.622397+0000 front 2023-12-16T21:39:05.622556+0000 (oldest deadline 2023-12-16T21:39:30.322546+0000) 2023-12-16T21:40:51.629 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:40:51 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:40:51.547+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:05.622506+0000 front 2023-12-16T21:39:05.622528+0000 (oldest deadline 2023-12-16T21:39:30.322546+0000) 2023-12-16T21:40:51.629 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:40:51 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:40:51.547+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:05.622479+0000 front 2023-12-16T21:39:05.622580+0000 (oldest deadline 2023-12-16T21:39:30.322546+0000) 2023-12-16T21:40:52.066 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:40:51 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:40:51.784+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:09.306583+0000 front 2023-12-16T21:39:09.306633+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:40:52.066 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:40:51 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:40:51.784+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:09.306551+0000 front 2023-12-16T21:39:09.306652+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:40:52.066 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:40:51 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:40:51.784+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:09.306605+0000 front 2023-12-16T21:39:09.306613+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:40:52.067 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:40:51 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:40:51.824+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:09.828911+0000 front 2023-12-16T21:39:09.828958+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:40:52.067 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:40:51 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:40:51.824+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:09.828868+0000 front 2023-12-16T21:39:09.828878+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:40:52.067 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:40:51 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:40:51.824+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:09.828947+0000 front 2023-12-16T21:39:09.828993+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:40:52.769 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:40:52 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:40:52.498+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:05.622397+0000 front 2023-12-16T21:39:05.622556+0000 (oldest deadline 2023-12-16T21:39:30.322546+0000) 2023-12-16T21:40:52.769 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:40:52 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:40:52.498+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:05.622506+0000 front 2023-12-16T21:39:05.622528+0000 (oldest deadline 2023-12-16T21:39:30.322546+0000) 2023-12-16T21:40:52.769 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:40:52 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:40:52.498+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:05.622479+0000 front 2023-12-16T21:39:05.622580+0000 (oldest deadline 2023-12-16T21:39:30.322546+0000) 2023-12-16T21:40:53.067 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:40:52 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:40:52.766+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:09.306583+0000 front 2023-12-16T21:39:09.306633+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:40:53.067 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:40:52 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:40:52.766+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:09.306551+0000 front 2023-12-16T21:39:09.306652+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:40:53.068 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:40:52 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:40:52.766+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:09.306605+0000 front 2023-12-16T21:39:09.306613+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:40:53.068 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:40:52 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:40:52.796+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:09.828911+0000 front 2023-12-16T21:39:09.828958+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:40:53.068 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:40:52 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:40:52.796+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:09.828868+0000 front 2023-12-16T21:39:09.828878+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:40:53.068 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:40:52 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:40:52.796+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:09.828947+0000 front 2023-12-16T21:39:09.828993+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:40:53.722 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:40:53 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:40:53.448+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:05.622397+0000 front 2023-12-16T21:39:05.622556+0000 (oldest deadline 2023-12-16T21:39:30.322546+0000) 2023-12-16T21:40:53.722 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:40:53 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:40:53.448+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:05.622506+0000 front 2023-12-16T21:39:05.622528+0000 (oldest deadline 2023-12-16T21:39:30.322546+0000) 2023-12-16T21:40:53.722 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:40:53 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:40:53.448+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:05.622479+0000 front 2023-12-16T21:39:05.622580+0000 (oldest deadline 2023-12-16T21:39:30.322546+0000) 2023-12-16T21:40:54.066 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:40:53 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:40:53.777+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:09.828911+0000 front 2023-12-16T21:39:09.828958+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:40:54.066 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:40:53 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:40:53.777+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:09.828868+0000 front 2023-12-16T21:39:09.828878+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:40:54.066 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:40:53 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:40:53.777+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:09.828947+0000 front 2023-12-16T21:39:09.828993+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:40:54.067 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:40:53 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:40:53.718+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:09.306583+0000 front 2023-12-16T21:39:09.306633+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:40:54.067 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:40:53 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:40:53.718+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:09.306551+0000 front 2023-12-16T21:39:09.306652+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:40:54.067 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:40:53 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:40:53.718+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:09.306605+0000 front 2023-12-16T21:39:09.306613+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:40:54.686 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:40:54 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:40:54.399+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:05.622397+0000 front 2023-12-16T21:39:05.622556+0000 (oldest deadline 2023-12-16T21:39:30.322546+0000) 2023-12-16T21:40:54.686 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:40:54 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:40:54.399+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:05.622506+0000 front 2023-12-16T21:39:05.622528+0000 (oldest deadline 2023-12-16T21:39:30.322546+0000) 2023-12-16T21:40:54.686 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:40:54 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:40:54.399+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:05.622479+0000 front 2023-12-16T21:39:05.622580+0000 (oldest deadline 2023-12-16T21:39:30.322546+0000) 2023-12-16T21:40:55.066 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:40:54 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:40:54.683+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:09.306583+0000 front 2023-12-16T21:39:09.306633+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:40:55.066 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:40:54 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:40:54.683+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:09.306551+0000 front 2023-12-16T21:39:09.306652+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:40:55.066 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:40:54 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:40:54.683+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:09.306605+0000 front 2023-12-16T21:39:09.306613+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:40:55.067 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:40:54 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:40:54.758+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:09.828911+0000 front 2023-12-16T21:39:09.828958+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:40:55.067 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:40:54 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:40:54.758+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:09.828868+0000 front 2023-12-16T21:39:09.828878+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:40:55.067 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:40:54 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:40:54.758+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:09.828947+0000 front 2023-12-16T21:39:09.828993+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:40:55.656 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:40:55 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:40:55.398+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:05.622397+0000 front 2023-12-16T21:39:05.622556+0000 (oldest deadline 2023-12-16T21:39:30.322546+0000) 2023-12-16T21:40:55.657 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:40:55 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:40:55.398+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:05.622506+0000 front 2023-12-16T21:39:05.622528+0000 (oldest deadline 2023-12-16T21:39:30.322546+0000) 2023-12-16T21:40:55.657 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:40:55 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:40:55.398+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:05.622479+0000 front 2023-12-16T21:39:05.622580+0000 (oldest deadline 2023-12-16T21:39:30.322546+0000) 2023-12-16T21:40:56.066 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:40:55 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:40:55.654+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:09.306583+0000 front 2023-12-16T21:39:09.306633+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:40:56.066 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:40:55 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:40:55.654+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:09.306551+0000 front 2023-12-16T21:39:09.306652+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:40:56.066 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:40:55 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:40:55.654+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:09.306605+0000 front 2023-12-16T21:39:09.306613+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:40:56.067 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:40:55 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:40:55.725+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:09.828911+0000 front 2023-12-16T21:39:09.828958+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:40:56.067 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:40:55 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:40:55.725+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:09.828868+0000 front 2023-12-16T21:39:09.828878+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:40:56.067 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:40:55 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:40:55.725+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:09.828947+0000 front 2023-12-16T21:39:09.828993+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:40:56.635 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:40:56 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:40:56.380+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:05.622397+0000 front 2023-12-16T21:39:05.622556+0000 (oldest deadline 2023-12-16T21:39:30.322546+0000) 2023-12-16T21:40:56.635 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:40:56 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:40:56.380+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:05.622506+0000 front 2023-12-16T21:39:05.622528+0000 (oldest deadline 2023-12-16T21:39:30.322546+0000) 2023-12-16T21:40:56.636 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:40:56 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:40:56.380+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:05.622479+0000 front 2023-12-16T21:39:05.622580+0000 (oldest deadline 2023-12-16T21:39:30.322546+0000) 2023-12-16T21:40:57.066 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:40:56 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:40:56.632+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:09.306583+0000 front 2023-12-16T21:39:09.306633+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:40:57.066 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:40:56 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:40:56.632+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:09.306551+0000 front 2023-12-16T21:39:09.306652+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:40:57.066 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:40:56 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:40:56.632+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:09.306605+0000 front 2023-12-16T21:39:09.306613+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:40:57.067 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:40:56 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:40:56.710+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:09.828911+0000 front 2023-12-16T21:39:09.828958+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:40:57.067 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:40:56 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:40:56.710+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:09.828868+0000 front 2023-12-16T21:39:09.828878+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:40:57.067 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:40:56 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:40:56.710+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:09.828947+0000 front 2023-12-16T21:39:09.828993+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:40:57.714 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:40:57 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:40:57.608+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:09.306583+0000 front 2023-12-16T21:39:09.306633+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:40:57.714 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:40:57 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:40:57.608+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:09.306551+0000 front 2023-12-16T21:39:09.306652+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:40:57.714 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:40:57 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:40:57.608+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:09.306605+0000 front 2023-12-16T21:39:09.306613+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:40:57.714 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:40:57 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:40:57.410+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:05.622397+0000 front 2023-12-16T21:39:05.622556+0000 (oldest deadline 2023-12-16T21:39:30.322546+0000) 2023-12-16T21:40:57.714 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:40:57 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:40:57.410+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:05.622506+0000 front 2023-12-16T21:39:05.622528+0000 (oldest deadline 2023-12-16T21:39:30.322546+0000) 2023-12-16T21:40:57.715 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:40:57 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:40:57.410+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:05.622479+0000 front 2023-12-16T21:39:05.622580+0000 (oldest deadline 2023-12-16T21:39:30.322546+0000) 2023-12-16T21:40:58.066 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:40:57 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:40:57.711+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:09.828911+0000 front 2023-12-16T21:39:09.828958+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:40:58.066 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:40:57 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:40:57.711+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:09.828868+0000 front 2023-12-16T21:39:09.828878+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:40:58.066 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:40:57 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:40:57.711+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:09.828947+0000 front 2023-12-16T21:39:09.828993+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:40:58.668 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:40:58 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:40:58.618+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:09.306583+0000 front 2023-12-16T21:39:09.306633+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:40:58.669 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:40:58 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:40:58.618+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:09.306551+0000 front 2023-12-16T21:39:09.306652+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:40:58.669 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:40:58 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:40:58.618+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:09.306605+0000 front 2023-12-16T21:39:09.306613+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:40:58.669 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:40:58 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:40:58.371+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:05.622397+0000 front 2023-12-16T21:39:05.622556+0000 (oldest deadline 2023-12-16T21:39:30.322546+0000) 2023-12-16T21:40:58.669 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:40:58 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:40:58.371+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:05.622506+0000 front 2023-12-16T21:39:05.622528+0000 (oldest deadline 2023-12-16T21:39:30.322546+0000) 2023-12-16T21:40:58.669 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:40:58 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:40:58.371+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:05.622479+0000 front 2023-12-16T21:39:05.622580+0000 (oldest deadline 2023-12-16T21:39:30.322546+0000) 2023-12-16T21:40:59.066 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:40:58 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:40:58.666+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:09.828911+0000 front 2023-12-16T21:39:09.828958+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:40:59.066 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:40:58 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:40:58.666+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:09.828868+0000 front 2023-12-16T21:39:09.828878+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:40:59.066 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:40:58 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:40:58.666+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:09.828947+0000 front 2023-12-16T21:39:09.828993+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:40:59.660 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:40:59 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:40:59.574+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:09.306583+0000 front 2023-12-16T21:39:09.306633+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:40:59.661 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:40:59 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:40:59.574+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:09.306551+0000 front 2023-12-16T21:39:09.306652+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:40:59.661 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:40:59 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:40:59.574+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:09.306605+0000 front 2023-12-16T21:39:09.306613+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:40:59.662 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:40:59 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:40:59.402+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:05.622397+0000 front 2023-12-16T21:39:05.622556+0000 (oldest deadline 2023-12-16T21:39:30.322546+0000) 2023-12-16T21:40:59.662 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:40:59 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:40:59.402+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:05.622506+0000 front 2023-12-16T21:39:05.622528+0000 (oldest deadline 2023-12-16T21:39:30.322546+0000) 2023-12-16T21:40:59.662 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:40:59 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:40:59.402+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:05.622479+0000 front 2023-12-16T21:39:05.622580+0000 (oldest deadline 2023-12-16T21:39:30.322546+0000) 2023-12-16T21:41:00.066 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:40:59 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:40:59.658+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:09.828911+0000 front 2023-12-16T21:39:09.828958+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:41:00.066 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:40:59 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:40:59.658+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:09.828868+0000 front 2023-12-16T21:39:09.828878+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:41:00.066 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:40:59 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:40:59.658+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:09.828947+0000 front 2023-12-16T21:39:09.828993+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:41:00.816 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:41:00 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:41:00.611+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:09.306583+0000 front 2023-12-16T21:39:09.306633+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:41:00.816 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:41:00 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:41:00.611+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:09.306551+0000 front 2023-12-16T21:39:09.306652+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:41:00.816 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:41:00 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:41:00.611+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:09.306605+0000 front 2023-12-16T21:39:09.306613+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:41:00.817 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:41:00 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:41:00.425+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:05.622397+0000 front 2023-12-16T21:39:05.622556+0000 (oldest deadline 2023-12-16T21:39:30.322546+0000) 2023-12-16T21:41:00.817 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:41:00 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:41:00.425+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:05.622506+0000 front 2023-12-16T21:39:05.622528+0000 (oldest deadline 2023-12-16T21:39:30.322546+0000) 2023-12-16T21:41:00.817 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:41:00 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:41:00.425+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:05.622479+0000 front 2023-12-16T21:39:05.622580+0000 (oldest deadline 2023-12-16T21:39:30.322546+0000) 2023-12-16T21:41:00.817 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:41:00 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:41:00.629+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:09.828911+0000 front 2023-12-16T21:39:09.828958+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:41:00.817 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:41:00 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:41:00.629+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:09.828868+0000 front 2023-12-16T21:39:09.828878+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:41:00.817 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:41:00 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:41:00.629+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:09.828947+0000 front 2023-12-16T21:39:09.828993+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:41:01.629 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:41:01 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:41:01.387+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:05.622397+0000 front 2023-12-16T21:39:05.622556+0000 (oldest deadline 2023-12-16T21:39:30.322546+0000) 2023-12-16T21:41:01.629 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:41:01 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:41:01.387+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:05.622506+0000 front 2023-12-16T21:39:05.622528+0000 (oldest deadline 2023-12-16T21:39:30.322546+0000) 2023-12-16T21:41:01.629 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:41:01 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:41:01.387+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:05.622479+0000 front 2023-12-16T21:39:05.622580+0000 (oldest deadline 2023-12-16T21:39:30.322546+0000) 2023-12-16T21:41:02.066 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:41:01 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:41:01.654+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:09.306583+0000 front 2023-12-16T21:39:09.306633+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:41:02.066 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:41:01 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:41:01.654+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:09.306551+0000 front 2023-12-16T21:39:09.306652+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:41:02.066 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:41:01 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:41:01.655+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:09.306605+0000 front 2023-12-16T21:39:09.306613+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:41:02.067 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:41:01 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:41:01.668+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:09.828911+0000 front 2023-12-16T21:39:09.828958+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:41:02.067 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:41:01 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:41:01.668+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:09.828868+0000 front 2023-12-16T21:39:09.828878+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:41:02.067 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:41:01 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:41:01.668+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:09.828947+0000 front 2023-12-16T21:39:09.828993+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:41:02.639 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:41:02 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:41:02.380+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:05.622397+0000 front 2023-12-16T21:39:05.622556+0000 (oldest deadline 2023-12-16T21:39:30.322546+0000) 2023-12-16T21:41:02.639 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:41:02 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:41:02.380+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:05.622506+0000 front 2023-12-16T21:39:05.622528+0000 (oldest deadline 2023-12-16T21:39:30.322546+0000) 2023-12-16T21:41:02.639 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:41:02 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:41:02.380+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:05.622479+0000 front 2023-12-16T21:39:05.622580+0000 (oldest deadline 2023-12-16T21:39:30.322546+0000) 2023-12-16T21:41:03.066 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:41:02 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:41:02.659+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:09.828911+0000 front 2023-12-16T21:39:09.828958+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:41:03.066 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:41:02 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:41:02.659+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:09.828868+0000 front 2023-12-16T21:39:09.828878+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:41:03.066 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:41:02 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:41:02.659+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:09.828947+0000 front 2023-12-16T21:39:09.828993+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:41:03.067 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:41:02 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:41:02.636+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:09.306583+0000 front 2023-12-16T21:39:09.306633+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:41:03.067 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:41:02 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:41:02.636+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:09.306551+0000 front 2023-12-16T21:39:09.306652+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:41:03.067 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:41:02 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:41:02.636+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:09.306605+0000 front 2023-12-16T21:39:09.306613+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:41:03.645 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:41:03 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:41:03.389+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:05.622397+0000 front 2023-12-16T21:39:05.622556+0000 (oldest deadline 2023-12-16T21:39:30.322546+0000) 2023-12-16T21:41:03.645 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:41:03 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:41:03.389+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:05.622506+0000 front 2023-12-16T21:39:05.622528+0000 (oldest deadline 2023-12-16T21:39:30.322546+0000) 2023-12-16T21:41:03.645 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:41:03 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:41:03.389+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:05.622479+0000 front 2023-12-16T21:39:05.622580+0000 (oldest deadline 2023-12-16T21:39:30.322546+0000) 2023-12-16T21:41:04.066 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:41:03 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:41:03.682+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:09.828911+0000 front 2023-12-16T21:39:09.828958+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:41:04.066 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:41:03 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:41:03.682+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:09.828868+0000 front 2023-12-16T21:39:09.828878+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:41:04.066 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:41:03 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:41:03.682+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:09.828947+0000 front 2023-12-16T21:39:09.828993+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:41:04.067 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:41:03 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:41:03.641+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:09.306583+0000 front 2023-12-16T21:39:09.306633+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:41:04.067 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:41:03 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:41:03.641+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:09.306551+0000 front 2023-12-16T21:39:09.306652+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:41:04.067 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:41:03 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:41:03.641+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:09.306605+0000 front 2023-12-16T21:39:09.306613+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:41:04.638 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:41:04 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:41:04.593+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:09.306583+0000 front 2023-12-16T21:39:09.306633+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:41:04.638 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:41:04 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:41:04.593+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:09.306551+0000 front 2023-12-16T21:39:09.306652+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:41:04.638 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:41:04 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:41:04.593+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:09.306605+0000 front 2023-12-16T21:39:09.306613+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:41:04.639 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:41:04 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:41:04.373+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:05.622397+0000 front 2023-12-16T21:39:05.622556+0000 (oldest deadline 2023-12-16T21:39:30.322546+0000) 2023-12-16T21:41:04.639 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:41:04 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:41:04.373+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:05.622506+0000 front 2023-12-16T21:39:05.622528+0000 (oldest deadline 2023-12-16T21:39:30.322546+0000) 2023-12-16T21:41:04.639 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:41:04 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:41:04.373+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:05.622479+0000 front 2023-12-16T21:39:05.622580+0000 (oldest deadline 2023-12-16T21:39:30.322546+0000) 2023-12-16T21:41:05.066 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:41:04 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:41:04.635+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:09.828911+0000 front 2023-12-16T21:39:09.828958+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:41:05.066 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:41:04 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:41:04.635+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:09.828868+0000 front 2023-12-16T21:39:09.828878+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:41:05.066 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:41:04 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:41:04.635+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:09.828947+0000 front 2023-12-16T21:39:09.828993+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:41:05.621 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:41:05 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:41:05.357+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:05.622397+0000 front 2023-12-16T21:39:05.622556+0000 (oldest deadline 2023-12-16T21:39:30.322546+0000) 2023-12-16T21:41:05.621 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:41:05 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:41:05.357+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:05.622506+0000 front 2023-12-16T21:39:05.622528+0000 (oldest deadline 2023-12-16T21:39:30.322546+0000) 2023-12-16T21:41:05.621 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:41:05 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:41:05.357+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:05.622479+0000 front 2023-12-16T21:39:05.622580+0000 (oldest deadline 2023-12-16T21:39:30.322546+0000) 2023-12-16T21:41:05.937 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:41:05 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:41:05.630+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:09.306583+0000 front 2023-12-16T21:39:09.306633+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:41:05.937 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:41:05 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:41:05.630+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:09.306551+0000 front 2023-12-16T21:39:09.306652+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:41:05.937 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:41:05 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:41:05.630+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:09.306605+0000 front 2023-12-16T21:39:09.306613+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:41:05.938 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:41:05 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:41:05.618+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:09.828911+0000 front 2023-12-16T21:39:09.828958+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:41:05.938 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:41:05 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:41:05.618+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:09.828868+0000 front 2023-12-16T21:39:09.828878+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:41:05.938 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:41:05 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:41:05.618+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:09.828947+0000 front 2023-12-16T21:39:09.828993+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:41:06.641 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:41:06 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:41:06.574+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:09.828911+0000 front 2023-12-16T21:39:09.828958+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:41:06.641 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:41:06 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:41:06.574+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:09.828868+0000 front 2023-12-16T21:39:09.828878+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:41:06.641 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:41:06 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:41:06.574+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:09.828947+0000 front 2023-12-16T21:39:09.828993+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:41:06.642 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:41:06 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:41:06.340+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:05.622397+0000 front 2023-12-16T21:39:05.622556+0000 (oldest deadline 2023-12-16T21:39:30.322546+0000) 2023-12-16T21:41:06.642 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:41:06 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:41:06.340+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:05.622506+0000 front 2023-12-16T21:39:05.622528+0000 (oldest deadline 2023-12-16T21:39:30.322546+0000) 2023-12-16T21:41:06.642 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:41:06 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:41:06.340+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:05.622479+0000 front 2023-12-16T21:39:05.622580+0000 (oldest deadline 2023-12-16T21:39:30.322546+0000) 2023-12-16T21:41:07.066 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:41:06 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:41:06.638+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:09.306583+0000 front 2023-12-16T21:39:09.306633+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:41:07.066 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:41:06 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:41:06.638+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:09.306551+0000 front 2023-12-16T21:39:09.306652+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:41:07.066 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:41:06 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:41:06.638+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:09.306605+0000 front 2023-12-16T21:39:09.306613+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:41:07.622 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:41:07 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:41:07.326+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:05.622397+0000 front 2023-12-16T21:39:05.622556+0000 (oldest deadline 2023-12-16T21:39:30.322546+0000) 2023-12-16T21:41:07.622 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:41:07 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:41:07.326+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:05.622506+0000 front 2023-12-16T21:39:05.622528+0000 (oldest deadline 2023-12-16T21:39:30.322546+0000) 2023-12-16T21:41:07.623 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:41:07 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:41:07.326+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:05.622479+0000 front 2023-12-16T21:39:05.622580+0000 (oldest deadline 2023-12-16T21:39:30.322546+0000) 2023-12-16T21:41:08.066 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:41:07 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:41:07.619+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:09.828911+0000 front 2023-12-16T21:39:09.828958+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:41:08.066 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:41:07 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:41:07.620+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:09.828868+0000 front 2023-12-16T21:39:09.828878+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:41:08.066 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:41:07 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:41:07.620+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:09.828947+0000 front 2023-12-16T21:39:09.828993+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:41:08.066 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:41:07 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:41:07.631+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:09.306583+0000 front 2023-12-16T21:39:09.306633+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:41:08.067 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:41:07 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:41:07.631+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:09.306551+0000 front 2023-12-16T21:39:09.306652+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:41:08.067 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:41:07 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:41:07.631+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:09.306605+0000 front 2023-12-16T21:39:09.306613+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:41:08.614 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:41:08 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:41:08.353+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:05.622397+0000 front 2023-12-16T21:39:05.622556+0000 (oldest deadline 2023-12-16T21:39:30.322546+0000) 2023-12-16T21:41:08.615 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:41:08 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:41:08.353+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:05.622506+0000 front 2023-12-16T21:39:05.622528+0000 (oldest deadline 2023-12-16T21:39:30.322546+0000) 2023-12-16T21:41:08.615 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:41:08 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:41:08.353+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:05.622479+0000 front 2023-12-16T21:39:05.622580+0000 (oldest deadline 2023-12-16T21:39:30.322546+0000) 2023-12-16T21:41:09.066 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:41:08 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:41:08.668+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:09.828911+0000 front 2023-12-16T21:39:09.828958+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:41:09.067 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:41:08 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:41:08.668+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:09.828868+0000 front 2023-12-16T21:39:09.828878+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:41:09.067 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:41:08 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:41:08.668+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:09.828947+0000 front 2023-12-16T21:39:09.828993+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:41:09.067 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:41:08 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:41:08.612+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:09.306583+0000 front 2023-12-16T21:39:09.306633+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:41:09.067 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:41:08 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:41:08.612+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:09.306551+0000 front 2023-12-16T21:39:09.306652+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:41:09.067 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:41:08 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:41:08.612+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:09.306605+0000 front 2023-12-16T21:39:09.306613+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:41:09.652 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:41:09 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:41:09.345+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:05.622397+0000 front 2023-12-16T21:39:05.622556+0000 (oldest deadline 2023-12-16T21:39:30.322546+0000) 2023-12-16T21:41:09.652 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:41:09 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:41:09.345+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:05.622506+0000 front 2023-12-16T21:39:05.622528+0000 (oldest deadline 2023-12-16T21:39:30.322546+0000) 2023-12-16T21:41:09.652 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:41:09 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:41:09.345+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:05.622479+0000 front 2023-12-16T21:39:05.622580+0000 (oldest deadline 2023-12-16T21:39:30.322546+0000) 2023-12-16T21:41:10.066 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:41:09 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:41:09.649+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:09.306583+0000 front 2023-12-16T21:39:09.306633+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:41:10.066 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:41:09 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:41:09.649+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:09.306551+0000 front 2023-12-16T21:39:09.306652+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:41:10.066 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:41:09 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:41:09.649+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:09.306605+0000 front 2023-12-16T21:39:09.306613+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:41:10.067 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:41:09 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:41:09.664+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:09.828911+0000 front 2023-12-16T21:39:09.828958+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:41:10.067 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:41:09 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:41:09.664+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:09.828868+0000 front 2023-12-16T21:39:09.828878+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:41:10.067 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:41:09 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:41:09.664+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:09.828947+0000 front 2023-12-16T21:39:09.828993+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:41:10.566 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:41:10 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:41:10.303+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:05.622397+0000 front 2023-12-16T21:39:05.622556+0000 (oldest deadline 2023-12-16T21:39:30.322546+0000) 2023-12-16T21:41:10.592 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:41:10 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:41:10.303+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:05.622506+0000 front 2023-12-16T21:39:05.622528+0000 (oldest deadline 2023-12-16T21:39:30.322546+0000) 2023-12-16T21:41:10.592 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:41:10 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:41:10.303+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:05.622479+0000 front 2023-12-16T21:39:05.622580+0000 (oldest deadline 2023-12-16T21:39:30.322546+0000) 2023-12-16T21:41:11.066 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:41:10 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:41:10.633+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:09.306583+0000 front 2023-12-16T21:39:09.306633+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:41:11.066 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:41:10 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:41:10.633+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:09.306551+0000 front 2023-12-16T21:39:09.306652+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:41:11.066 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:41:10 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:41:10.633+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:09.306605+0000 front 2023-12-16T21:39:09.306613+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:41:11.067 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:41:10 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:41:10.694+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:09.828911+0000 front 2023-12-16T21:39:09.828958+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:41:11.067 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:41:10 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:41:10.694+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:09.828868+0000 front 2023-12-16T21:39:09.828878+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:41:11.067 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:41:10 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:41:10.694+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:09.828947+0000 front 2023-12-16T21:39:09.828993+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:41:11.630 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:41:11 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:41:11.337+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:05.622397+0000 front 2023-12-16T21:39:05.622556+0000 (oldest deadline 2023-12-16T21:39:30.322546+0000) 2023-12-16T21:41:11.630 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:41:11 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:41:11.337+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:05.622506+0000 front 2023-12-16T21:39:05.622528+0000 (oldest deadline 2023-12-16T21:39:30.322546+0000) 2023-12-16T21:41:11.630 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:41:11 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:41:11.337+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:05.622479+0000 front 2023-12-16T21:39:05.622580+0000 (oldest deadline 2023-12-16T21:39:30.322546+0000) 2023-12-16T21:41:12.066 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:41:11 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:41:11.639+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:09.306583+0000 front 2023-12-16T21:39:09.306633+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:41:12.066 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:41:11 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:41:11.639+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:09.306551+0000 front 2023-12-16T21:39:09.306652+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:41:12.066 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:41:11 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:41:11.639+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:09.306605+0000 front 2023-12-16T21:39:09.306613+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:41:12.067 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:41:11 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:41:11.700+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:09.828911+0000 front 2023-12-16T21:39:09.828958+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:41:12.067 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:41:11 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:41:11.700+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:09.828868+0000 front 2023-12-16T21:39:09.828878+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:41:12.067 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:41:11 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:41:11.700+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:09.828947+0000 front 2023-12-16T21:39:09.828993+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:41:12.566 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:41:12 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:41:12.291+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:05.622397+0000 front 2023-12-16T21:39:05.622556+0000 (oldest deadline 2023-12-16T21:39:30.322546+0000) 2023-12-16T21:41:12.566 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:41:12 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:41:12.291+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:05.622506+0000 front 2023-12-16T21:39:05.622528+0000 (oldest deadline 2023-12-16T21:39:30.322546+0000) 2023-12-16T21:41:12.566 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:41:12 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:41:12.291+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:05.622479+0000 front 2023-12-16T21:39:05.622580+0000 (oldest deadline 2023-12-16T21:39:30.322546+0000) 2023-12-16T21:41:13.066 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:41:12 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:41:12.627+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:09.306583+0000 front 2023-12-16T21:39:09.306633+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:41:13.066 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:41:12 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:41:12.627+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:09.306551+0000 front 2023-12-16T21:39:09.306652+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:41:13.066 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:41:12 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:41:12.627+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:09.306605+0000 front 2023-12-16T21:39:09.306613+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:41:13.067 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:41:12 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:41:12.669+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:09.828911+0000 front 2023-12-16T21:39:09.828958+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:41:13.067 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:41:12 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:41:12.669+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:09.828868+0000 front 2023-12-16T21:39:09.828878+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:41:13.067 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:41:12 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:41:12.669+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:09.828947+0000 front 2023-12-16T21:39:09.828993+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:41:13.566 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:41:13 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:41:13.309+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:05.622397+0000 front 2023-12-16T21:39:05.622556+0000 (oldest deadline 2023-12-16T21:39:30.322546+0000) 2023-12-16T21:41:13.566 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:41:13 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:41:13.309+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:05.622506+0000 front 2023-12-16T21:39:05.622528+0000 (oldest deadline 2023-12-16T21:39:30.322546+0000) 2023-12-16T21:41:13.566 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:41:13 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:41:13.309+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:05.622479+0000 front 2023-12-16T21:39:05.622580+0000 (oldest deadline 2023-12-16T21:39:30.322546+0000) 2023-12-16T21:41:14.066 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:41:13 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:41:13.693+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:09.828911+0000 front 2023-12-16T21:39:09.828958+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:41:14.066 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:41:13 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:41:13.693+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:09.828868+0000 front 2023-12-16T21:39:09.828878+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:41:14.067 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:41:13 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:41:13.693+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:09.828947+0000 front 2023-12-16T21:39:09.828993+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:41:14.067 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:41:13 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:41:13.598+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:09.306583+0000 front 2023-12-16T21:39:09.306633+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:41:14.067 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:41:13 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:41:13.598+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:09.306551+0000 front 2023-12-16T21:39:09.306652+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:41:14.067 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:41:13 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:41:13.598+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:09.306605+0000 front 2023-12-16T21:39:09.306613+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:41:14.566 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:41:14 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:41:14.302+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:05.622397+0000 front 2023-12-16T21:39:05.622556+0000 (oldest deadline 2023-12-16T21:39:30.322546+0000) 2023-12-16T21:41:14.566 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:41:14 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:41:14.302+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:05.622506+0000 front 2023-12-16T21:39:05.622528+0000 (oldest deadline 2023-12-16T21:39:30.322546+0000) 2023-12-16T21:41:14.566 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:41:14 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:41:14.302+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:05.622479+0000 front 2023-12-16T21:39:05.622580+0000 (oldest deadline 2023-12-16T21:39:30.322546+0000) 2023-12-16T21:41:15.066 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:41:14 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:41:14.565+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:09.306583+0000 front 2023-12-16T21:39:09.306633+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:41:15.066 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:41:14 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:41:14.565+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:09.306551+0000 front 2023-12-16T21:39:09.306652+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:41:15.066 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:41:14 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:41:14.565+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:09.306605+0000 front 2023-12-16T21:39:09.306613+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:41:15.067 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:41:14 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:41:14.733+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:09.828911+0000 front 2023-12-16T21:39:09.828958+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:41:15.067 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:41:14 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:41:14.733+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:09.828868+0000 front 2023-12-16T21:39:09.828878+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:41:15.067 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:41:14 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:41:14.733+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:09.828947+0000 front 2023-12-16T21:39:09.828993+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:41:15.563 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:41:15 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:41:15.280+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:05.622397+0000 front 2023-12-16T21:39:05.622556+0000 (oldest deadline 2023-12-16T21:39:30.322546+0000) 2023-12-16T21:41:15.563 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:41:15 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:41:15.280+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:05.622506+0000 front 2023-12-16T21:39:05.622528+0000 (oldest deadline 2023-12-16T21:39:30.322546+0000) 2023-12-16T21:41:15.563 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:41:15 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:41:15.280+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:05.622479+0000 front 2023-12-16T21:39:05.622580+0000 (oldest deadline 2023-12-16T21:39:30.322546+0000) 2023-12-16T21:41:15.816 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:41:15 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:41:15.740+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:09.828911+0000 front 2023-12-16T21:39:09.828958+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:41:15.816 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:41:15 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:41:15.740+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:09.828868+0000 front 2023-12-16T21:39:09.828878+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:41:15.816 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:41:15 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:41:15.740+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:09.828947+0000 front 2023-12-16T21:39:09.828993+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:41:15.817 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:41:15 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:41:15.559+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:09.306583+0000 front 2023-12-16T21:39:09.306633+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:41:15.817 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:41:15 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:41:15.559+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:09.306551+0000 front 2023-12-16T21:39:09.306652+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:41:15.817 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:41:15 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:41:15.559+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:09.306605+0000 front 2023-12-16T21:39:09.306613+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:41:16.566 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Dec 16 21:41:16 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-mon-smithi045[80759]: 2023-12-16T21:41:16.328+0000 7f45a1f8f700 -1 mon.smithi045@0(probing) e2 get_health_metrics reporting 7 slow ops, oldest is monmgrreport(gid 14214, 0 checks, 1 progress events) 2023-12-16T21:41:16.567 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:41:16 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:41:16.275+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:05.622397+0000 front 2023-12-16T21:39:05.622556+0000 (oldest deadline 2023-12-16T21:39:30.322546+0000) 2023-12-16T21:41:16.567 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:41:16 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:41:16.275+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:05.622506+0000 front 2023-12-16T21:39:05.622528+0000 (oldest deadline 2023-12-16T21:39:30.322546+0000) 2023-12-16T21:41:16.567 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:41:16 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:41:16.275+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:05.622479+0000 front 2023-12-16T21:39:05.622580+0000 (oldest deadline 2023-12-16T21:39:30.322546+0000) 2023-12-16T21:41:17.066 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:41:16 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:41:16.598+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:09.306583+0000 front 2023-12-16T21:39:09.306633+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:41:17.066 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:41:16 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:41:16.598+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:09.306551+0000 front 2023-12-16T21:39:09.306652+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:41:17.067 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:41:16 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:41:16.598+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:09.306605+0000 front 2023-12-16T21:39:09.306613+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:41:17.067 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:41:16 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:41:16.730+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:09.828911+0000 front 2023-12-16T21:39:09.828958+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:41:17.067 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:41:16 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:41:16.730+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:09.828868+0000 front 2023-12-16T21:39:09.828878+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:41:17.067 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:41:16 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:41:16.730+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:09.828947+0000 front 2023-12-16T21:39:09.828993+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:41:17.566 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:41:17 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:41:17.301+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:05.622397+0000 front 2023-12-16T21:39:05.622556+0000 (oldest deadline 2023-12-16T21:39:30.322546+0000) 2023-12-16T21:41:17.566 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:41:17 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:41:17.301+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:05.622506+0000 front 2023-12-16T21:39:05.622528+0000 (oldest deadline 2023-12-16T21:39:30.322546+0000) 2023-12-16T21:41:17.566 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:41:17 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:41:17.301+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:05.622479+0000 front 2023-12-16T21:39:05.622580+0000 (oldest deadline 2023-12-16T21:39:30.322546+0000) 2023-12-16T21:41:18.066 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:41:17 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:41:17.592+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:09.306583+0000 front 2023-12-16T21:39:09.306633+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:41:18.066 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:41:17 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:41:17.592+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:09.306551+0000 front 2023-12-16T21:39:09.306652+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:41:18.066 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:41:17 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:41:17.592+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:09.306605+0000 front 2023-12-16T21:39:09.306613+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:41:18.067 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:41:17 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:41:17.735+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:09.828911+0000 front 2023-12-16T21:39:09.828958+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:41:18.067 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:41:17 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:41:17.735+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:09.828868+0000 front 2023-12-16T21:39:09.828878+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:41:18.067 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:41:17 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:41:17.735+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:09.828947+0000 front 2023-12-16T21:39:09.828993+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:41:18.566 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:41:18 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:41:18.267+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:05.622397+0000 front 2023-12-16T21:39:05.622556+0000 (oldest deadline 2023-12-16T21:39:30.322546+0000) 2023-12-16T21:41:18.566 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:41:18 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:41:18.267+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:05.622506+0000 front 2023-12-16T21:39:05.622528+0000 (oldest deadline 2023-12-16T21:39:30.322546+0000) 2023-12-16T21:41:18.566 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:41:18 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:41:18.267+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:05.622479+0000 front 2023-12-16T21:39:05.622580+0000 (oldest deadline 2023-12-16T21:39:30.322546+0000) 2023-12-16T21:41:19.066 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:41:18 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:41:18.609+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:09.306583+0000 front 2023-12-16T21:39:09.306633+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:41:19.066 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:41:18 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:41:18.609+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:09.306551+0000 front 2023-12-16T21:39:09.306652+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:41:19.066 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:41:18 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:41:18.609+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:09.306605+0000 front 2023-12-16T21:39:09.306613+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:41:19.067 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:41:18 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:41:18.783+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:09.828911+0000 front 2023-12-16T21:39:09.828958+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:41:19.067 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:41:18 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:41:18.783+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:09.828868+0000 front 2023-12-16T21:39:09.828878+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:41:19.067 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:41:18 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:41:18.783+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:09.828947+0000 front 2023-12-16T21:39:09.828993+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:41:19.566 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:41:19 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:41:19.292+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:05.622397+0000 front 2023-12-16T21:39:05.622556+0000 (oldest deadline 2023-12-16T21:39:30.322546+0000) 2023-12-16T21:41:19.566 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:41:19 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:41:19.292+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:05.622506+0000 front 2023-12-16T21:39:05.622528+0000 (oldest deadline 2023-12-16T21:39:30.322546+0000) 2023-12-16T21:41:19.566 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:41:19 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:41:19.292+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:05.622479+0000 front 2023-12-16T21:39:05.622580+0000 (oldest deadline 2023-12-16T21:39:30.322546+0000) 2023-12-16T21:41:20.066 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:41:19 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:41:19.657+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:09.306583+0000 front 2023-12-16T21:39:09.306633+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:41:20.066 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:41:19 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:41:19.657+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:09.306551+0000 front 2023-12-16T21:39:09.306652+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:41:20.066 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:41:19 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:41:19.657+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:09.306605+0000 front 2023-12-16T21:39:09.306613+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:41:20.067 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:41:19 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:41:19.748+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:09.828911+0000 front 2023-12-16T21:39:09.828958+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:41:20.067 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:41:19 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:41:19.748+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:09.828868+0000 front 2023-12-16T21:39:09.828878+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:41:20.067 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:41:19 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:41:19.748+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:09.828947+0000 front 2023-12-16T21:39:09.828993+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:41:20.566 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:41:20 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:41:20.309+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:05.622397+0000 front 2023-12-16T21:39:05.622556+0000 (oldest deadline 2023-12-16T21:39:30.322546+0000) 2023-12-16T21:41:20.566 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:41:20 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:41:20.309+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:05.622506+0000 front 2023-12-16T21:39:05.622528+0000 (oldest deadline 2023-12-16T21:39:30.322546+0000) 2023-12-16T21:41:20.566 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:41:20 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:41:20.309+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:05.622479+0000 front 2023-12-16T21:39:05.622580+0000 (oldest deadline 2023-12-16T21:39:30.322546+0000) 2023-12-16T21:41:21.066 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:41:20 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:41:20.723+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:09.828911+0000 front 2023-12-16T21:39:09.828958+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:41:21.067 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:41:20 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:41:20.723+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:09.828868+0000 front 2023-12-16T21:39:09.828878+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:41:21.067 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:41:20 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:41:20.723+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:09.828947+0000 front 2023-12-16T21:39:09.828993+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:41:21.067 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:41:20 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:41:20.663+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:09.306583+0000 front 2023-12-16T21:39:09.306633+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:41:21.067 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:41:20 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:41:20.663+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:09.306551+0000 front 2023-12-16T21:39:09.306652+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:41:21.067 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:41:20 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:41:20.663+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:09.306605+0000 front 2023-12-16T21:39:09.306613+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:41:21.566 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Dec 16 21:41:21 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-mon-smithi045[80759]: 2023-12-16T21:41:21.329+0000 7f45a1f8f700 -1 mon.smithi045@0(probing) e2 get_health_metrics reporting 18 slow ops, oldest is monmgrreport(gid 14214, 0 checks, 1 progress events) 2023-12-16T21:41:21.566 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:41:21 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:41:21.302+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:05.622397+0000 front 2023-12-16T21:39:05.622556+0000 (oldest deadline 2023-12-16T21:39:30.322546+0000) 2023-12-16T21:41:21.566 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:41:21 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:41:21.302+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:05.622506+0000 front 2023-12-16T21:39:05.622528+0000 (oldest deadline 2023-12-16T21:39:30.322546+0000) 2023-12-16T21:41:21.566 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:41:21 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:41:21.302+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:05.622479+0000 front 2023-12-16T21:39:05.622580+0000 (oldest deadline 2023-12-16T21:39:30.322546+0000) 2023-12-16T21:41:22.066 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:41:21 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:41:21.706+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:09.828911+0000 front 2023-12-16T21:39:09.828958+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:41:22.066 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:41:21 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:41:21.706+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:09.828868+0000 front 2023-12-16T21:39:09.828878+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:41:22.066 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:41:21 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:41:21.706+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:09.828947+0000 front 2023-12-16T21:39:09.828993+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:41:22.067 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:41:21 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:41:21.697+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:09.306583+0000 front 2023-12-16T21:39:09.306633+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:41:22.067 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:41:21 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:41:21.698+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:09.306551+0000 front 2023-12-16T21:39:09.306652+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:41:22.067 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:41:21 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:41:21.698+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:09.306605+0000 front 2023-12-16T21:39:09.306613+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:41:22.566 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:41:22 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:41:22.312+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:05.622397+0000 front 2023-12-16T21:39:05.622556+0000 (oldest deadline 2023-12-16T21:39:30.322546+0000) 2023-12-16T21:41:22.566 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:41:22 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:41:22.312+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:05.622506+0000 front 2023-12-16T21:39:05.622528+0000 (oldest deadline 2023-12-16T21:39:30.322546+0000) 2023-12-16T21:41:22.566 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:41:22 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:41:22.313+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:05.622479+0000 front 2023-12-16T21:39:05.622580+0000 (oldest deadline 2023-12-16T21:39:30.322546+0000) 2023-12-16T21:41:23.067 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:41:22 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:41:22.719+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:09.828911+0000 front 2023-12-16T21:39:09.828958+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:41:23.067 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:41:22 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:41:22.719+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:09.828868+0000 front 2023-12-16T21:39:09.828878+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:41:23.067 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:41:22 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:41:22.719+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:09.828947+0000 front 2023-12-16T21:39:09.828993+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:41:23.067 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:41:22 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:41:22.694+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:09.306583+0000 front 2023-12-16T21:39:09.306633+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:41:23.067 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:41:22 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:41:22.694+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:09.306551+0000 front 2023-12-16T21:39:09.306652+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:41:23.067 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:41:22 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:41:22.694+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:09.306605+0000 front 2023-12-16T21:39:09.306613+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:41:23.705 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:41:23 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:41:23.320+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:05.622397+0000 front 2023-12-16T21:39:05.622556+0000 (oldest deadline 2023-12-16T21:39:30.322546+0000) 2023-12-16T21:41:23.705 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:41:23 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:41:23.320+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:05.622506+0000 front 2023-12-16T21:39:05.622528+0000 (oldest deadline 2023-12-16T21:39:30.322546+0000) 2023-12-16T21:41:23.705 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:41:23 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:41:23.320+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:05.622479+0000 front 2023-12-16T21:39:05.622580+0000 (oldest deadline 2023-12-16T21:39:30.322546+0000) 2023-12-16T21:41:24.066 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:41:23 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:41:23.715+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:09.306583+0000 front 2023-12-16T21:39:09.306633+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:41:24.066 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:41:23 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:41:23.715+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:09.306551+0000 front 2023-12-16T21:39:09.306652+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:41:24.067 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:41:23 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:41:23.715+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:09.306605+0000 front 2023-12-16T21:39:09.306613+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:41:24.067 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:41:23 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:41:23.702+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:09.828911+0000 front 2023-12-16T21:39:09.828958+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:41:24.067 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:41:23 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:41:23.702+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:09.828868+0000 front 2023-12-16T21:39:09.828878+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:41:24.067 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:41:23 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:41:23.702+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:09.828947+0000 front 2023-12-16T21:39:09.828993+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:41:24.566 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:41:24 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:41:24.275+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:05.622397+0000 front 2023-12-16T21:39:05.622556+0000 (oldest deadline 2023-12-16T21:39:30.322546+0000) 2023-12-16T21:41:24.566 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:41:24 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:41:24.275+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:05.622506+0000 front 2023-12-16T21:39:05.622528+0000 (oldest deadline 2023-12-16T21:39:30.322546+0000) 2023-12-16T21:41:24.567 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:41:24 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:41:24.275+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:05.622479+0000 front 2023-12-16T21:39:05.622580+0000 (oldest deadline 2023-12-16T21:39:30.322546+0000) 2023-12-16T21:41:25.066 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:41:24 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:41:24.670+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:09.306583+0000 front 2023-12-16T21:39:09.306633+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:41:25.066 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:41:24 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:41:24.670+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:09.306551+0000 front 2023-12-16T21:39:09.306652+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:41:25.066 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:41:24 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:41:24.670+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:09.306605+0000 front 2023-12-16T21:39:09.306613+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:41:25.067 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:41:24 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:41:24.746+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:09.828911+0000 front 2023-12-16T21:39:09.828958+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:41:25.067 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:41:24 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:41:24.746+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:09.828868+0000 front 2023-12-16T21:39:09.828878+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:41:25.067 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:41:24 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:41:24.746+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:09.828947+0000 front 2023-12-16T21:39:09.828993+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:41:25.566 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:41:25 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:41:25.289+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:05.622397+0000 front 2023-12-16T21:39:05.622556+0000 (oldest deadline 2023-12-16T21:39:30.322546+0000) 2023-12-16T21:41:25.566 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:41:25 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:41:25.289+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:05.622506+0000 front 2023-12-16T21:39:05.622528+0000 (oldest deadline 2023-12-16T21:39:30.322546+0000) 2023-12-16T21:41:25.566 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:41:25 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:41:25.289+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:05.622479+0000 front 2023-12-16T21:39:05.622580+0000 (oldest deadline 2023-12-16T21:39:30.322546+0000) 2023-12-16T21:41:26.066 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:41:25 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:41:25.703+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:09.306583+0000 front 2023-12-16T21:39:09.306633+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:41:26.066 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:41:25 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:41:25.703+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:09.306551+0000 front 2023-12-16T21:39:09.306652+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:41:26.066 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:41:25 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:41:25.703+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:09.306605+0000 front 2023-12-16T21:39:09.306613+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:41:26.067 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:41:25 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:41:25.713+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:09.828911+0000 front 2023-12-16T21:39:09.828958+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:41:26.067 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:41:25 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:41:25.713+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:09.828868+0000 front 2023-12-16T21:39:09.828878+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:41:26.067 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:41:25 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:41:25.713+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:09.828947+0000 front 2023-12-16T21:39:09.828993+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:41:26.566 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Dec 16 21:41:26 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-mon-smithi045[80759]: 2023-12-16T21:41:26.330+0000 7f45a1f8f700 -1 mon.smithi045@0(probing) e2 get_health_metrics reporting 27 slow ops, oldest is monmgrreport(gid 14214, 0 checks, 1 progress events) 2023-12-16T21:41:26.566 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:41:26 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:41:26.294+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:05.622397+0000 front 2023-12-16T21:39:05.622556+0000 (oldest deadline 2023-12-16T21:39:30.322546+0000) 2023-12-16T21:41:26.566 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:41:26 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:41:26.294+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:05.622506+0000 front 2023-12-16T21:39:05.622528+0000 (oldest deadline 2023-12-16T21:39:30.322546+0000) 2023-12-16T21:41:26.567 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:41:26 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:41:26.294+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:05.622479+0000 front 2023-12-16T21:39:05.622580+0000 (oldest deadline 2023-12-16T21:39:30.322546+0000) 2023-12-16T21:41:27.066 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:41:26 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:41:26.725+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:09.306583+0000 front 2023-12-16T21:39:09.306633+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:41:27.066 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:41:26 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:41:26.725+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:09.306551+0000 front 2023-12-16T21:39:09.306652+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:41:27.067 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:41:26 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:41:26.725+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:09.306605+0000 front 2023-12-16T21:39:09.306613+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:41:27.067 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:41:26 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:41:26.684+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:09.828911+0000 front 2023-12-16T21:39:09.828958+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:41:27.067 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:41:26 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:41:26.684+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:09.828868+0000 front 2023-12-16T21:39:09.828878+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:41:27.067 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:41:26 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:41:26.684+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:09.828947+0000 front 2023-12-16T21:39:09.828993+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:41:27.566 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:41:27 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:41:27.276+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:05.622397+0000 front 2023-12-16T21:39:05.622556+0000 (oldest deadline 2023-12-16T21:39:30.322546+0000) 2023-12-16T21:41:27.566 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:41:27 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:41:27.276+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:05.622506+0000 front 2023-12-16T21:39:05.622528+0000 (oldest deadline 2023-12-16T21:39:30.322546+0000) 2023-12-16T21:41:27.566 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:41:27 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:41:27.276+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:05.622479+0000 front 2023-12-16T21:39:05.622580+0000 (oldest deadline 2023-12-16T21:39:30.322546+0000) 2023-12-16T21:41:28.066 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:41:27 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:41:27.717+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:09.306583+0000 front 2023-12-16T21:39:09.306633+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:41:28.067 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:41:27 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:41:27.717+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:09.306551+0000 front 2023-12-16T21:39:09.306652+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:41:28.067 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:41:27 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:41:27.717+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:09.306605+0000 front 2023-12-16T21:39:09.306613+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:41:28.067 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:41:27 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:41:27.689+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:09.828911+0000 front 2023-12-16T21:39:09.828958+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:41:28.067 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:41:27 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:41:27.689+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:09.828868+0000 front 2023-12-16T21:39:09.828878+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:41:28.067 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:41:27 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:41:27.689+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:09.828947+0000 front 2023-12-16T21:39:09.828993+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:41:28.722 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:41:28 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:41:28.319+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:05.622397+0000 front 2023-12-16T21:39:05.622556+0000 (oldest deadline 2023-12-16T21:39:30.322546+0000) 2023-12-16T21:41:28.722 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:41:28 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:41:28.319+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:05.622506+0000 front 2023-12-16T21:39:05.622528+0000 (oldest deadline 2023-12-16T21:39:30.322546+0000) 2023-12-16T21:41:28.722 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:41:28 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:41:28.319+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:05.622479+0000 front 2023-12-16T21:39:05.622580+0000 (oldest deadline 2023-12-16T21:39:30.322546+0000) 2023-12-16T21:41:29.066 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:41:28 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:41:28.721+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:09.306583+0000 front 2023-12-16T21:39:09.306633+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:41:29.066 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:41:28 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:41:28.721+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:09.306551+0000 front 2023-12-16T21:39:09.306652+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:41:29.066 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:41:28 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:41:28.721+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:09.306605+0000 front 2023-12-16T21:39:09.306613+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:41:29.067 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:41:28 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:41:28.734+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:09.828911+0000 front 2023-12-16T21:39:09.828958+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:41:29.067 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:41:28 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:41:28.734+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:09.828868+0000 front 2023-12-16T21:39:09.828878+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:41:29.067 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:41:28 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:41:28.734+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:09.828947+0000 front 2023-12-16T21:39:09.828993+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:41:29.566 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:41:29 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:41:29.299+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:05.622397+0000 front 2023-12-16T21:39:05.622556+0000 (oldest deadline 2023-12-16T21:39:30.322546+0000) 2023-12-16T21:41:29.566 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:41:29 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:41:29.299+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:05.622506+0000 front 2023-12-16T21:39:05.622528+0000 (oldest deadline 2023-12-16T21:39:30.322546+0000) 2023-12-16T21:41:29.566 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:41:29 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:41:29.299+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:05.622479+0000 front 2023-12-16T21:39:05.622580+0000 (oldest deadline 2023-12-16T21:39:30.322546+0000) 2023-12-16T21:41:30.066 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:41:29 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:41:29.697+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:09.306583+0000 front 2023-12-16T21:39:09.306633+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:41:30.067 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:41:29 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:41:29.697+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:09.306551+0000 front 2023-12-16T21:39:09.306652+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:41:30.067 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:41:29 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:41:29.697+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:09.306605+0000 front 2023-12-16T21:39:09.306613+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:41:30.067 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:41:29 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:41:29.765+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:09.828911+0000 front 2023-12-16T21:39:09.828958+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:41:30.067 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:41:29 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:41:29.765+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:09.828868+0000 front 2023-12-16T21:39:09.828878+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:41:30.067 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:41:29 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:41:29.765+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:09.828947+0000 front 2023-12-16T21:39:09.828993+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:41:30.746 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:41:30 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:41:30.316+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:05.622397+0000 front 2023-12-16T21:39:05.622556+0000 (oldest deadline 2023-12-16T21:39:30.322546+0000) 2023-12-16T21:41:30.746 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:41:30 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:41:30.316+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:05.622506+0000 front 2023-12-16T21:39:05.622528+0000 (oldest deadline 2023-12-16T21:39:30.322546+0000) 2023-12-16T21:41:30.746 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:41:30 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:41:30.316+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:05.622479+0000 front 2023-12-16T21:39:05.622580+0000 (oldest deadline 2023-12-16T21:39:30.322546+0000) 2023-12-16T21:41:31.066 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:41:30 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:41:30.743+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:09.306583+0000 front 2023-12-16T21:39:09.306633+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:41:31.067 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:41:30 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:41:30.743+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:09.306551+0000 front 2023-12-16T21:39:09.306652+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:41:31.067 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:41:30 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:41:30.743+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:09.306605+0000 front 2023-12-16T21:39:09.306613+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:41:31.067 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:41:30 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:41:30.743+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:09.828911+0000 front 2023-12-16T21:39:09.828958+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:41:31.067 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:41:30 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:41:30.743+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:09.828868+0000 front 2023-12-16T21:39:09.828878+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:41:31.067 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:41:30 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:41:30.743+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:09.828947+0000 front 2023-12-16T21:39:09.828993+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:41:31.566 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Dec 16 21:41:31 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-mon-smithi045[80759]: 2023-12-16T21:41:31.330+0000 7f45a1f8f700 -1 mon.smithi045@0(probing) e2 get_health_metrics reporting 53 slow ops, oldest is monmgrreport(gid 14214, 0 checks, 1 progress events) 2023-12-16T21:41:31.567 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:41:31 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:41:31.284+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:05.622397+0000 front 2023-12-16T21:39:05.622556+0000 (oldest deadline 2023-12-16T21:39:30.322546+0000) 2023-12-16T21:41:31.567 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:41:31 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:41:31.284+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:05.622506+0000 front 2023-12-16T21:39:05.622528+0000 (oldest deadline 2023-12-16T21:39:30.322546+0000) 2023-12-16T21:41:31.567 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:41:31 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:41:31.284+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:05.622479+0000 front 2023-12-16T21:39:05.622580+0000 (oldest deadline 2023-12-16T21:39:30.322546+0000) 2023-12-16T21:41:32.066 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:41:31 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:41:31.765+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:09.306583+0000 front 2023-12-16T21:39:09.306633+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:41:32.066 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:41:31 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:41:31.765+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:09.306551+0000 front 2023-12-16T21:39:09.306652+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:41:32.066 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:41:31 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:41:31.765+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:09.306605+0000 front 2023-12-16T21:39:09.306613+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:41:32.067 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:41:31 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:41:31.723+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:09.828911+0000 front 2023-12-16T21:39:09.828958+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:41:32.067 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:41:31 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:41:31.723+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:09.828868+0000 front 2023-12-16T21:39:09.828878+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:41:32.067 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:41:31 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:41:31.723+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:09.828947+0000 front 2023-12-16T21:39:09.828993+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:41:32.566 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:41:32 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:41:32.296+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:05.622397+0000 front 2023-12-16T21:39:05.622556+0000 (oldest deadline 2023-12-16T21:39:30.322546+0000) 2023-12-16T21:41:32.566 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:41:32 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:41:32.296+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:05.622506+0000 front 2023-12-16T21:39:05.622528+0000 (oldest deadline 2023-12-16T21:39:30.322546+0000) 2023-12-16T21:41:32.566 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:41:32 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:41:32.296+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:05.622479+0000 front 2023-12-16T21:39:05.622580+0000 (oldest deadline 2023-12-16T21:39:30.322546+0000) 2023-12-16T21:41:33.066 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:41:32 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:41:32.735+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:09.828911+0000 front 2023-12-16T21:39:09.828958+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:41:33.067 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:41:32 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:41:32.735+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:09.828868+0000 front 2023-12-16T21:39:09.828878+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:41:33.067 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:41:32 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:41:32.735+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:09.828947+0000 front 2023-12-16T21:39:09.828993+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:41:33.067 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:41:32 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:41:32.797+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:09.306583+0000 front 2023-12-16T21:39:09.306633+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:41:33.067 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:41:32 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:41:32.797+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:09.306551+0000 front 2023-12-16T21:39:09.306652+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:41:33.067 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:41:32 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:41:32.797+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:09.306605+0000 front 2023-12-16T21:39:09.306613+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:41:33.585 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:41:33 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:41:33.328+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:05.622397+0000 front 2023-12-16T21:39:05.622556+0000 (oldest deadline 2023-12-16T21:39:30.322546+0000) 2023-12-16T21:41:33.585 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:41:33 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:41:33.328+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:05.622506+0000 front 2023-12-16T21:39:05.622528+0000 (oldest deadline 2023-12-16T21:39:30.322546+0000) 2023-12-16T21:41:33.585 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:41:33 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:41:33.328+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:05.622479+0000 front 2023-12-16T21:39:05.622580+0000 (oldest deadline 2023-12-16T21:39:30.322546+0000) 2023-12-16T21:41:34.066 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:41:33 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:41:33.823+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:09.306583+0000 front 2023-12-16T21:39:09.306633+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:41:34.066 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:41:33 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:41:33.823+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:09.306551+0000 front 2023-12-16T21:39:09.306652+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:41:34.066 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:41:33 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:41:33.823+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:09.306605+0000 front 2023-12-16T21:39:09.306613+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:41:34.067 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:41:33 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:41:33.735+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:09.828911+0000 front 2023-12-16T21:39:09.828958+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:41:34.067 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:41:33 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:41:33.735+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:09.828868+0000 front 2023-12-16T21:39:09.828878+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:41:34.067 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:41:33 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:41:33.735+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:09.828947+0000 front 2023-12-16T21:39:09.828993+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:41:34.566 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:41:34 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:41:34.299+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:05.622397+0000 front 2023-12-16T21:39:05.622556+0000 (oldest deadline 2023-12-16T21:39:30.322546+0000) 2023-12-16T21:41:34.566 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:41:34 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:41:34.300+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:05.622506+0000 front 2023-12-16T21:39:05.622528+0000 (oldest deadline 2023-12-16T21:39:30.322546+0000) 2023-12-16T21:41:34.566 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:41:34 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:41:34.300+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:05.622479+0000 front 2023-12-16T21:39:05.622580+0000 (oldest deadline 2023-12-16T21:39:30.322546+0000) 2023-12-16T21:41:35.066 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:41:34 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:41:34.780+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:09.306583+0000 front 2023-12-16T21:39:09.306633+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:41:35.066 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:41:34 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:41:34.780+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:09.306551+0000 front 2023-12-16T21:39:09.306652+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:41:35.067 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:41:34 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:41:34.780+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:09.306605+0000 front 2023-12-16T21:39:09.306613+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:41:35.067 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:41:34 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:41:34.719+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:09.828911+0000 front 2023-12-16T21:39:09.828958+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:41:35.067 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:41:34 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:41:34.719+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:09.828868+0000 front 2023-12-16T21:39:09.828878+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:41:35.067 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:41:34 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:41:34.719+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:09.828947+0000 front 2023-12-16T21:39:09.828993+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:41:35.566 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:41:35 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:41:35.253+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:05.622397+0000 front 2023-12-16T21:39:05.622556+0000 (oldest deadline 2023-12-16T21:39:30.322546+0000) 2023-12-16T21:41:35.566 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:41:35 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:41:35.253+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:05.622506+0000 front 2023-12-16T21:39:05.622528+0000 (oldest deadline 2023-12-16T21:39:30.322546+0000) 2023-12-16T21:41:35.566 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:41:35 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:41:35.253+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:05.622479+0000 front 2023-12-16T21:39:05.622580+0000 (oldest deadline 2023-12-16T21:39:30.322546+0000) 2023-12-16T21:41:36.066 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:41:35 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:41:35.830+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:09.306583+0000 front 2023-12-16T21:39:09.306633+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:41:36.066 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:41:35 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:41:35.830+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:09.306551+0000 front 2023-12-16T21:39:09.306652+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:41:36.066 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:41:35 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:41:35.830+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:09.306605+0000 front 2023-12-16T21:39:09.306613+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:41:36.067 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:41:35 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:41:35.748+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:09.828911+0000 front 2023-12-16T21:39:09.828958+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:41:36.067 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:41:35 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:41:35.748+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:09.828868+0000 front 2023-12-16T21:39:09.828878+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:41:36.067 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:41:35 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:41:35.748+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:09.828947+0000 front 2023-12-16T21:39:09.828993+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:41:36.566 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Dec 16 21:41:36 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-mon-smithi045[80759]: 2023-12-16T21:41:36.331+0000 7f45a1f8f700 -1 mon.smithi045@0(probing) e2 get_health_metrics reporting 80 slow ops, oldest is monmgrreport(gid 14214, 0 checks, 1 progress events) 2023-12-16T21:41:36.566 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:41:36 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:41:36.208+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:05.622397+0000 front 2023-12-16T21:39:05.622556+0000 (oldest deadline 2023-12-16T21:39:30.322546+0000) 2023-12-16T21:41:36.566 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:41:36 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:41:36.208+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:05.622506+0000 front 2023-12-16T21:39:05.622528+0000 (oldest deadline 2023-12-16T21:39:30.322546+0000) 2023-12-16T21:41:36.567 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:41:36 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:41:36.208+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:05.622479+0000 front 2023-12-16T21:39:05.622580+0000 (oldest deadline 2023-12-16T21:39:30.322546+0000) 2023-12-16T21:41:37.066 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:41:36 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:41:36.828+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:09.306583+0000 front 2023-12-16T21:39:09.306633+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:41:37.066 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:41:36 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:41:36.828+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:09.306551+0000 front 2023-12-16T21:39:09.306652+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:41:37.066 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:41:36 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:41:36.828+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:09.306605+0000 front 2023-12-16T21:39:09.306613+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:41:37.067 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:41:36 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:41:36.742+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:09.828911+0000 front 2023-12-16T21:39:09.828958+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:41:37.067 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:41:36 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:41:36.742+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:09.828868+0000 front 2023-12-16T21:39:09.828878+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:41:37.067 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:41:36 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:41:36.742+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:09.828947+0000 front 2023-12-16T21:39:09.828993+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:41:37.566 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:41:37 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:41:37.192+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:05.622397+0000 front 2023-12-16T21:39:05.622556+0000 (oldest deadline 2023-12-16T21:39:30.322546+0000) 2023-12-16T21:41:37.566 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:41:37 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:41:37.192+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:05.622506+0000 front 2023-12-16T21:39:05.622528+0000 (oldest deadline 2023-12-16T21:39:30.322546+0000) 2023-12-16T21:41:37.566 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:41:37 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:41:37.192+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:05.622479+0000 front 2023-12-16T21:39:05.622580+0000 (oldest deadline 2023-12-16T21:39:30.322546+0000) 2023-12-16T21:41:38.066 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:41:37 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:41:37.771+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:09.828911+0000 front 2023-12-16T21:39:09.828958+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:41:38.066 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:41:37 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:41:37.771+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:09.828868+0000 front 2023-12-16T21:39:09.828878+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:41:38.066 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:41:37 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:41:37.771+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:09.828947+0000 front 2023-12-16T21:39:09.828993+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:41:38.067 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:41:37 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:41:37.829+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:09.306583+0000 front 2023-12-16T21:39:09.306633+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:41:38.067 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:41:37 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:41:37.829+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:09.306551+0000 front 2023-12-16T21:39:09.306652+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:41:38.067 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:41:37 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:41:37.829+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:09.306605+0000 front 2023-12-16T21:39:09.306613+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:41:38.566 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:41:38 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:41:38.155+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:05.622397+0000 front 2023-12-16T21:39:05.622556+0000 (oldest deadline 2023-12-16T21:39:30.322546+0000) 2023-12-16T21:41:38.566 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:41:38 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:41:38.155+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:05.622506+0000 front 2023-12-16T21:39:05.622528+0000 (oldest deadline 2023-12-16T21:39:30.322546+0000) 2023-12-16T21:41:38.566 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:41:38 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:41:38.155+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:05.622479+0000 front 2023-12-16T21:39:05.622580+0000 (oldest deadline 2023-12-16T21:39:30.322546+0000) 2023-12-16T21:41:39.066 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:41:38 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:41:38.763+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:09.828911+0000 front 2023-12-16T21:39:09.828958+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:41:39.066 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:41:38 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:41:38.763+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:09.828868+0000 front 2023-12-16T21:39:09.828878+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:41:39.066 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:41:38 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:41:38.763+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:09.828947+0000 front 2023-12-16T21:39:09.828993+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:41:39.066 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:41:38 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:41:38.841+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:09.306583+0000 front 2023-12-16T21:39:09.306633+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:41:39.066 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:41:38 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:41:38.841+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:09.306551+0000 front 2023-12-16T21:39:09.306652+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:41:39.067 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:41:38 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:41:38.841+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:09.306605+0000 front 2023-12-16T21:39:09.306613+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:41:39.566 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:41:39 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:41:39.205+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:05.622397+0000 front 2023-12-16T21:39:05.622556+0000 (oldest deadline 2023-12-16T21:39:30.322546+0000) 2023-12-16T21:41:39.567 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:41:39 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:41:39.205+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:05.622506+0000 front 2023-12-16T21:39:05.622528+0000 (oldest deadline 2023-12-16T21:39:30.322546+0000) 2023-12-16T21:41:39.567 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:41:39 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:41:39.205+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:05.622479+0000 front 2023-12-16T21:39:05.622580+0000 (oldest deadline 2023-12-16T21:39:30.322546+0000) 2023-12-16T21:41:40.066 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:41:39 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:41:39.817+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:09.306583+0000 front 2023-12-16T21:39:09.306633+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:41:40.066 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:41:39 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:41:39.817+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:09.306551+0000 front 2023-12-16T21:39:09.306652+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:41:40.066 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:41:39 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:41:39.817+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:09.306605+0000 front 2023-12-16T21:39:09.306613+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:41:40.067 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:41:39 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:41:39.738+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:09.828911+0000 front 2023-12-16T21:39:09.828958+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:41:40.067 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:41:39 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:41:39.738+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:09.828868+0000 front 2023-12-16T21:39:09.828878+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:41:40.067 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:41:39 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:41:39.738+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:09.828947+0000 front 2023-12-16T21:39:09.828993+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:41:40.566 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:41:40 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:41:40.181+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:05.622397+0000 front 2023-12-16T21:39:05.622556+0000 (oldest deadline 2023-12-16T21:39:30.322546+0000) 2023-12-16T21:41:40.566 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:41:40 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:41:40.181+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:05.622506+0000 front 2023-12-16T21:39:05.622528+0000 (oldest deadline 2023-12-16T21:39:30.322546+0000) 2023-12-16T21:41:40.566 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:41:40 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:41:40.181+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:05.622479+0000 front 2023-12-16T21:39:05.622580+0000 (oldest deadline 2023-12-16T21:39:30.322546+0000) 2023-12-16T21:41:41.066 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:41:40 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:41:40.840+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:09.306583+0000 front 2023-12-16T21:39:09.306633+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:41:41.066 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:41:40 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:41:40.840+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:09.306551+0000 front 2023-12-16T21:39:09.306652+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:41:41.066 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:41:40 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:41:40.840+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:09.306605+0000 front 2023-12-16T21:39:09.306613+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:41:41.067 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:41:40 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:41:40.734+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:09.828911+0000 front 2023-12-16T21:39:09.828958+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:41:41.067 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:41:40 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:41:40.734+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:09.828868+0000 front 2023-12-16T21:39:09.828878+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:41:41.067 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:41:40 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:41:40.734+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:09.828947+0000 front 2023-12-16T21:39:09.828993+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:41:41.456 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Dec 16 21:41:41 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-mon-smithi045[80759]: 2023-12-16T21:41:41.331+0000 7f45a1f8f700 -1 mon.smithi045@0(probing) e2 get_health_metrics reporting 113 slow ops, oldest is monmgrreport(gid 14214, 0 checks, 1 progress events) 2023-12-16T21:41:41.456 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:41:41 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:41:41.194+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:05.622397+0000 front 2023-12-16T21:39:05.622556+0000 (oldest deadline 2023-12-16T21:39:30.322546+0000) 2023-12-16T21:41:41.456 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:41:41 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:41:41.194+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:05.622506+0000 front 2023-12-16T21:39:05.622528+0000 (oldest deadline 2023-12-16T21:39:30.322546+0000) 2023-12-16T21:41:41.456 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:41:41 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:41:41.194+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:05.622479+0000 front 2023-12-16T21:39:05.622580+0000 (oldest deadline 2023-12-16T21:39:30.322546+0000) 2023-12-16T21:41:42.066 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:41:41 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:41:41.715+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:09.828911+0000 front 2023-12-16T21:39:09.828958+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:41:42.066 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:41:41 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:41:41.715+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:09.828868+0000 front 2023-12-16T21:39:09.828878+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:41:42.066 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:41:41 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:41:41.715+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:09.828947+0000 front 2023-12-16T21:39:09.828993+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:41:42.067 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:41:41 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:41:41.881+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:09.306583+0000 front 2023-12-16T21:39:09.306633+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:41:42.067 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:41:41 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:41:41.881+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:09.306551+0000 front 2023-12-16T21:39:09.306652+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:41:42.067 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:41:41 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:41:41.881+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:09.306605+0000 front 2023-12-16T21:39:09.306613+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:41:42.566 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:41:42 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:41:42.229+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:05.622397+0000 front 2023-12-16T21:39:05.622556+0000 (oldest deadline 2023-12-16T21:39:30.322546+0000) 2023-12-16T21:41:42.566 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:41:42 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:41:42.229+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:05.622506+0000 front 2023-12-16T21:39:05.622528+0000 (oldest deadline 2023-12-16T21:39:30.322546+0000) 2023-12-16T21:41:42.566 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:41:42 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:41:42.229+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:05.622479+0000 front 2023-12-16T21:39:05.622580+0000 (oldest deadline 2023-12-16T21:39:30.322546+0000) 2023-12-16T21:41:43.066 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:41:42 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:41:42.851+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:09.306583+0000 front 2023-12-16T21:39:09.306633+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:41:43.066 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:41:42 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:41:42.851+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:09.306551+0000 front 2023-12-16T21:39:09.306652+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:41:43.066 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:41:42 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:41:42.851+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:09.306605+0000 front 2023-12-16T21:39:09.306613+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:41:43.067 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:41:42 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:41:42.694+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:09.828911+0000 front 2023-12-16T21:39:09.828958+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:41:43.067 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:41:42 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:41:42.694+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:09.828868+0000 front 2023-12-16T21:39:09.828878+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:41:43.067 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:41:42 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:41:42.694+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:09.828947+0000 front 2023-12-16T21:39:09.828993+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:41:43.522 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:41:43 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:41:43.242+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:05.622397+0000 front 2023-12-16T21:39:05.622556+0000 (oldest deadline 2023-12-16T21:39:30.322546+0000) 2023-12-16T21:41:43.522 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:41:43 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:41:43.242+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:05.622506+0000 front 2023-12-16T21:39:05.622528+0000 (oldest deadline 2023-12-16T21:39:30.322546+0000) 2023-12-16T21:41:43.522 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:41:43 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:41:43.242+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:05.622479+0000 front 2023-12-16T21:39:05.622580+0000 (oldest deadline 2023-12-16T21:39:30.322546+0000) 2023-12-16T21:41:43.816 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:41:43 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:41:43.734+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:09.828911+0000 front 2023-12-16T21:39:09.828958+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:41:43.816 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:41:43 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:41:43.734+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:09.828868+0000 front 2023-12-16T21:39:09.828878+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:41:43.816 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:41:43 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:41:43.734+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:09.828947+0000 front 2023-12-16T21:39:09.828993+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:41:44.221 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:41:43 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:41:43.848+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:09.306583+0000 front 2023-12-16T21:39:09.306633+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:41:44.221 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:41:43 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:41:43.848+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:09.306551+0000 front 2023-12-16T21:39:09.306652+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:41:44.221 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:41:43 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:41:43.848+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:09.306605+0000 front 2023-12-16T21:39:09.306613+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:41:44.566 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:41:44 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:41:44.218+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:05.622397+0000 front 2023-12-16T21:39:05.622556+0000 (oldest deadline 2023-12-16T21:39:30.322546+0000) 2023-12-16T21:41:44.566 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:41:44 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:41:44.218+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:05.622506+0000 front 2023-12-16T21:39:05.622528+0000 (oldest deadline 2023-12-16T21:39:30.322546+0000) 2023-12-16T21:41:44.566 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:41:44 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:41:44.218+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:05.622479+0000 front 2023-12-16T21:39:05.622580+0000 (oldest deadline 2023-12-16T21:39:30.322546+0000) 2023-12-16T21:41:45.066 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:41:44 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:41:44.885+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:09.306583+0000 front 2023-12-16T21:39:09.306633+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:41:45.066 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:41:44 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:41:44.885+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:09.306551+0000 front 2023-12-16T21:39:09.306652+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:41:45.066 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:41:44 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:41:44.885+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:09.306605+0000 front 2023-12-16T21:39:09.306613+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:41:45.067 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:41:44 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:41:44.760+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:09.828911+0000 front 2023-12-16T21:39:09.828958+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:41:45.067 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:41:44 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:41:44.760+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:09.828868+0000 front 2023-12-16T21:39:09.828878+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:41:45.067 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:41:44 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:41:44.760+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:09.828947+0000 front 2023-12-16T21:39:09.828993+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:41:45.566 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:41:45 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:41:45.182+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:05.622397+0000 front 2023-12-16T21:39:05.622556+0000 (oldest deadline 2023-12-16T21:39:30.322546+0000) 2023-12-16T21:41:45.566 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:41:45 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:41:45.182+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:05.622506+0000 front 2023-12-16T21:39:05.622528+0000 (oldest deadline 2023-12-16T21:39:30.322546+0000) 2023-12-16T21:41:45.566 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:41:45 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:41:45.182+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:05.622479+0000 front 2023-12-16T21:39:05.622580+0000 (oldest deadline 2023-12-16T21:39:30.322546+0000) 2023-12-16T21:41:46.066 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:41:45 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:41:45.927+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:09.306583+0000 front 2023-12-16T21:39:09.306633+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:41:46.066 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:41:45 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:41:45.927+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:09.306551+0000 front 2023-12-16T21:39:09.306652+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:41:46.066 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:41:45 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:41:45.927+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:09.306605+0000 front 2023-12-16T21:39:09.306613+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:41:46.067 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:41:45 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:41:45.796+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:09.828911+0000 front 2023-12-16T21:39:09.828958+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:41:46.067 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:41:45 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:41:45.796+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:09.828868+0000 front 2023-12-16T21:39:09.828878+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:41:46.067 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:41:45 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:41:45.796+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:09.828947+0000 front 2023-12-16T21:39:09.828993+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:41:46.457 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Dec 16 21:41:46 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-mon-smithi045[80759]: 2023-12-16T21:41:46.331+0000 7f45a1f8f700 -1 mon.smithi045@0(probing) e2 get_health_metrics reporting 141 slow ops, oldest is monmgrreport(gid 14214, 0 checks, 1 progress events) 2023-12-16T21:41:46.457 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:41:46 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:41:46.175+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:05.622397+0000 front 2023-12-16T21:39:05.622556+0000 (oldest deadline 2023-12-16T21:39:30.322546+0000) 2023-12-16T21:41:46.457 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:41:46 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:41:46.175+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:05.622506+0000 front 2023-12-16T21:39:05.622528+0000 (oldest deadline 2023-12-16T21:39:30.322546+0000) 2023-12-16T21:41:46.457 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:41:46 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:41:46.175+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:05.622479+0000 front 2023-12-16T21:39:05.622580+0000 (oldest deadline 2023-12-16T21:39:30.322546+0000) 2023-12-16T21:41:47.066 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:41:46 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:41:46.758+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:09.828911+0000 front 2023-12-16T21:39:09.828958+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:41:47.066 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:41:46 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:41:46.758+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:09.828868+0000 front 2023-12-16T21:39:09.828878+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:41:47.066 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:41:46 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:41:46.758+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:09.828947+0000 front 2023-12-16T21:39:09.828993+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:41:47.066 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:41:46 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:41:46.918+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:09.306583+0000 front 2023-12-16T21:39:09.306633+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:41:47.067 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:41:46 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:41:46.918+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:09.306551+0000 front 2023-12-16T21:39:09.306652+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:41:47.067 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:41:46 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:41:46.918+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:09.306605+0000 front 2023-12-16T21:39:09.306613+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:41:47.566 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:41:47 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:41:47.193+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:05.622397+0000 front 2023-12-16T21:39:05.622556+0000 (oldest deadline 2023-12-16T21:39:30.322546+0000) 2023-12-16T21:41:47.566 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:41:47 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:41:47.193+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:05.622506+0000 front 2023-12-16T21:39:05.622528+0000 (oldest deadline 2023-12-16T21:39:30.322546+0000) 2023-12-16T21:41:47.566 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:41:47 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:41:47.193+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:05.622479+0000 front 2023-12-16T21:39:05.622580+0000 (oldest deadline 2023-12-16T21:39:30.322546+0000) 2023-12-16T21:41:48.066 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:41:47 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:41:47.954+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:09.306583+0000 front 2023-12-16T21:39:09.306633+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:41:48.067 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:41:47 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:41:47.954+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:09.306551+0000 front 2023-12-16T21:39:09.306652+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:41:48.067 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:41:47 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:41:47.954+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:09.306605+0000 front 2023-12-16T21:39:09.306613+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:41:48.067 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:41:47 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:41:47.777+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:09.828911+0000 front 2023-12-16T21:39:09.828958+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:41:48.067 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:41:47 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:41:47.777+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:09.828868+0000 front 2023-12-16T21:39:09.828878+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:41:48.067 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:41:47 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:41:47.777+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:09.828947+0000 front 2023-12-16T21:39:09.828993+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:41:48.566 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:41:48 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:41:48.147+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:05.622397+0000 front 2023-12-16T21:39:05.622556+0000 (oldest deadline 2023-12-16T21:39:30.322546+0000) 2023-12-16T21:41:48.567 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:41:48 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:41:48.147+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:05.622506+0000 front 2023-12-16T21:39:05.622528+0000 (oldest deadline 2023-12-16T21:39:30.322546+0000) 2023-12-16T21:41:48.567 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:41:48 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:41:48.147+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:05.622479+0000 front 2023-12-16T21:39:05.622580+0000 (oldest deadline 2023-12-16T21:39:30.322546+0000) 2023-12-16T21:41:49.066 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:41:48 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:41:48.923+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:09.306583+0000 front 2023-12-16T21:39:09.306633+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:41:49.066 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:41:48 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:41:48.923+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:09.306551+0000 front 2023-12-16T21:39:09.306652+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:41:49.066 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:41:48 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:41:48.923+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:09.306605+0000 front 2023-12-16T21:39:09.306613+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:41:49.067 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:41:48 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:41:48.823+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:09.828911+0000 front 2023-12-16T21:39:09.828958+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:41:49.067 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:41:48 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:41:48.823+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:09.828868+0000 front 2023-12-16T21:39:09.828878+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:41:49.067 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:41:48 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:41:48.823+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:09.828947+0000 front 2023-12-16T21:39:09.828993+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:41:49.566 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:41:49 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:41:49.098+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:05.622397+0000 front 2023-12-16T21:39:05.622556+0000 (oldest deadline 2023-12-16T21:39:30.322546+0000) 2023-12-16T21:41:49.570 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:41:49 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:41:49.098+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:05.622506+0000 front 2023-12-16T21:39:05.622528+0000 (oldest deadline 2023-12-16T21:39:30.322546+0000) 2023-12-16T21:41:49.570 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:41:49 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:41:49.098+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:05.622479+0000 front 2023-12-16T21:39:05.622580+0000 (oldest deadline 2023-12-16T21:39:30.322546+0000) 2023-12-16T21:41:50.066 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:41:49 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:41:49.930+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:09.306583+0000 front 2023-12-16T21:39:09.306633+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:41:50.067 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:41:49 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:41:49.930+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:09.306551+0000 front 2023-12-16T21:39:09.306652+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:41:50.067 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:41:49 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:41:49.930+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:09.306605+0000 front 2023-12-16T21:39:09.306613+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:41:50.067 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:41:49 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:41:49.774+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:09.828911+0000 front 2023-12-16T21:39:09.828958+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:41:50.067 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:41:49 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:41:49.774+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:09.828868+0000 front 2023-12-16T21:39:09.828878+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:41:50.067 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:41:49 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:41:49.774+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:09.828947+0000 front 2023-12-16T21:39:09.828993+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:41:50.566 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:41:50 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:41:50.119+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:05.622397+0000 front 2023-12-16T21:39:05.622556+0000 (oldest deadline 2023-12-16T21:39:30.322546+0000) 2023-12-16T21:41:50.567 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:41:50 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:41:50.119+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:05.622506+0000 front 2023-12-16T21:39:05.622528+0000 (oldest deadline 2023-12-16T21:39:30.322546+0000) 2023-12-16T21:41:50.567 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:41:50 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:41:50.119+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:05.622479+0000 front 2023-12-16T21:39:05.622580+0000 (oldest deadline 2023-12-16T21:39:30.322546+0000) 2023-12-16T21:41:51.163 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:41:50 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:41:50.888+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:09.306583+0000 front 2023-12-16T21:39:09.306633+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:41:51.163 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:41:50 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:41:50.888+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:09.306551+0000 front 2023-12-16T21:39:09.306652+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:41:51.163 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:41:50 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:41:50.888+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:09.306605+0000 front 2023-12-16T21:39:09.306613+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:41:51.163 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:41:50 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:41:50.824+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:09.828911+0000 front 2023-12-16T21:39:09.828958+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:41:51.163 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:41:50 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:41:50.824+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:09.828868+0000 front 2023-12-16T21:39:09.828878+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:41:51.163 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:41:50 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:41:50.824+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:09.828947+0000 front 2023-12-16T21:39:09.828993+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:41:51.460 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Dec 16 21:41:51 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-mon-smithi045[80759]: 2023-12-16T21:41:51.332+0000 7f45a1f8f700 -1 mon.smithi045@0(probing) e2 get_health_metrics reporting 170 slow ops, oldest is monmgrreport(gid 14214, 0 checks, 1 progress events) 2023-12-16T21:41:51.460 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:41:51 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:41:51.160+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:05.622397+0000 front 2023-12-16T21:39:05.622556+0000 (oldest deadline 2023-12-16T21:39:30.322546+0000) 2023-12-16T21:41:51.461 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:41:51 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:41:51.160+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:05.622506+0000 front 2023-12-16T21:39:05.622528+0000 (oldest deadline 2023-12-16T21:39:30.322546+0000) 2023-12-16T21:41:51.461 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:41:51 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:41:51.160+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:05.622479+0000 front 2023-12-16T21:39:05.622580+0000 (oldest deadline 2023-12-16T21:39:30.322546+0000) 2023-12-16T21:41:52.066 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:41:51 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:41:51.911+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:09.306583+0000 front 2023-12-16T21:39:09.306633+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:41:52.066 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:41:51 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:41:51.911+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:09.306551+0000 front 2023-12-16T21:39:09.306652+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:41:52.066 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:41:51 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:41:51.911+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:09.306605+0000 front 2023-12-16T21:39:09.306613+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:41:52.067 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:41:51 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:41:51.779+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:09.828911+0000 front 2023-12-16T21:39:09.828958+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:41:52.067 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:41:51 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:41:51.779+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:09.828868+0000 front 2023-12-16T21:39:09.828878+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:41:52.067 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:41:51 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:41:51.779+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:09.828947+0000 front 2023-12-16T21:39:09.828993+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:41:52.566 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:41:52 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:41:52.176+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:05.622397+0000 front 2023-12-16T21:39:05.622556+0000 (oldest deadline 2023-12-16T21:39:30.322546+0000) 2023-12-16T21:41:52.566 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:41:52 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:41:52.177+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:05.622506+0000 front 2023-12-16T21:39:05.622528+0000 (oldest deadline 2023-12-16T21:39:30.322546+0000) 2023-12-16T21:41:52.566 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:41:52 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:41:52.177+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:05.622479+0000 front 2023-12-16T21:39:05.622580+0000 (oldest deadline 2023-12-16T21:39:30.322546+0000) 2023-12-16T21:41:53.066 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:41:52 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:41:52.937+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:09.306583+0000 front 2023-12-16T21:39:09.306633+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:41:53.066 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:41:52 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:41:52.937+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:09.306551+0000 front 2023-12-16T21:39:09.306652+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:41:53.066 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:41:52 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:41:52.937+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:09.306605+0000 front 2023-12-16T21:39:09.306613+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:41:53.067 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:41:52 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:41:52.732+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:09.828911+0000 front 2023-12-16T21:39:09.828958+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:41:53.067 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:41:52 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:41:52.732+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:09.828868+0000 front 2023-12-16T21:39:09.828878+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:41:53.067 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:41:52 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:41:52.732+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:09.828947+0000 front 2023-12-16T21:39:09.828993+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:41:53.523 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:41:53 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:41:53.208+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:05.622397+0000 front 2023-12-16T21:39:05.622556+0000 (oldest deadline 2023-12-16T21:39:30.322546+0000) 2023-12-16T21:41:53.523 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:41:53 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:41:53.208+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:05.622506+0000 front 2023-12-16T21:39:05.622528+0000 (oldest deadline 2023-12-16T21:39:30.322546+0000) 2023-12-16T21:41:53.523 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:41:53 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:41:53.208+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:05.622479+0000 front 2023-12-16T21:39:05.622580+0000 (oldest deadline 2023-12-16T21:39:30.322546+0000) 2023-12-16T21:41:54.066 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:41:53 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:41:53.888+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:09.306583+0000 front 2023-12-16T21:39:09.306633+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:41:54.066 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:41:53 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:41:53.888+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:09.306551+0000 front 2023-12-16T21:39:09.306652+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:41:54.067 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:41:53 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:41:53.888+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:09.306605+0000 front 2023-12-16T21:39:09.306613+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:41:54.067 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:41:53 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:41:53.779+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:09.828911+0000 front 2023-12-16T21:39:09.828958+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:41:54.067 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:41:53 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:41:53.779+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:09.828868+0000 front 2023-12-16T21:39:09.828878+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:41:54.067 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:41:53 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:41:53.779+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:09.828947+0000 front 2023-12-16T21:39:09.828993+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:41:54.566 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:41:54 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:41:54.202+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:05.622397+0000 front 2023-12-16T21:39:05.622556+0000 (oldest deadline 2023-12-16T21:39:30.322546+0000) 2023-12-16T21:41:54.566 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:41:54 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:41:54.202+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:05.622506+0000 front 2023-12-16T21:39:05.622528+0000 (oldest deadline 2023-12-16T21:39:30.322546+0000) 2023-12-16T21:41:54.566 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:41:54 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:41:54.202+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:05.622479+0000 front 2023-12-16T21:39:05.622580+0000 (oldest deadline 2023-12-16T21:39:30.322546+0000) 2023-12-16T21:41:55.067 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:41:54 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:41:54.852+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:09.306583+0000 front 2023-12-16T21:39:09.306633+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:41:55.067 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:41:54 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:41:54.852+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:09.306551+0000 front 2023-12-16T21:39:09.306652+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:41:55.067 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:41:54 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:41:54.852+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:09.306605+0000 front 2023-12-16T21:39:09.306613+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:41:55.067 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:41:54 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:41:54.776+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:09.828911+0000 front 2023-12-16T21:39:09.828958+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:41:55.067 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:41:54 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:41:54.776+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:09.828868+0000 front 2023-12-16T21:39:09.828878+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:41:55.067 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:41:54 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:41:54.776+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:09.828947+0000 front 2023-12-16T21:39:09.828993+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:41:55.566 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:41:55 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:41:55.177+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:05.622397+0000 front 2023-12-16T21:39:05.622556+0000 (oldest deadline 2023-12-16T21:39:30.322546+0000) 2023-12-16T21:41:55.566 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:41:55 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:41:55.177+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:05.622506+0000 front 2023-12-16T21:39:05.622528+0000 (oldest deadline 2023-12-16T21:39:30.322546+0000) 2023-12-16T21:41:55.566 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:41:55 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:41:55.177+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:05.622479+0000 front 2023-12-16T21:39:05.622580+0000 (oldest deadline 2023-12-16T21:39:30.322546+0000) 2023-12-16T21:41:56.066 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:41:55 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:41:55.854+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:09.306583+0000 front 2023-12-16T21:39:09.306633+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:41:56.066 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:41:55 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:41:55.854+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:09.306551+0000 front 2023-12-16T21:39:09.306652+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:41:56.066 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:41:55 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:41:55.854+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:09.306605+0000 front 2023-12-16T21:39:09.306613+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:41:56.067 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:41:55 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:41:55.803+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:09.828911+0000 front 2023-12-16T21:39:09.828958+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:41:56.067 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:41:55 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:41:55.803+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:09.828868+0000 front 2023-12-16T21:39:09.828878+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:41:56.067 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:41:55 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:41:55.803+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:09.828947+0000 front 2023-12-16T21:39:09.828993+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:41:56.459 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Dec 16 21:41:56 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-mon-smithi045[80759]: 2023-12-16T21:41:56.332+0000 7f45a1f8f700 -1 mon.smithi045@0(probing) e2 get_health_metrics reporting 198 slow ops, oldest is monmgrreport(gid 14214, 0 checks, 1 progress events) 2023-12-16T21:41:56.460 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:41:56 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:41:56.202+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:05.622397+0000 front 2023-12-16T21:39:05.622556+0000 (oldest deadline 2023-12-16T21:39:30.322546+0000) 2023-12-16T21:41:56.460 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:41:56 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:41:56.202+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:05.622506+0000 front 2023-12-16T21:39:05.622528+0000 (oldest deadline 2023-12-16T21:39:30.322546+0000) 2023-12-16T21:41:56.460 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:41:56 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:41:56.202+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:05.622479+0000 front 2023-12-16T21:39:05.622580+0000 (oldest deadline 2023-12-16T21:39:30.322546+0000) 2023-12-16T21:41:57.225 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:41:56 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:41:56.853+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:09.306583+0000 front 2023-12-16T21:39:09.306633+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:41:57.225 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:41:56 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:41:56.853+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:09.306551+0000 front 2023-12-16T21:39:09.306652+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:41:57.225 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:41:56 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:41:56.853+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:09.306605+0000 front 2023-12-16T21:39:09.306613+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:41:57.225 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:41:56 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:41:56.845+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:09.828911+0000 front 2023-12-16T21:39:09.828958+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:41:57.225 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:41:56 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:41:56.845+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:09.828868+0000 front 2023-12-16T21:39:09.828878+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:41:57.225 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:41:56 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:41:56.845+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:09.828947+0000 front 2023-12-16T21:39:09.828993+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:41:57.566 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:41:57 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:41:57.221+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:05.622397+0000 front 2023-12-16T21:39:05.622556+0000 (oldest deadline 2023-12-16T21:39:30.322546+0000) 2023-12-16T21:41:57.566 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:41:57 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:41:57.221+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:05.622506+0000 front 2023-12-16T21:39:05.622528+0000 (oldest deadline 2023-12-16T21:39:30.322546+0000) 2023-12-16T21:41:57.566 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:41:57 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:41:57.221+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:05.622479+0000 front 2023-12-16T21:39:05.622580+0000 (oldest deadline 2023-12-16T21:39:30.322546+0000) 2023-12-16T21:41:58.162 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:41:57 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:41:57.863+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:09.828911+0000 front 2023-12-16T21:39:09.828958+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:41:58.162 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:41:57 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:41:57.863+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:09.828868+0000 front 2023-12-16T21:39:09.828878+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:41:58.162 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:41:57 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:41:57.863+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:09.828947+0000 front 2023-12-16T21:39:09.828993+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:41:58.162 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:41:57 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:41:57.842+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:09.306583+0000 front 2023-12-16T21:39:09.306633+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:41:58.162 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:41:57 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:41:57.842+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:09.306551+0000 front 2023-12-16T21:39:09.306652+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:41:58.162 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:41:57 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:41:57.842+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:09.306605+0000 front 2023-12-16T21:39:09.306613+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:41:58.566 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:41:58 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:41:58.268+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:05.622397+0000 front 2023-12-16T21:39:05.622556+0000 (oldest deadline 2023-12-16T21:39:30.322546+0000) 2023-12-16T21:41:58.566 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:41:58 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:41:58.268+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:05.622506+0000 front 2023-12-16T21:39:05.622528+0000 (oldest deadline 2023-12-16T21:39:30.322546+0000) 2023-12-16T21:41:58.566 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:41:58 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:41:58.268+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:05.622479+0000 front 2023-12-16T21:39:05.622580+0000 (oldest deadline 2023-12-16T21:39:30.322546+0000) 2023-12-16T21:41:59.066 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:41:58 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:41:58.870+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:09.306583+0000 front 2023-12-16T21:39:09.306633+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:41:59.066 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:41:58 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:41:58.870+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:09.306551+0000 front 2023-12-16T21:39:09.306652+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:41:59.066 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:41:58 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:41:58.870+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:09.306605+0000 front 2023-12-16T21:39:09.306613+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:41:59.067 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:41:58 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:41:58.859+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:09.828911+0000 front 2023-12-16T21:39:09.828958+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:41:59.067 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:41:58 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:41:58.859+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:09.828868+0000 front 2023-12-16T21:39:09.828878+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:41:59.067 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:41:58 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:41:58.859+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:09.828947+0000 front 2023-12-16T21:39:09.828993+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:41:59.566 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:41:59 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:41:59.270+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:05.622397+0000 front 2023-12-16T21:39:05.622556+0000 (oldest deadline 2023-12-16T21:39:30.322546+0000) 2023-12-16T21:41:59.566 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:41:59 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:41:59.270+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:05.622506+0000 front 2023-12-16T21:39:05.622528+0000 (oldest deadline 2023-12-16T21:39:30.322546+0000) 2023-12-16T21:41:59.566 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:41:59 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:41:59.270+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:05.622479+0000 front 2023-12-16T21:39:05.622580+0000 (oldest deadline 2023-12-16T21:39:30.322546+0000) 2023-12-16T21:42:00.310 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:41:59 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:41:59.907+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:09.828911+0000 front 2023-12-16T21:39:09.828958+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:42:00.311 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:41:59 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:41:59.907+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:09.828868+0000 front 2023-12-16T21:39:09.828878+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:42:00.311 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:41:59 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:41:59.907+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:09.828947+0000 front 2023-12-16T21:39:09.828993+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:42:00.311 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:41:59 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:41:59.846+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:09.306583+0000 front 2023-12-16T21:39:09.306633+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:42:00.311 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:41:59 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:41:59.846+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:09.306551+0000 front 2023-12-16T21:39:09.306652+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:42:00.311 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:41:59 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:41:59.846+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:09.306605+0000 front 2023-12-16T21:39:09.306613+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:42:00.566 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:42:00 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:42:00.308+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:05.622397+0000 front 2023-12-16T21:39:05.622556+0000 (oldest deadline 2023-12-16T21:39:30.322546+0000) 2023-12-16T21:42:00.566 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:42:00 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:42:00.308+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:05.622506+0000 front 2023-12-16T21:39:05.622528+0000 (oldest deadline 2023-12-16T21:39:30.322546+0000) 2023-12-16T21:42:00.566 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:42:00 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:42:00.308+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:05.622479+0000 front 2023-12-16T21:39:05.622580+0000 (oldest deadline 2023-12-16T21:39:30.322546+0000) 2023-12-16T21:42:01.234 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:42:00 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:42:00.817+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:09.306583+0000 front 2023-12-16T21:39:09.306633+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:42:01.235 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:42:00 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:42:00.817+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:09.306551+0000 front 2023-12-16T21:39:09.306652+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:42:01.235 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:42:00 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:42:00.817+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:09.306605+0000 front 2023-12-16T21:39:09.306613+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:42:01.235 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:42:00 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:42:00.937+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:09.828911+0000 front 2023-12-16T21:39:09.828958+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:42:01.235 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:42:00 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:42:00.937+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:09.828868+0000 front 2023-12-16T21:39:09.828878+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:42:01.235 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:42:00 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:42:00.937+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:09.828947+0000 front 2023-12-16T21:39:09.828993+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:42:01.632 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:42:01 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:42:01.344+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:05.622397+0000 front 2023-12-16T21:39:05.622556+0000 (oldest deadline 2023-12-16T21:39:30.322546+0000) 2023-12-16T21:42:01.633 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:42:01 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:42:01.344+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:05.622506+0000 front 2023-12-16T21:39:05.622528+0000 (oldest deadline 2023-12-16T21:39:30.322546+0000) 2023-12-16T21:42:01.633 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:42:01 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:42:01.344+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:05.622479+0000 front 2023-12-16T21:39:05.622580+0000 (oldest deadline 2023-12-16T21:39:30.322546+0000) 2023-12-16T21:42:01.633 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Dec 16 21:42:01 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-mon-smithi045[80759]: 2023-12-16T21:42:01.333+0000 7f45a1f8f700 -1 mon.smithi045@0(probing) e2 get_health_metrics reporting 229 slow ops, oldest is monmgrreport(gid 14214, 0 checks, 1 progress events) 2023-12-16T21:42:01.986 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:42:01 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:42:01.863+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:09.306583+0000 front 2023-12-16T21:39:09.306633+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:42:01.986 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:42:01 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:42:01.863+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:09.306551+0000 front 2023-12-16T21:39:09.306652+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:42:01.986 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:42:01 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:42:01.863+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:09.306605+0000 front 2023-12-16T21:39:09.306613+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:42:02.316 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:42:01 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:42:01.983+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:09.828911+0000 front 2023-12-16T21:39:09.828958+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:42:02.316 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:42:01 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:42:01.983+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:09.828868+0000 front 2023-12-16T21:39:09.828878+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:42:02.316 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:42:01 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:42:01.983+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:09.828947+0000 front 2023-12-16T21:39:09.828993+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:42:02.816 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:42:02 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:42:02.375+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:05.622397+0000 front 2023-12-16T21:39:05.622556+0000 (oldest deadline 2023-12-16T21:39:30.322546+0000) 2023-12-16T21:42:02.816 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:42:02 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:42:02.375+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:05.622506+0000 front 2023-12-16T21:39:05.622528+0000 (oldest deadline 2023-12-16T21:39:30.322546+0000) 2023-12-16T21:42:02.816 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:42:02 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:42:02.375+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:05.622479+0000 front 2023-12-16T21:39:05.622580+0000 (oldest deadline 2023-12-16T21:39:30.322546+0000) 2023-12-16T21:42:03.316 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:42:02 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:42:02.843+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:09.306583+0000 front 2023-12-16T21:39:09.306633+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:42:03.316 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:42:02 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:42:02.843+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:09.306551+0000 front 2023-12-16T21:39:09.306652+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:42:03.316 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:42:02 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:42:02.843+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:09.306605+0000 front 2023-12-16T21:39:09.306613+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:42:03.317 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:42:02 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:42:02.998+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:09.828911+0000 front 2023-12-16T21:39:09.828958+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:42:03.317 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:42:02 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:42:02.998+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:09.828868+0000 front 2023-12-16T21:39:09.828878+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:42:03.317 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:42:02 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:42:02.998+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:09.828947+0000 front 2023-12-16T21:39:09.828993+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:42:03.816 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:42:03 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:42:03.332+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:05.622397+0000 front 2023-12-16T21:39:05.622556+0000 (oldest deadline 2023-12-16T21:39:30.322546+0000) 2023-12-16T21:42:03.816 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:42:03 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:42:03.332+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:05.622506+0000 front 2023-12-16T21:39:05.622528+0000 (oldest deadline 2023-12-16T21:39:30.322546+0000) 2023-12-16T21:42:03.816 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:42:03 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:42:03.332+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:05.622479+0000 front 2023-12-16T21:39:05.622580+0000 (oldest deadline 2023-12-16T21:39:30.322546+0000) 2023-12-16T21:42:04.304 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:42:03 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:42:03.893+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:09.306583+0000 front 2023-12-16T21:39:09.306633+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:42:04.305 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:42:03 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:42:03.893+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:09.306551+0000 front 2023-12-16T21:39:09.306652+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:42:04.305 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:42:03 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:42:03.893+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:09.306605+0000 front 2023-12-16T21:39:09.306613+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:42:04.305 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:42:04 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:42:04.014+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:09.828911+0000 front 2023-12-16T21:39:09.828958+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:42:04.305 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:42:04 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:42:04.014+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:09.828868+0000 front 2023-12-16T21:39:09.828878+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:42:04.305 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:42:04 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:42:04.014+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:09.828947+0000 front 2023-12-16T21:39:09.828993+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:42:04.566 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:42:04 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:42:04.360+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:05.622397+0000 front 2023-12-16T21:39:05.622556+0000 (oldest deadline 2023-12-16T21:39:30.322546+0000) 2023-12-16T21:42:04.566 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:42:04 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:42:04.360+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:05.622506+0000 front 2023-12-16T21:39:05.622528+0000 (oldest deadline 2023-12-16T21:39:30.322546+0000) 2023-12-16T21:42:04.566 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:42:04 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:42:04.360+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:05.622479+0000 front 2023-12-16T21:39:05.622580+0000 (oldest deadline 2023-12-16T21:39:30.322546+0000) 2023-12-16T21:42:05.316 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:42:04 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:42:04.979+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:09.828911+0000 front 2023-12-16T21:39:09.828958+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:42:05.316 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:42:04 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:42:04.979+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:09.828868+0000 front 2023-12-16T21:39:09.828878+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:42:05.316 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:42:04 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:42:04.979+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:09.828947+0000 front 2023-12-16T21:39:09.828993+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:42:05.317 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:42:04 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:42:04.918+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:09.306583+0000 front 2023-12-16T21:39:09.306633+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:42:05.317 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:42:04 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:42:04.918+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:09.306551+0000 front 2023-12-16T21:39:09.306652+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:42:05.317 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:42:04 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:42:04.918+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:09.306605+0000 front 2023-12-16T21:39:09.306613+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:42:05.816 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:42:05 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:42:05.400+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:05.622397+0000 front 2023-12-16T21:39:05.622556+0000 (oldest deadline 2023-12-16T21:39:30.322546+0000) 2023-12-16T21:42:05.816 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:42:05 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:42:05.400+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:05.622506+0000 front 2023-12-16T21:39:05.622528+0000 (oldest deadline 2023-12-16T21:39:30.322546+0000) 2023-12-16T21:42:05.816 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:42:05 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:42:05.400+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:05.622479+0000 front 2023-12-16T21:39:05.622580+0000 (oldest deadline 2023-12-16T21:39:30.322546+0000) 2023-12-16T21:42:06.316 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:42:05 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:42:05.949+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:09.828911+0000 front 2023-12-16T21:39:09.828958+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:42:06.316 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:42:05 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:42:05.949+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:09.828868+0000 front 2023-12-16T21:39:09.828878+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:42:06.316 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:42:05 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:42:05.949+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:09.828947+0000 front 2023-12-16T21:39:09.828993+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:42:06.317 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:42:05 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:42:05.945+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:09.306583+0000 front 2023-12-16T21:39:09.306633+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:42:06.317 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:42:05 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:42:05.945+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:09.306551+0000 front 2023-12-16T21:39:09.306652+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:42:06.317 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:42:05 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:42:05.945+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:09.306605+0000 front 2023-12-16T21:39:09.306613+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:42:06.816 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Dec 16 21:42:06 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-mon-smithi045[80759]: 2023-12-16T21:42:06.333+0000 7f45a1f8f700 -1 mon.smithi045@0(probing) e2 get_health_metrics reporting 256 slow ops, oldest is monmgrreport(gid 14214, 0 checks, 1 progress events) 2023-12-16T21:42:06.816 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:42:06 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:42:06.360+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:05.622397+0000 front 2023-12-16T21:39:05.622556+0000 (oldest deadline 2023-12-16T21:39:30.322546+0000) 2023-12-16T21:42:06.817 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:42:06 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:42:06.360+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:05.622506+0000 front 2023-12-16T21:39:05.622528+0000 (oldest deadline 2023-12-16T21:39:30.322546+0000) 2023-12-16T21:42:06.817 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:42:06 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:42:06.360+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:05.622479+0000 front 2023-12-16T21:39:05.622580+0000 (oldest deadline 2023-12-16T21:39:30.322546+0000) 2023-12-16T21:42:07.316 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:42:06 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:42:06.992+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:09.306583+0000 front 2023-12-16T21:39:09.306633+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:42:07.316 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:42:06 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:42:06.992+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:09.306551+0000 front 2023-12-16T21:39:09.306652+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:42:07.317 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:42:06 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:42:06.992+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:09.306605+0000 front 2023-12-16T21:39:09.306613+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:42:07.317 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:42:06 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:42:06.944+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:09.828911+0000 front 2023-12-16T21:39:09.828958+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:42:07.317 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:42:06 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:42:06.944+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:09.828868+0000 front 2023-12-16T21:39:09.828878+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:42:07.317 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:42:06 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:42:06.944+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:09.828947+0000 front 2023-12-16T21:39:09.828993+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:42:07.817 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:42:07 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:42:07.347+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:05.622397+0000 front 2023-12-16T21:39:05.622556+0000 (oldest deadline 2023-12-16T21:39:30.322546+0000) 2023-12-16T21:42:07.817 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:42:07 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:42:07.347+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:05.622506+0000 front 2023-12-16T21:39:05.622528+0000 (oldest deadline 2023-12-16T21:39:30.322546+0000) 2023-12-16T21:42:07.817 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:42:07 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:42:07.347+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:05.622479+0000 front 2023-12-16T21:39:05.622580+0000 (oldest deadline 2023-12-16T21:39:30.322546+0000) 2023-12-16T21:42:08.316 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:42:07 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:42:07.988+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:09.306583+0000 front 2023-12-16T21:39:09.306633+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:42:08.316 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:42:07 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:42:07.988+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:09.306551+0000 front 2023-12-16T21:39:09.306652+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:42:08.316 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:42:07 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:42:07.988+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:09.306605+0000 front 2023-12-16T21:39:09.306613+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:42:08.317 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:42:07 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:42:07.935+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:09.828911+0000 front 2023-12-16T21:39:09.828958+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:42:08.317 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:42:07 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:42:07.935+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:09.828868+0000 front 2023-12-16T21:39:09.828878+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:42:08.317 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:42:07 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:42:07.935+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:09.828947+0000 front 2023-12-16T21:39:09.828993+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:42:08.722 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:42:08 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:42:08.362+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:05.622397+0000 front 2023-12-16T21:39:05.622556+0000 (oldest deadline 2023-12-16T21:39:30.322546+0000) 2023-12-16T21:42:08.722 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:42:08 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:42:08.362+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:05.622506+0000 front 2023-12-16T21:39:05.622528+0000 (oldest deadline 2023-12-16T21:39:30.322546+0000) 2023-12-16T21:42:08.722 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:42:08 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:42:08.362+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:05.622479+0000 front 2023-12-16T21:39:05.622580+0000 (oldest deadline 2023-12-16T21:39:30.322546+0000) 2023-12-16T21:42:09.316 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:42:09 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:42:09.003+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:09.306583+0000 front 2023-12-16T21:39:09.306633+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:42:09.316 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:42:09 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:42:09.003+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:09.306551+0000 front 2023-12-16T21:39:09.306652+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:42:09.316 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:42:09 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:42:09.003+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:09.306605+0000 front 2023-12-16T21:39:09.306613+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:42:09.317 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:42:08 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:42:08.980+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:09.828911+0000 front 2023-12-16T21:39:09.828958+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:42:09.317 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:42:08 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:42:08.980+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:09.828868+0000 front 2023-12-16T21:39:09.828878+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:42:09.317 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:42:08 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:42:08.980+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:09.828947+0000 front 2023-12-16T21:39:09.828993+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:42:09.816 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:42:09 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:42:09.348+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:05.622397+0000 front 2023-12-16T21:39:05.622556+0000 (oldest deadline 2023-12-16T21:39:30.322546+0000) 2023-12-16T21:42:09.816 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:42:09 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:42:09.348+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:05.622506+0000 front 2023-12-16T21:39:05.622528+0000 (oldest deadline 2023-12-16T21:39:30.322546+0000) 2023-12-16T21:42:09.816 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:42:09 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:42:09.348+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:05.622479+0000 front 2023-12-16T21:39:05.622580+0000 (oldest deadline 2023-12-16T21:39:30.322546+0000) 2023-12-16T21:42:10.316 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:42:09 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:42:09.958+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:09.828911+0000 front 2023-12-16T21:39:09.828958+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:42:10.316 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:42:09 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:42:09.958+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:09.828868+0000 front 2023-12-16T21:39:09.828878+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:42:10.316 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:42:09 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:42:09.958+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:09.828947+0000 front 2023-12-16T21:39:09.828993+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:42:10.317 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:42:09 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:42:09.990+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:09.306583+0000 front 2023-12-16T21:39:09.306633+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:42:10.317 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:42:09 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:42:09.990+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:09.306551+0000 front 2023-12-16T21:39:09.306652+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:42:10.317 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:42:09 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:42:09.990+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:09.306605+0000 front 2023-12-16T21:39:09.306613+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:42:10.816 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:42:10 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:42:10.338+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:05.622397+0000 front 2023-12-16T21:39:05.622556+0000 (oldest deadline 2023-12-16T21:39:30.322546+0000) 2023-12-16T21:42:10.817 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:42:10 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:42:10.338+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:05.622506+0000 front 2023-12-16T21:39:05.622528+0000 (oldest deadline 2023-12-16T21:39:30.322546+0000) 2023-12-16T21:42:10.817 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:42:10 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:42:10.338+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:05.622479+0000 front 2023-12-16T21:39:05.622580+0000 (oldest deadline 2023-12-16T21:39:30.322546+0000) 2023-12-16T21:42:11.317 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:42:10 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:42:10.962+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:09.828911+0000 front 2023-12-16T21:39:09.828958+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:42:11.317 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:42:10 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:42:10.962+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:09.828868+0000 front 2023-12-16T21:39:09.828878+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:42:11.317 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:42:10 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:42:10.962+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:09.828947+0000 front 2023-12-16T21:39:09.828993+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:42:11.317 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:42:10 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:42:10.958+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:09.306583+0000 front 2023-12-16T21:39:09.306633+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:42:11.317 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:42:10 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:42:10.958+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:09.306551+0000 front 2023-12-16T21:39:09.306652+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:42:11.317 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:42:10 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:42:10.958+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:09.306605+0000 front 2023-12-16T21:39:09.306613+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:42:11.633 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:42:11 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:42:11.334+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:05.622397+0000 front 2023-12-16T21:39:05.622556+0000 (oldest deadline 2023-12-16T21:39:30.322546+0000) 2023-12-16T21:42:11.633 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:42:11 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:42:11.334+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:05.622506+0000 front 2023-12-16T21:39:05.622528+0000 (oldest deadline 2023-12-16T21:39:30.322546+0000) 2023-12-16T21:42:11.633 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:42:11 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:42:11.334+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:05.622479+0000 front 2023-12-16T21:39:05.622580+0000 (oldest deadline 2023-12-16T21:39:30.322546+0000) 2023-12-16T21:42:11.634 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Dec 16 21:42:11 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-mon-smithi045[80759]: 2023-12-16T21:42:11.334+0000 7f45a1f8f700 -1 mon.smithi045@0(probing) e2 get_health_metrics reporting 286 slow ops, oldest is monmgrreport(gid 14214, 0 checks, 1 progress events) 2023-12-16T21:42:12.310 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:42:12 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:42:12.007+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:09.828911+0000 front 2023-12-16T21:39:09.828958+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:42:12.310 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:42:12 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:42:12.007+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:09.828868+0000 front 2023-12-16T21:39:09.828878+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:42:12.310 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:42:12 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:42:12.007+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:09.828947+0000 front 2023-12-16T21:39:09.828993+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:42:12.310 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:42:11 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:42:11.967+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:09.306583+0000 front 2023-12-16T21:39:09.306633+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:42:12.310 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:42:11 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:42:11.967+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:09.306551+0000 front 2023-12-16T21:39:09.306652+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:42:12.310 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:42:11 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:42:11.967+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:09.306605+0000 front 2023-12-16T21:39:09.306613+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:42:12.566 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:42:12 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:42:12.307+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:05.622397+0000 front 2023-12-16T21:39:05.622556+0000 (oldest deadline 2023-12-16T21:39:30.322546+0000) 2023-12-16T21:42:12.566 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:42:12 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:42:12.307+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:05.622506+0000 front 2023-12-16T21:39:05.622528+0000 (oldest deadline 2023-12-16T21:39:30.322546+0000) 2023-12-16T21:42:12.566 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:42:12 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:42:12.307+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:05.622479+0000 front 2023-12-16T21:39:05.622580+0000 (oldest deadline 2023-12-16T21:39:30.322546+0000) 2023-12-16T21:42:13.294 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:42:12 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:42:12.951+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:09.306583+0000 front 2023-12-16T21:39:09.306633+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:42:13.294 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:42:12 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:42:12.951+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:09.306551+0000 front 2023-12-16T21:39:09.306652+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:42:13.294 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:42:12 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:42:12.951+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:09.306605+0000 front 2023-12-16T21:39:09.306613+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:42:13.294 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:42:12 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:42:12.962+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:09.828911+0000 front 2023-12-16T21:39:09.828958+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:42:13.294 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:42:12 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:42:12.962+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:09.828868+0000 front 2023-12-16T21:39:09.828878+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:42:13.294 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:42:12 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:42:12.962+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:09.828947+0000 front 2023-12-16T21:39:09.828993+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:42:13.566 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:42:13 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:42:13.291+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:05.622397+0000 front 2023-12-16T21:39:05.622556+0000 (oldest deadline 2023-12-16T21:39:30.322546+0000) 2023-12-16T21:42:13.566 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:42:13 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:42:13.291+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:05.622506+0000 front 2023-12-16T21:39:05.622528+0000 (oldest deadline 2023-12-16T21:39:30.322546+0000) 2023-12-16T21:42:13.566 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:42:13 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:42:13.291+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:05.622479+0000 front 2023-12-16T21:39:05.622580+0000 (oldest deadline 2023-12-16T21:39:30.322546+0000) 2023-12-16T21:42:14.295 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:42:13 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:42:13.924+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:09.828911+0000 front 2023-12-16T21:39:09.828958+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:42:14.296 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:42:13 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:42:13.924+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:09.828868+0000 front 2023-12-16T21:39:09.828878+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:42:14.296 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:42:13 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:42:13.924+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:09.828947+0000 front 2023-12-16T21:39:09.828993+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:42:14.296 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:42:13 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:42:13.971+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:09.306583+0000 front 2023-12-16T21:39:09.306633+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:42:14.296 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:42:13 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:42:13.971+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:09.306551+0000 front 2023-12-16T21:39:09.306652+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:42:14.296 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:42:13 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:42:13.971+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:09.306605+0000 front 2023-12-16T21:39:09.306613+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:42:14.566 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:42:14 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:42:14.292+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:05.622397+0000 front 2023-12-16T21:39:05.622556+0000 (oldest deadline 2023-12-16T21:39:30.322546+0000) 2023-12-16T21:42:14.566 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:42:14 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:42:14.292+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:05.622506+0000 front 2023-12-16T21:39:05.622528+0000 (oldest deadline 2023-12-16T21:39:30.322546+0000) 2023-12-16T21:42:14.566 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:42:14 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:42:14.292+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:05.622479+0000 front 2023-12-16T21:39:05.622580+0000 (oldest deadline 2023-12-16T21:39:30.322546+0000) 2023-12-16T21:42:15.273 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:42:14 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:42:14.997+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:09.306583+0000 front 2023-12-16T21:39:09.306633+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:42:15.273 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:42:14 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:42:14.997+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:09.306551+0000 front 2023-12-16T21:39:09.306652+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:42:15.273 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:42:14 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:42:14.997+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:09.306605+0000 front 2023-12-16T21:39:09.306613+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:42:15.274 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:42:14 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:42:14.882+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:09.828911+0000 front 2023-12-16T21:39:09.828958+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:42:15.274 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:42:14 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:42:14.882+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:09.828868+0000 front 2023-12-16T21:39:09.828878+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:42:15.274 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:42:14 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:42:14.882+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:09.828947+0000 front 2023-12-16T21:39:09.828993+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:42:15.566 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:42:15 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:42:15.270+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:05.622397+0000 front 2023-12-16T21:39:05.622556+0000 (oldest deadline 2023-12-16T21:39:30.322546+0000) 2023-12-16T21:42:15.566 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:42:15 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:42:15.270+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:05.622506+0000 front 2023-12-16T21:39:05.622528+0000 (oldest deadline 2023-12-16T21:39:30.322546+0000) 2023-12-16T21:42:15.566 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:42:15 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:42:15.270+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:05.622479+0000 front 2023-12-16T21:39:05.622580+0000 (oldest deadline 2023-12-16T21:39:30.322546+0000) 2023-12-16T21:42:16.263 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:42:15 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:42:15.991+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:09.306583+0000 front 2023-12-16T21:39:09.306633+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:42:16.264 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:42:15 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:42:15.991+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:09.306551+0000 front 2023-12-16T21:39:09.306652+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:42:16.264 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:42:15 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:42:15.991+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:09.306605+0000 front 2023-12-16T21:39:09.306613+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:42:16.264 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:42:15 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:42:15.856+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:09.828911+0000 front 2023-12-16T21:39:09.828958+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:42:16.264 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:42:15 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:42:15.856+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:09.828868+0000 front 2023-12-16T21:39:09.828878+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:42:16.264 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:42:15 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:42:15.856+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:09.828947+0000 front 2023-12-16T21:39:09.828993+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:42:16.566 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Dec 16 21:42:16 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-mon-smithi045[80759]: 2023-12-16T21:42:16.333+0000 7f45a1f8f700 -1 mon.smithi045@0(probing) e2 get_health_metrics reporting 314 slow ops, oldest is monmgrreport(gid 14214, 0 checks, 1 progress events) 2023-12-16T21:42:16.566 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:42:16 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:42:16.259+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:05.622397+0000 front 2023-12-16T21:39:05.622556+0000 (oldest deadline 2023-12-16T21:39:30.322546+0000) 2023-12-16T21:42:16.566 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:42:16 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:42:16.259+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:05.622506+0000 front 2023-12-16T21:39:05.622528+0000 (oldest deadline 2023-12-16T21:39:30.322546+0000) 2023-12-16T21:42:16.567 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:42:16 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:42:16.259+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:05.622479+0000 front 2023-12-16T21:39:05.622580+0000 (oldest deadline 2023-12-16T21:39:30.322546+0000) 2023-12-16T21:42:17.312 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:42:16 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:42:16.942+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:09.306583+0000 front 2023-12-16T21:39:09.306633+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:42:17.312 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:42:16 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:42:16.942+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:09.306551+0000 front 2023-12-16T21:39:09.306652+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:42:17.312 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:42:16 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:42:16.942+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:09.306605+0000 front 2023-12-16T21:39:09.306613+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:42:17.312 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:42:16 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:42:16.884+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:09.828911+0000 front 2023-12-16T21:39:09.828958+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:42:17.312 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:42:16 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:42:16.884+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:09.828868+0000 front 2023-12-16T21:39:09.828878+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:42:17.312 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:42:16 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:42:16.884+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:09.828947+0000 front 2023-12-16T21:39:09.828993+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:42:17.566 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:42:17 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:42:17.307+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:05.622397+0000 front 2023-12-16T21:39:05.622556+0000 (oldest deadline 2023-12-16T21:39:30.322546+0000) 2023-12-16T21:42:17.566 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:42:17 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:42:17.307+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:05.622506+0000 front 2023-12-16T21:39:05.622528+0000 (oldest deadline 2023-12-16T21:39:30.322546+0000) 2023-12-16T21:42:17.566 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:42:17 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:42:17.307+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:05.622479+0000 front 2023-12-16T21:39:05.622580+0000 (oldest deadline 2023-12-16T21:39:30.322546+0000) 2023-12-16T21:42:18.278 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:42:17 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:42:17.894+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:09.828911+0000 front 2023-12-16T21:39:09.828958+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:42:18.278 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:42:17 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:42:17.894+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:09.828868+0000 front 2023-12-16T21:39:09.828878+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:42:18.278 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:42:17 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:42:17.894+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:09.828947+0000 front 2023-12-16T21:39:09.828993+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:42:18.279 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:42:17 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:42:17.976+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:09.306583+0000 front 2023-12-16T21:39:09.306633+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:42:18.279 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:42:17 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:42:17.976+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:09.306551+0000 front 2023-12-16T21:39:09.306652+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:42:18.279 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:42:17 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:42:17.976+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:09.306605+0000 front 2023-12-16T21:39:09.306613+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:42:18.566 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:42:18 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:42:18.274+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:05.622397+0000 front 2023-12-16T21:39:05.622556+0000 (oldest deadline 2023-12-16T21:39:30.322546+0000) 2023-12-16T21:42:18.566 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:42:18 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:42:18.274+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:05.622506+0000 front 2023-12-16T21:39:05.622528+0000 (oldest deadline 2023-12-16T21:39:30.322546+0000) 2023-12-16T21:42:18.566 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:42:18 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:42:18.274+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:05.622479+0000 front 2023-12-16T21:39:05.622580+0000 (oldest deadline 2023-12-16T21:39:30.322546+0000) 2023-12-16T21:42:19.004 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:42:18 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:42:18.883+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:09.828911+0000 front 2023-12-16T21:39:09.828958+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:42:19.004 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:42:18 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:42:18.883+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:09.828868+0000 front 2023-12-16T21:39:09.828878+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:42:19.004 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:42:18 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:42:18.883+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:09.828947+0000 front 2023-12-16T21:39:09.828993+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:42:19.258 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:42:19 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:42:19.001+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:09.306583+0000 front 2023-12-16T21:39:09.306633+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:42:19.258 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:42:19 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:42:19.001+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:09.306551+0000 front 2023-12-16T21:39:09.306652+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:42:19.258 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:42:19 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:42:19.001+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:09.306605+0000 front 2023-12-16T21:39:09.306613+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:42:19.566 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:42:19 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:42:19.255+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:05.622397+0000 front 2023-12-16T21:39:05.622556+0000 (oldest deadline 2023-12-16T21:39:30.322546+0000) 2023-12-16T21:42:19.566 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:42:19 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:42:19.255+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:05.622506+0000 front 2023-12-16T21:39:05.622528+0000 (oldest deadline 2023-12-16T21:39:30.322546+0000) 2023-12-16T21:42:19.566 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:42:19 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:42:19.255+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:05.622479+0000 front 2023-12-16T21:39:05.622580+0000 (oldest deadline 2023-12-16T21:39:30.322546+0000) 2023-12-16T21:42:20.269 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:42:19 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:42:19.974+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:09.306583+0000 front 2023-12-16T21:39:09.306633+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:42:20.269 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:42:19 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:42:19.974+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:09.306551+0000 front 2023-12-16T21:39:09.306652+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:42:20.270 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:42:19 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:42:19.974+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:09.306605+0000 front 2023-12-16T21:39:09.306613+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:42:20.270 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:42:19 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:42:19.839+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:09.828911+0000 front 2023-12-16T21:39:09.828958+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:42:20.270 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:42:19 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:42:19.839+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:09.828868+0000 front 2023-12-16T21:39:09.828878+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:42:20.270 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:42:19 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:42:19.839+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:09.828947+0000 front 2023-12-16T21:39:09.828993+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:42:20.566 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:42:20 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:42:20.266+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:05.622397+0000 front 2023-12-16T21:39:05.622556+0000 (oldest deadline 2023-12-16T21:39:30.322546+0000) 2023-12-16T21:42:20.566 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:42:20 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:42:20.266+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:05.622506+0000 front 2023-12-16T21:39:05.622528+0000 (oldest deadline 2023-12-16T21:39:30.322546+0000) 2023-12-16T21:42:20.566 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:42:20 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:42:20.266+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:05.622479+0000 front 2023-12-16T21:39:05.622580+0000 (oldest deadline 2023-12-16T21:39:30.322546+0000) 2023-12-16T21:42:21.316 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:42:20 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:42:20.973+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:09.306583+0000 front 2023-12-16T21:39:09.306633+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:42:21.316 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:42:20 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:42:20.973+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:09.306551+0000 front 2023-12-16T21:39:09.306652+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:42:21.317 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:42:20 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:42:20.973+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:09.306605+0000 front 2023-12-16T21:39:09.306613+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:42:21.317 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:42:20 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:42:20.974+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:42:21.317 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:42:20 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:42:20.872+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:09.828911+0000 front 2023-12-16T21:39:09.828958+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:42:21.317 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:42:20 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:42:20.872+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:09.828868+0000 front 2023-12-16T21:39:09.828878+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:42:21.317 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:42:20 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:42:20.872+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:09.828947+0000 front 2023-12-16T21:39:09.828993+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:42:21.317 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:42:20 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:42:20.874+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:42:21.633 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Dec 16 21:42:21 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-mon-smithi045[80759]: 2023-12-16T21:42:21.335+0000 7f45a1f8f700 -1 mon.smithi045@0(probing) e2 get_health_metrics reporting 345 slow ops, oldest is monmgrreport(gid 14214, 0 checks, 1 progress events) 2023-12-16T21:42:21.634 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:42:21 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:42:21.316+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:05.622397+0000 front 2023-12-16T21:39:05.622556+0000 (oldest deadline 2023-12-16T21:39:30.322546+0000) 2023-12-16T21:42:21.634 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:42:21 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:42:21.316+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:05.622506+0000 front 2023-12-16T21:39:05.622528+0000 (oldest deadline 2023-12-16T21:39:30.322546+0000) 2023-12-16T21:42:21.634 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:42:21 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:42:21.316+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:05.622479+0000 front 2023-12-16T21:39:05.622580+0000 (oldest deadline 2023-12-16T21:39:30.322546+0000) 2023-12-16T21:42:21.634 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:42:21 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:42:21.317+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:42:22.306 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:42:22 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:42:22.020+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:09.306583+0000 front 2023-12-16T21:39:09.306633+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:42:22.306 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:42:22 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:42:22.020+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:09.306551+0000 front 2023-12-16T21:39:09.306652+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:42:22.306 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:42:22 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:42:22.020+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:09.306605+0000 front 2023-12-16T21:39:09.306613+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:42:22.306 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:42:22 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:42:22.020+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:42:22.306 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:42:21 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:42:21.905+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:09.828911+0000 front 2023-12-16T21:39:09.828958+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:42:22.306 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:42:21 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:42:21.905+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:09.828868+0000 front 2023-12-16T21:39:09.828878+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:42:22.307 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:42:21 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:42:21.905+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:09.828947+0000 front 2023-12-16T21:39:09.828993+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:42:22.307 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:42:21 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:42:21.906+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:42:22.566 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:42:22 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:42:22.302+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:05.622397+0000 front 2023-12-16T21:39:05.622556+0000 (oldest deadline 2023-12-16T21:39:30.322546+0000) 2023-12-16T21:42:22.566 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:42:22 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:42:22.302+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:05.622506+0000 front 2023-12-16T21:39:05.622528+0000 (oldest deadline 2023-12-16T21:39:30.322546+0000) 2023-12-16T21:42:22.566 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:42:22 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:42:22.302+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:05.622479+0000 front 2023-12-16T21:39:05.622580+0000 (oldest deadline 2023-12-16T21:39:30.322546+0000) 2023-12-16T21:42:22.566 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:42:22 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:42:22.304+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:42:22.996 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:42:22 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:42:22.902+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:09.828911+0000 front 2023-12-16T21:39:09.828958+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:42:22.996 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:42:22 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:42:22.902+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:09.828868+0000 front 2023-12-16T21:39:09.828878+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:42:22.996 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:42:22 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:42:22.902+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:09.828947+0000 front 2023-12-16T21:39:09.828993+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:42:22.996 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:42:22 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:42:22.902+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:42:23.316 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:42:22 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:42:22.993+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:09.306583+0000 front 2023-12-16T21:39:09.306633+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:42:23.316 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:42:22 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:42:22.993+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:09.306551+0000 front 2023-12-16T21:39:09.306652+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:42:23.316 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:42:22 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:42:22.993+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:09.306605+0000 front 2023-12-16T21:39:09.306613+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:42:23.317 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:42:22 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:42:22.994+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:42:23.816 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:42:23 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:42:23.336+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:05.622397+0000 front 2023-12-16T21:39:05.622556+0000 (oldest deadline 2023-12-16T21:39:30.322546+0000) 2023-12-16T21:42:23.816 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:42:23 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:42:23.336+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:05.622506+0000 front 2023-12-16T21:39:05.622528+0000 (oldest deadline 2023-12-16T21:39:30.322546+0000) 2023-12-16T21:42:23.816 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:42:23 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:42:23.336+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:05.622479+0000 front 2023-12-16T21:39:05.622580+0000 (oldest deadline 2023-12-16T21:39:30.322546+0000) 2023-12-16T21:42:23.816 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:42:23 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:42:23.337+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:42:24.311 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:42:23 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:42:23.945+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:09.306583+0000 front 2023-12-16T21:39:09.306633+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:42:24.311 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:42:23 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:42:23.945+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:09.306551+0000 front 2023-12-16T21:39:09.306652+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:42:24.311 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:42:23 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:42:23.945+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:09.306605+0000 front 2023-12-16T21:39:09.306613+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:42:24.311 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:42:23 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:42:23.946+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:42:24.312 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:42:23 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:42:23.941+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:09.828911+0000 front 2023-12-16T21:39:09.828958+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:42:24.312 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:42:23 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:42:23.941+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:09.828868+0000 front 2023-12-16T21:39:09.828878+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:42:24.312 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:42:23 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:42:23.941+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:09.828947+0000 front 2023-12-16T21:39:09.828993+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:42:24.312 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:42:23 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:42:23.942+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:42:24.566 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:42:24 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:42:24.308+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:05.622397+0000 front 2023-12-16T21:39:05.622556+0000 (oldest deadline 2023-12-16T21:39:30.322546+0000) 2023-12-16T21:42:24.566 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:42:24 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:42:24.308+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:05.622506+0000 front 2023-12-16T21:39:05.622528+0000 (oldest deadline 2023-12-16T21:39:30.322546+0000) 2023-12-16T21:42:24.566 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:42:24 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:42:24.308+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:05.622479+0000 front 2023-12-16T21:39:05.622580+0000 (oldest deadline 2023-12-16T21:39:30.322546+0000) 2023-12-16T21:42:24.566 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:42:24 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:42:24.309+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:42:25.266 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:42:24 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:42:24.909+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:09.306583+0000 front 2023-12-16T21:39:09.306633+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:42:25.266 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:42:24 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:42:24.909+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:09.306551+0000 front 2023-12-16T21:39:09.306652+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:42:25.266 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:42:24 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:42:24.909+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:09.306605+0000 front 2023-12-16T21:39:09.306613+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:42:25.266 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:42:24 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:42:24.910+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:42:25.266 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:42:24 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:42:24.915+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:09.828911+0000 front 2023-12-16T21:39:09.828958+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:42:25.266 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:42:24 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:42:24.915+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:09.828868+0000 front 2023-12-16T21:39:09.828878+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:42:25.267 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:42:24 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:42:24.915+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:09.828947+0000 front 2023-12-16T21:39:09.828993+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:42:25.267 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:42:24 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:42:24.916+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:42:25.566 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:42:25 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:42:25.263+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:05.622397+0000 front 2023-12-16T21:39:05.622556+0000 (oldest deadline 2023-12-16T21:39:30.322546+0000) 2023-12-16T21:42:25.566 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:42:25 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:42:25.263+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:05.622506+0000 front 2023-12-16T21:39:05.622528+0000 (oldest deadline 2023-12-16T21:39:30.322546+0000) 2023-12-16T21:42:25.566 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:42:25 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:42:25.263+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:05.622479+0000 front 2023-12-16T21:39:05.622580+0000 (oldest deadline 2023-12-16T21:39:30.322546+0000) 2023-12-16T21:42:25.566 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:42:25 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:42:25.264+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:42:26.066 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:42:25 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:42:25.864+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:09.306583+0000 front 2023-12-16T21:39:09.306633+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:42:26.066 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:42:25 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:42:25.864+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:09.306551+0000 front 2023-12-16T21:39:09.306652+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:42:26.066 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:42:25 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:42:25.864+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:09.306605+0000 front 2023-12-16T21:39:09.306613+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:42:26.066 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:42:25 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:42:25.865+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:42:26.067 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:42:25 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:42:25.892+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:09.828911+0000 front 2023-12-16T21:39:09.828958+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:42:26.067 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:42:25 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:42:25.892+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:09.828868+0000 front 2023-12-16T21:39:09.828878+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:42:26.067 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:42:25 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:42:25.892+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:09.828947+0000 front 2023-12-16T21:39:09.828993+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:42:26.067 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:42:25 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:42:25.893+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:42:26.566 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Dec 16 21:42:26 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-mon-smithi045[80759]: 2023-12-16T21:42:26.335+0000 7f45a1f8f700 -1 mon.smithi045@0(probing) e2 get_health_metrics reporting 379 slow ops, oldest is monmgrreport(gid 14214, 0 checks, 1 progress events) 2023-12-16T21:42:26.566 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:42:26 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:42:26.255+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:05.622397+0000 front 2023-12-16T21:39:05.622556+0000 (oldest deadline 2023-12-16T21:39:30.322546+0000) 2023-12-16T21:42:26.567 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:42:26 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:42:26.255+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:05.622506+0000 front 2023-12-16T21:39:05.622528+0000 (oldest deadline 2023-12-16T21:39:30.322546+0000) 2023-12-16T21:42:26.567 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:42:26 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:42:26.255+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:05.622479+0000 front 2023-12-16T21:39:05.622580+0000 (oldest deadline 2023-12-16T21:39:30.322546+0000) 2023-12-16T21:42:26.567 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:42:26 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:42:26.256+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:42:27.213 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:42:26 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:42:26.845+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:09.306583+0000 front 2023-12-16T21:39:09.306633+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:42:27.213 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:42:26 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:42:26.845+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:09.306551+0000 front 2023-12-16T21:39:09.306652+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:42:27.213 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:42:26 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:42:26.845+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:09.306605+0000 front 2023-12-16T21:39:09.306613+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:42:27.213 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:42:26 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:42:26.846+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:42:27.214 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:42:26 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:42:26.934+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:09.828911+0000 front 2023-12-16T21:39:09.828958+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:42:27.214 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:42:26 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:42:26.934+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:09.828868+0000 front 2023-12-16T21:39:09.828878+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:42:27.214 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:42:26 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:42:26.934+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:09.828947+0000 front 2023-12-16T21:39:09.828993+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:42:27.214 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:42:26 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:42:26.935+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:42:27.566 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:42:27 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:42:27.210+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:05.622397+0000 front 2023-12-16T21:39:05.622556+0000 (oldest deadline 2023-12-16T21:39:30.322546+0000) 2023-12-16T21:42:27.566 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:42:27 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:42:27.210+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:05.622506+0000 front 2023-12-16T21:39:05.622528+0000 (oldest deadline 2023-12-16T21:39:30.322546+0000) 2023-12-16T21:42:27.566 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:42:27 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:42:27.210+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:05.622479+0000 front 2023-12-16T21:39:05.622580+0000 (oldest deadline 2023-12-16T21:39:30.322546+0000) 2023-12-16T21:42:27.566 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:42:27 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:42:27.211+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:42:28.243 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:42:27 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:42:27.868+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:09.306583+0000 front 2023-12-16T21:39:09.306633+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:42:28.244 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:42:27 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:42:27.868+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:09.306551+0000 front 2023-12-16T21:39:09.306652+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:42:28.244 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:42:27 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:42:27.868+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:09.306605+0000 front 2023-12-16T21:39:09.306613+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:42:28.244 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:42:27 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:42:27.870+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:42:28.244 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:42:27 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:42:27.907+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:09.828911+0000 front 2023-12-16T21:39:09.828958+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:42:28.244 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:42:27 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:42:27.907+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:09.828868+0000 front 2023-12-16T21:39:09.828878+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:42:28.244 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:42:27 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:42:27.907+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:09.828947+0000 front 2023-12-16T21:39:09.828993+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:42:28.244 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:42:27 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:42:27.908+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:42:28.566 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:42:28 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:42:28.240+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:05.622397+0000 front 2023-12-16T21:39:05.622556+0000 (oldest deadline 2023-12-16T21:39:30.322546+0000) 2023-12-16T21:42:28.566 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:42:28 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:42:28.240+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:05.622506+0000 front 2023-12-16T21:39:05.622528+0000 (oldest deadline 2023-12-16T21:39:30.322546+0000) 2023-12-16T21:42:28.566 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:42:28 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:42:28.240+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:05.622479+0000 front 2023-12-16T21:39:05.622580+0000 (oldest deadline 2023-12-16T21:39:30.322546+0000) 2023-12-16T21:42:28.566 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:42:28 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:42:28.242+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:42:29.066 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:42:28 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:42:28.847+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:09.306583+0000 front 2023-12-16T21:39:09.306633+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:42:29.066 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:42:28 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:42:28.847+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:09.306551+0000 front 2023-12-16T21:39:09.306652+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:42:29.066 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:42:28 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:42:28.847+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:09.306605+0000 front 2023-12-16T21:39:09.306613+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:42:29.066 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:42:28 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:42:28.848+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:42:29.067 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:42:28 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:42:28.876+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:09.828911+0000 front 2023-12-16T21:39:09.828958+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:42:29.067 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:42:28 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:42:28.876+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:09.828868+0000 front 2023-12-16T21:39:09.828878+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:42:29.067 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:42:28 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:42:28.876+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:09.828947+0000 front 2023-12-16T21:39:09.828993+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:42:29.067 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:42:28 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:42:28.877+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:42:29.566 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:42:29 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:42:29.282+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:05.622397+0000 front 2023-12-16T21:39:05.622556+0000 (oldest deadline 2023-12-16T21:39:30.322546+0000) 2023-12-16T21:42:29.566 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:42:29 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:42:29.282+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:05.622506+0000 front 2023-12-16T21:39:05.622528+0000 (oldest deadline 2023-12-16T21:39:30.322546+0000) 2023-12-16T21:42:29.566 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:42:29 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:42:29.282+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:05.622479+0000 front 2023-12-16T21:39:05.622580+0000 (oldest deadline 2023-12-16T21:39:30.322546+0000) 2023-12-16T21:42:29.566 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:42:29 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:42:29.282+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:42:30.290 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:42:29 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:42:29.867+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:09.828911+0000 front 2023-12-16T21:39:09.828958+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:42:30.290 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:42:29 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:42:29.867+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:09.828868+0000 front 2023-12-16T21:39:09.828878+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:42:30.290 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:42:29 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:42:29.867+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:09.828947+0000 front 2023-12-16T21:39:09.828993+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:42:30.290 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:42:29 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:42:29.868+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:42:30.290 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:42:29 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:42:29.845+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:09.306583+0000 front 2023-12-16T21:39:09.306633+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:42:30.290 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:42:29 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:42:29.845+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:09.306551+0000 front 2023-12-16T21:39:09.306652+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:42:30.291 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:42:29 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:42:29.845+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:09.306605+0000 front 2023-12-16T21:39:09.306613+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:42:30.291 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:42:29 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:42:29.846+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:42:30.566 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:42:30 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:42:30.287+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:05.622397+0000 front 2023-12-16T21:39:05.622556+0000 (oldest deadline 2023-12-16T21:39:30.322546+0000) 2023-12-16T21:42:30.566 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:42:30 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:42:30.287+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:05.622506+0000 front 2023-12-16T21:39:05.622528+0000 (oldest deadline 2023-12-16T21:39:30.322546+0000) 2023-12-16T21:42:30.566 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:42:30 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:42:30.287+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:05.622479+0000 front 2023-12-16T21:39:05.622580+0000 (oldest deadline 2023-12-16T21:39:30.322546+0000) 2023-12-16T21:42:30.566 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:42:30 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:42:30.288+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:42:31.316 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:42:30 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:42:30.909+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:09.828911+0000 front 2023-12-16T21:39:09.828958+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:42:31.316 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:42:30 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:42:30.909+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:09.828868+0000 front 2023-12-16T21:39:09.828878+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:42:31.316 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:42:30 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:42:30.909+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:09.828947+0000 front 2023-12-16T21:39:09.828993+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:42:31.317 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:42:30 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:42:30.910+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:42:31.317 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:42:30 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:42:30.842+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:09.306583+0000 front 2023-12-16T21:39:09.306633+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:42:31.317 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:42:30 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:42:30.842+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:09.306551+0000 front 2023-12-16T21:39:09.306652+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:42:31.317 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:42:30 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:42:30.842+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:09.306605+0000 front 2023-12-16T21:39:09.306613+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:42:31.317 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:42:30 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:42:30.844+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:42:31.634 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Dec 16 21:42:31 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-mon-smithi045[80759]: 2023-12-16T21:42:31.336+0000 7f45a1f8f700 -1 mon.smithi045@0(probing) e2 get_health_metrics reporting 422 slow ops, oldest is monmgrreport(gid 14214, 0 checks, 1 progress events) 2023-12-16T21:42:31.634 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:42:31 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:42:31.325+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:05.622397+0000 front 2023-12-16T21:39:05.622556+0000 (oldest deadline 2023-12-16T21:39:30.322546+0000) 2023-12-16T21:42:31.634 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:42:31 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:42:31.325+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:05.622506+0000 front 2023-12-16T21:39:05.622528+0000 (oldest deadline 2023-12-16T21:39:30.322546+0000) 2023-12-16T21:42:31.634 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:42:31 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:42:31.325+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:05.622479+0000 front 2023-12-16T21:39:05.622580+0000 (oldest deadline 2023-12-16T21:39:30.322546+0000) 2023-12-16T21:42:31.634 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:42:31 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:42:31.326+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:42:31.946 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:42:31 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:42:31.804+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:09.306583+0000 front 2023-12-16T21:39:09.306633+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:42:31.947 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:42:31 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:42:31.804+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:09.306551+0000 front 2023-12-16T21:39:09.306652+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:42:31.947 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:42:31 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:42:31.804+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:09.306605+0000 front 2023-12-16T21:39:09.306613+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:42:31.947 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:42:31 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:42:31.805+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:42:32.316 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:42:31 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:42:31.943+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:09.828911+0000 front 2023-12-16T21:39:09.828958+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:42:32.316 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:42:31 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:42:31.943+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:09.828868+0000 front 2023-12-16T21:39:09.828878+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:42:32.316 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:42:31 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:42:31.943+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:09.828947+0000 front 2023-12-16T21:39:09.828993+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:42:32.316 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:42:31 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:42:31.944+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:42:32.764 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:42:32 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:42:32.314+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:05.622397+0000 front 2023-12-16T21:39:05.622556+0000 (oldest deadline 2023-12-16T21:39:30.322546+0000) 2023-12-16T21:42:32.764 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:42:32 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:42:32.314+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:05.622506+0000 front 2023-12-16T21:39:05.622528+0000 (oldest deadline 2023-12-16T21:39:30.322546+0000) 2023-12-16T21:42:32.764 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:42:32 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:42:32.314+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:05.622479+0000 front 2023-12-16T21:39:05.622580+0000 (oldest deadline 2023-12-16T21:39:30.322546+0000) 2023-12-16T21:42:32.764 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:42:32 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:42:32.315+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:42:33.066 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:42:32 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:42:32.975+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:09.828911+0000 front 2023-12-16T21:39:09.828958+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:42:33.066 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:42:32 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:42:32.975+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:09.828868+0000 front 2023-12-16T21:39:09.828878+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:42:33.067 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:42:32 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:42:32.975+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:09.828947+0000 front 2023-12-16T21:39:09.828993+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:42:33.067 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:42:32 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:42:32.976+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:42:33.067 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:42:32 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:42:32.761+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:09.306583+0000 front 2023-12-16T21:39:09.306633+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:42:33.067 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:42:32 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:42:32.761+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:09.306551+0000 front 2023-12-16T21:39:09.306652+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:42:33.067 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:42:32 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:42:32.761+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:09.306605+0000 front 2023-12-16T21:39:09.306613+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:42:33.067 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:42:32 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:42:32.762+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:42:33.798 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:42:33 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:42:33.325+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:05.622397+0000 front 2023-12-16T21:39:05.622556+0000 (oldest deadline 2023-12-16T21:39:30.322546+0000) 2023-12-16T21:42:33.798 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:42:33 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:42:33.325+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:05.622506+0000 front 2023-12-16T21:39:05.622528+0000 (oldest deadline 2023-12-16T21:39:30.322546+0000) 2023-12-16T21:42:33.798 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:42:33 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:42:33.325+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:05.622479+0000 front 2023-12-16T21:39:05.622580+0000 (oldest deadline 2023-12-16T21:39:30.322546+0000) 2023-12-16T21:42:33.798 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:42:33 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:42:33.326+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:42:34.066 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:42:33 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:42:33.795+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:09.306583+0000 front 2023-12-16T21:39:09.306633+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:42:34.067 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:42:33 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:42:33.795+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:09.306551+0000 front 2023-12-16T21:39:09.306652+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:42:34.067 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:42:33 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:42:33.795+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:09.306605+0000 front 2023-12-16T21:39:09.306613+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:42:34.067 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:42:33 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:42:33.796+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:42:34.067 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:42:33 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:42:33.990+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:09.828911+0000 front 2023-12-16T21:39:09.828958+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:42:34.067 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:42:33 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:42:33.990+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:09.828868+0000 front 2023-12-16T21:39:09.828878+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:42:34.067 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:42:33 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:42:33.990+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:09.828947+0000 front 2023-12-16T21:39:09.828993+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:42:34.067 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:42:33 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:42:33.990+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:42:34.816 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:42:34 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:42:34.366+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:05.622397+0000 front 2023-12-16T21:39:05.622556+0000 (oldest deadline 2023-12-16T21:39:30.322546+0000) 2023-12-16T21:42:34.816 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:42:34 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:42:34.366+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:05.622506+0000 front 2023-12-16T21:39:05.622528+0000 (oldest deadline 2023-12-16T21:39:30.322546+0000) 2023-12-16T21:42:34.817 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:42:34 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:42:34.366+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:05.622479+0000 front 2023-12-16T21:39:05.622580+0000 (oldest deadline 2023-12-16T21:39:30.322546+0000) 2023-12-16T21:42:34.817 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:42:34 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:42:34.367+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:42:35.316 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:42:35 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:42:35.025+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:09.828911+0000 front 2023-12-16T21:39:09.828958+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:42:35.316 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:42:35 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:42:35.025+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:09.828868+0000 front 2023-12-16T21:39:09.828878+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:42:35.316 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:42:35 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:42:35.025+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:09.828947+0000 front 2023-12-16T21:39:09.828993+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:42:35.317 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:42:35 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:42:35.026+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:42:35.317 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:42:34 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:42:34.821+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:09.306583+0000 front 2023-12-16T21:39:09.306633+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:42:35.317 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:42:34 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:42:34.821+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:09.306551+0000 front 2023-12-16T21:39:09.306652+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:42:35.317 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:42:34 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:42:34.821+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:09.306605+0000 front 2023-12-16T21:39:09.306613+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:42:35.317 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:42:34 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:42:34.822+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:42:35.816 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:42:35 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:42:35.411+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:05.622397+0000 front 2023-12-16T21:39:05.622556+0000 (oldest deadline 2023-12-16T21:39:30.322546+0000) 2023-12-16T21:42:35.816 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:42:35 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:42:35.411+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:05.622506+0000 front 2023-12-16T21:39:05.622528+0000 (oldest deadline 2023-12-16T21:39:30.322546+0000) 2023-12-16T21:42:35.816 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:42:35 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:42:35.411+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:05.622479+0000 front 2023-12-16T21:39:05.622580+0000 (oldest deadline 2023-12-16T21:39:30.322546+0000) 2023-12-16T21:42:35.817 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:42:35 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:42:35.412+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:42:36.316 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:42:36 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:42:36.000+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:09.828911+0000 front 2023-12-16T21:39:09.828958+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:42:36.316 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:42:36 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:42:36.000+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:09.828868+0000 front 2023-12-16T21:39:09.828878+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:42:36.316 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:42:36 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:42:36.000+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:09.828947+0000 front 2023-12-16T21:39:09.828993+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:42:36.316 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:42:36 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:42:36.002+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:42:36.317 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:42:35 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:42:35.835+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:09.306583+0000 front 2023-12-16T21:39:09.306633+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:42:36.317 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:42:35 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:42:35.835+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:09.306551+0000 front 2023-12-16T21:39:09.306652+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:42:36.317 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:42:35 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:42:35.835+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:09.306605+0000 front 2023-12-16T21:39:09.306613+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:42:36.317 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:42:35 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:42:35.836+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:42:36.816 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Dec 16 21:42:36 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-mon-smithi045[80759]: 2023-12-16T21:42:36.336+0000 7f45a1f8f700 -1 mon.smithi045@0(probing) e2 get_health_metrics reporting 450 slow ops, oldest is monmgrreport(gid 14214, 0 checks, 1 progress events) 2023-12-16T21:42:36.816 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:42:36 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:42:36.382+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:05.622397+0000 front 2023-12-16T21:39:05.622556+0000 (oldest deadline 2023-12-16T21:39:30.322546+0000) 2023-12-16T21:42:36.817 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:42:36 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:42:36.382+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:05.622506+0000 front 2023-12-16T21:39:05.622528+0000 (oldest deadline 2023-12-16T21:39:30.322546+0000) 2023-12-16T21:42:36.817 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:42:36 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:42:36.382+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:05.622479+0000 front 2023-12-16T21:39:05.622580+0000 (oldest deadline 2023-12-16T21:39:30.322546+0000) 2023-12-16T21:42:36.817 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:42:36 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:42:36.382+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:42:37.316 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:42:36 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:42:36.880+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:09.306583+0000 front 2023-12-16T21:39:09.306633+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:42:37.316 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:42:36 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:42:36.880+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:09.306551+0000 front 2023-12-16T21:39:09.306652+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:42:37.316 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:42:36 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:42:36.880+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:09.306605+0000 front 2023-12-16T21:39:09.306613+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:42:37.317 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:42:36 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:42:36.881+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:42:37.317 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:42:36 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:42:36.994+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:09.828911+0000 front 2023-12-16T21:39:09.828958+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:42:37.317 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:42:36 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:42:36.994+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:09.828868+0000 front 2023-12-16T21:39:09.828878+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:42:37.317 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:42:36 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:42:36.994+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:09.828947+0000 front 2023-12-16T21:39:09.828993+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:42:37.317 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:42:36 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:42:36.995+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:42:37.816 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:42:37 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:42:37.397+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:05.622397+0000 front 2023-12-16T21:39:05.622556+0000 (oldest deadline 2023-12-16T21:39:30.322546+0000) 2023-12-16T21:42:37.816 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:42:37 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:42:37.397+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:05.622506+0000 front 2023-12-16T21:39:05.622528+0000 (oldest deadline 2023-12-16T21:39:30.322546+0000) 2023-12-16T21:42:37.816 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:42:37 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:42:37.397+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:05.622479+0000 front 2023-12-16T21:39:05.622580+0000 (oldest deadline 2023-12-16T21:39:30.322546+0000) 2023-12-16T21:42:37.816 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:42:37 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:42:37.398+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:42:38.316 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:42:37 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:42:37.973+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:09.828911+0000 front 2023-12-16T21:39:09.828958+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:42:38.316 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:42:37 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:42:37.973+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:09.828868+0000 front 2023-12-16T21:39:09.828878+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:42:38.317 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:42:37 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:42:37.973+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:09.828947+0000 front 2023-12-16T21:39:09.828993+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:42:38.317 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:42:37 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:42:37.974+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:42:38.317 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:42:37 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:42:37.907+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:09.306583+0000 front 2023-12-16T21:39:09.306633+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:42:38.317 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:42:37 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:42:37.907+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:09.306551+0000 front 2023-12-16T21:39:09.306652+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:42:38.317 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:42:37 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:42:37.907+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:09.306605+0000 front 2023-12-16T21:39:09.306613+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:42:38.317 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:42:37 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:42:37.908+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:42:38.722 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:42:38 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:42:38.366+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:05.622397+0000 front 2023-12-16T21:39:05.622556+0000 (oldest deadline 2023-12-16T21:39:30.322546+0000) 2023-12-16T21:42:38.722 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:42:38 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:42:38.366+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:05.622506+0000 front 2023-12-16T21:39:05.622528+0000 (oldest deadline 2023-12-16T21:39:30.322546+0000) 2023-12-16T21:42:38.722 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:42:38 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:42:38.366+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:05.622479+0000 front 2023-12-16T21:39:05.622580+0000 (oldest deadline 2023-12-16T21:39:30.322546+0000) 2023-12-16T21:42:38.722 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:42:38 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:42:38.366+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:42:39.066 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:42:38 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:42:38.947+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:09.306583+0000 front 2023-12-16T21:39:09.306633+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:42:39.067 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:42:38 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:42:38.947+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:09.306551+0000 front 2023-12-16T21:39:09.306652+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:42:39.067 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:42:38 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:42:38.947+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:09.306605+0000 front 2023-12-16T21:39:09.306613+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:42:39.067 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:42:38 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:42:38.948+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:42:39.067 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:42:38 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:42:38.933+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:09.828911+0000 front 2023-12-16T21:39:09.828958+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:42:39.067 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:42:38 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:42:38.933+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:09.828868+0000 front 2023-12-16T21:39:09.828878+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:42:39.067 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:42:38 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:42:38.933+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:09.828947+0000 front 2023-12-16T21:39:09.828993+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:42:39.067 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:42:38 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:42:38.934+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:42:39.816 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:42:39 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:42:39.414+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:05.622397+0000 front 2023-12-16T21:39:05.622556+0000 (oldest deadline 2023-12-16T21:39:30.322546+0000) 2023-12-16T21:42:39.816 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:42:39 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:42:39.414+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:05.622506+0000 front 2023-12-16T21:39:05.622528+0000 (oldest deadline 2023-12-16T21:39:30.322546+0000) 2023-12-16T21:42:39.816 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:42:39 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:42:39.414+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:05.622479+0000 front 2023-12-16T21:39:05.622580+0000 (oldest deadline 2023-12-16T21:39:30.322546+0000) 2023-12-16T21:42:39.817 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:42:39 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:42:39.416+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:42:40.316 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:42:39 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:42:39.971+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:09.306583+0000 front 2023-12-16T21:39:09.306633+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:42:40.316 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:42:39 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:42:39.971+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:09.306551+0000 front 2023-12-16T21:39:09.306652+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:42:40.317 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:42:39 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:42:39.971+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:09.306605+0000 front 2023-12-16T21:39:09.306613+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:42:40.317 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:42:39 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:42:39.972+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:42:40.317 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:42:39 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:42:39.907+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:09.828911+0000 front 2023-12-16T21:39:09.828958+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:42:40.317 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:42:39 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:42:39.907+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:09.828868+0000 front 2023-12-16T21:39:09.828878+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:42:40.317 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:42:39 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:42:39.907+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:09.828947+0000 front 2023-12-16T21:39:09.828993+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:42:40.317 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:42:39 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:42:39.908+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:42:40.816 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:42:40 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:42:40.433+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:05.622397+0000 front 2023-12-16T21:39:05.622556+0000 (oldest deadline 2023-12-16T21:39:30.322546+0000) 2023-12-16T21:42:40.816 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:42:40 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:42:40.433+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:05.622506+0000 front 2023-12-16T21:39:05.622528+0000 (oldest deadline 2023-12-16T21:39:30.322546+0000) 2023-12-16T21:42:40.817 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:42:40 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:42:40.433+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:05.622479+0000 front 2023-12-16T21:39:05.622580+0000 (oldest deadline 2023-12-16T21:39:30.322546+0000) 2023-12-16T21:42:40.817 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:42:40 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:42:40.434+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:42:41.316 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:42:40 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:42:40.891+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:09.828911+0000 front 2023-12-16T21:39:09.828958+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:42:41.316 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:42:40 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:42:40.891+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:09.828868+0000 front 2023-12-16T21:39:09.828878+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:42:41.317 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:42:40 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:42:40.891+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:09.828947+0000 front 2023-12-16T21:39:09.828993+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:42:41.317 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:42:40 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:42:40.892+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:42:41.317 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:42:41 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:42:41.003+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:09.306583+0000 front 2023-12-16T21:39:09.306633+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:42:41.317 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:42:41 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:42:41.003+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:09.306551+0000 front 2023-12-16T21:39:09.306652+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:42:41.317 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:42:41 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:42:41.003+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:09.306605+0000 front 2023-12-16T21:39:09.306613+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:42:41.317 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:42:41 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:42:41.004+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:42:41.634 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Dec 16 21:42:41 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-mon-smithi045[80759]: 2023-12-16T21:42:41.337+0000 7f45a1f8f700 -1 mon.smithi045@0(probing) e2 get_health_metrics reporting 481 slow ops, oldest is monmgrreport(gid 14214, 0 checks, 1 progress events) 2023-12-16T21:42:41.635 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:42:41 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:42:41.398+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:05.622397+0000 front 2023-12-16T21:39:05.622556+0000 (oldest deadline 2023-12-16T21:39:30.322546+0000) 2023-12-16T21:42:41.635 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:42:41 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:42:41.398+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:05.622506+0000 front 2023-12-16T21:39:05.622528+0000 (oldest deadline 2023-12-16T21:39:30.322546+0000) 2023-12-16T21:42:41.635 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:42:41 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:42:41.398+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:05.622479+0000 front 2023-12-16T21:39:05.622580+0000 (oldest deadline 2023-12-16T21:39:30.322546+0000) 2023-12-16T21:42:41.635 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:42:41 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:42:41.399+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:42:42.316 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:42:42 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:42:42.028+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:09.306583+0000 front 2023-12-16T21:39:09.306633+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:42:42.316 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:42:42 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:42:42.028+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:09.306551+0000 front 2023-12-16T21:39:09.306652+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:42:42.316 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:42:42 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:42:42.028+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:09.306605+0000 front 2023-12-16T21:39:09.306613+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:42:42.316 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:42:42 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:42:42.029+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:42:42.317 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:42:41 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:42:41.893+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:09.828911+0000 front 2023-12-16T21:39:09.828958+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:42:42.317 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:42:41 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:42:41.893+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:09.828868+0000 front 2023-12-16T21:39:09.828878+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:42:42.317 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:42:41 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:42:41.893+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:09.828947+0000 front 2023-12-16T21:39:09.828993+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:42:42.317 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:42:41 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:42:41.894+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:42:42.816 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:42:42 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:42:42.392+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:05.622397+0000 front 2023-12-16T21:39:05.622556+0000 (oldest deadline 2023-12-16T21:39:30.322546+0000) 2023-12-16T21:42:42.816 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:42:42 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:42:42.392+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:05.622506+0000 front 2023-12-16T21:39:05.622528+0000 (oldest deadline 2023-12-16T21:39:30.322546+0000) 2023-12-16T21:42:42.816 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:42:42 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:42:42.392+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:05.622479+0000 front 2023-12-16T21:39:05.622580+0000 (oldest deadline 2023-12-16T21:39:30.322546+0000) 2023-12-16T21:42:42.817 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:42:42 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:42:42.394+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:42:43.316 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:42:42 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:42:42.997+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:09.306583+0000 front 2023-12-16T21:39:09.306633+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:42:43.317 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:42:42 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:42:42.997+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:09.306551+0000 front 2023-12-16T21:39:09.306652+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:42:43.317 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:42:42 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:42:42.997+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:09.306605+0000 front 2023-12-16T21:39:09.306613+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:42:43.317 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:42:42 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:42:42.998+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:42:43.317 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:42:42 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:42:42.879+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:09.828911+0000 front 2023-12-16T21:39:09.828958+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:42:43.317 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:42:42 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:42:42.879+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:09.828868+0000 front 2023-12-16T21:39:09.828878+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:42:43.317 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:42:42 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:42:42.879+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:09.828947+0000 front 2023-12-16T21:39:09.828993+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:42:43.317 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:42:42 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:42:42.880+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:42:43.816 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:42:43 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:42:43.379+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:05.622397+0000 front 2023-12-16T21:39:05.622556+0000 (oldest deadline 2023-12-16T21:39:30.322546+0000) 2023-12-16T21:42:43.816 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:42:43 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:42:43.379+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:05.622506+0000 front 2023-12-16T21:39:05.622528+0000 (oldest deadline 2023-12-16T21:39:30.322546+0000) 2023-12-16T21:42:43.816 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:42:43 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:42:43.379+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:05.622479+0000 front 2023-12-16T21:39:05.622580+0000 (oldest deadline 2023-12-16T21:39:30.322546+0000) 2023-12-16T21:42:43.817 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:42:43 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:42:43.380+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:42:44.241 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:42:43 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:42:43.833+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:09.828911+0000 front 2023-12-16T21:39:09.828958+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:42:44.241 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:42:43 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:42:43.833+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:09.828868+0000 front 2023-12-16T21:39:09.828878+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:42:44.241 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:42:43 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:42:43.833+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:09.828947+0000 front 2023-12-16T21:39:09.828993+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:42:44.241 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:42:43 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:42:43.834+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:42:44.241 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:42:44 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:42:43.999+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:09.306583+0000 front 2023-12-16T21:39:09.306633+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:42:44.241 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:42:44 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:42:43.999+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:09.306551+0000 front 2023-12-16T21:39:09.306652+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:42:44.241 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:42:44 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:42:43.999+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:09.306605+0000 front 2023-12-16T21:39:09.306613+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:42:44.242 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:42:44 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:42:44.000+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:42:44.566 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:42:44 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:42:44.330+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:05.622397+0000 front 2023-12-16T21:39:05.622556+0000 (oldest deadline 2023-12-16T21:39:30.322546+0000) 2023-12-16T21:42:44.566 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:42:44 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:42:44.330+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:05.622506+0000 front 2023-12-16T21:39:05.622528+0000 (oldest deadline 2023-12-16T21:39:30.322546+0000) 2023-12-16T21:42:44.567 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:42:44 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:42:44.330+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:05.622479+0000 front 2023-12-16T21:39:05.622580+0000 (oldest deadline 2023-12-16T21:39:30.322546+0000) 2023-12-16T21:42:44.567 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:42:44 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:42:44.331+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:42:45.066 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:42:44 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:42:44.801+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:09.828911+0000 front 2023-12-16T21:39:09.828958+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:42:45.066 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:42:44 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:42:44.801+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:09.828868+0000 front 2023-12-16T21:39:09.828878+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:42:45.067 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:42:44 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:42:44.801+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:09.828947+0000 front 2023-12-16T21:39:09.828993+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:42:45.067 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:42:44 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:42:44.801+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:42:45.067 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:42:45 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:42:45.035+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:09.306583+0000 front 2023-12-16T21:39:09.306633+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:42:45.067 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:42:45 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:42:45.035+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:09.306551+0000 front 2023-12-16T21:39:09.306652+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:42:45.067 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:42:45 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:42:45.035+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:09.306605+0000 front 2023-12-16T21:39:09.306613+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:42:45.067 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:42:45 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:42:45.036+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:42:45.566 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:42:45 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:42:45.308+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:05.622397+0000 front 2023-12-16T21:39:05.622556+0000 (oldest deadline 2023-12-16T21:39:30.322546+0000) 2023-12-16T21:42:45.566 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:42:45 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:42:45.308+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:05.622506+0000 front 2023-12-16T21:39:05.622528+0000 (oldest deadline 2023-12-16T21:39:30.322546+0000) 2023-12-16T21:42:45.566 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:42:45 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:42:45.308+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:05.622479+0000 front 2023-12-16T21:39:05.622580+0000 (oldest deadline 2023-12-16T21:39:30.322546+0000) 2023-12-16T21:42:45.567 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:42:45 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:42:45.309+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:42:46.311 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:42:45 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:42:45.849+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:09.828911+0000 front 2023-12-16T21:39:09.828958+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:42:46.311 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:42:45 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:42:45.849+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:09.828868+0000 front 2023-12-16T21:39:09.828878+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:42:46.311 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:42:45 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:42:45.849+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:09.828947+0000 front 2023-12-16T21:39:09.828993+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:42:46.311 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:42:45 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:42:45.850+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:42:46.311 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:42:46 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:42:46.073+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:09.306583+0000 front 2023-12-16T21:39:09.306633+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:42:46.311 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:42:46 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:42:46.073+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:09.306551+0000 front 2023-12-16T21:39:09.306652+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:42:46.311 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:42:46 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:42:46.073+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:09.306605+0000 front 2023-12-16T21:39:09.306613+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:42:46.312 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:42:46 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:42:46.074+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:42:46.566 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Dec 16 21:42:46 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-mon-smithi045[80759]: 2023-12-16T21:42:46.337+0000 7f45a1f8f700 -1 mon.smithi045@0(probing) e2 get_health_metrics reporting 508 slow ops, oldest is monmgrreport(gid 14214, 0 checks, 1 progress events) 2023-12-16T21:42:46.567 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:42:46 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:42:46.308+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:05.622397+0000 front 2023-12-16T21:39:05.622556+0000 (oldest deadline 2023-12-16T21:39:30.322546+0000) 2023-12-16T21:42:46.567 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:42:46 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:42:46.308+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:05.622506+0000 front 2023-12-16T21:39:05.622528+0000 (oldest deadline 2023-12-16T21:39:30.322546+0000) 2023-12-16T21:42:46.567 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:42:46 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:42:46.308+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:05.622479+0000 front 2023-12-16T21:39:05.622580+0000 (oldest deadline 2023-12-16T21:39:30.322546+0000) 2023-12-16T21:42:46.567 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:42:46 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:42:46.309+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:42:47.317 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:42:47 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:42:47.041+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:09.306583+0000 front 2023-12-16T21:39:09.306633+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:42:47.317 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:42:47 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:42:47.041+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:09.306551+0000 front 2023-12-16T21:39:09.306652+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:42:47.317 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:42:47 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:42:47.041+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:09.306605+0000 front 2023-12-16T21:39:09.306613+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:42:47.317 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:42:47 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:42:47.042+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:42:47.317 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:42:46 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:42:46.845+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:09.828911+0000 front 2023-12-16T21:39:09.828958+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:42:47.317 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:42:46 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:42:46.845+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:09.828868+0000 front 2023-12-16T21:39:09.828878+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:42:47.317 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:42:46 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:42:46.845+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:09.828947+0000 front 2023-12-16T21:39:09.828993+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:42:47.317 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:42:46 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:42:46.845+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:42:47.817 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:42:47 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:42:47.332+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:05.622397+0000 front 2023-12-16T21:39:05.622556+0000 (oldest deadline 2023-12-16T21:39:30.322546+0000) 2023-12-16T21:42:47.817 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:42:47 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:42:47.332+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:05.622506+0000 front 2023-12-16T21:39:05.622528+0000 (oldest deadline 2023-12-16T21:39:30.322546+0000) 2023-12-16T21:42:47.817 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:42:47 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:42:47.332+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:05.622479+0000 front 2023-12-16T21:39:05.622580+0000 (oldest deadline 2023-12-16T21:39:30.322546+0000) 2023-12-16T21:42:47.818 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:42:47 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:42:47.333+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:42:48.316 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:42:48 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:42:48.033+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:09.306583+0000 front 2023-12-16T21:39:09.306633+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:42:48.316 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:42:48 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:42:48.033+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:09.306551+0000 front 2023-12-16T21:39:09.306652+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:42:48.316 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:42:48 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:42:48.033+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:09.306605+0000 front 2023-12-16T21:39:09.306613+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:42:48.316 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:42:48 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:42:48.034+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:42:48.317 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:42:47 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:42:47.813+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:09.828911+0000 front 2023-12-16T21:39:09.828958+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:42:48.317 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:42:47 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:42:47.813+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:09.828868+0000 front 2023-12-16T21:39:09.828878+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:42:48.317 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:42:47 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:42:47.813+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:09.828947+0000 front 2023-12-16T21:39:09.828993+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:42:48.317 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:42:47 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:42:47.815+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:42:48.722 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:42:48 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:42:48.326+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:05.622397+0000 front 2023-12-16T21:39:05.622556+0000 (oldest deadline 2023-12-16T21:39:30.322546+0000) 2023-12-16T21:42:48.722 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:42:48 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:42:48.326+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:05.622506+0000 front 2023-12-16T21:39:05.622528+0000 (oldest deadline 2023-12-16T21:39:30.322546+0000) 2023-12-16T21:42:48.722 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:42:48 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:42:48.326+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:05.622479+0000 front 2023-12-16T21:39:05.622580+0000 (oldest deadline 2023-12-16T21:39:30.322546+0000) 2023-12-16T21:42:48.722 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:42:48 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:42:48.327+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:42:49.049 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:42:48 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:42:48.792+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:09.828911+0000 front 2023-12-16T21:39:09.828958+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:42:49.049 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:42:48 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:42:48.792+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:09.828868+0000 front 2023-12-16T21:39:09.828878+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:42:49.049 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:42:48 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:42:48.792+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:09.828947+0000 front 2023-12-16T21:39:09.828993+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:42:49.049 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:42:48 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:42:48.793+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:42:49.316 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:42:49 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:42:49.043+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:09.306583+0000 front 2023-12-16T21:39:09.306633+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:42:49.316 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:42:49 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:42:49.043+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:09.306551+0000 front 2023-12-16T21:39:09.306652+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:42:49.316 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:42:49 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:42:49.043+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:09.306605+0000 front 2023-12-16T21:39:09.306613+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:42:49.316 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:42:49 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:42:49.044+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:42:49.816 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:42:49 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:42:49.364+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:05.622397+0000 front 2023-12-16T21:39:05.622556+0000 (oldest deadline 2023-12-16T21:39:30.322546+0000) 2023-12-16T21:42:49.816 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:42:49 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:42:49.364+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:05.622506+0000 front 2023-12-16T21:39:05.622528+0000 (oldest deadline 2023-12-16T21:39:30.322546+0000) 2023-12-16T21:42:49.817 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:42:49 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:42:49.364+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:05.622479+0000 front 2023-12-16T21:39:05.622580+0000 (oldest deadline 2023-12-16T21:39:30.322546+0000) 2023-12-16T21:42:49.817 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:42:49 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:42:49.365+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:42:50.085 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:42:49 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:42:49.815+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:09.828911+0000 front 2023-12-16T21:39:09.828958+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:42:50.085 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:42:49 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:42:49.815+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:09.828868+0000 front 2023-12-16T21:39:09.828878+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:42:50.085 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:42:49 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:42:49.815+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:09.828947+0000 front 2023-12-16T21:39:09.828993+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:42:50.085 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:42:49 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:42:49.815+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:42:50.370 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:42:50 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:42:50.082+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:09.306583+0000 front 2023-12-16T21:39:09.306633+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:42:50.370 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:42:50 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:42:50.082+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:09.306551+0000 front 2023-12-16T21:39:09.306652+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:42:50.370 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:42:50 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:42:50.082+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:09.306605+0000 front 2023-12-16T21:39:09.306613+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:42:50.370 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:42:50 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:42:50.083+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:42:50.778 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:42:50 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:42:50.367+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:05.622397+0000 front 2023-12-16T21:39:05.622556+0000 (oldest deadline 2023-12-16T21:39:30.322546+0000) 2023-12-16T21:42:50.778 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:42:50 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:42:50.367+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:05.622506+0000 front 2023-12-16T21:39:05.622528+0000 (oldest deadline 2023-12-16T21:39:30.322546+0000) 2023-12-16T21:42:50.779 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:42:50 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:42:50.367+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:05.622479+0000 front 2023-12-16T21:39:05.622580+0000 (oldest deadline 2023-12-16T21:39:30.322546+0000) 2023-12-16T21:42:50.779 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:42:50 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:42:50.367+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:42:51.066 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:42:50 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:42:50.775+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:09.828911+0000 front 2023-12-16T21:39:09.828958+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:42:51.067 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:42:50 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:42:50.775+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:09.828868+0000 front 2023-12-16T21:39:09.828878+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:42:51.067 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:42:50 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:42:50.775+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:09.828947+0000 front 2023-12-16T21:39:09.828993+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:42:51.067 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:42:50 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:42:50.776+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:42:51.409 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Dec 16 21:42:51 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-mon-smithi045[80759]: 2023-12-16T21:42:51.338+0000 7f45a1f8f700 -1 mon.smithi045@0(probing) e2 get_health_metrics reporting 538 slow ops, oldest is monmgrreport(gid 14214, 0 checks, 1 progress events) 2023-12-16T21:42:51.409 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:42:51 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:42:51.112+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:09.306583+0000 front 2023-12-16T21:39:09.306633+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:42:51.409 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:42:51 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:42:51.112+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:09.306551+0000 front 2023-12-16T21:39:09.306652+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:42:51.409 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:42:51 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:42:51.112+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:09.306605+0000 front 2023-12-16T21:39:09.306613+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:42:51.409 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:42:51 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:42:51.113+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:42:51.758 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:42:51 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:42:51.405+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:05.622397+0000 front 2023-12-16T21:39:05.622556+0000 (oldest deadline 2023-12-16T21:39:30.322546+0000) 2023-12-16T21:42:51.759 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:42:51 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:42:51.405+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:05.622506+0000 front 2023-12-16T21:39:05.622528+0000 (oldest deadline 2023-12-16T21:39:30.322546+0000) 2023-12-16T21:42:51.759 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:42:51 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:42:51.405+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:05.622479+0000 front 2023-12-16T21:39:05.622580+0000 (oldest deadline 2023-12-16T21:39:30.322546+0000) 2023-12-16T21:42:51.759 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:42:51 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:42:51.407+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:42:52.066 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:42:51 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:42:51.755+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:09.828911+0000 front 2023-12-16T21:39:09.828958+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:42:52.066 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:42:51 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:42:51.755+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:09.828868+0000 front 2023-12-16T21:39:09.828878+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:42:52.066 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:42:51 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:42:51.755+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:09.828947+0000 front 2023-12-16T21:39:09.828993+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:42:52.067 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:42:51 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:42:51.757+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:42:52.390 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:42:52 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:42:52.114+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:09.306583+0000 front 2023-12-16T21:39:09.306633+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:42:52.390 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:42:52 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:42:52.114+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:09.306551+0000 front 2023-12-16T21:39:09.306652+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:42:52.390 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:42:52 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:42:52.114+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:09.306605+0000 front 2023-12-16T21:39:09.306613+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:42:52.390 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:42:52 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:42:52.115+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:42:52.736 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:42:52 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:42:52.387+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:05.622397+0000 front 2023-12-16T21:39:05.622556+0000 (oldest deadline 2023-12-16T21:39:30.322546+0000) 2023-12-16T21:42:52.736 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:42:52 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:42:52.387+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:05.622506+0000 front 2023-12-16T21:39:05.622528+0000 (oldest deadline 2023-12-16T21:39:30.322546+0000) 2023-12-16T21:42:52.736 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:42:52 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:42:52.387+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:05.622479+0000 front 2023-12-16T21:39:05.622580+0000 (oldest deadline 2023-12-16T21:39:30.322546+0000) 2023-12-16T21:42:52.736 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:42:52 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:42:52.388+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:42:53.066 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:42:52 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:42:52.733+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:09.828911+0000 front 2023-12-16T21:39:09.828958+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:42:53.066 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:42:52 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:42:52.733+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:09.828868+0000 front 2023-12-16T21:39:09.828878+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:42:53.066 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:42:52 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:42:52.733+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:09.828947+0000 front 2023-12-16T21:39:09.828993+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:42:53.066 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:42:52 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:42:52.733+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:42:53.413 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:42:53 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:42:53.106+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:09.306583+0000 front 2023-12-16T21:39:09.306633+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:42:53.414 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:42:53 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:42:53.106+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:09.306551+0000 front 2023-12-16T21:39:09.306652+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:42:53.414 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:42:53 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:42:53.106+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:09.306605+0000 front 2023-12-16T21:39:09.306613+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:42:53.414 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:42:53 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:42:53.107+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:42:53.699 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:42:53 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:42:53.410+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:05.622397+0000 front 2023-12-16T21:39:05.622556+0000 (oldest deadline 2023-12-16T21:39:30.322546+0000) 2023-12-16T21:42:53.699 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:42:53 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:42:53.410+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:05.622506+0000 front 2023-12-16T21:39:05.622528+0000 (oldest deadline 2023-12-16T21:39:30.322546+0000) 2023-12-16T21:42:53.699 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:42:53 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:42:53.410+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:05.622479+0000 front 2023-12-16T21:39:05.622580+0000 (oldest deadline 2023-12-16T21:39:30.322546+0000) 2023-12-16T21:42:53.699 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:42:53 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:42:53.411+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:42:54.067 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:42:53 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:42:53.695+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:09.828911+0000 front 2023-12-16T21:39:09.828958+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:42:54.067 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:42:53 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:42:53.695+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:09.828868+0000 front 2023-12-16T21:39:09.828878+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:42:54.067 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:42:53 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:42:53.695+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:09.828947+0000 front 2023-12-16T21:39:09.828993+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:42:54.067 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:42:53 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:42:53.697+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:42:54.393 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:42:54 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:42:54.096+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:09.306583+0000 front 2023-12-16T21:39:09.306633+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:42:54.393 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:42:54 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:42:54.096+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:09.306551+0000 front 2023-12-16T21:39:09.306652+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:42:54.393 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:42:54 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:42:54.096+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:09.306605+0000 front 2023-12-16T21:39:09.306613+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:42:54.393 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:42:54 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:42:54.096+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:42:54.678 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:42:54 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:42:54.390+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:05.622397+0000 front 2023-12-16T21:39:05.622556+0000 (oldest deadline 2023-12-16T21:39:30.322546+0000) 2023-12-16T21:42:54.679 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:42:54 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:42:54.390+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:05.622506+0000 front 2023-12-16T21:39:05.622528+0000 (oldest deadline 2023-12-16T21:39:30.322546+0000) 2023-12-16T21:42:54.679 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:42:54 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:42:54.390+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:05.622479+0000 front 2023-12-16T21:39:05.622580+0000 (oldest deadline 2023-12-16T21:39:30.322546+0000) 2023-12-16T21:42:54.679 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:42:54 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:42:54.391+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:42:55.066 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:42:54 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:42:54.675+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:09.828911+0000 front 2023-12-16T21:39:09.828958+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:42:55.066 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:42:54 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:42:54.675+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:09.828868+0000 front 2023-12-16T21:39:09.828878+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:42:55.066 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:42:54 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:42:54.675+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:09.828947+0000 front 2023-12-16T21:39:09.828993+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:42:55.067 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:42:54 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:42:54.677+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:42:55.363 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:42:55 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:42:55.065+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:09.306583+0000 front 2023-12-16T21:39:09.306633+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:42:55.364 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:42:55 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:42:55.065+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:09.306551+0000 front 2023-12-16T21:39:09.306652+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:42:55.364 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:42:55 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:42:55.065+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:09.306605+0000 front 2023-12-16T21:39:09.306613+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:42:55.364 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:42:55 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:42:55.066+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:42:55.707 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:42:55 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:42:55.361+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:05.622397+0000 front 2023-12-16T21:39:05.622556+0000 (oldest deadline 2023-12-16T21:39:30.322546+0000) 2023-12-16T21:42:55.707 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:42:55 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:42:55.361+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:05.622506+0000 front 2023-12-16T21:39:05.622528+0000 (oldest deadline 2023-12-16T21:39:30.322546+0000) 2023-12-16T21:42:55.707 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:42:55 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:42:55.361+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:05.622479+0000 front 2023-12-16T21:39:05.622580+0000 (oldest deadline 2023-12-16T21:39:30.322546+0000) 2023-12-16T21:42:55.708 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:42:55 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:42:55.361+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:42:56.054 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:42:55 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:42:55.704+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:09.828911+0000 front 2023-12-16T21:39:09.828958+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:42:56.054 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:42:55 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:42:55.704+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:09.828868+0000 front 2023-12-16T21:39:09.828878+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:42:56.054 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:42:55 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:42:55.704+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:09.828947+0000 front 2023-12-16T21:39:09.828993+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:42:56.054 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:42:55 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:42:55.705+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:42:56.316 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:42:56 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:42:56.051+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:09.306583+0000 front 2023-12-16T21:39:09.306633+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:42:56.317 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:42:56 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:42:56.051+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:09.306551+0000 front 2023-12-16T21:39:09.306652+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:42:56.317 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:42:56 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:42:56.051+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:09.306605+0000 front 2023-12-16T21:39:09.306613+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:42:56.317 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:42:56 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:42:56.051+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:42:56.690 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Dec 16 21:42:56 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-mon-smithi045[80759]: 2023-12-16T21:42:56.338+0000 7f45a1f8f700 -1 mon.smithi045@0(probing) e2 get_health_metrics reporting 566 slow ops, oldest is monmgrreport(gid 14214, 0 checks, 1 progress events) 2023-12-16T21:42:56.690 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:42:56 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:42:56.367+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:05.622397+0000 front 2023-12-16T21:39:05.622556+0000 (oldest deadline 2023-12-16T21:39:30.322546+0000) 2023-12-16T21:42:56.690 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:42:56 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:42:56.367+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:05.622506+0000 front 2023-12-16T21:39:05.622528+0000 (oldest deadline 2023-12-16T21:39:30.322546+0000) 2023-12-16T21:42:56.691 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:42:56 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:42:56.367+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:05.622479+0000 front 2023-12-16T21:39:05.622580+0000 (oldest deadline 2023-12-16T21:39:30.322546+0000) 2023-12-16T21:42:56.691 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:42:56 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:42:56.368+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:42:57.054 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:42:56 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:42:56.687+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:09.828911+0000 front 2023-12-16T21:39:09.828958+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:42:57.054 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:42:56 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:42:56.687+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:09.828868+0000 front 2023-12-16T21:39:09.828878+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:42:57.054 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:42:56 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:42:56.687+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:09.828947+0000 front 2023-12-16T21:39:09.828993+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:42:57.054 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:42:56 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:42:56.688+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:42:57.316 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:42:57 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:42:57.051+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:09.306583+0000 front 2023-12-16T21:39:09.306633+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:42:57.316 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:42:57 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:42:57.051+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:09.306551+0000 front 2023-12-16T21:39:09.306652+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:42:57.316 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:42:57 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:42:57.051+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:09.306605+0000 front 2023-12-16T21:39:09.306613+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:42:57.316 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:42:57 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:42:57.051+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:42:57.665 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:42:57 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:42:57.365+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:05.622397+0000 front 2023-12-16T21:39:05.622556+0000 (oldest deadline 2023-12-16T21:39:30.322546+0000) 2023-12-16T21:42:57.665 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:42:57 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:42:57.365+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:05.622506+0000 front 2023-12-16T21:39:05.622528+0000 (oldest deadline 2023-12-16T21:39:30.322546+0000) 2023-12-16T21:42:57.665 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:42:57 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:42:57.365+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:05.622479+0000 front 2023-12-16T21:39:05.622580+0000 (oldest deadline 2023-12-16T21:39:30.322546+0000) 2023-12-16T21:42:57.665 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:42:57 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:42:57.365+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:42:58.039 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:42:57 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:42:57.662+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:09.828911+0000 front 2023-12-16T21:39:09.828958+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:42:58.039 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:42:57 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:42:57.662+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:09.828868+0000 front 2023-12-16T21:39:09.828878+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:42:58.039 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:42:57 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:42:57.662+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:09.828947+0000 front 2023-12-16T21:39:09.828993+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:42:58.039 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:42:57 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:42:57.663+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:42:58.316 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:42:58 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:42:58.036+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:09.306583+0000 front 2023-12-16T21:39:09.306633+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:42:58.317 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:42:58 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:42:58.036+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:09.306551+0000 front 2023-12-16T21:39:09.306652+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:42:58.317 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:42:58 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:42:58.036+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:09.306605+0000 front 2023-12-16T21:39:09.306613+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:42:58.317 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:42:58 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:42:58.037+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:42:58.693 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:42:58 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:42:58.355+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:05.622397+0000 front 2023-12-16T21:39:05.622556+0000 (oldest deadline 2023-12-16T21:39:30.322546+0000) 2023-12-16T21:42:58.693 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:42:58 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:42:58.355+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:05.622506+0000 front 2023-12-16T21:39:05.622528+0000 (oldest deadline 2023-12-16T21:39:30.322546+0000) 2023-12-16T21:42:58.693 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:42:58 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:42:58.355+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:05.622479+0000 front 2023-12-16T21:39:05.622580+0000 (oldest deadline 2023-12-16T21:39:30.322546+0000) 2023-12-16T21:42:58.693 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:42:58 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:42:58.356+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:42:59.066 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:42:58 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:42:58.687+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:09.828911+0000 front 2023-12-16T21:39:09.828958+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:42:59.066 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:42:58 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:42:58.687+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:09.828868+0000 front 2023-12-16T21:39:09.828878+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:42:59.066 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:42:58 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:42:58.687+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:09.828947+0000 front 2023-12-16T21:39:09.828993+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:42:59.066 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:42:58 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:42:58.689+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:42:59.385 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:42:59 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:42:59.086+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:09.306583+0000 front 2023-12-16T21:39:09.306633+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:42:59.385 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:42:59 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:42:59.086+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:09.306551+0000 front 2023-12-16T21:39:09.306652+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:42:59.385 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:42:59 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:42:59.086+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:09.306605+0000 front 2023-12-16T21:39:09.306613+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:42:59.385 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:42:59 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:42:59.087+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:42:59.737 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:42:59 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:42:59.382+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:05.622397+0000 front 2023-12-16T21:39:05.622556+0000 (oldest deadline 2023-12-16T21:39:30.322546+0000) 2023-12-16T21:42:59.737 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:42:59 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:42:59.382+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:05.622506+0000 front 2023-12-16T21:39:05.622528+0000 (oldest deadline 2023-12-16T21:39:30.322546+0000) 2023-12-16T21:42:59.737 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:42:59 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:42:59.382+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:05.622479+0000 front 2023-12-16T21:39:05.622580+0000 (oldest deadline 2023-12-16T21:39:30.322546+0000) 2023-12-16T21:42:59.737 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:42:59 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:42:59.383+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:43:00.066 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:42:59 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:42:59.734+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:09.828911+0000 front 2023-12-16T21:39:09.828958+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:43:00.067 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:42:59 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:42:59.734+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:09.828868+0000 front 2023-12-16T21:39:09.828878+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:43:00.067 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:42:59 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:42:59.734+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:09.828947+0000 front 2023-12-16T21:39:09.828993+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:43:00.067 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:42:59 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:42:59.735+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:43:00.402 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:43:00 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:43:00.137+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:09.306583+0000 front 2023-12-16T21:39:09.306633+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:43:00.402 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:43:00 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:43:00.137+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:09.306551+0000 front 2023-12-16T21:39:09.306652+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:43:00.402 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:43:00 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:43:00.137+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:09.306605+0000 front 2023-12-16T21:39:09.306613+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:43:00.402 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:43:00 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:43:00.137+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:43:00.703 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:43:00 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:43:00.399+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:05.622397+0000 front 2023-12-16T21:39:05.622556+0000 (oldest deadline 2023-12-16T21:39:30.322546+0000) 2023-12-16T21:43:00.704 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:43:00 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:43:00.399+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:05.622506+0000 front 2023-12-16T21:39:05.622528+0000 (oldest deadline 2023-12-16T21:39:30.322546+0000) 2023-12-16T21:43:00.704 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:43:00 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:43:00.399+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:05.622479+0000 front 2023-12-16T21:39:05.622580+0000 (oldest deadline 2023-12-16T21:39:30.322546+0000) 2023-12-16T21:43:00.704 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:43:00 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:43:00.400+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:43:01.066 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:43:00 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:43:00.700+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:09.828911+0000 front 2023-12-16T21:39:09.828958+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:43:01.067 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:43:00 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:43:00.700+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:09.828868+0000 front 2023-12-16T21:39:09.828878+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:43:01.067 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:43:00 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:43:00.700+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:09.828947+0000 front 2023-12-16T21:39:09.828993+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:43:01.067 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:43:00 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:43:00.701+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:43:01.366 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Dec 16 21:43:01 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-mon-smithi045[80759]: 2023-12-16T21:43:01.339+0000 7f45a1f8f700 -1 mon.smithi045@0(probing) e2 get_health_metrics reporting 597 slow ops, oldest is monmgrreport(gid 14214, 0 checks, 1 progress events) 2023-12-16T21:43:01.366 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:43:01 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:43:01.353+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:05.622397+0000 front 2023-12-16T21:39:05.622556+0000 (oldest deadline 2023-12-16T21:39:30.322546+0000) 2023-12-16T21:43:01.367 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:43:01 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:43:01.353+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:05.622506+0000 front 2023-12-16T21:39:05.622528+0000 (oldest deadline 2023-12-16T21:39:30.322546+0000) 2023-12-16T21:43:01.367 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:43:01 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:43:01.353+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:05.622479+0000 front 2023-12-16T21:39:05.622580+0000 (oldest deadline 2023-12-16T21:39:30.322546+0000) 2023-12-16T21:43:01.367 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:43:01 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:43:01.355+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:43:01.367 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:43:01 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:43:01.108+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:09.306583+0000 front 2023-12-16T21:39:09.306633+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:43:01.367 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:43:01 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:43:01.108+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:09.306551+0000 front 2023-12-16T21:39:09.306652+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:43:01.367 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:43:01 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:43:01.108+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:09.306605+0000 front 2023-12-16T21:39:09.306613+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:43:01.367 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:43:01 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:43:01.109+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:43:02.066 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:43:01 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:43:01.672+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:09.828911+0000 front 2023-12-16T21:39:09.828958+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:43:02.066 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:43:01 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:43:01.672+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:09.828868+0000 front 2023-12-16T21:39:09.828878+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:43:02.066 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:43:01 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:43:01.672+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:09.828947+0000 front 2023-12-16T21:39:09.828993+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:43:02.066 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:43:01 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:43:01.673+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:43:02.566 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:43:02 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:43:02.156+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:09.306583+0000 front 2023-12-16T21:39:09.306633+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:43:02.566 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:43:02 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:43:02.156+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:09.306551+0000 front 2023-12-16T21:39:09.306652+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:43:02.566 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:43:02 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:43:02.156+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:09.306605+0000 front 2023-12-16T21:39:09.306613+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:43:02.566 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:43:02 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:43:02.157+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:43:02.567 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:43:02 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:43:02.382+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:05.622397+0000 front 2023-12-16T21:39:05.622556+0000 (oldest deadline 2023-12-16T21:39:30.322546+0000) 2023-12-16T21:43:02.567 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:43:02 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:43:02.382+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:05.622506+0000 front 2023-12-16T21:39:05.622528+0000 (oldest deadline 2023-12-16T21:39:30.322546+0000) 2023-12-16T21:43:02.567 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:43:02 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:43:02.382+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:05.622479+0000 front 2023-12-16T21:39:05.622580+0000 (oldest deadline 2023-12-16T21:39:30.322546+0000) 2023-12-16T21:43:02.567 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:43:02 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:43:02.383+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:43:03.067 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:43:02 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:43:02.683+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:09.828911+0000 front 2023-12-16T21:39:09.828958+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:43:03.067 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:43:02 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:43:02.683+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:09.828868+0000 front 2023-12-16T21:39:09.828878+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:43:03.067 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:43:02 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:43:02.683+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:09.828947+0000 front 2023-12-16T21:39:09.828993+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:43:03.067 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:43:02 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:43:02.684+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:43:03.527 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:43:03 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:43:03.155+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:09.306583+0000 front 2023-12-16T21:39:09.306633+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:43:03.527 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:43:03 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:43:03.155+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:09.306551+0000 front 2023-12-16T21:39:09.306652+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:43:03.527 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:43:03 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:43:03.155+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:09.306605+0000 front 2023-12-16T21:39:09.306613+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:43:03.528 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:43:03 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:43:03.156+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:43:03.528 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:43:03 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:43:03.383+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:05.622397+0000 front 2023-12-16T21:39:05.622556+0000 (oldest deadline 2023-12-16T21:39:30.322546+0000) 2023-12-16T21:43:03.528 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:43:03 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:43:03.384+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:05.622506+0000 front 2023-12-16T21:39:05.622528+0000 (oldest deadline 2023-12-16T21:39:30.322546+0000) 2023-12-16T21:43:03.528 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:43:03 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:43:03.384+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:05.622479+0000 front 2023-12-16T21:39:05.622580+0000 (oldest deadline 2023-12-16T21:39:30.322546+0000) 2023-12-16T21:43:03.529 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:43:03 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:43:03.385+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:43:03.816 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:43:03 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:43:03.711+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:09.828911+0000 front 2023-12-16T21:39:09.828958+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:43:03.816 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:43:03 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:43:03.711+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:09.828868+0000 front 2023-12-16T21:39:09.828878+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:43:03.816 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:43:03 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:43:03.711+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:09.828947+0000 front 2023-12-16T21:39:09.828993+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:43:03.816 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:43:03 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:43:03.712+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:43:04.566 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:43:04 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:43:04.108+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:09.306583+0000 front 2023-12-16T21:39:09.306633+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:43:04.566 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:43:04 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:43:04.108+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:09.306551+0000 front 2023-12-16T21:39:09.306652+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:43:04.566 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:43:04 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:43:04.108+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:09.306605+0000 front 2023-12-16T21:39:09.306613+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:43:04.567 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:43:04 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:43:04.108+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:43:04.567 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:43:04 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:43:04.345+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:05.622397+0000 front 2023-12-16T21:39:05.622556+0000 (oldest deadline 2023-12-16T21:39:30.322546+0000) 2023-12-16T21:43:04.567 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:43:04 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:43:04.345+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:05.622506+0000 front 2023-12-16T21:39:05.622528+0000 (oldest deadline 2023-12-16T21:39:30.322546+0000) 2023-12-16T21:43:04.567 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:43:04 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:43:04.345+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:05.622479+0000 front 2023-12-16T21:39:05.622580+0000 (oldest deadline 2023-12-16T21:39:30.322546+0000) 2023-12-16T21:43:04.567 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:43:04 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:43:04.346+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:43:05.066 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:43:04 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:43:04.729+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:09.828911+0000 front 2023-12-16T21:39:09.828958+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:43:05.066 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:43:04 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:43:04.729+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:09.828868+0000 front 2023-12-16T21:39:09.828878+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:43:05.066 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:43:04 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:43:04.729+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:09.828947+0000 front 2023-12-16T21:39:09.828993+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:43:05.067 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:43:04 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:43:04.730+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:43:05.566 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:43:05 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:43:05.320+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:05.622397+0000 front 2023-12-16T21:39:05.622556+0000 (oldest deadline 2023-12-16T21:39:30.322546+0000) 2023-12-16T21:43:05.567 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:43:05 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:43:05.320+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:05.622506+0000 front 2023-12-16T21:39:05.622528+0000 (oldest deadline 2023-12-16T21:39:30.322546+0000) 2023-12-16T21:43:05.567 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:43:05 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:43:05.320+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:05.622479+0000 front 2023-12-16T21:39:05.622580+0000 (oldest deadline 2023-12-16T21:39:30.322546+0000) 2023-12-16T21:43:05.567 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:43:05 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:43:05.321+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:43:05.567 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:43:05 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:43:05.104+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:09.306583+0000 front 2023-12-16T21:39:09.306633+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:43:05.567 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:43:05 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:43:05.104+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:09.306551+0000 front 2023-12-16T21:39:09.306652+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:43:05.567 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:43:05 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:43:05.104+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:09.306605+0000 front 2023-12-16T21:39:09.306613+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:43:05.567 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:43:05 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:43:05.105+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:43:06.066 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:43:05 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:43:05.715+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:09.828911+0000 front 2023-12-16T21:39:09.828958+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:43:06.067 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:43:05 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:43:05.715+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:09.828868+0000 front 2023-12-16T21:39:09.828878+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:43:06.067 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:43:05 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:43:05.715+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:09.828947+0000 front 2023-12-16T21:39:09.828993+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:43:06.067 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:43:05 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:43:05.717+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:43:06.465 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Dec 16 21:43:06 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-mon-smithi045[80759]: 2023-12-16T21:43:06.339+0000 7f45a1f8f700 -1 mon.smithi045@0(probing) e2 get_health_metrics reporting 624 slow ops, oldest is monmgrreport(gid 14214, 0 checks, 1 progress events) 2023-12-16T21:43:06.466 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:43:06 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:43:06.128+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:09.306583+0000 front 2023-12-16T21:39:09.306633+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:43:06.466 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:43:06 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:43:06.128+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:09.306551+0000 front 2023-12-16T21:39:09.306652+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:43:06.466 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:43:06 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:43:06.128+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:09.306605+0000 front 2023-12-16T21:39:09.306613+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:43:06.466 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:43:06 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:43:06.129+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:43:06.466 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:43:06 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:43:06.300+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:05.622397+0000 front 2023-12-16T21:39:05.622556+0000 (oldest deadline 2023-12-16T21:39:30.322546+0000) 2023-12-16T21:43:06.466 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:43:06 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:43:06.300+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:05.622506+0000 front 2023-12-16T21:39:05.622528+0000 (oldest deadline 2023-12-16T21:39:30.322546+0000) 2023-12-16T21:43:06.466 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:43:06 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:43:06.300+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:05.622479+0000 front 2023-12-16T21:39:05.622580+0000 (oldest deadline 2023-12-16T21:39:30.322546+0000) 2023-12-16T21:43:06.467 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:43:06 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:43:06.301+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:43:07.067 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:43:06 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:43:06.717+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:09.828911+0000 front 2023-12-16T21:39:09.828958+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:43:07.067 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:43:06 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:43:06.717+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:09.828868+0000 front 2023-12-16T21:39:09.828878+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:43:07.067 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:43:06 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:43:06.717+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:09.828947+0000 front 2023-12-16T21:39:09.828993+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:43:07.067 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:43:06 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:43:06.718+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:43:07.566 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:43:07 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:43:07.169+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:09.306583+0000 front 2023-12-16T21:39:09.306633+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:43:07.567 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:43:07 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:43:07.169+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:09.306551+0000 front 2023-12-16T21:39:09.306652+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:43:07.567 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:43:07 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:43:07.169+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:09.306605+0000 front 2023-12-16T21:39:09.306613+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:43:07.567 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:43:07 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:43:07.170+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:43:07.567 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:43:07 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:43:07.281+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:05.622397+0000 front 2023-12-16T21:39:05.622556+0000 (oldest deadline 2023-12-16T21:39:30.322546+0000) 2023-12-16T21:43:07.567 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:43:07 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:43:07.281+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:05.622506+0000 front 2023-12-16T21:39:05.622528+0000 (oldest deadline 2023-12-16T21:39:30.322546+0000) 2023-12-16T21:43:07.567 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:43:07 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:43:07.281+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:05.622479+0000 front 2023-12-16T21:39:05.622580+0000 (oldest deadline 2023-12-16T21:39:30.322546+0000) 2023-12-16T21:43:07.567 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:43:07 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:43:07.281+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:43:08.066 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:43:07 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:43:07.675+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:09.828911+0000 front 2023-12-16T21:39:09.828958+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:43:08.066 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:43:07 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:43:07.675+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:09.828868+0000 front 2023-12-16T21:39:09.828878+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:43:08.066 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:43:07 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:43:07.675+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:09.828947+0000 front 2023-12-16T21:39:09.828993+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:43:08.066 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:43:07 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:43:07.676+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:43:08.567 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:43:08 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:43:08.287+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:05.622397+0000 front 2023-12-16T21:39:05.622556+0000 (oldest deadline 2023-12-16T21:39:30.322546+0000) 2023-12-16T21:43:08.567 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:43:08 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:43:08.287+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:05.622506+0000 front 2023-12-16T21:39:05.622528+0000 (oldest deadline 2023-12-16T21:39:30.322546+0000) 2023-12-16T21:43:08.567 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:43:08 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:43:08.287+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:05.622479+0000 front 2023-12-16T21:39:05.622580+0000 (oldest deadline 2023-12-16T21:39:30.322546+0000) 2023-12-16T21:43:08.567 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:43:08 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:43:08.287+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:43:08.567 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:43:08 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:43:08.129+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:09.306583+0000 front 2023-12-16T21:39:09.306633+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:43:08.567 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:43:08 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:43:08.129+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:09.306551+0000 front 2023-12-16T21:39:09.306652+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:43:08.567 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:43:08 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:43:08.129+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:09.306605+0000 front 2023-12-16T21:39:09.306613+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:43:08.568 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:43:08 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:43:08.130+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:43:09.066 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:43:08 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:43:08.653+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:09.828911+0000 front 2023-12-16T21:39:09.828958+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:43:09.066 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:43:08 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:43:08.653+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:09.828868+0000 front 2023-12-16T21:39:09.828878+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:43:09.066 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:43:08 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:43:08.653+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:09.828947+0000 front 2023-12-16T21:39:09.828993+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:43:09.066 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:43:08 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:43:08.654+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:43:09.566 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:43:09 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:43:09.283+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:05.622397+0000 front 2023-12-16T21:39:05.622556+0000 (oldest deadline 2023-12-16T21:39:30.322546+0000) 2023-12-16T21:43:09.566 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:43:09 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:43:09.283+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:05.622506+0000 front 2023-12-16T21:39:05.622528+0000 (oldest deadline 2023-12-16T21:39:30.322546+0000) 2023-12-16T21:43:09.567 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:43:09 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:43:09.283+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:05.622479+0000 front 2023-12-16T21:39:05.622580+0000 (oldest deadline 2023-12-16T21:39:30.322546+0000) 2023-12-16T21:43:09.567 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:43:09 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:43:09.283+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:43:09.567 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:43:09 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:43:09.107+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:09.306583+0000 front 2023-12-16T21:39:09.306633+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:43:09.567 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:43:09 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:43:09.107+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:09.306551+0000 front 2023-12-16T21:39:09.306652+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:43:09.567 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:43:09 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:43:09.107+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:09.306605+0000 front 2023-12-16T21:39:09.306613+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:43:09.567 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:43:09 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:43:09.108+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:43:10.064 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:43:09 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:43:09.604+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:09.828911+0000 front 2023-12-16T21:39:09.828958+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:43:10.065 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:43:09 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:43:09.604+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:09.828868+0000 front 2023-12-16T21:39:09.828878+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:43:10.065 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:43:09 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:43:09.604+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:09.828947+0000 front 2023-12-16T21:39:09.828993+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:43:10.065 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:43:09 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:43:09.605+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:43:10.316 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:43:10 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:43:10.256+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:05.622397+0000 front 2023-12-16T21:39:05.622556+0000 (oldest deadline 2023-12-16T21:39:30.322546+0000) 2023-12-16T21:43:10.317 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:43:10 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:43:10.256+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:05.622506+0000 front 2023-12-16T21:39:05.622528+0000 (oldest deadline 2023-12-16T21:39:30.322546+0000) 2023-12-16T21:43:10.317 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:43:10 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:43:10.256+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:05.622479+0000 front 2023-12-16T21:39:05.622580+0000 (oldest deadline 2023-12-16T21:39:30.322546+0000) 2023-12-16T21:43:10.317 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:43:10 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:43:10.257+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:43:10.317 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:43:10 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:43:10.061+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:09.306583+0000 front 2023-12-16T21:39:09.306633+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:43:10.317 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:43:10 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:43:10.061+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:09.306551+0000 front 2023-12-16T21:39:09.306652+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:43:10.318 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:43:10 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:43:10.061+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:09.306605+0000 front 2023-12-16T21:39:09.306613+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:43:10.318 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:43:10 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:43:10.063+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:43:10.816 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:43:10 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:43:10.558+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:09.828911+0000 front 2023-12-16T21:39:09.828958+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:43:10.816 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:43:10 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:43:10.558+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:09.828868+0000 front 2023-12-16T21:39:09.828878+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:43:10.816 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:43:10 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:43:10.558+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:09.828947+0000 front 2023-12-16T21:39:09.828993+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:43:10.816 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:43:10 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:43:10.559+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:43:11.316 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:43:11 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:43:11.216+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:05.622397+0000 front 2023-12-16T21:39:05.622556+0000 (oldest deadline 2023-12-16T21:39:30.322546+0000) 2023-12-16T21:43:11.317 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:43:11 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:43:11.216+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:05.622506+0000 front 2023-12-16T21:39:05.622528+0000 (oldest deadline 2023-12-16T21:39:30.322546+0000) 2023-12-16T21:43:11.317 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:43:11 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:43:11.216+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:05.622479+0000 front 2023-12-16T21:39:05.622580+0000 (oldest deadline 2023-12-16T21:39:30.322546+0000) 2023-12-16T21:43:11.317 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:43:11 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:43:11.216+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:43:11.317 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:43:11 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:43:11.034+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:09.306583+0000 front 2023-12-16T21:39:09.306633+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:43:11.317 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:43:11 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:43:11.034+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:09.306551+0000 front 2023-12-16T21:39:09.306652+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:43:11.317 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:43:11 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:43:11.034+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:09.306605+0000 front 2023-12-16T21:39:09.306613+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:43:11.317 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:43:11 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:43:11.035+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:43:11.637 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Dec 16 21:43:11 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-mon-smithi045[80759]: 2023-12-16T21:43:11.340+0000 7f45a1f8f700 -1 mon.smithi045@0(probing) e2 get_health_metrics reporting 654 slow ops, oldest is monmgrreport(gid 14214, 0 checks, 1 progress events) 2023-12-16T21:43:11.637 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:43:11 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:43:11.524+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:09.828911+0000 front 2023-12-16T21:39:09.828958+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:43:11.637 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:43:11 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:43:11.524+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:09.828868+0000 front 2023-12-16T21:39:09.828878+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:43:11.637 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:43:11 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:43:11.524+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:09.828947+0000 front 2023-12-16T21:39:09.828993+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:43:11.637 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:43:11 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:43:11.524+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:43:12.198 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:43:12 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:43:12.070+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:09.306583+0000 front 2023-12-16T21:39:09.306633+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:43:12.198 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:43:12 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:43:12.070+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:09.306551+0000 front 2023-12-16T21:39:09.306652+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:43:12.198 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:43:12 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:43:12.070+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:09.306605+0000 front 2023-12-16T21:39:09.306613+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:43:12.199 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:43:12 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:43:12.071+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:43:12.483 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:43:12 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:43:12.195+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:05.622397+0000 front 2023-12-16T21:39:05.622556+0000 (oldest deadline 2023-12-16T21:39:30.322546+0000) 2023-12-16T21:43:12.483 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:43:12 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:43:12.195+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:05.622506+0000 front 2023-12-16T21:39:05.622528+0000 (oldest deadline 2023-12-16T21:39:30.322546+0000) 2023-12-16T21:43:12.483 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:43:12 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:43:12.195+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:05.622479+0000 front 2023-12-16T21:39:05.622580+0000 (oldest deadline 2023-12-16T21:39:30.322546+0000) 2023-12-16T21:43:12.483 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:43:12 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:43:12.196+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:43:12.816 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:43:12 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:43:12.480+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:09.828911+0000 front 2023-12-16T21:39:09.828958+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:43:12.816 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:43:12 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:43:12.480+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:09.828868+0000 front 2023-12-16T21:39:09.828878+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:43:12.816 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:43:12 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:43:12.480+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:09.828947+0000 front 2023-12-16T21:39:09.828993+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:43:12.816 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:43:12 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:43:12.480+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:43:13.528 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:43:13 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:43:13.090+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:09.306583+0000 front 2023-12-16T21:39:09.306633+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:43:13.528 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:43:13 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:43:13.090+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:09.306551+0000 front 2023-12-16T21:39:09.306652+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:43:13.528 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:43:13 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:43:13.090+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:09.306605+0000 front 2023-12-16T21:39:09.306613+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:43:13.528 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:43:13 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:43:13.090+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:43:13.528 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:43:13 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:43:13.173+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:05.622397+0000 front 2023-12-16T21:39:05.622556+0000 (oldest deadline 2023-12-16T21:39:30.322546+0000) 2023-12-16T21:43:13.528 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:43:13 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:43:13.173+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:05.622506+0000 front 2023-12-16T21:39:05.622528+0000 (oldest deadline 2023-12-16T21:39:30.322546+0000) 2023-12-16T21:43:13.529 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:43:13 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:43:13.173+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:05.622479+0000 front 2023-12-16T21:39:05.622580+0000 (oldest deadline 2023-12-16T21:39:30.322546+0000) 2023-12-16T21:43:13.529 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:43:13 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:43:13.174+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:43:13.816 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:43:13 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:43:13.526+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:09.828911+0000 front 2023-12-16T21:39:09.828958+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:43:13.816 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:43:13 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:43:13.527+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:09.828868+0000 front 2023-12-16T21:39:09.828878+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:43:13.816 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:43:13 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:43:13.527+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:09.828947+0000 front 2023-12-16T21:39:09.828993+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:43:13.816 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:43:13 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:43:13.528+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:43:14.517 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:43:14 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:43:14.217+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:05.622397+0000 front 2023-12-16T21:39:05.622556+0000 (oldest deadline 2023-12-16T21:39:30.322546+0000) 2023-12-16T21:43:14.517 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:43:14 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:43:14.217+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:05.622506+0000 front 2023-12-16T21:39:05.622528+0000 (oldest deadline 2023-12-16T21:39:30.322546+0000) 2023-12-16T21:43:14.517 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:43:14 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:43:14.217+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:05.622479+0000 front 2023-12-16T21:39:05.622580+0000 (oldest deadline 2023-12-16T21:39:30.322546+0000) 2023-12-16T21:43:14.517 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:43:14 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:43:14.218+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:43:14.517 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:43:14 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:43:14.116+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:09.306583+0000 front 2023-12-16T21:39:09.306633+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:43:14.517 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:43:14 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:43:14.116+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:09.306551+0000 front 2023-12-16T21:39:09.306652+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:43:14.517 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:43:14 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:43:14.116+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:09.306605+0000 front 2023-12-16T21:39:09.306613+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:43:14.518 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:43:14 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:43:14.117+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:43:14.816 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:43:14 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:43:14.514+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:09.828911+0000 front 2023-12-16T21:39:09.828958+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:43:14.816 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:43:14 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:43:14.514+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:09.828868+0000 front 2023-12-16T21:39:09.828878+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:43:14.816 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:43:14 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:43:14.514+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:09.828947+0000 front 2023-12-16T21:39:09.828993+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:43:14.816 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:43:14 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:43:14.515+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:43:15.316 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:43:15 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:43:15.082+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:09.306583+0000 front 2023-12-16T21:39:09.306633+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:43:15.316 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:43:15 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:43:15.082+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:09.306551+0000 front 2023-12-16T21:39:09.306652+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:43:15.317 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:43:15 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:43:15.082+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:09.306605+0000 front 2023-12-16T21:39:09.306613+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:43:15.317 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:43:15 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:43:15.082+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:43:15.317 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:43:15 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:43:15.187+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:05.622397+0000 front 2023-12-16T21:39:05.622556+0000 (oldest deadline 2023-12-16T21:39:30.322546+0000) 2023-12-16T21:43:15.317 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:43:15 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:43:15.188+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:05.622506+0000 front 2023-12-16T21:39:05.622528+0000 (oldest deadline 2023-12-16T21:39:30.322546+0000) 2023-12-16T21:43:15.317 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:43:15 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:43:15.188+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:05.622479+0000 front 2023-12-16T21:39:05.622580+0000 (oldest deadline 2023-12-16T21:39:30.322546+0000) 2023-12-16T21:43:15.317 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:43:15 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:43:15.188+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:43:15.816 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:43:15 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:43:15.487+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:09.828911+0000 front 2023-12-16T21:39:09.828958+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:43:15.816 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:43:15 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:43:15.487+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:09.828868+0000 front 2023-12-16T21:39:09.828878+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:43:15.816 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:43:15 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:43:15.487+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:09.828947+0000 front 2023-12-16T21:39:09.828993+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:43:15.816 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:43:15 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:43:15.488+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:43:16.469 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Dec 16 21:43:16 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-mon-smithi045[80759]: 2023-12-16T21:43:16.341+0000 7f45a1f8f700 -1 mon.smithi045@0(probing) e2 get_health_metrics reporting 684 slow ops, oldest is monmgrreport(gid 14214, 0 checks, 1 progress events) 2023-12-16T21:43:16.469 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:43:16 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:43:16.158+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:05.622397+0000 front 2023-12-16T21:39:05.622556+0000 (oldest deadline 2023-12-16T21:39:30.322546+0000) 2023-12-16T21:43:16.469 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:43:16 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:43:16.158+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:05.622506+0000 front 2023-12-16T21:39:05.622528+0000 (oldest deadline 2023-12-16T21:39:30.322546+0000) 2023-12-16T21:43:16.469 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:43:16 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:43:16.158+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:05.622479+0000 front 2023-12-16T21:39:05.622580+0000 (oldest deadline 2023-12-16T21:39:30.322546+0000) 2023-12-16T21:43:16.469 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:43:16 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:43:16.159+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:43:16.470 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:43:16 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:43:16.127+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:09.306583+0000 front 2023-12-16T21:39:09.306633+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:43:16.470 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:43:16 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:43:16.127+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:09.306551+0000 front 2023-12-16T21:39:09.306652+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:43:16.470 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:43:16 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:43:16.127+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:09.306605+0000 front 2023-12-16T21:39:09.306613+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:43:16.470 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:43:16 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:43:16.128+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:43:16.816 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:43:16 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:43:16.487+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:09.828911+0000 front 2023-12-16T21:39:09.828958+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:43:16.816 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:43:16 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:43:16.487+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:09.828868+0000 front 2023-12-16T21:39:09.828878+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:43:16.816 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:43:16 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:43:16.487+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:09.828947+0000 front 2023-12-16T21:39:09.828993+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:43:16.816 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:43:16 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:43:16.488+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:43:17.483 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:43:17 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:43:17.117+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:05.622397+0000 front 2023-12-16T21:39:05.622556+0000 (oldest deadline 2023-12-16T21:39:30.322546+0000) 2023-12-16T21:43:17.483 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:43:17 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:43:17.117+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:05.622506+0000 front 2023-12-16T21:39:05.622528+0000 (oldest deadline 2023-12-16T21:39:30.322546+0000) 2023-12-16T21:43:17.483 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:43:17 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:43:17.117+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:05.622479+0000 front 2023-12-16T21:39:05.622580+0000 (oldest deadline 2023-12-16T21:39:30.322546+0000) 2023-12-16T21:43:17.483 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:43:17 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:43:17.118+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:43:17.483 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:43:17 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:43:17.131+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:09.306583+0000 front 2023-12-16T21:39:09.306633+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:43:17.484 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:43:17 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:43:17.131+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:09.306551+0000 front 2023-12-16T21:39:09.306652+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:43:17.484 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:43:17 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:43:17.131+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:09.306605+0000 front 2023-12-16T21:39:09.306613+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:43:17.484 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:43:17 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:43:17.132+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:43:17.816 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:43:17 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:43:17.480+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:09.828911+0000 front 2023-12-16T21:39:09.828958+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:43:17.816 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:43:17 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:43:17.480+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:09.828868+0000 front 2023-12-16T21:39:09.828878+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:43:17.816 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:43:17 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:43:17.480+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:09.828947+0000 front 2023-12-16T21:39:09.828993+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:43:17.816 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:43:17 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:43:17.481+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:43:18.316 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:43:18 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:43:18.157+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:05.622397+0000 front 2023-12-16T21:39:05.622556+0000 (oldest deadline 2023-12-16T21:39:30.322546+0000) 2023-12-16T21:43:18.316 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:43:18 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:43:18.157+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:05.622506+0000 front 2023-12-16T21:39:05.622528+0000 (oldest deadline 2023-12-16T21:39:30.322546+0000) 2023-12-16T21:43:18.316 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:43:18 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:43:18.157+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:05.622479+0000 front 2023-12-16T21:39:05.622580+0000 (oldest deadline 2023-12-16T21:39:30.322546+0000) 2023-12-16T21:43:18.316 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:43:18 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:43:18.158+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:43:18.317 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:43:18 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:43:18.109+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:09.306583+0000 front 2023-12-16T21:39:09.306633+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:43:18.317 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:43:18 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:43:18.109+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:09.306551+0000 front 2023-12-16T21:39:09.306652+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:43:18.317 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:43:18 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:43:18.109+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:09.306605+0000 front 2023-12-16T21:39:09.306613+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:43:18.317 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:43:18 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:43:18.110+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:43:18.816 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:43:18 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:43:18.474+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:09.828911+0000 front 2023-12-16T21:39:09.828958+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:43:18.816 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:43:18 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:43:18.474+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:09.828868+0000 front 2023-12-16T21:39:09.828878+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:43:18.816 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:43:18 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:43:18.474+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:09.828947+0000 front 2023-12-16T21:39:09.828993+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:43:18.817 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:43:18 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:43:18.475+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:43:19.464 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:43:19 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:43:19.075+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:09.306583+0000 front 2023-12-16T21:39:09.306633+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:43:19.464 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:43:19 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:43:19.075+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:09.306551+0000 front 2023-12-16T21:39:09.306652+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:43:19.464 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:43:19 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:43:19.075+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:09.306605+0000 front 2023-12-16T21:39:09.306613+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:43:19.464 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:43:19 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:43:19.076+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:43:19.465 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:43:19 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:43:19.176+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:05.622397+0000 front 2023-12-16T21:39:05.622556+0000 (oldest deadline 2023-12-16T21:39:30.322546+0000) 2023-12-16T21:43:19.465 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:43:19 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:43:19.176+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:05.622506+0000 front 2023-12-16T21:39:05.622528+0000 (oldest deadline 2023-12-16T21:39:30.322546+0000) 2023-12-16T21:43:19.465 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:43:19 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:43:19.176+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:05.622479+0000 front 2023-12-16T21:39:05.622580+0000 (oldest deadline 2023-12-16T21:39:30.322546+0000) 2023-12-16T21:43:19.465 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:43:19 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:43:19.176+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:43:19.816 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:43:19 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:43:19.461+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:09.828911+0000 front 2023-12-16T21:39:09.828958+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:43:19.816 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:43:19 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:43:19.461+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:09.828868+0000 front 2023-12-16T21:39:09.828878+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:43:19.816 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:43:19 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:43:19.461+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:09.828947+0000 front 2023-12-16T21:39:09.828993+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:43:19.817 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:43:19 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:43:19.462+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:43:20.477 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:43:20 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:43:20.074+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:09.306583+0000 front 2023-12-16T21:39:09.306633+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:43:20.477 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:43:20 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:43:20.074+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:09.306551+0000 front 2023-12-16T21:39:09.306652+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:43:20.477 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:43:20 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:43:20.074+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:09.306605+0000 front 2023-12-16T21:39:09.306613+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:43:20.477 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:43:20 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:43:20.075+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:43:20.478 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:43:20 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:43:20.214+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:05.622397+0000 front 2023-12-16T21:39:05.622556+0000 (oldest deadline 2023-12-16T21:39:30.322546+0000) 2023-12-16T21:43:20.478 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:43:20 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:43:20.214+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:05.622506+0000 front 2023-12-16T21:39:05.622528+0000 (oldest deadline 2023-12-16T21:39:30.322546+0000) 2023-12-16T21:43:20.478 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:43:20 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:43:20.214+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:05.622479+0000 front 2023-12-16T21:39:05.622580+0000 (oldest deadline 2023-12-16T21:39:30.322546+0000) 2023-12-16T21:43:20.478 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:43:20 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:43:20.215+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:43:20.816 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:43:20 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:43:20.473+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:09.828911+0000 front 2023-12-16T21:39:09.828958+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:43:20.816 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:43:20 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:43:20.473+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:09.828868+0000 front 2023-12-16T21:39:09.828878+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:43:20.816 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:43:20 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:43:20.473+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:09.828947+0000 front 2023-12-16T21:39:09.828993+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:43:20.817 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:43:20 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:43:20.474+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:43:21.316 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:43:21 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:43:21.177+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:05.622397+0000 front 2023-12-16T21:39:05.622556+0000 (oldest deadline 2023-12-16T21:39:30.322546+0000) 2023-12-16T21:43:21.316 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:43:21 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:43:21.177+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:05.622506+0000 front 2023-12-16T21:39:05.622528+0000 (oldest deadline 2023-12-16T21:39:30.322546+0000) 2023-12-16T21:43:21.316 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:43:21 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:43:21.177+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:05.622479+0000 front 2023-12-16T21:39:05.622580+0000 (oldest deadline 2023-12-16T21:39:30.322546+0000) 2023-12-16T21:43:21.317 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:43:21 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:43:21.178+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:43:21.317 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:43:21 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:43:21.036+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:09.306583+0000 front 2023-12-16T21:39:09.306633+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:43:21.317 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:43:21 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:43:21.036+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:09.306551+0000 front 2023-12-16T21:39:09.306652+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:43:21.317 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:43:21 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:43:21.036+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:09.306605+0000 front 2023-12-16T21:39:09.306613+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:43:21.317 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:43:21 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:43:21.038+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:43:21.637 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Dec 16 21:43:21 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-mon-smithi045[80759]: 2023-12-16T21:43:21.340+0000 7f45a1f8f700 -1 mon.smithi045@0(probing) e2 get_health_metrics reporting 754 slow ops, oldest is monmgrreport(gid 14214, 0 checks, 1 progress events) 2023-12-16T21:43:21.637 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:43:21 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:43:21.426+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:09.828911+0000 front 2023-12-16T21:39:09.828958+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:43:21.638 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:43:21 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:43:21.426+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:09.828868+0000 front 2023-12-16T21:39:09.828878+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:43:21.638 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:43:21 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:43:21.426+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:09.828947+0000 front 2023-12-16T21:39:09.828993+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:43:21.638 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:43:21 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:43:21.427+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:43:22.410 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:43:22 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:43:22.070+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:09.306583+0000 front 2023-12-16T21:39:09.306633+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:43:22.411 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:43:22 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:43:22.070+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:09.306551+0000 front 2023-12-16T21:39:09.306652+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:43:22.411 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:43:22 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:43:22.070+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:09.306605+0000 front 2023-12-16T21:39:09.306613+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:43:22.411 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:43:22 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:43:22.071+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:43:22.411 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:43:22 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:43:22.133+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:05.622397+0000 front 2023-12-16T21:39:05.622556+0000 (oldest deadline 2023-12-16T21:39:30.322546+0000) 2023-12-16T21:43:22.411 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:43:22 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:43:22.133+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:05.622506+0000 front 2023-12-16T21:39:05.622528+0000 (oldest deadline 2023-12-16T21:39:30.322546+0000) 2023-12-16T21:43:22.411 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:43:22 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:43:22.133+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:05.622479+0000 front 2023-12-16T21:39:05.622580+0000 (oldest deadline 2023-12-16T21:39:30.322546+0000) 2023-12-16T21:43:22.411 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:43:22 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:43:22.134+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:43:22.816 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:43:22 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:43:22.406+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:09.828911+0000 front 2023-12-16T21:39:09.828958+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:43:22.816 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:43:22 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:43:22.406+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:09.828868+0000 front 2023-12-16T21:39:09.828878+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:43:22.816 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:43:22 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:43:22.406+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:09.828947+0000 front 2023-12-16T21:39:09.828993+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:43:22.816 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:43:22 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:43:22.407+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:43:23.443 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:43:23 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:43:23.083+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:09.306583+0000 front 2023-12-16T21:39:09.306633+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:43:23.444 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:43:23 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:43:23.083+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:09.306551+0000 front 2023-12-16T21:39:09.306652+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:43:23.444 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:43:23 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:43:23.083+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:09.306605+0000 front 2023-12-16T21:39:09.306613+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:43:23.444 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:43:23 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:43:23.083+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:43:23.444 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:43:23 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:43:23.147+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:05.622397+0000 front 2023-12-16T21:39:05.622556+0000 (oldest deadline 2023-12-16T21:39:30.322546+0000) 2023-12-16T21:43:23.444 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:43:23 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:43:23.147+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:05.622506+0000 front 2023-12-16T21:39:05.622528+0000 (oldest deadline 2023-12-16T21:39:30.322546+0000) 2023-12-16T21:43:23.444 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:43:23 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:43:23.147+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:05.622479+0000 front 2023-12-16T21:39:05.622580+0000 (oldest deadline 2023-12-16T21:39:30.322546+0000) 2023-12-16T21:43:23.444 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:43:23 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:43:23.148+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:43:23.816 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:43:23 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:43:23.439+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:09.828911+0000 front 2023-12-16T21:39:09.828958+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:43:23.817 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:43:23 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:43:23.439+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:09.828868+0000 front 2023-12-16T21:39:09.828878+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:43:23.817 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:43:23 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:43:23.439+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:09.828947+0000 front 2023-12-16T21:39:09.828993+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:43:23.817 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:43:23 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:43:23.440+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:43:24.487 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:43:24 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:43:24.109+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:09.306583+0000 front 2023-12-16T21:39:09.306633+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:43:24.488 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:43:24 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:43:24.109+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:09.306551+0000 front 2023-12-16T21:39:09.306652+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:43:24.488 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:43:24 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:43:24.109+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:09.306605+0000 front 2023-12-16T21:39:09.306613+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:43:24.488 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:43:24 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:43:24.109+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:43:24.488 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:43:24 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:43:24.170+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:05.622397+0000 front 2023-12-16T21:39:05.622556+0000 (oldest deadline 2023-12-16T21:39:30.322546+0000) 2023-12-16T21:43:24.488 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:43:24 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:43:24.170+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:05.622506+0000 front 2023-12-16T21:39:05.622528+0000 (oldest deadline 2023-12-16T21:39:30.322546+0000) 2023-12-16T21:43:24.488 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:43:24 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:43:24.170+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:05.622479+0000 front 2023-12-16T21:39:05.622580+0000 (oldest deadline 2023-12-16T21:39:30.322546+0000) 2023-12-16T21:43:24.488 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:43:24 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:43:24.170+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:43:24.816 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:43:24 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:43:24.483+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:09.828911+0000 front 2023-12-16T21:39:09.828958+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:43:24.816 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:43:24 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:43:24.483+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:09.828868+0000 front 2023-12-16T21:39:09.828878+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:43:24.816 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:43:24 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:43:24.483+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:09.828947+0000 front 2023-12-16T21:39:09.828993+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:43:24.817 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:43:24 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:43:24.484+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:43:25.525 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:43:25 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:43:25.079+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:09.306583+0000 front 2023-12-16T21:39:09.306633+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:43:25.525 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:43:25 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:43:25.079+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:09.306551+0000 front 2023-12-16T21:39:09.306652+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:43:25.525 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:43:25 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:43:25.079+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:09.306605+0000 front 2023-12-16T21:39:09.306613+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:43:25.525 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:43:25 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:43:25.080+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:43:25.525 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:43:25 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:43:25.215+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:05.622397+0000 front 2023-12-16T21:39:05.622556+0000 (oldest deadline 2023-12-16T21:39:30.322546+0000) 2023-12-16T21:43:25.525 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:43:25 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:43:25.215+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:05.622506+0000 front 2023-12-16T21:39:05.622528+0000 (oldest deadline 2023-12-16T21:39:30.322546+0000) 2023-12-16T21:43:25.526 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:43:25 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:43:25.215+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:05.622479+0000 front 2023-12-16T21:39:05.622580+0000 (oldest deadline 2023-12-16T21:39:30.322546+0000) 2023-12-16T21:43:25.526 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:43:25 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:43:25.215+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:43:25.816 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:43:25 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:43:25.522+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:09.828911+0000 front 2023-12-16T21:39:09.828958+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:43:25.817 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:43:25 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:43:25.522+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:09.828868+0000 front 2023-12-16T21:39:09.828878+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:43:25.817 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:43:25 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:43:25.522+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:09.828947+0000 front 2023-12-16T21:39:09.828993+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:43:25.817 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:43:25 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:43:25.523+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:43:26.469 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Dec 16 21:43:26 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-mon-smithi045[80759]: 2023-12-16T21:43:26.342+0000 7f45a1f8f700 -1 mon.smithi045@0(probing) e2 get_health_metrics reporting 831 slow ops, oldest is monmgrreport(gid 14214, 0 checks, 1 progress events) 2023-12-16T21:43:26.470 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:43:26 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:43:26.123+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:09.306583+0000 front 2023-12-16T21:39:09.306633+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:43:26.470 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:43:26 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:43:26.123+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:09.306551+0000 front 2023-12-16T21:39:09.306652+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:43:26.470 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:43:26 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:43:26.123+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:09.306605+0000 front 2023-12-16T21:39:09.306613+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:43:26.470 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:43:26 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:43:26.124+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:43:26.470 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:43:26 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:43:26.179+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:05.622397+0000 front 2023-12-16T21:39:05.622556+0000 (oldest deadline 2023-12-16T21:39:30.322546+0000) 2023-12-16T21:43:26.470 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:43:26 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:43:26.179+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:05.622506+0000 front 2023-12-16T21:39:05.622528+0000 (oldest deadline 2023-12-16T21:39:30.322546+0000) 2023-12-16T21:43:26.470 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:43:26 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:43:26.179+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:05.622479+0000 front 2023-12-16T21:39:05.622580+0000 (oldest deadline 2023-12-16T21:39:30.322546+0000) 2023-12-16T21:43:26.470 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:43:26 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:43:26.180+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:43:26.816 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:43:26 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:43:26.561+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:09.828911+0000 front 2023-12-16T21:39:09.828958+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:43:26.816 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:43:26 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:43:26.561+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:09.828868+0000 front 2023-12-16T21:39:09.828878+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:43:26.817 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:43:26 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:43:26.561+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:09.828947+0000 front 2023-12-16T21:39:09.828993+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:43:26.817 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:43:26 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:43:26.563+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:43:27.566 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:43:27 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:43:27.140+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:09.306583+0000 front 2023-12-16T21:39:09.306633+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:43:27.566 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:43:27 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:43:27.140+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:09.306551+0000 front 2023-12-16T21:39:09.306652+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:43:27.566 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:43:27 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:43:27.140+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:09.306605+0000 front 2023-12-16T21:39:09.306613+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:43:27.567 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:43:27 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:43:27.141+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:43:27.567 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:43:27 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:43:27.226+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:05.622397+0000 front 2023-12-16T21:39:05.622556+0000 (oldest deadline 2023-12-16T21:39:30.322546+0000) 2023-12-16T21:43:27.567 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:43:27 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:43:27.226+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:05.622506+0000 front 2023-12-16T21:39:05.622528+0000 (oldest deadline 2023-12-16T21:39:30.322546+0000) 2023-12-16T21:43:27.567 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:43:27 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:43:27.226+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:05.622479+0000 front 2023-12-16T21:39:05.622580+0000 (oldest deadline 2023-12-16T21:39:30.322546+0000) 2023-12-16T21:43:27.567 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:43:27 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:43:27.227+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:43:28.066 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:43:27 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:43:27.595+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:09.828911+0000 front 2023-12-16T21:39:09.828958+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:43:28.066 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:43:27 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:43:27.595+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:09.828868+0000 front 2023-12-16T21:39:09.828878+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:43:28.066 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:43:27 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:43:27.595+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:09.828947+0000 front 2023-12-16T21:39:09.828993+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:43:28.066 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:43:27 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:43:27.597+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:43:28.559 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:43:28 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:43:28.181+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:09.306583+0000 front 2023-12-16T21:39:09.306633+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:43:28.559 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:43:28 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:43:28.181+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:09.306551+0000 front 2023-12-16T21:39:09.306652+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:43:28.559 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:43:28 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:43:28.181+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:09.306605+0000 front 2023-12-16T21:39:09.306613+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:43:28.559 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:43:28 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:43:28.181+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:43:28.559 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:43:28 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:43:28.250+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:05.622397+0000 front 2023-12-16T21:39:05.622556+0000 (oldest deadline 2023-12-16T21:39:30.322546+0000) 2023-12-16T21:43:28.559 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:43:28 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:43:28.250+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:05.622506+0000 front 2023-12-16T21:39:05.622528+0000 (oldest deadline 2023-12-16T21:39:30.322546+0000) 2023-12-16T21:43:28.560 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:43:28 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:43:28.250+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:05.622479+0000 front 2023-12-16T21:39:05.622580+0000 (oldest deadline 2023-12-16T21:39:30.322546+0000) 2023-12-16T21:43:28.560 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:43:28 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:43:28.251+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:43:28.816 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:43:28 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:43:28.555+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:09.828911+0000 front 2023-12-16T21:39:09.828958+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:43:28.883 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:43:28 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:43:28.555+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:09.828868+0000 front 2023-12-16T21:39:09.828878+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:43:28.883 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:43:28 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:43:28.556+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:09.828947+0000 front 2023-12-16T21:39:09.828993+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:43:28.883 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:43:28 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:43:28.557+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:43:29.550 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:43:29 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:43:29.232+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:05.622397+0000 front 2023-12-16T21:39:05.622556+0000 (oldest deadline 2023-12-16T21:39:30.322546+0000) 2023-12-16T21:43:29.550 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:43:29 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:43:29.232+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:05.622506+0000 front 2023-12-16T21:39:05.622528+0000 (oldest deadline 2023-12-16T21:39:30.322546+0000) 2023-12-16T21:43:29.550 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:43:29 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:43:29.232+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:05.622479+0000 front 2023-12-16T21:39:05.622580+0000 (oldest deadline 2023-12-16T21:39:30.322546+0000) 2023-12-16T21:43:29.550 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:43:29 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:43:29.233+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:43:29.551 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:43:29 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:43:29.153+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:09.306583+0000 front 2023-12-16T21:39:09.306633+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:43:29.551 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:43:29 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:43:29.153+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:09.306551+0000 front 2023-12-16T21:39:09.306652+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:43:29.551 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:43:29 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:43:29.153+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:09.306605+0000 front 2023-12-16T21:39:09.306613+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:43:29.551 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:43:29 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:43:29.154+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:43:29.816 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:43:29 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:43:29.547+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:09.828911+0000 front 2023-12-16T21:39:09.828958+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:43:29.816 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:43:29 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:43:29.547+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:09.828868+0000 front 2023-12-16T21:39:09.828878+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:43:29.816 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:43:29 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:43:29.547+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:09.828947+0000 front 2023-12-16T21:39:09.828993+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:43:29.816 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:43:29 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:43:29.548+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:43:30.524 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:43:30 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:43:30.158+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:09.306583+0000 front 2023-12-16T21:39:09.306633+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:43:30.524 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:43:30 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:43:30.158+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:09.306551+0000 front 2023-12-16T21:39:09.306652+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:43:30.524 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:43:30 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:43:30.158+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:09.306605+0000 front 2023-12-16T21:39:09.306613+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:43:30.525 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:43:30 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:43:30.159+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:43:30.525 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:43:30 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:43:30.222+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:05.622397+0000 front 2023-12-16T21:39:05.622556+0000 (oldest deadline 2023-12-16T21:39:30.322546+0000) 2023-12-16T21:43:30.525 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:43:30 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:43:30.222+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:05.622506+0000 front 2023-12-16T21:39:05.622528+0000 (oldest deadline 2023-12-16T21:39:30.322546+0000) 2023-12-16T21:43:30.525 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:43:30 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:43:30.222+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:05.622479+0000 front 2023-12-16T21:39:05.622580+0000 (oldest deadline 2023-12-16T21:39:30.322546+0000) 2023-12-16T21:43:30.525 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:43:30 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:43:30.223+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:43:30.816 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:43:30 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:43:30.521+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:09.828911+0000 front 2023-12-16T21:39:09.828958+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:43:30.816 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:43:30 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:43:30.521+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:09.828868+0000 front 2023-12-16T21:39:09.828878+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:43:30.816 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:43:30 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:43:30.521+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:09.828947+0000 front 2023-12-16T21:39:09.828993+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:43:30.816 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:43:30 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:43:30.522+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:43:31.471 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Dec 16 21:43:31 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-mon-smithi045[80759]: 2023-12-16T21:43:31.342+0000 7f45a1f8f700 -1 mon.smithi045@0(probing) e2 get_health_metrics reporting 911 slow ops, oldest is monmgrreport(gid 14214, 0 checks, 1 progress events) 2023-12-16T21:43:31.471 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:43:31 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:43:31.202+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:05.622397+0000 front 2023-12-16T21:39:05.622556+0000 (oldest deadline 2023-12-16T21:39:30.322546+0000) 2023-12-16T21:43:31.471 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:43:31 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:43:31.202+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:05.622506+0000 front 2023-12-16T21:39:05.622528+0000 (oldest deadline 2023-12-16T21:39:30.322546+0000) 2023-12-16T21:43:31.472 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:43:31 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:43:31.202+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:05.622479+0000 front 2023-12-16T21:39:05.622580+0000 (oldest deadline 2023-12-16T21:39:30.322546+0000) 2023-12-16T21:43:31.472 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:43:31 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:43:31.203+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:43:31.472 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:43:31 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:43:31.207+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:09.306583+0000 front 2023-12-16T21:39:09.306633+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:43:31.472 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:43:31 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:43:31.207+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:09.306551+0000 front 2023-12-16T21:39:09.306652+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:43:31.472 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:43:31 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:43:31.207+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:09.306605+0000 front 2023-12-16T21:39:09.306613+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:43:31.472 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:43:31 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:43:31.208+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:43:31.816 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:43:31 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:43:31.509+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:09.828911+0000 front 2023-12-16T21:39:09.828958+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:43:31.817 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:43:31 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:43:31.509+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:09.828868+0000 front 2023-12-16T21:39:09.828878+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:43:31.817 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:43:31 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:43:31.509+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:09.828947+0000 front 2023-12-16T21:39:09.828993+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:43:31.817 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:43:31 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:43:31.510+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:43:32.514 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:43:32 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:43:32.228+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:09.306583+0000 front 2023-12-16T21:39:09.306633+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:43:32.515 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:43:32 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:43:32.228+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:09.306551+0000 front 2023-12-16T21:39:09.306652+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:43:32.515 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:43:32 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:43:32.228+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:09.306605+0000 front 2023-12-16T21:39:09.306613+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:43:32.515 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:43:32 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:43:32.229+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:43:32.515 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:43:32 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:43:32.206+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:05.622397+0000 front 2023-12-16T21:39:05.622556+0000 (oldest deadline 2023-12-16T21:39:30.322546+0000) 2023-12-16T21:43:32.515 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:43:32 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:43:32.206+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:05.622506+0000 front 2023-12-16T21:39:05.622528+0000 (oldest deadline 2023-12-16T21:39:30.322546+0000) 2023-12-16T21:43:32.515 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:43:32 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:43:32.206+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:05.622479+0000 front 2023-12-16T21:39:05.622580+0000 (oldest deadline 2023-12-16T21:39:30.322546+0000) 2023-12-16T21:43:32.515 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:43:32 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:43:32.207+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:43:32.816 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:43:32 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:43:32.512+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:09.828911+0000 front 2023-12-16T21:39:09.828958+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:43:32.816 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:43:32 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:43:32.512+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:09.828868+0000 front 2023-12-16T21:39:09.828878+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:43:32.816 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:43:32 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:43:32.512+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:09.828947+0000 front 2023-12-16T21:39:09.828993+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:43:32.816 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:43:32 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:43:32.513+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:43:33.475 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:43:33 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:43:33.267+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:09.306583+0000 front 2023-12-16T21:39:09.306633+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:43:33.475 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:43:33 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:43:33.267+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:09.306551+0000 front 2023-12-16T21:39:09.306652+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:43:33.475 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:43:33 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:43:33.267+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:09.306605+0000 front 2023-12-16T21:39:09.306613+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:43:33.475 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:43:33 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:43:33.268+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:43:33.475 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:43:33 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:43:33.208+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:05.622397+0000 front 2023-12-16T21:39:05.622556+0000 (oldest deadline 2023-12-16T21:39:30.322546+0000) 2023-12-16T21:43:33.475 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:43:33 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:43:33.208+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:05.622506+0000 front 2023-12-16T21:39:05.622528+0000 (oldest deadline 2023-12-16T21:39:30.322546+0000) 2023-12-16T21:43:33.475 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:43:33 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:43:33.208+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:05.622479+0000 front 2023-12-16T21:39:05.622580+0000 (oldest deadline 2023-12-16T21:39:30.322546+0000) 2023-12-16T21:43:33.475 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:43:33 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:43:33.209+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:43:33.816 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:43:33 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:43:33.471+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:09.828911+0000 front 2023-12-16T21:39:09.828958+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:43:33.816 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:43:33 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:43:33.471+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:09.828868+0000 front 2023-12-16T21:39:09.828878+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:43:33.816 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:43:33 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:43:33.471+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:09.828947+0000 front 2023-12-16T21:39:09.828993+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:43:33.817 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:43:33 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:43:33.472+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:43:34.501 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:43:34 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:43:34.244+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:09.306583+0000 front 2023-12-16T21:39:09.306633+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:43:34.501 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:43:34 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:43:34.245+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:09.306551+0000 front 2023-12-16T21:39:09.306652+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:43:34.502 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:43:34 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:43:34.245+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:09.306605+0000 front 2023-12-16T21:39:09.306613+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:43:34.502 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:43:34 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:43:34.246+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:43:34.502 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:43:34 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:43:34.246+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:05.622397+0000 front 2023-12-16T21:39:05.622556+0000 (oldest deadline 2023-12-16T21:39:30.322546+0000) 2023-12-16T21:43:34.502 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:43:34 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:43:34.246+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:05.622506+0000 front 2023-12-16T21:39:05.622528+0000 (oldest deadline 2023-12-16T21:39:30.322546+0000) 2023-12-16T21:43:34.502 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:43:34 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:43:34.246+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:05.622479+0000 front 2023-12-16T21:39:05.622580+0000 (oldest deadline 2023-12-16T21:39:30.322546+0000) 2023-12-16T21:43:34.502 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:43:34 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:43:34.247+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:43:34.816 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:43:34 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:43:34.497+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:09.828911+0000 front 2023-12-16T21:39:09.828958+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:43:34.816 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:43:34 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:43:34.497+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:09.828868+0000 front 2023-12-16T21:39:09.828878+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:43:34.816 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:43:34 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:43:34.497+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:09.828947+0000 front 2023-12-16T21:39:09.828993+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:43:34.817 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:43:34 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:43:34.498+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:43:35.549 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:43:35 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:43:35.202+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:09.306583+0000 front 2023-12-16T21:39:09.306633+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:43:35.549 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:43:35 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:43:35.202+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:09.306551+0000 front 2023-12-16T21:39:09.306652+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:43:35.549 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:43:35 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:43:35.202+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:09.306605+0000 front 2023-12-16T21:39:09.306613+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:43:35.549 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:43:35 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:43:35.204+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:43:35.549 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:43:35 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:43:35.270+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:05.622397+0000 front 2023-12-16T21:39:05.622556+0000 (oldest deadline 2023-12-16T21:39:30.322546+0000) 2023-12-16T21:43:35.549 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:43:35 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:43:35.270+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:05.622506+0000 front 2023-12-16T21:39:05.622528+0000 (oldest deadline 2023-12-16T21:39:30.322546+0000) 2023-12-16T21:43:35.549 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:43:35 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:43:35.270+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:05.622479+0000 front 2023-12-16T21:39:05.622580+0000 (oldest deadline 2023-12-16T21:39:30.322546+0000) 2023-12-16T21:43:35.549 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:43:35 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:43:35.271+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:43:35.816 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:43:35 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:43:35.545+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:09.828911+0000 front 2023-12-16T21:39:09.828958+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:43:35.816 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:43:35 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:43:35.545+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:09.828868+0000 front 2023-12-16T21:39:09.828878+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:43:35.816 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:43:35 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:43:35.545+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:09.828947+0000 front 2023-12-16T21:39:09.828993+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:43:35.817 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:43:35 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:43:35.546+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:43:36.465 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Dec 16 21:43:36 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-mon-smithi045[80759]: 2023-12-16T21:43:36.343+0000 7f45a1f8f700 -1 mon.smithi045@0(probing) e2 get_health_metrics reporting 993 slow ops, oldest is monmgrreport(gid 14214, 0 checks, 1 progress events) 2023-12-16T21:43:36.465 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:43:36 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:43:36.312+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:05.622397+0000 front 2023-12-16T21:39:05.622556+0000 (oldest deadline 2023-12-16T21:39:30.322546+0000) 2023-12-16T21:43:36.465 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:43:36 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:43:36.312+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:05.622506+0000 front 2023-12-16T21:39:05.622528+0000 (oldest deadline 2023-12-16T21:39:30.322546+0000) 2023-12-16T21:43:36.465 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:43:36 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:43:36.312+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:05.622479+0000 front 2023-12-16T21:39:05.622580+0000 (oldest deadline 2023-12-16T21:39:30.322546+0000) 2023-12-16T21:43:36.465 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:43:36 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:43:36.313+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:43:36.465 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:43:36 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:43:36.181+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:09.306583+0000 front 2023-12-16T21:39:09.306633+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:43:36.466 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:43:36 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:43:36.181+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:09.306551+0000 front 2023-12-16T21:39:09.306652+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:43:36.466 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:43:36 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:43:36.181+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:09.306605+0000 front 2023-12-16T21:39:09.306613+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:43:36.466 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:43:36 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:43:36.182+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:43:36.816 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:43:36 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:43:36.525+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:09.828911+0000 front 2023-12-16T21:39:09.828958+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:43:36.816 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:43:36 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:43:36.525+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:09.828868+0000 front 2023-12-16T21:39:09.828878+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:43:36.816 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:43:36 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:43:36.525+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:09.828947+0000 front 2023-12-16T21:39:09.828993+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:43:36.816 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:43:36 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:43:36.526+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:43:37.516 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:43:37 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:43:37.229+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:09.306583+0000 front 2023-12-16T21:39:09.306633+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:43:37.516 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:43:37 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:43:37.229+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:09.306551+0000 front 2023-12-16T21:39:09.306652+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:43:37.516 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:43:37 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:43:37.229+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:09.306605+0000 front 2023-12-16T21:39:09.306613+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:43:37.516 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:43:37 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:43:37.230+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:43:37.517 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:43:37 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:43:37.271+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:05.622397+0000 front 2023-12-16T21:39:05.622556+0000 (oldest deadline 2023-12-16T21:39:30.322546+0000) 2023-12-16T21:43:37.517 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:43:37 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:43:37.271+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:05.622506+0000 front 2023-12-16T21:39:05.622528+0000 (oldest deadline 2023-12-16T21:39:30.322546+0000) 2023-12-16T21:43:37.517 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:43:37 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:43:37.271+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:05.622479+0000 front 2023-12-16T21:39:05.622580+0000 (oldest deadline 2023-12-16T21:39:30.322546+0000) 2023-12-16T21:43:37.517 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:43:37 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:43:37.272+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:43:37.816 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:43:37 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:43:37.513+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:09.828911+0000 front 2023-12-16T21:39:09.828958+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:43:37.816 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:43:37 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:43:37.513+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:09.828868+0000 front 2023-12-16T21:39:09.828878+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:43:37.816 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:43:37 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:43:37.513+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:09.828947+0000 front 2023-12-16T21:39:09.828993+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:43:37.817 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:43:37 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:43:37.514+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:43:38.542 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:43:38 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:43:38.280+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:05.622397+0000 front 2023-12-16T21:39:05.622556+0000 (oldest deadline 2023-12-16T21:39:30.322546+0000) 2023-12-16T21:43:38.542 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:43:38 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:43:38.280+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:05.622506+0000 front 2023-12-16T21:39:05.622528+0000 (oldest deadline 2023-12-16T21:39:30.322546+0000) 2023-12-16T21:43:38.542 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:43:38 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:43:38.280+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:05.622479+0000 front 2023-12-16T21:39:05.622580+0000 (oldest deadline 2023-12-16T21:39:30.322546+0000) 2023-12-16T21:43:38.542 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:43:38 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:43:38.280+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:43:38.542 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:43:38 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:43:38.192+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:09.306583+0000 front 2023-12-16T21:39:09.306633+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:43:38.542 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:43:38 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:43:38.192+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:09.306551+0000 front 2023-12-16T21:39:09.306652+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:43:38.542 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:43:38 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:43:38.192+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:09.306605+0000 front 2023-12-16T21:39:09.306613+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:43:38.542 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:43:38 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:43:38.193+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:43:38.816 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:43:38 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:43:38.538+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:09.828911+0000 front 2023-12-16T21:39:09.828958+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:43:38.816 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:43:38 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:43:38.538+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:09.828868+0000 front 2023-12-16T21:39:09.828878+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:43:38.816 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:43:38 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:43:38.538+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:09.828947+0000 front 2023-12-16T21:39:09.828993+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:43:38.817 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:43:38 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:43:38.539+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:43:39.525 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:43:39 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:43:39.167+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:09.306583+0000 front 2023-12-16T21:39:09.306633+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:43:39.526 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:43:39 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:43:39.167+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:09.306551+0000 front 2023-12-16T21:39:09.306652+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:43:39.526 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:43:39 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:43:39.167+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:09.306605+0000 front 2023-12-16T21:39:09.306613+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:43:39.526 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:43:39 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:43:39.168+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:43:39.526 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:43:39 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:43:39.256+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:05.622397+0000 front 2023-12-16T21:39:05.622556+0000 (oldest deadline 2023-12-16T21:39:30.322546+0000) 2023-12-16T21:43:39.526 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:43:39 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:43:39.256+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:05.622506+0000 front 2023-12-16T21:39:05.622528+0000 (oldest deadline 2023-12-16T21:39:30.322546+0000) 2023-12-16T21:43:39.526 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:43:39 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:43:39.256+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:05.622479+0000 front 2023-12-16T21:39:05.622580+0000 (oldest deadline 2023-12-16T21:39:30.322546+0000) 2023-12-16T21:43:39.526 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:43:39 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:43:39.257+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:43:39.816 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:43:39 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:43:39.522+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:09.828911+0000 front 2023-12-16T21:39:09.828958+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:43:39.816 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:43:39 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:43:39.522+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:09.828868+0000 front 2023-12-16T21:39:09.828878+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:43:39.816 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:43:39 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:43:39.522+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:09.828947+0000 front 2023-12-16T21:39:09.828993+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:43:39.816 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:43:39 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:43:39.523+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:43:40.500 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:43:40 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:43:40.143+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:09.306583+0000 front 2023-12-16T21:39:09.306633+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:43:40.500 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:43:40 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:43:40.143+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:09.306551+0000 front 2023-12-16T21:39:09.306652+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:43:40.500 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:43:40 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:43:40.143+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:09.306605+0000 front 2023-12-16T21:39:09.306613+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:43:40.500 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:43:40 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:43:40.144+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:43:40.501 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:43:40 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:43:40.247+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:05.622397+0000 front 2023-12-16T21:39:05.622556+0000 (oldest deadline 2023-12-16T21:39:30.322546+0000) 2023-12-16T21:43:40.501 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:43:40 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:43:40.247+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:05.622506+0000 front 2023-12-16T21:39:05.622528+0000 (oldest deadline 2023-12-16T21:39:30.322546+0000) 2023-12-16T21:43:40.501 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:43:40 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:43:40.247+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:05.622479+0000 front 2023-12-16T21:39:05.622580+0000 (oldest deadline 2023-12-16T21:39:30.322546+0000) 2023-12-16T21:43:40.501 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:43:40 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:43:40.248+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:43:40.816 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:43:40 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:43:40.497+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:09.828911+0000 front 2023-12-16T21:39:09.828958+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:43:40.816 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:43:40 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:43:40.497+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:09.828868+0000 front 2023-12-16T21:39:09.828878+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:43:40.816 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:43:40 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:43:40.497+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:09.828947+0000 front 2023-12-16T21:39:09.828993+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:43:40.817 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:43:40 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:43:40.498+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:43:41.467 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Dec 16 21:43:41 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-mon-smithi045[80759]: 2023-12-16T21:43:41.343+0000 7f45a1f8f700 -1 mon.smithi045@0(probing) e2 get_health_metrics reporting 1074 slow ops, oldest is monmgrreport(gid 14214, 0 checks, 1 progress events) 2023-12-16T21:43:41.468 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:43:41 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:43:41.177+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:09.306583+0000 front 2023-12-16T21:39:09.306633+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:43:41.469 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:43:41 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:43:41.177+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:09.306551+0000 front 2023-12-16T21:39:09.306652+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:43:41.469 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:43:41 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:43:41.177+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:09.306605+0000 front 2023-12-16T21:39:09.306613+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:43:41.469 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:43:41 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:43:41.178+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:43:41.469 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:43:41 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:43:41.270+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:05.622397+0000 front 2023-12-16T21:39:05.622556+0000 (oldest deadline 2023-12-16T21:39:30.322546+0000) 2023-12-16T21:43:41.469 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:43:41 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:43:41.270+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:05.622506+0000 front 2023-12-16T21:39:05.622528+0000 (oldest deadline 2023-12-16T21:39:30.322546+0000) 2023-12-16T21:43:41.469 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:43:41 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:43:41.270+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:05.622479+0000 front 2023-12-16T21:39:05.622580+0000 (oldest deadline 2023-12-16T21:39:30.322546+0000) 2023-12-16T21:43:41.469 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:43:41 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:43:41.271+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:43:41.816 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:43:41 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:43:41.464+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:09.828911+0000 front 2023-12-16T21:39:09.828958+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:43:41.816 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:43:41 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:43:41.464+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:09.828868+0000 front 2023-12-16T21:39:09.828878+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:43:41.816 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:43:41 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:43:41.464+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:09.828947+0000 front 2023-12-16T21:39:09.828993+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:43:41.816 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:43:41 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:43:41.465+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:43:42.566 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:43:42 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:43:42.210+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:09.306583+0000 front 2023-12-16T21:39:09.306633+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:43:42.566 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:43:42 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:43:42.210+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:09.306551+0000 front 2023-12-16T21:39:09.306652+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:43:42.566 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:43:42 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:43:42.210+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:09.306605+0000 front 2023-12-16T21:39:09.306613+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:43:42.566 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:43:42 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:43:42.211+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:43:42.567 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:43:42 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:43:42.285+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:05.622397+0000 front 2023-12-16T21:39:05.622556+0000 (oldest deadline 2023-12-16T21:39:30.322546+0000) 2023-12-16T21:43:42.567 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:43:42 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:43:42.285+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:05.622506+0000 front 2023-12-16T21:39:05.622528+0000 (oldest deadline 2023-12-16T21:39:30.322546+0000) 2023-12-16T21:43:42.567 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:43:42 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:43:42.285+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:05.622479+0000 front 2023-12-16T21:39:05.622580+0000 (oldest deadline 2023-12-16T21:39:30.322546+0000) 2023-12-16T21:43:42.567 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:43:42 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:43:42.285+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:43:42.567 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:43:42 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:43:42.424+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:09.828911+0000 front 2023-12-16T21:39:09.828958+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:43:42.567 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:43:42 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:43:42.424+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:09.828868+0000 front 2023-12-16T21:39:09.828878+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:43:42.567 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:43:42 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:43:42.424+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:09.828947+0000 front 2023-12-16T21:39:09.828993+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:43:42.567 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:43:42 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:43:42.425+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:43:43.530 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:43:43 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:43:43.161+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:09.306583+0000 front 2023-12-16T21:39:09.306633+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:43:43.530 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:43:43 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:43:43.161+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:09.306551+0000 front 2023-12-16T21:39:09.306652+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:43:43.530 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:43:43 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:43:43.161+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:09.306605+0000 front 2023-12-16T21:39:09.306613+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:43:43.530 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:43:43 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:43:43.162+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:43:43.531 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:43:43 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:43:43.410+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:09.828911+0000 front 2023-12-16T21:39:09.828958+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:43:43.531 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:43:43 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:43:43.410+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:09.828868+0000 front 2023-12-16T21:39:09.828878+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:43:43.531 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:43:43 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:43:43.410+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:09.828947+0000 front 2023-12-16T21:39:09.828993+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:43:43.531 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:43:43 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:43:43.411+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:43:43.531 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:43:43 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:43:43.280+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:05.622397+0000 front 2023-12-16T21:39:05.622556+0000 (oldest deadline 2023-12-16T21:39:30.322546+0000) 2023-12-16T21:43:43.531 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:43:43 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:43:43.280+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:05.622506+0000 front 2023-12-16T21:39:05.622528+0000 (oldest deadline 2023-12-16T21:39:30.322546+0000) 2023-12-16T21:43:43.531 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:43:43 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:43:43.280+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:05.622479+0000 front 2023-12-16T21:39:05.622580+0000 (oldest deadline 2023-12-16T21:39:30.322546+0000) 2023-12-16T21:43:43.531 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:43:43 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:43:43.281+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:43:44.403 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:43:44 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:43:44.147+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:09.306583+0000 front 2023-12-16T21:39:09.306633+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:43:44.403 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:43:44 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:43:44.147+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:09.306551+0000 front 2023-12-16T21:39:09.306652+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:43:44.404 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:43:44 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:43:44.147+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:09.306605+0000 front 2023-12-16T21:39:09.306613+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:43:44.404 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:43:44 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:43:44.148+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:43:44.404 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:43:44 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:43:44.300+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:05.622397+0000 front 2023-12-16T21:39:05.622556+0000 (oldest deadline 2023-12-16T21:39:30.322546+0000) 2023-12-16T21:43:44.404 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:43:44 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:43:44.300+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:05.622506+0000 front 2023-12-16T21:39:05.622528+0000 (oldest deadline 2023-12-16T21:39:30.322546+0000) 2023-12-16T21:43:44.404 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:43:44 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:43:44.300+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:05.622479+0000 front 2023-12-16T21:39:05.622580+0000 (oldest deadline 2023-12-16T21:39:30.322546+0000) 2023-12-16T21:43:44.404 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:43:44 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:43:44.301+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:43:44.816 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:43:44 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:43:44.400+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:09.828911+0000 front 2023-12-16T21:39:09.828958+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:43:44.817 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:43:44 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:43:44.400+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:09.828868+0000 front 2023-12-16T21:39:09.828878+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:43:44.817 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:43:44 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:43:44.400+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:09.828947+0000 front 2023-12-16T21:39:09.828993+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:43:44.817 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:43:44 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:43:44.401+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:43:45.392 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:43:45 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:43:45.326+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:05.622397+0000 front 2023-12-16T21:39:05.622556+0000 (oldest deadline 2023-12-16T21:39:30.322546+0000) 2023-12-16T21:43:45.392 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:43:45 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:43:45.326+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:05.622506+0000 front 2023-12-16T21:39:05.622528+0000 (oldest deadline 2023-12-16T21:39:30.322546+0000) 2023-12-16T21:43:45.392 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:43:45 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:43:45.326+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:05.622479+0000 front 2023-12-16T21:39:05.622580+0000 (oldest deadline 2023-12-16T21:39:30.322546+0000) 2023-12-16T21:43:45.392 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:43:45 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:43:45.327+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:43:45.392 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:43:45 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:43:45.099+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:09.306583+0000 front 2023-12-16T21:39:09.306633+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:43:45.392 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:43:45 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:43:45.099+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:09.306551+0000 front 2023-12-16T21:39:09.306652+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:43:45.392 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:43:45 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:43:45.099+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:09.306605+0000 front 2023-12-16T21:39:09.306613+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:43:45.393 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:43:45 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:43:45.101+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:43:45.681 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:43:45 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:43:45.389+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:09.828911+0000 front 2023-12-16T21:39:09.828958+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:43:45.681 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:43:45 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:43:45.389+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:09.828868+0000 front 2023-12-16T21:39:09.828878+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:43:45.681 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:43:45 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:43:45.389+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:09.828947+0000 front 2023-12-16T21:39:09.828993+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:43:45.681 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:43:45 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:43:45.390+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:43:46.420 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Dec 16 21:43:46 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-mon-smithi045[80759]: 2023-12-16T21:43:46.344+0000 7f45a1f8f700 -1 mon.smithi045@0(probing) e2 get_health_metrics reporting 1151 slow ops, oldest is monmgrreport(gid 14214, 0 checks, 1 progress events) 2023-12-16T21:43:46.421 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:43:46 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:43:46.130+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:09.306583+0000 front 2023-12-16T21:39:09.306633+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:43:46.421 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:43:46 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:43:46.130+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:09.306551+0000 front 2023-12-16T21:39:09.306652+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:43:46.421 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:43:46 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:43:46.130+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:09.306605+0000 front 2023-12-16T21:39:09.306613+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:43:46.421 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:43:46 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:43:46.131+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:43:46.421 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:43:46 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:43:46.294+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:05.622397+0000 front 2023-12-16T21:39:05.622556+0000 (oldest deadline 2023-12-16T21:39:30.322546+0000) 2023-12-16T21:43:46.421 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:43:46 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:43:46.294+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:05.622506+0000 front 2023-12-16T21:39:05.622528+0000 (oldest deadline 2023-12-16T21:39:30.322546+0000) 2023-12-16T21:43:46.421 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:43:46 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:43:46.294+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:05.622479+0000 front 2023-12-16T21:39:05.622580+0000 (oldest deadline 2023-12-16T21:39:30.322546+0000) 2023-12-16T21:43:46.421 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:43:46 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:43:46.295+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:43:46.816 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:43:46 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:43:46.417+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:09.828911+0000 front 2023-12-16T21:39:09.828958+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:43:46.817 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:43:46 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:43:46.417+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:09.828868+0000 front 2023-12-16T21:39:09.828878+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:43:46.817 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:43:46 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:43:46.417+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:09.828947+0000 front 2023-12-16T21:39:09.828993+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:43:46.817 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:43:46 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:43:46.418+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:43:47.443 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:43:47 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:43:47.315+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:05.622397+0000 front 2023-12-16T21:39:05.622556+0000 (oldest deadline 2023-12-16T21:39:30.322546+0000) 2023-12-16T21:43:47.443 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:43:47 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:43:47.315+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:05.622506+0000 front 2023-12-16T21:39:05.622528+0000 (oldest deadline 2023-12-16T21:39:30.322546+0000) 2023-12-16T21:43:47.444 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:43:47 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:43:47.315+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:05.622479+0000 front 2023-12-16T21:39:05.622580+0000 (oldest deadline 2023-12-16T21:39:30.322546+0000) 2023-12-16T21:43:47.444 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:43:47 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:43:47.317+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:43:47.444 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:43:47 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:43:47.162+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:09.306583+0000 front 2023-12-16T21:39:09.306633+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:43:47.444 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:43:47 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:43:47.162+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:09.306551+0000 front 2023-12-16T21:39:09.306652+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:43:47.444 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:43:47 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:43:47.162+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:09.306605+0000 front 2023-12-16T21:39:09.306613+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:43:47.444 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:43:47 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:43:47.163+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:43:47.817 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:43:47 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:43:47.440+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:09.828911+0000 front 2023-12-16T21:39:09.828958+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:43:47.817 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:43:47 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:43:47.440+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:09.828868+0000 front 2023-12-16T21:39:09.828878+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:43:47.817 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:43:47 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:43:47.440+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:09.828947+0000 front 2023-12-16T21:39:09.828993+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:43:47.817 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:43:47 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:43:47.441+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:43:48.446 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:43:48 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:43:48.114+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:09.306583+0000 front 2023-12-16T21:39:09.306633+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:43:48.447 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:43:48 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:43:48.114+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:09.306551+0000 front 2023-12-16T21:39:09.306652+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:43:48.447 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:43:48 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:43:48.114+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:09.306605+0000 front 2023-12-16T21:39:09.306613+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:43:48.447 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:43:48 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:43:48.115+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:43:48.447 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:43:48 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:43:48.321+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:05.622397+0000 front 2023-12-16T21:39:05.622556+0000 (oldest deadline 2023-12-16T21:39:30.322546+0000) 2023-12-16T21:43:48.447 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:43:48 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:43:48.321+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:05.622506+0000 front 2023-12-16T21:39:05.622528+0000 (oldest deadline 2023-12-16T21:39:30.322546+0000) 2023-12-16T21:43:48.447 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:43:48 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:43:48.321+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:05.622479+0000 front 2023-12-16T21:39:05.622580+0000 (oldest deadline 2023-12-16T21:39:30.322546+0000) 2023-12-16T21:43:48.447 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:43:48 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:43:48.321+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:43:48.722 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:43:48 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:43:48.443+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:09.828911+0000 front 2023-12-16T21:39:09.828958+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:43:48.722 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:43:48 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:43:48.443+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:09.828868+0000 front 2023-12-16T21:39:09.828878+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:43:48.722 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:43:48 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:43:48.443+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:09.828947+0000 front 2023-12-16T21:39:09.828993+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:43:48.722 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:43:48 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:43:48.444+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:43:49.324 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:43:49 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:43:49.086+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:09.306583+0000 front 2023-12-16T21:39:09.306633+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:43:49.324 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:43:49 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:43:49.086+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:09.306551+0000 front 2023-12-16T21:39:09.306652+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:43:49.324 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:43:49 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:43:49.086+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:09.306605+0000 front 2023-12-16T21:39:09.306613+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:43:49.324 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:43:49 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:43:49.087+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:43:49.816 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:43:49 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:43:49.410+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:09.828911+0000 front 2023-12-16T21:39:09.828958+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:43:49.816 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:43:49 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:43:49.410+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:09.828868+0000 front 2023-12-16T21:39:09.828878+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:43:49.816 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:43:49 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:43:49.410+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:09.828947+0000 front 2023-12-16T21:39:09.828993+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:43:49.817 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:43:49 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:43:49.411+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:43:49.817 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:43:49 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:43:49.321+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:05.622397+0000 front 2023-12-16T21:39:05.622556+0000 (oldest deadline 2023-12-16T21:39:30.322546+0000) 2023-12-16T21:43:49.817 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:43:49 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:43:49.321+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:05.622506+0000 front 2023-12-16T21:39:05.622528+0000 (oldest deadline 2023-12-16T21:39:30.322546+0000) 2023-12-16T21:43:49.817 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:43:49 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:43:49.321+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:05.622479+0000 front 2023-12-16T21:39:05.622580+0000 (oldest deadline 2023-12-16T21:39:30.322546+0000) 2023-12-16T21:43:49.817 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:43:49 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:43:49.322+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:43:50.453 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:43:50 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:43:50.121+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:09.306583+0000 front 2023-12-16T21:39:09.306633+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:43:50.454 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:43:50 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:43:50.121+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:09.306551+0000 front 2023-12-16T21:39:09.306652+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:43:50.454 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:43:50 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:43:50.121+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:09.306605+0000 front 2023-12-16T21:39:09.306613+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:43:50.454 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:43:50 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:43:50.121+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:43:50.454 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:43:50 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:43:50.333+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:05.622397+0000 front 2023-12-16T21:39:05.622556+0000 (oldest deadline 2023-12-16T21:39:30.322546+0000) 2023-12-16T21:43:50.454 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:43:50 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:43:50.333+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:05.622506+0000 front 2023-12-16T21:39:05.622528+0000 (oldest deadline 2023-12-16T21:39:30.322546+0000) 2023-12-16T21:43:50.454 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:43:50 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:43:50.334+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:05.622479+0000 front 2023-12-16T21:39:05.622580+0000 (oldest deadline 2023-12-16T21:39:30.322546+0000) 2023-12-16T21:43:50.454 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:43:50 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:43:50.334+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:43:50.816 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:43:50 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:43:50.450+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:09.828911+0000 front 2023-12-16T21:39:09.828958+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:43:50.816 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:43:50 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:43:50.450+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:09.828868+0000 front 2023-12-16T21:39:09.828878+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:43:50.816 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:43:50 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:43:50.450+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:09.828947+0000 front 2023-12-16T21:39:09.828993+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:43:50.817 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:43:50 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:43:50.451+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:43:51.348 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:43:51 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:43:51.093+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:09.306583+0000 front 2023-12-16T21:39:09.306633+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:43:51.348 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:43:51 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:43:51.093+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:09.306551+0000 front 2023-12-16T21:39:09.306652+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:43:51.348 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:43:51 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:43:51.093+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:09.306605+0000 front 2023-12-16T21:39:09.306613+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:43:51.348 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:43:51 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:43:51.094+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:43:51.349 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:43:51 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:43:51.308+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:05.622397+0000 front 2023-12-16T21:39:05.622556+0000 (oldest deadline 2023-12-16T21:39:30.322546+0000) 2023-12-16T21:43:51.349 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:43:51 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:43:51.308+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:05.622506+0000 front 2023-12-16T21:39:05.622528+0000 (oldest deadline 2023-12-16T21:39:30.322546+0000) 2023-12-16T21:43:51.349 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:43:51 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:43:51.308+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:05.622479+0000 front 2023-12-16T21:39:05.622580+0000 (oldest deadline 2023-12-16T21:39:30.322546+0000) 2023-12-16T21:43:51.349 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:43:51 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:43:51.309+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:43:51.639 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Dec 16 21:43:51 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-mon-smithi045[80759]: 2023-12-16T21:43:51.345+0000 7f45a1f8f700 -1 mon.smithi045@0(probing) e2 get_health_metrics reporting 1234 slow ops, oldest is monmgrreport(gid 14214, 0 checks, 1 progress events) 2023-12-16T21:43:51.639 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:43:51 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:43:51.488+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:09.828911+0000 front 2023-12-16T21:39:09.828958+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:43:51.639 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:43:51 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:43:51.488+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:09.828868+0000 front 2023-12-16T21:39:09.828878+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:43:51.639 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:43:51 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:43:51.488+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:09.828947+0000 front 2023-12-16T21:39:09.828993+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:43:51.639 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:43:51 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:43:51.489+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:43:52.345 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:43:52 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:43:52.097+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:09.306583+0000 front 2023-12-16T21:39:09.306633+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:43:52.345 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:43:52 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:43:52.097+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:09.306551+0000 front 2023-12-16T21:39:09.306652+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:43:52.345 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:43:52 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:43:52.097+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:09.306605+0000 front 2023-12-16T21:39:09.306613+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:43:52.345 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:43:52 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:43:52.098+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:43:52.816 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:43:52 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:43:52.522+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:09.828911+0000 front 2023-12-16T21:39:09.828958+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:43:52.816 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:43:52 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:43:52.522+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:09.828868+0000 front 2023-12-16T21:39:09.828878+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:43:52.816 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:43:52 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:43:52.522+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:09.828947+0000 front 2023-12-16T21:39:09.828993+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:43:52.816 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:43:52 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:43:52.523+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:43:52.817 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:43:52 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:43:52.342+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:05.622397+0000 front 2023-12-16T21:39:05.622556+0000 (oldest deadline 2023-12-16T21:39:30.322546+0000) 2023-12-16T21:43:52.817 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:43:52 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:43:52.342+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:05.622506+0000 front 2023-12-16T21:39:05.622528+0000 (oldest deadline 2023-12-16T21:39:30.322546+0000) 2023-12-16T21:43:52.817 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:43:52 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:43:52.342+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:05.622479+0000 front 2023-12-16T21:39:05.622580+0000 (oldest deadline 2023-12-16T21:39:30.322546+0000) 2023-12-16T21:43:52.817 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:43:52 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:43:52.343+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:43:53.528 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:43:53 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:43:53.121+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:09.306583+0000 front 2023-12-16T21:39:09.306633+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:43:53.529 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:43:53 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:43:53.121+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:09.306551+0000 front 2023-12-16T21:39:09.306652+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:43:53.529 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:43:53 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:43:53.121+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:09.306605+0000 front 2023-12-16T21:39:09.306613+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:43:53.529 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:43:53 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:43:53.122+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:43:53.529 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:43:53 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:43:53.313+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:05.622397+0000 front 2023-12-16T21:39:05.622556+0000 (oldest deadline 2023-12-16T21:39:30.322546+0000) 2023-12-16T21:43:53.529 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:43:53 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:43:53.313+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:05.622506+0000 front 2023-12-16T21:39:05.622528+0000 (oldest deadline 2023-12-16T21:39:30.322546+0000) 2023-12-16T21:43:53.529 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:43:53 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:43:53.313+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:05.622479+0000 front 2023-12-16T21:39:05.622580+0000 (oldest deadline 2023-12-16T21:39:30.322546+0000) 2023-12-16T21:43:53.529 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:43:53 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:43:53.313+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:43:53.816 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:43:53 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:43:53.525+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:09.828911+0000 front 2023-12-16T21:39:09.828958+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:43:53.816 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:43:53 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:43:53.525+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:09.828868+0000 front 2023-12-16T21:39:09.828878+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:43:53.816 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:43:53 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:43:53.525+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:09.828947+0000 front 2023-12-16T21:39:09.828993+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:43:53.816 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:43:53 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:43:53.527+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:43:54.531 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:43:54 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:43:54.297+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:05.622397+0000 front 2023-12-16T21:39:05.622556+0000 (oldest deadline 2023-12-16T21:39:30.322546+0000) 2023-12-16T21:43:54.531 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:43:54 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:43:54.297+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:05.622506+0000 front 2023-12-16T21:39:05.622528+0000 (oldest deadline 2023-12-16T21:39:30.322546+0000) 2023-12-16T21:43:54.531 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:43:54 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:43:54.297+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:05.622479+0000 front 2023-12-16T21:39:05.622580+0000 (oldest deadline 2023-12-16T21:39:30.322546+0000) 2023-12-16T21:43:54.531 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:43:54 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:43:54.298+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:43:54.531 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:43:54 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:43:54.113+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:09.306583+0000 front 2023-12-16T21:39:09.306633+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:43:54.532 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:43:54 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:43:54.113+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:09.306551+0000 front 2023-12-16T21:39:09.306652+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:43:54.532 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:43:54 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:43:54.113+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:09.306605+0000 front 2023-12-16T21:39:09.306613+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:43:54.532 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:43:54 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:43:54.114+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:43:54.816 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:43:54 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:43:54.527+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:09.828911+0000 front 2023-12-16T21:39:09.828958+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:43:54.816 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:43:54 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:43:54.527+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:09.828868+0000 front 2023-12-16T21:39:09.828878+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:43:54.816 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:43:54 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:43:54.527+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:09.828947+0000 front 2023-12-16T21:39:09.828993+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:43:54.817 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:43:54 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:43:54.528+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:43:55.316 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:43:55 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:43:55.084+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:09.306583+0000 front 2023-12-16T21:39:09.306633+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:43:55.316 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:43:55 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:43:55.084+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:09.306551+0000 front 2023-12-16T21:39:09.306652+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:43:55.316 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:43:55 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:43:55.084+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:09.306605+0000 front 2023-12-16T21:39:09.306613+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:43:55.316 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:43:55 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:43:55.084+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:43:55.816 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:43:55 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:43:55.496+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:09.828911+0000 front 2023-12-16T21:39:09.828958+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:43:55.816 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:43:55 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:43:55.496+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:09.828868+0000 front 2023-12-16T21:39:09.828878+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:43:55.817 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:43:55 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:43:55.496+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:09.828947+0000 front 2023-12-16T21:39:09.828993+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:43:55.817 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:43:55 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:43:55.497+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:43:55.817 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:43:55 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:43:55.314+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:05.622397+0000 front 2023-12-16T21:39:05.622556+0000 (oldest deadline 2023-12-16T21:39:30.322546+0000) 2023-12-16T21:43:55.817 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:43:55 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:43:55.314+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:05.622506+0000 front 2023-12-16T21:39:05.622528+0000 (oldest deadline 2023-12-16T21:39:30.322546+0000) 2023-12-16T21:43:55.817 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:43:55 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:43:55.314+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:05.622479+0000 front 2023-12-16T21:39:05.622580+0000 (oldest deadline 2023-12-16T21:39:30.322546+0000) 2023-12-16T21:43:55.817 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:43:55 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:43:55.314+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:43:56.294 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:43:56 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:43:56.040+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:09.306583+0000 front 2023-12-16T21:39:09.306633+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:43:56.294 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:43:56 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:43:56.040+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:09.306551+0000 front 2023-12-16T21:39:09.306652+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:43:56.294 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:43:56 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:43:56.040+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:09.306605+0000 front 2023-12-16T21:39:09.306613+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:43:56.294 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:43:56 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:43:56.040+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:43:56.566 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Dec 16 21:43:56 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-mon-smithi045[80759]: 2023-12-16T21:43:56.345+0000 7f45a1f8f700 -1 mon.smithi045@0(probing) e2 get_health_metrics reporting 1314 slow ops, oldest is monmgrreport(gid 14214, 0 checks, 1 progress events) 2023-12-16T21:43:56.566 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:43:56 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:43:56.500+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:09.828911+0000 front 2023-12-16T21:39:09.828958+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:43:56.567 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:43:56 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:43:56.500+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:09.828868+0000 front 2023-12-16T21:39:09.828878+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:43:56.567 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:43:56 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:43:56.500+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:09.828947+0000 front 2023-12-16T21:39:09.828993+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:43:56.567 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:43:56 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:43:56.501+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:43:56.567 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:43:56 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:43:56.291+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:05.622397+0000 front 2023-12-16T21:39:05.622556+0000 (oldest deadline 2023-12-16T21:39:30.322546+0000) 2023-12-16T21:43:56.567 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:43:56 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:43:56.291+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:05.622506+0000 front 2023-12-16T21:39:05.622528+0000 (oldest deadline 2023-12-16T21:39:30.322546+0000) 2023-12-16T21:43:56.567 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:43:56 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:43:56.291+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:05.622479+0000 front 2023-12-16T21:39:05.622580+0000 (oldest deadline 2023-12-16T21:39:30.322546+0000) 2023-12-16T21:43:56.567 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:43:56 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:43:56.292+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:43:57.316 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:43:57 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:43:57.021+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:09.306583+0000 front 2023-12-16T21:39:09.306633+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:43:57.316 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:43:57 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:43:57.021+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:09.306551+0000 front 2023-12-16T21:39:09.306652+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:43:57.316 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:43:57 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:43:57.021+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:09.306605+0000 front 2023-12-16T21:39:09.306613+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:43:57.317 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:43:57 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:43:57.022+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:43:57.317 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:43:57 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:43:57.254+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:05.622397+0000 front 2023-12-16T21:39:05.622556+0000 (oldest deadline 2023-12-16T21:39:30.322546+0000) 2023-12-16T21:43:57.317 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:43:57 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:43:57.254+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:05.622506+0000 front 2023-12-16T21:39:05.622528+0000 (oldest deadline 2023-12-16T21:39:30.322546+0000) 2023-12-16T21:43:57.317 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:43:57 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:43:57.254+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:05.622479+0000 front 2023-12-16T21:39:05.622580+0000 (oldest deadline 2023-12-16T21:39:30.322546+0000) 2023-12-16T21:43:57.317 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:43:57 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:43:57.254+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:43:57.816 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:43:57 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:43:57.527+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:09.828911+0000 front 2023-12-16T21:39:09.828958+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:43:57.816 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:43:57 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:43:57.527+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:09.828868+0000 front 2023-12-16T21:39:09.828878+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:43:57.816 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:43:57 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:43:57.527+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:09.828947+0000 front 2023-12-16T21:39:09.828993+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:43:57.816 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:43:57 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:43:57.528+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:43:58.248 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:43:58 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:43:58.040+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:09.306583+0000 front 2023-12-16T21:39:09.306633+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:43:58.248 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:43:58 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:43:58.040+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:09.306551+0000 front 2023-12-16T21:39:09.306652+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:43:58.248 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:43:58 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:43:58.040+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:09.306605+0000 front 2023-12-16T21:39:09.306613+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:43:58.248 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:43:58 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:43:58.042+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:43:58.566 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:43:58 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:43:58.244+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:05.622397+0000 front 2023-12-16T21:39:05.622556+0000 (oldest deadline 2023-12-16T21:39:30.322546+0000) 2023-12-16T21:43:58.566 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:43:58 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:43:58.244+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:05.622506+0000 front 2023-12-16T21:39:05.622528+0000 (oldest deadline 2023-12-16T21:39:30.322546+0000) 2023-12-16T21:43:58.567 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:43:58 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:43:58.244+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:05.622479+0000 front 2023-12-16T21:39:05.622580+0000 (oldest deadline 2023-12-16T21:39:30.322546+0000) 2023-12-16T21:43:58.567 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:43:58 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:43:58.246+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:43:59.047 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:43:58 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:43:58.574+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:09.828911+0000 front 2023-12-16T21:39:09.828958+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:43:59.048 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:43:58 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:43:58.574+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:09.828868+0000 front 2023-12-16T21:39:09.828878+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:43:59.048 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:43:58 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:43:58.574+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:09.828947+0000 front 2023-12-16T21:39:09.828993+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:43:59.048 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:43:58 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:43:58.574+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:43:59.317 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:43:59 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:43:59.044+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:09.306583+0000 front 2023-12-16T21:39:09.306633+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:43:59.317 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:43:59 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:43:59.044+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:09.306551+0000 front 2023-12-16T21:39:09.306652+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:43:59.317 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:43:59 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:43:59.044+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:09.306605+0000 front 2023-12-16T21:39:09.306613+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:43:59.317 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:43:59 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:43:59.045+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:43:59.317 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:43:59 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:43:59.230+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:05.622397+0000 front 2023-12-16T21:39:05.622556+0000 (oldest deadline 2023-12-16T21:39:30.322546+0000) 2023-12-16T21:43:59.317 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:43:59 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:43:59.230+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:05.622506+0000 front 2023-12-16T21:39:05.622528+0000 (oldest deadline 2023-12-16T21:39:30.322546+0000) 2023-12-16T21:43:59.317 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:43:59 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:43:59.230+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:05.622479+0000 front 2023-12-16T21:39:05.622580+0000 (oldest deadline 2023-12-16T21:39:30.322546+0000) 2023-12-16T21:43:59.318 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:43:59 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:43:59.230+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:44:00.044 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:43:59 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:43:59.598+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:09.828911+0000 front 2023-12-16T21:39:09.828958+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:44:00.044 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:43:59 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:43:59.598+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:09.828868+0000 front 2023-12-16T21:39:09.828878+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:44:00.045 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:43:59 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:43:59.598+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:09.828947+0000 front 2023-12-16T21:39:09.828993+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:44:00.045 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:43:59 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:43:59.599+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:44:00.317 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:44:00 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:44:00.270+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:05.622397+0000 front 2023-12-16T21:39:05.622556+0000 (oldest deadline 2023-12-16T21:39:30.322546+0000) 2023-12-16T21:44:00.317 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:44:00 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:44:00.271+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:05.622506+0000 front 2023-12-16T21:39:05.622528+0000 (oldest deadline 2023-12-16T21:39:30.322546+0000) 2023-12-16T21:44:00.317 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:44:00 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:44:00.271+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:05.622479+0000 front 2023-12-16T21:39:05.622580+0000 (oldest deadline 2023-12-16T21:39:30.322546+0000) 2023-12-16T21:44:00.317 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:44:00 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:44:00.271+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:44:00.317 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:44:00 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:44:00.041+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:09.306583+0000 front 2023-12-16T21:39:09.306633+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:44:00.317 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:44:00 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:44:00.041+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:09.306551+0000 front 2023-12-16T21:39:09.306652+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:44:00.317 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:44:00 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:44:00.041+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:09.306605+0000 front 2023-12-16T21:39:09.306613+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:44:00.318 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:44:00 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:44:00.042+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:44:01.040 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:44:00 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:44:00.576+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:09.828911+0000 front 2023-12-16T21:39:09.828958+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:44:01.040 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:44:00 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:44:00.576+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:09.828868+0000 front 2023-12-16T21:39:09.828878+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:44:01.040 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:44:00 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:44:00.576+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:09.828947+0000 front 2023-12-16T21:39:09.828993+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:44:01.040 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:44:00 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:44:00.577+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:44:01.316 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:44:01 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:44:01.278+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:05.622397+0000 front 2023-12-16T21:39:05.622556+0000 (oldest deadline 2023-12-16T21:39:30.322546+0000) 2023-12-16T21:44:01.316 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:44:01 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:44:01.278+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:05.622506+0000 front 2023-12-16T21:39:05.622528+0000 (oldest deadline 2023-12-16T21:39:30.322546+0000) 2023-12-16T21:44:01.316 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:44:01 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:44:01.278+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:05.622479+0000 front 2023-12-16T21:39:05.622580+0000 (oldest deadline 2023-12-16T21:39:30.322546+0000) 2023-12-16T21:44:01.317 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:44:01 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:44:01.280+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:44:01.317 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:44:01 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:44:01.036+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:09.306583+0000 front 2023-12-16T21:39:09.306633+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:44:01.317 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:44:01 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:44:01.036+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:09.306551+0000 front 2023-12-16T21:39:09.306652+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:44:01.317 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:44:01 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:44:01.037+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:09.306605+0000 front 2023-12-16T21:39:09.306613+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:44:01.317 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:44:01 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:44:01.037+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:44:01.639 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Dec 16 21:44:01 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-mon-smithi045[80759]: 2023-12-16T21:44:01.346+0000 7f45a1f8f700 -1 mon.smithi045@0(probing) e2 get_health_metrics reporting 1395 slow ops, oldest is monmgrreport(gid 14214, 0 checks, 1 progress events) 2023-12-16T21:44:01.640 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:44:01 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:44:01.561+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:09.828911+0000 front 2023-12-16T21:39:09.828958+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:44:01.640 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:44:01 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:44:01.561+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:09.828868+0000 front 2023-12-16T21:39:09.828878+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:44:01.640 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:44:01 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:44:01.561+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:09.828947+0000 front 2023-12-16T21:39:09.828993+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:44:01.640 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:44:01 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:44:01.562+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:44:02.566 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:44:02 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:44:02.084+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:09.306583+0000 front 2023-12-16T21:39:09.306633+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:44:02.567 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:44:02 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:44:02.084+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:09.306551+0000 front 2023-12-16T21:39:09.306652+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:44:02.567 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:44:02 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:44:02.084+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:09.306605+0000 front 2023-12-16T21:39:09.306613+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:44:02.567 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:44:02 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:44:02.084+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:44:02.567 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:44:02 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:44:02.273+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:05.622397+0000 front 2023-12-16T21:39:05.622556+0000 (oldest deadline 2023-12-16T21:39:30.322546+0000) 2023-12-16T21:44:02.567 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:44:02 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:44:02.273+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:05.622506+0000 front 2023-12-16T21:39:05.622528+0000 (oldest deadline 2023-12-16T21:39:30.322546+0000) 2023-12-16T21:44:02.567 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:44:02 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:44:02.273+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:05.622479+0000 front 2023-12-16T21:39:05.622580+0000 (oldest deadline 2023-12-16T21:39:30.322546+0000) 2023-12-16T21:44:02.567 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:44:02 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:44:02.274+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:44:03.056 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:44:02 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:44:02.585+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:09.828911+0000 front 2023-12-16T21:39:09.828958+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:44:03.056 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:44:02 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:44:02.585+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:09.828868+0000 front 2023-12-16T21:39:09.828878+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:44:03.056 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:44:02 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:44:02.585+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:09.828947+0000 front 2023-12-16T21:39:09.828993+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:44:03.056 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:44:02 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:44:02.586+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:44:03.316 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:44:03 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:44:03.288+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:05.622397+0000 front 2023-12-16T21:39:05.622556+0000 (oldest deadline 2023-12-16T21:39:30.322546+0000) 2023-12-16T21:44:03.316 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:44:03 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:44:03.288+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:05.622506+0000 front 2023-12-16T21:39:05.622528+0000 (oldest deadline 2023-12-16T21:39:30.322546+0000) 2023-12-16T21:44:03.316 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:44:03 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:44:03.288+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:05.622479+0000 front 2023-12-16T21:39:05.622580+0000 (oldest deadline 2023-12-16T21:39:30.322546+0000) 2023-12-16T21:44:03.317 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:44:03 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:44:03.289+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:44:03.317 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:44:03 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:44:03.053+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:09.306583+0000 front 2023-12-16T21:39:09.306633+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:44:03.317 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:44:03 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:44:03.053+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:09.306551+0000 front 2023-12-16T21:39:09.306652+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:44:03.317 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:44:03 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:44:03.053+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:09.306605+0000 front 2023-12-16T21:39:09.306613+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:44:03.317 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:44:03 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:44:03.054+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:44:03.816 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:44:03 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:44:03.538+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:09.828911+0000 front 2023-12-16T21:39:09.828958+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:44:03.816 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:44:03 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:44:03.538+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:09.828868+0000 front 2023-12-16T21:39:09.828878+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:44:03.816 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:44:03 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:44:03.538+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:09.828947+0000 front 2023-12-16T21:39:09.828993+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:44:03.817 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:44:03 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:44:03.540+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:44:04.275 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:44:04 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:44:04.008+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:09.306583+0000 front 2023-12-16T21:39:09.306633+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:44:04.276 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:44:04 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:44:04.008+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:09.306551+0000 front 2023-12-16T21:39:09.306652+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:44:04.276 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:44:04 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:44:04.008+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:09.306605+0000 front 2023-12-16T21:39:09.306613+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:44:04.276 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:44:04 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:44:04.009+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:44:04.566 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:44:04 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:44:04.272+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:05.622397+0000 front 2023-12-16T21:39:05.622556+0000 (oldest deadline 2023-12-16T21:39:30.322546+0000) 2023-12-16T21:44:04.566 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:44:04 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:44:04.272+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:05.622506+0000 front 2023-12-16T21:39:05.622528+0000 (oldest deadline 2023-12-16T21:39:30.322546+0000) 2023-12-16T21:44:04.566 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:44:04 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:44:04.272+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:05.622479+0000 front 2023-12-16T21:39:05.622580+0000 (oldest deadline 2023-12-16T21:39:30.322546+0000) 2023-12-16T21:44:04.567 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:44:04 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:44:04.273+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:44:04.567 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:44:04 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:44:04.491+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:09.828911+0000 front 2023-12-16T21:39:09.828958+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:44:04.567 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:44:04 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:44:04.491+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:09.828868+0000 front 2023-12-16T21:39:09.828878+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:44:04.567 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:44:04 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:44:04.491+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:09.828947+0000 front 2023-12-16T21:39:09.828993+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:44:04.567 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:44:04 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:44:04.492+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:44:05.314 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:44:05 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:44:05.012+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:09.306583+0000 front 2023-12-16T21:39:09.306633+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:44:05.314 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:44:05 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:44:05.012+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:09.306551+0000 front 2023-12-16T21:39:09.306652+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:44:05.314 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:44:05 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:44:05.012+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:09.306605+0000 front 2023-12-16T21:39:09.306613+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:44:05.314 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:44:05 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:44:05.013+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:44:05.566 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:44:05 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:44:05.311+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:05.622397+0000 front 2023-12-16T21:39:05.622556+0000 (oldest deadline 2023-12-16T21:39:30.322546+0000) 2023-12-16T21:44:05.566 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:44:05 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:44:05.311+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:05.622506+0000 front 2023-12-16T21:39:05.622528+0000 (oldest deadline 2023-12-16T21:39:30.322546+0000) 2023-12-16T21:44:05.566 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:44:05 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:44:05.311+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:05.622479+0000 front 2023-12-16T21:39:05.622580+0000 (oldest deadline 2023-12-16T21:39:30.322546+0000) 2023-12-16T21:44:05.566 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:44:05 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:44:05.312+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:44:05.567 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:44:05 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:44:05.458+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:09.828911+0000 front 2023-12-16T21:39:09.828958+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:44:05.567 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:44:05 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:44:05.458+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:09.828868+0000 front 2023-12-16T21:39:09.828878+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:44:05.567 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:44:05 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:44:05.458+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:09.828947+0000 front 2023-12-16T21:39:09.828993+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:44:05.567 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:44:05 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:44:05.459+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:44:06.316 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:44:05 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:44:05.991+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:09.306583+0000 front 2023-12-16T21:39:09.306633+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:44:06.316 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:44:05 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:44:05.991+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:09.306551+0000 front 2023-12-16T21:39:09.306652+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:44:06.317 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:44:05 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:44:05.991+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:09.306605+0000 front 2023-12-16T21:39:09.306613+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:44:06.317 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:44:05 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:44:05.991+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:44:06.816 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Dec 16 21:44:06 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-mon-smithi045[80759]: 2023-12-16T21:44:06.346+0000 7f45a1f8f700 -1 mon.smithi045@0(probing) e2 get_health_metrics reporting 1472 slow ops, oldest is monmgrreport(gid 14214, 0 checks, 1 progress events) 2023-12-16T21:44:06.817 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:44:06 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:44:06.346+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:05.622397+0000 front 2023-12-16T21:39:05.622556+0000 (oldest deadline 2023-12-16T21:39:30.322546+0000) 2023-12-16T21:44:06.817 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:44:06 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:44:06.346+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:05.622506+0000 front 2023-12-16T21:39:05.622528+0000 (oldest deadline 2023-12-16T21:39:30.322546+0000) 2023-12-16T21:44:06.817 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:44:06 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:44:06.346+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:05.622479+0000 front 2023-12-16T21:39:05.622580+0000 (oldest deadline 2023-12-16T21:39:30.322546+0000) 2023-12-16T21:44:06.817 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:44:06 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:44:06.348+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:44:06.817 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:44:06 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:44:06.492+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:09.828911+0000 front 2023-12-16T21:39:09.828958+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:44:06.817 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:44:06 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:44:06.492+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:09.828868+0000 front 2023-12-16T21:39:09.828878+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:44:06.817 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:44:06 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:44:06.492+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:09.828947+0000 front 2023-12-16T21:39:09.828993+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:44:06.817 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:44:06 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:44:06.493+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:44:07.313 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:44:06 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:44:06.946+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:09.306583+0000 front 2023-12-16T21:39:09.306633+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:44:07.313 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:44:06 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:44:06.946+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:09.306551+0000 front 2023-12-16T21:39:09.306652+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:44:07.313 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:44:06 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:44:06.946+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:09.306605+0000 front 2023-12-16T21:39:09.306613+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:44:07.313 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:44:06 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:44:06.946+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:44:07.566 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:44:07 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:44:07.310+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:05.622397+0000 front 2023-12-16T21:39:05.622556+0000 (oldest deadline 2023-12-16T21:39:30.322546+0000) 2023-12-16T21:44:07.567 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:44:07 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:44:07.310+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:05.622506+0000 front 2023-12-16T21:39:05.622528+0000 (oldest deadline 2023-12-16T21:39:30.322546+0000) 2023-12-16T21:44:07.567 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:44:07 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:44:07.310+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:05.622479+0000 front 2023-12-16T21:39:05.622580+0000 (oldest deadline 2023-12-16T21:39:30.322546+0000) 2023-12-16T21:44:07.567 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:44:07 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:44:07.310+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:44:07.567 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:44:07 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:44:07.485+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:09.828911+0000 front 2023-12-16T21:39:09.828958+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:44:07.567 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:44:07 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:44:07.485+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:09.828868+0000 front 2023-12-16T21:39:09.828878+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:44:07.567 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:44:07 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:44:07.485+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:09.828947+0000 front 2023-12-16T21:39:09.828993+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:44:07.567 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:44:07 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:44:07.485+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:44:08.316 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:44:07 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:44:07.966+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:09.306583+0000 front 2023-12-16T21:39:09.306633+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:44:08.316 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:44:07 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:44:07.966+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:09.306551+0000 front 2023-12-16T21:39:09.306652+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:44:08.316 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:44:07 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:44:07.966+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:09.306605+0000 front 2023-12-16T21:39:09.306613+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:44:08.316 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:44:07 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:44:07.967+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:44:08.722 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:44:08 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:44:08.452+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:09.828911+0000 front 2023-12-16T21:39:09.828958+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:44:08.723 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:44:08 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:44:08.452+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:09.828868+0000 front 2023-12-16T21:39:09.828878+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:44:08.723 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:44:08 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:44:08.452+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:09.828947+0000 front 2023-12-16T21:39:09.828993+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:44:08.723 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:44:08 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:44:08.453+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:44:08.723 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:44:08 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:44:08.354+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:05.622397+0000 front 2023-12-16T21:39:05.622556+0000 (oldest deadline 2023-12-16T21:39:30.322546+0000) 2023-12-16T21:44:08.723 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:44:08 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:44:08.354+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:05.622506+0000 front 2023-12-16T21:39:05.622528+0000 (oldest deadline 2023-12-16T21:39:30.322546+0000) 2023-12-16T21:44:08.723 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:44:08 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:44:08.354+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:05.622479+0000 front 2023-12-16T21:39:05.622580+0000 (oldest deadline 2023-12-16T21:39:30.322546+0000) 2023-12-16T21:44:08.723 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:44:08 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:44:08.355+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:44:09.069 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:44:08 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:44:08.945+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:09.306583+0000 front 2023-12-16T21:39:09.306633+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:44:09.069 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:44:08 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:44:08.945+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:09.306551+0000 front 2023-12-16T21:39:09.306652+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:44:09.069 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:44:08 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:44:08.945+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:09.306605+0000 front 2023-12-16T21:39:09.306613+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:44:09.069 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:44:08 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:44:08.945+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:44:09.816 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:44:09 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:44:09.379+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:05.622397+0000 front 2023-12-16T21:39:05.622556+0000 (oldest deadline 2023-12-16T21:39:30.322546+0000) 2023-12-16T21:44:09.817 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:44:09 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:44:09.379+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:05.622506+0000 front 2023-12-16T21:39:05.622528+0000 (oldest deadline 2023-12-16T21:39:30.322546+0000) 2023-12-16T21:44:09.817 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:44:09 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:44:09.379+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:05.622479+0000 front 2023-12-16T21:39:05.622580+0000 (oldest deadline 2023-12-16T21:39:30.322546+0000) 2023-12-16T21:44:09.817 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:44:09 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:44:09.381+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:44:09.817 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:44:09 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:44:09.440+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:09.828911+0000 front 2023-12-16T21:39:09.828958+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:44:09.817 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:44:09 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:44:09.440+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:09.828868+0000 front 2023-12-16T21:39:09.828878+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:44:09.817 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:44:09 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:44:09.440+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:09.828947+0000 front 2023-12-16T21:39:09.828993+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:44:09.817 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:44:09 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:44:09.441+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:44:10.260 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:44:09 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:44:09.948+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:09.306583+0000 front 2023-12-16T21:39:09.306633+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:44:10.260 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:44:09 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:44:09.948+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:09.306551+0000 front 2023-12-16T21:39:09.306652+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:44:10.260 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:44:09 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:44:09.948+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:09.306605+0000 front 2023-12-16T21:39:09.306613+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:44:10.260 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:44:09 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:44:09.949+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:44:10.817 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:44:10 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:44:10.367+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:05.622397+0000 front 2023-12-16T21:39:05.622556+0000 (oldest deadline 2023-12-16T21:39:30.322546+0000) 2023-12-16T21:44:10.817 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:44:10 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:44:10.367+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:05.622506+0000 front 2023-12-16T21:39:05.622528+0000 (oldest deadline 2023-12-16T21:39:30.322546+0000) 2023-12-16T21:44:10.817 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:44:10 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:44:10.367+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:05.622479+0000 front 2023-12-16T21:39:05.622580+0000 (oldest deadline 2023-12-16T21:39:30.322546+0000) 2023-12-16T21:44:10.817 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:44:10 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:44:10.368+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:44:10.818 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:44:10 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:44:10.459+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:09.828911+0000 front 2023-12-16T21:39:09.828958+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:44:10.818 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:44:10 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:44:10.459+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:09.828868+0000 front 2023-12-16T21:39:09.828878+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:44:10.818 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:44:10 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:44:10.459+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:09.828947+0000 front 2023-12-16T21:39:09.828993+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:44:10.818 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:44:10 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:44:10.460+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:44:11.316 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:44:10 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:44:10.908+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:09.306583+0000 front 2023-12-16T21:39:09.306633+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:44:11.317 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:44:10 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:44:10.908+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:09.306551+0000 front 2023-12-16T21:39:09.306652+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:44:11.317 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:44:10 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:44:10.908+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:09.306605+0000 front 2023-12-16T21:39:09.306613+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:44:11.317 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:44:10 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:44:10.909+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:44:11.640 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Dec 16 21:44:11 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-mon-smithi045[80759]: 2023-12-16T21:44:11.347+0000 7f45a1f8f700 -1 mon.smithi045@0(probing) e2 get_health_metrics reporting 1552 slow ops, oldest is monmgrreport(gid 14214, 0 checks, 1 progress events) 2023-12-16T21:44:11.640 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:44:11 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:44:11.435+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:09.828911+0000 front 2023-12-16T21:39:09.828958+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:44:11.640 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:44:11 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:44:11.435+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:09.828868+0000 front 2023-12-16T21:39:09.828878+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:44:11.640 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:44:11 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:44:11.435+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:09.828947+0000 front 2023-12-16T21:39:09.828993+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:44:11.640 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:44:11 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:44:11.436+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:44:11.641 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:44:11 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:44:11.323+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:05.622397+0000 front 2023-12-16T21:39:05.622556+0000 (oldest deadline 2023-12-16T21:39:30.322546+0000) 2023-12-16T21:44:11.641 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:44:11 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:44:11.323+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:05.622506+0000 front 2023-12-16T21:39:05.622528+0000 (oldest deadline 2023-12-16T21:39:30.322546+0000) 2023-12-16T21:44:11.641 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:44:11 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:44:11.323+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:05.622479+0000 front 2023-12-16T21:39:05.622580+0000 (oldest deadline 2023-12-16T21:39:30.322546+0000) 2023-12-16T21:44:11.641 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:44:11 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:44:11.323+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:44:12.316 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:44:11 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:44:11.923+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:09.306583+0000 front 2023-12-16T21:39:09.306633+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:44:12.317 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:44:11 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:44:11.923+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:09.306551+0000 front 2023-12-16T21:39:09.306652+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:44:12.317 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:44:11 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:44:11.923+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:09.306605+0000 front 2023-12-16T21:39:09.306613+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:44:12.317 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:44:11 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:44:11.924+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:44:12.817 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:44:12 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:44:12.346+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:05.622397+0000 front 2023-12-16T21:39:05.622556+0000 (oldest deadline 2023-12-16T21:39:30.322546+0000) 2023-12-16T21:44:12.817 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:44:12 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:44:12.346+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:05.622506+0000 front 2023-12-16T21:39:05.622528+0000 (oldest deadline 2023-12-16T21:39:30.322546+0000) 2023-12-16T21:44:12.817 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:44:12 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:44:12.346+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:05.622479+0000 front 2023-12-16T21:39:05.622580+0000 (oldest deadline 2023-12-16T21:39:30.322546+0000) 2023-12-16T21:44:12.817 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:44:12 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:44:12.347+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:44:12.817 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:44:12 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:44:12.455+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:09.828911+0000 front 2023-12-16T21:39:09.828958+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:44:12.817 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:44:12 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:44:12.455+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:09.828868+0000 front 2023-12-16T21:39:09.828878+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:44:12.818 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:44:12 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:44:12.455+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:09.828947+0000 front 2023-12-16T21:39:09.828993+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:44:12.818 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:44:12 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:44:12.457+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:44:13.316 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:44:12 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:44:12.965+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:09.306583+0000 front 2023-12-16T21:39:09.306633+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:44:13.317 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:44:12 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:44:12.965+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:09.306551+0000 front 2023-12-16T21:39:09.306652+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:44:13.317 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:44:12 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:44:12.965+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:09.306605+0000 front 2023-12-16T21:39:09.306613+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:44:13.317 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:44:12 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:44:12.966+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:44:13.817 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:44:13 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:44:13.366+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:05.622397+0000 front 2023-12-16T21:39:05.622556+0000 (oldest deadline 2023-12-16T21:39:30.322546+0000) 2023-12-16T21:44:13.817 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:44:13 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:44:13.366+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:05.622506+0000 front 2023-12-16T21:39:05.622528+0000 (oldest deadline 2023-12-16T21:39:30.322546+0000) 2023-12-16T21:44:13.817 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:44:13 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:44:13.366+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:05.622479+0000 front 2023-12-16T21:39:05.622580+0000 (oldest deadline 2023-12-16T21:39:30.322546+0000) 2023-12-16T21:44:13.817 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:44:13 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:44:13.367+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:44:13.817 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:44:13 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:44:13.471+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:09.828911+0000 front 2023-12-16T21:39:09.828958+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:44:13.817 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:44:13 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:44:13.471+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:09.828868+0000 front 2023-12-16T21:39:09.828878+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:44:13.818 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:44:13 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:44:13.471+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:09.828947+0000 front 2023-12-16T21:39:09.828993+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:44:13.818 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:44:13 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:44:13.471+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:44:14.316 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:44:13 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:44:13.932+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:09.306583+0000 front 2023-12-16T21:39:09.306633+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:44:14.317 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:44:13 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:44:13.932+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:09.306551+0000 front 2023-12-16T21:39:09.306652+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:44:14.317 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:44:13 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:44:13.932+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:09.306605+0000 front 2023-12-16T21:39:09.306613+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:44:14.318 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:44:13 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:44:13.933+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:44:14.817 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:44:14 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:44:14.384+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:05.622397+0000 front 2023-12-16T21:39:05.622556+0000 (oldest deadline 2023-12-16T21:39:30.322546+0000) 2023-12-16T21:44:14.817 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:44:14 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:44:14.384+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:05.622506+0000 front 2023-12-16T21:39:05.622528+0000 (oldest deadline 2023-12-16T21:39:30.322546+0000) 2023-12-16T21:44:14.817 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:44:14 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:44:14.384+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:05.622479+0000 front 2023-12-16T21:39:05.622580+0000 (oldest deadline 2023-12-16T21:39:30.322546+0000) 2023-12-16T21:44:14.817 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:44:14 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:44:14.385+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:44:14.817 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:44:14 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:44:14.500+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:09.828911+0000 front 2023-12-16T21:39:09.828958+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:44:14.817 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:44:14 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:44:14.500+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:09.828868+0000 front 2023-12-16T21:39:09.828878+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:44:14.817 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:44:14 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:44:14.500+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:09.828947+0000 front 2023-12-16T21:39:09.828993+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:44:14.817 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:44:14 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:44:14.501+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:44:15.316 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:44:14 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:44:14.951+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:09.306583+0000 front 2023-12-16T21:39:09.306633+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:44:15.317 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:44:14 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:44:14.951+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:09.306551+0000 front 2023-12-16T21:39:09.306652+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:44:15.317 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:44:14 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:44:14.951+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:09.306605+0000 front 2023-12-16T21:39:09.306613+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:44:15.317 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:44:14 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:44:14.952+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:44:15.817 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:44:15 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:44:15.405+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:05.622397+0000 front 2023-12-16T21:39:05.622556+0000 (oldest deadline 2023-12-16T21:39:30.322546+0000) 2023-12-16T21:44:15.817 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:44:15 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:44:15.405+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:05.622506+0000 front 2023-12-16T21:39:05.622528+0000 (oldest deadline 2023-12-16T21:39:30.322546+0000) 2023-12-16T21:44:15.817 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:44:15 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:44:15.405+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:05.622479+0000 front 2023-12-16T21:39:05.622580+0000 (oldest deadline 2023-12-16T21:39:30.322546+0000) 2023-12-16T21:44:15.817 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:44:15 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:44:15.407+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:44:15.818 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:44:15 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:44:15.545+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:09.828911+0000 front 2023-12-16T21:39:09.828958+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:44:15.818 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:44:15 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:44:15.545+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:09.828868+0000 front 2023-12-16T21:39:09.828878+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:44:15.818 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:44:15 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:44:15.545+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:09.828947+0000 front 2023-12-16T21:39:09.828993+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:44:15.818 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:44:15 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:44:15.545+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:44:16.316 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:44:15 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:44:15.970+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:09.306583+0000 front 2023-12-16T21:39:09.306633+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:44:16.316 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:44:15 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:44:15.970+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:09.306551+0000 front 2023-12-16T21:39:09.306652+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:44:16.317 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:44:15 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:44:15.970+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:09.306605+0000 front 2023-12-16T21:39:09.306613+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:44:16.317 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:44:15 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:44:15.970+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:44:16.816 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Dec 16 21:44:16 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-mon-smithi045[80759]: 2023-12-16T21:44:16.348+0000 7f45a1f8f700 -1 mon.smithi045@0(probing) e2 get_health_metrics reporting 1630 slow ops, oldest is monmgrreport(gid 14214, 0 checks, 1 progress events) 2023-12-16T21:44:16.817 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:44:16 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:44:16.559+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:09.828911+0000 front 2023-12-16T21:39:09.828958+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:44:16.817 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:44:16 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:44:16.559+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:09.828868+0000 front 2023-12-16T21:39:09.828878+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:44:16.817 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:44:16 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:44:16.559+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:09.828947+0000 front 2023-12-16T21:39:09.828993+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:44:16.817 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:44:16 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:44:16.559+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:44:16.817 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:44:16 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:44:16.425+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:05.622397+0000 front 2023-12-16T21:39:05.622556+0000 (oldest deadline 2023-12-16T21:39:30.322546+0000) 2023-12-16T21:44:16.817 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:44:16 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:44:16.425+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:05.622506+0000 front 2023-12-16T21:39:05.622528+0000 (oldest deadline 2023-12-16T21:39:30.322546+0000) 2023-12-16T21:44:16.818 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:44:16 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:44:16.425+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:05.622479+0000 front 2023-12-16T21:39:05.622580+0000 (oldest deadline 2023-12-16T21:39:30.322546+0000) 2023-12-16T21:44:16.818 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:44:16 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:44:16.425+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:44:17.316 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:44:16 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:44:16.985+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:09.306583+0000 front 2023-12-16T21:39:09.306633+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:44:17.317 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:44:16 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:44:16.985+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:09.306551+0000 front 2023-12-16T21:39:09.306652+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:44:17.317 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:44:16 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:44:16.985+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:09.306605+0000 front 2023-12-16T21:39:09.306613+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:44:17.317 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:44:16 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:44:16.987+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:44:17.816 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:44:17 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:44:17.463+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:05.622397+0000 front 2023-12-16T21:39:05.622556+0000 (oldest deadline 2023-12-16T21:39:30.322546+0000) 2023-12-16T21:44:17.817 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:44:17 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:44:17.463+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:05.622506+0000 front 2023-12-16T21:39:05.622528+0000 (oldest deadline 2023-12-16T21:39:30.322546+0000) 2023-12-16T21:44:17.817 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:44:17 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:44:17.463+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:05.622479+0000 front 2023-12-16T21:39:05.622580+0000 (oldest deadline 2023-12-16T21:39:30.322546+0000) 2023-12-16T21:44:17.817 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:44:17 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:44:17.463+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:44:17.817 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:44:17 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:44:17.607+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:09.828911+0000 front 2023-12-16T21:39:09.828958+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:44:17.817 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:44:17 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:44:17.607+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:09.828868+0000 front 2023-12-16T21:39:09.828878+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:44:17.817 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:44:17 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:44:17.607+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:09.828947+0000 front 2023-12-16T21:39:09.828993+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:44:17.817 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:44:17 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:44:17.608+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:44:18.317 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:44:17 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:44:17.961+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:09.306583+0000 front 2023-12-16T21:39:09.306633+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:44:18.317 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:44:17 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:44:17.961+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:09.306551+0000 front 2023-12-16T21:39:09.306652+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:44:18.317 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:44:17 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:44:17.961+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:09.306605+0000 front 2023-12-16T21:39:09.306613+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:44:18.317 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:44:17 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:44:17.962+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:44:18.722 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:44:18 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:44:18.436+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:05.622397+0000 front 2023-12-16T21:39:05.622556+0000 (oldest deadline 2023-12-16T21:39:30.322546+0000) 2023-12-16T21:44:18.722 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:44:18 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:44:18.436+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:05.622506+0000 front 2023-12-16T21:39:05.622528+0000 (oldest deadline 2023-12-16T21:39:30.322546+0000) 2023-12-16T21:44:18.723 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:44:18 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:44:18.436+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:05.622479+0000 front 2023-12-16T21:39:05.622580+0000 (oldest deadline 2023-12-16T21:39:30.322546+0000) 2023-12-16T21:44:18.723 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:44:18 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:44:18.437+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:44:18.723 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:44:18 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:44:18.559+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:09.828911+0000 front 2023-12-16T21:39:09.828958+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:44:18.723 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:44:18 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:44:18.559+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:09.828868+0000 front 2023-12-16T21:39:09.828878+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:44:18.723 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:44:18 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:44:18.559+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:09.828947+0000 front 2023-12-16T21:39:09.828993+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:44:18.723 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:44:18 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:44:18.560+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:44:19.066 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:44:18 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:44:18.960+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:09.306583+0000 front 2023-12-16T21:39:09.306633+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:44:19.067 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:44:18 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:44:18.960+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:09.306551+0000 front 2023-12-16T21:39:09.306652+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:44:19.067 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:44:18 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:44:18.960+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:09.306605+0000 front 2023-12-16T21:39:09.306613+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:44:19.067 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:44:18 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:44:18.961+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:44:19.816 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:44:19 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:44:19.560+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:09.828911+0000 front 2023-12-16T21:39:09.828958+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:44:19.817 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:44:19 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:44:19.560+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:09.828868+0000 front 2023-12-16T21:39:09.828878+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:44:19.817 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:44:19 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:44:19.560+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:09.828947+0000 front 2023-12-16T21:39:09.828993+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:44:19.817 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:44:19 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:44:19.561+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:44:19.817 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:44:19 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:44:19.483+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:05.622397+0000 front 2023-12-16T21:39:05.622556+0000 (oldest deadline 2023-12-16T21:39:30.322546+0000) 2023-12-16T21:44:19.817 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:44:19 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:44:19.483+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:05.622506+0000 front 2023-12-16T21:39:05.622528+0000 (oldest deadline 2023-12-16T21:39:30.322546+0000) 2023-12-16T21:44:19.817 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:44:19 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:44:19.483+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:05.622479+0000 front 2023-12-16T21:39:05.622580+0000 (oldest deadline 2023-12-16T21:39:30.322546+0000) 2023-12-16T21:44:19.817 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:44:19 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:44:19.483+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:44:20.316 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:44:19 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:44:19.980+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:09.306583+0000 front 2023-12-16T21:39:09.306633+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:44:20.317 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:44:19 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:44:19.980+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:09.306551+0000 front 2023-12-16T21:39:09.306652+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:44:20.317 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:44:19 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:44:19.980+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:09.306605+0000 front 2023-12-16T21:39:09.306613+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:44:20.317 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:44:19 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:44:19.980+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:44:20.816 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:44:20 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:44:20.552+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:09.828911+0000 front 2023-12-16T21:39:09.828958+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:44:20.817 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:44:20 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:44:20.552+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:09.828868+0000 front 2023-12-16T21:39:09.828878+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:44:20.817 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:44:20 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:44:20.552+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:09.828947+0000 front 2023-12-16T21:39:09.828993+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:44:20.817 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:44:20 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:44:20.552+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:44:20.817 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:44:20 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:44:20.452+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:05.622397+0000 front 2023-12-16T21:39:05.622556+0000 (oldest deadline 2023-12-16T21:39:30.322546+0000) 2023-12-16T21:44:20.817 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:44:20 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:44:20.452+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:05.622506+0000 front 2023-12-16T21:39:05.622528+0000 (oldest deadline 2023-12-16T21:39:30.322546+0000) 2023-12-16T21:44:20.817 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:44:20 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:44:20.452+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:05.622479+0000 front 2023-12-16T21:39:05.622580+0000 (oldest deadline 2023-12-16T21:39:30.322546+0000) 2023-12-16T21:44:20.817 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:44:20 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:44:20.452+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:44:21.317 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:44:20 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:44:20.940+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:09.306583+0000 front 2023-12-16T21:39:09.306633+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:44:21.317 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:44:20 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:44:20.940+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:09.306551+0000 front 2023-12-16T21:39:09.306652+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:44:21.317 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:44:20 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:44:20.940+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:09.306605+0000 front 2023-12-16T21:39:09.306613+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:44:21.317 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:44:20 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:44:20.942+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:44:21.640 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Dec 16 21:44:21 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-mon-smithi045[80759]: 2023-12-16T21:44:21.348+0000 7f45a1f8f700 -1 mon.smithi045@0(probing) e2 get_health_metrics reporting 1713 slow ops, oldest is monmgrreport(gid 14214, 0 checks, 1 progress events) 2023-12-16T21:44:21.641 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:44:21 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:44:21.529+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:09.828911+0000 front 2023-12-16T21:39:09.828958+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:44:21.641 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:44:21 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:44:21.529+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:09.828868+0000 front 2023-12-16T21:39:09.828878+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:44:21.641 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:44:21 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:44:21.529+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:09.828947+0000 front 2023-12-16T21:39:09.828993+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:44:21.641 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:44:21 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:44:21.530+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:44:21.641 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:44:21 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:44:21.470+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:05.622397+0000 front 2023-12-16T21:39:05.622556+0000 (oldest deadline 2023-12-16T21:39:30.322546+0000) 2023-12-16T21:44:21.641 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:44:21 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:44:21.470+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:05.622506+0000 front 2023-12-16T21:39:05.622528+0000 (oldest deadline 2023-12-16T21:39:30.322546+0000) 2023-12-16T21:44:21.641 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:44:21 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:44:21.470+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:05.622479+0000 front 2023-12-16T21:39:05.622580+0000 (oldest deadline 2023-12-16T21:39:30.322546+0000) 2023-12-16T21:44:21.642 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:44:21 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:44:21.471+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:44:22.316 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:44:21 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:44:21.978+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:09.306583+0000 front 2023-12-16T21:39:09.306633+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:44:22.316 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:44:21 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:44:21.978+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:09.306551+0000 front 2023-12-16T21:39:09.306652+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:44:22.316 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:44:21 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:44:21.978+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:09.306605+0000 front 2023-12-16T21:39:09.306613+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:44:22.317 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:44:21 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:44:21.979+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:44:22.817 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:44:22 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:44:22.537+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:09.828911+0000 front 2023-12-16T21:39:09.828958+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:44:22.817 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:44:22 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:44:22.537+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:09.828868+0000 front 2023-12-16T21:39:09.828878+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:44:22.817 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:44:22 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:44:22.537+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:09.828947+0000 front 2023-12-16T21:39:09.828993+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:44:22.817 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:44:22 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:44:22.538+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:44:22.817 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:44:22 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:44:22.467+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:05.622397+0000 front 2023-12-16T21:39:05.622556+0000 (oldest deadline 2023-12-16T21:39:30.322546+0000) 2023-12-16T21:44:22.818 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:44:22 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:44:22.467+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:05.622506+0000 front 2023-12-16T21:39:05.622528+0000 (oldest deadline 2023-12-16T21:39:30.322546+0000) 2023-12-16T21:44:22.818 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:44:22 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:44:22.467+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:05.622479+0000 front 2023-12-16T21:39:05.622580+0000 (oldest deadline 2023-12-16T21:39:30.322546+0000) 2023-12-16T21:44:22.818 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:44:22 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:44:22.467+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:44:23.316 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:44:23 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:44:23.020+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:09.306583+0000 front 2023-12-16T21:39:09.306633+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:44:23.317 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:44:23 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:44:23.020+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:09.306551+0000 front 2023-12-16T21:39:09.306652+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:44:23.317 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:44:23 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:44:23.020+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:09.306605+0000 front 2023-12-16T21:39:09.306613+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:44:23.317 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:44:23 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:44:23.021+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:44:23.390 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:44:23.387+0000 7fbe3be6b700 0 monclient(hunting): authenticate timed out after 300 2023-12-16T21:44:23.390 INFO:teuthology.orchestra.run.smithi045.stderr:[errno 110] RADOS timed out (error connecting to the cluster) 2023-12-16T21:44:23.672 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:44:23 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:44:23.505+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:05.622397+0000 front 2023-12-16T21:39:05.622556+0000 (oldest deadline 2023-12-16T21:39:30.322546+0000) 2023-12-16T21:44:23.672 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:44:23 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:44:23.505+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:05.622506+0000 front 2023-12-16T21:39:05.622528+0000 (oldest deadline 2023-12-16T21:39:30.322546+0000) 2023-12-16T21:44:23.672 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:44:23 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:44:23.505+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:05.622479+0000 front 2023-12-16T21:39:05.622580+0000 (oldest deadline 2023-12-16T21:39:30.322546+0000) 2023-12-16T21:44:23.672 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:44:23 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:44:23.506+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:44:23.673 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:44:23 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:44:23.542+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:09.828911+0000 front 2023-12-16T21:39:09.828958+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:44:23.673 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:44:23 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:44:23.542+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:09.828868+0000 front 2023-12-16T21:39:09.828878+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:44:23.673 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:44:23 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:44:23.542+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:09.828947+0000 front 2023-12-16T21:39:09.828993+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:44:23.673 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:44:23 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:44:23.543+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:44:23.698 DEBUG:teuthology.orchestra.run.smithi045:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 88ff363c-9c5a-11ee-95a5-87774f69a715 -e sha1=c7586e95519b7be76f3b6e3a4b333888baf46bc9 -- bash -c 'ceph versions | jq -e '"'"'.mgr | length == 1'"'"'' 2023-12-16T21:44:24.031 INFO:teuthology.orchestra.run.smithi045.stderr:Inferring config /var/lib/ceph/88ff363c-9c5a-11ee-95a5-87774f69a715/mon.smithi045/config 2023-12-16T21:44:24.316 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:44:24 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:44:24.005+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:09.306583+0000 front 2023-12-16T21:39:09.306633+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:44:24.316 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:44:24 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:44:24.005+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:09.306551+0000 front 2023-12-16T21:39:09.306652+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:44:24.317 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:44:24 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:44:24.005+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:09.306605+0000 front 2023-12-16T21:39:09.306613+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:44:24.317 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:44:24 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:44:24.006+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:44:24.755 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:44:24 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:44:24.576+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:09.828911+0000 front 2023-12-16T21:39:09.828958+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:44:24.755 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:44:24 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:44:24.576+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:09.828868+0000 front 2023-12-16T21:39:09.828878+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:44:24.755 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:44:24 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:44:24.576+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:09.828947+0000 front 2023-12-16T21:39:09.828993+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:44:24.755 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:44:24 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:44:24.577+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:44:24.755 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:44:24 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:44:24.460+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:05.622397+0000 front 2023-12-16T21:39:05.622556+0000 (oldest deadline 2023-12-16T21:39:30.322546+0000) 2023-12-16T21:44:24.756 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:44:24 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:44:24.460+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:05.622506+0000 front 2023-12-16T21:39:05.622528+0000 (oldest deadline 2023-12-16T21:39:30.322546+0000) 2023-12-16T21:44:24.756 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:44:24 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:44:24.460+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:05.622479+0000 front 2023-12-16T21:39:05.622580+0000 (oldest deadline 2023-12-16T21:39:30.322546+0000) 2023-12-16T21:44:24.756 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:44:24 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:44:24.461+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:44:25.066 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:44:24 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:44:24.994+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:09.306583+0000 front 2023-12-16T21:39:09.306633+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:44:25.066 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:44:24 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:44:24.994+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:09.306551+0000 front 2023-12-16T21:39:09.306652+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:44:25.066 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:44:24 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:44:24.994+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:09.306605+0000 front 2023-12-16T21:39:09.306613+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:44:25.067 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:44:24 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:44:24.995+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:44:25.816 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:44:25 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:44:25.620+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:09.828911+0000 front 2023-12-16T21:39:09.828958+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:44:25.816 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:44:25 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:44:25.620+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:09.828868+0000 front 2023-12-16T21:39:09.828878+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:44:25.817 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:44:25 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:44:25.620+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:09.828947+0000 front 2023-12-16T21:39:09.828993+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:44:25.817 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:44:25 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:44:25.621+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:44:25.817 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:44:25 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:44:25.505+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:05.622397+0000 front 2023-12-16T21:39:05.622556+0000 (oldest deadline 2023-12-16T21:39:30.322546+0000) 2023-12-16T21:44:25.817 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:44:25 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:44:25.505+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:05.622506+0000 front 2023-12-16T21:39:05.622528+0000 (oldest deadline 2023-12-16T21:39:30.322546+0000) 2023-12-16T21:44:25.817 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:44:25 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:44:25.505+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:05.622479+0000 front 2023-12-16T21:39:05.622580+0000 (oldest deadline 2023-12-16T21:39:30.322546+0000) 2023-12-16T21:44:25.817 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:44:25 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:44:25.506+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:44:26.316 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:44:26 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:44:26.006+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:09.306583+0000 front 2023-12-16T21:39:09.306633+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:44:26.317 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:44:26 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:44:26.006+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:09.306551+0000 front 2023-12-16T21:39:09.306652+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:44:26.317 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:44:26 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:44:26.006+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:09.306605+0000 front 2023-12-16T21:39:09.306613+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:44:26.317 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:44:26 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:44:26.007+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:44:26.621 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:44:26 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:44:26.508+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:05.622397+0000 front 2023-12-16T21:39:05.622556+0000 (oldest deadline 2023-12-16T21:39:30.322546+0000) 2023-12-16T21:44:26.621 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:44:26 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:44:26.508+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:05.622506+0000 front 2023-12-16T21:39:05.622528+0000 (oldest deadline 2023-12-16T21:39:30.322546+0000) 2023-12-16T21:44:26.621 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:44:26 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:44:26.508+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:05.622479+0000 front 2023-12-16T21:39:05.622580+0000 (oldest deadline 2023-12-16T21:39:30.322546+0000) 2023-12-16T21:44:26.621 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:44:26 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:44:26.509+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:44:26.621 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Dec 16 21:44:26 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-mon-smithi045[80759]: 2023-12-16T21:44:26.348+0000 7f45a1f8f700 -1 mon.smithi045@0(probing) e2 get_health_metrics reporting 1807 slow ops, oldest is monmgrreport(gid 14214, 0 checks, 1 progress events) 2023-12-16T21:44:26.879 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:44:26 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:44:26.634+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:09.828911+0000 front 2023-12-16T21:39:09.828958+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:44:26.879 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:44:26 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:44:26.634+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:09.828868+0000 front 2023-12-16T21:39:09.828878+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:44:26.880 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:44:26 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:44:26.634+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:09.828947+0000 front 2023-12-16T21:39:09.828993+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:44:26.880 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:44:26 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:44:26.635+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:44:27.317 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:44:26 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:44:26.958+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:09.306583+0000 front 2023-12-16T21:39:09.306633+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:44:27.317 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:44:26 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:44:26.958+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:09.306551+0000 front 2023-12-16T21:39:09.306652+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:44:27.317 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:44:26 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:44:26.958+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:09.306605+0000 front 2023-12-16T21:39:09.306613+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:44:27.317 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:44:26 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:44:26.959+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:44:27.816 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:44:27 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:44:27.499+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:05.622397+0000 front 2023-12-16T21:39:05.622556+0000 (oldest deadline 2023-12-16T21:39:30.322546+0000) 2023-12-16T21:44:27.816 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:44:27 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:44:27.499+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:05.622506+0000 front 2023-12-16T21:39:05.622528+0000 (oldest deadline 2023-12-16T21:39:30.322546+0000) 2023-12-16T21:44:27.817 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:44:27 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:44:27.499+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:05.622479+0000 front 2023-12-16T21:39:05.622580+0000 (oldest deadline 2023-12-16T21:39:30.322546+0000) 2023-12-16T21:44:27.817 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:44:27 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:44:27.500+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:44:27.817 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:44:27 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:44:27.598+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:09.828911+0000 front 2023-12-16T21:39:09.828958+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:44:27.817 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:44:27 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:44:27.598+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:09.828868+0000 front 2023-12-16T21:39:09.828878+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:44:27.817 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:44:27 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:44:27.598+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:09.828947+0000 front 2023-12-16T21:39:09.828993+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:44:27.817 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:44:27 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:44:27.599+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:44:28.316 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:44:28 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:44:28.008+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:09.306583+0000 front 2023-12-16T21:39:09.306633+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:44:28.317 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:44:28 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:44:28.008+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:09.306551+0000 front 2023-12-16T21:39:09.306652+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:44:28.317 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:44:28 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:44:28.008+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:09.306605+0000 front 2023-12-16T21:39:09.306613+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:44:28.317 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:44:28 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:44:28.009+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:44:28.816 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:44:28 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:44:28.498+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:05.622397+0000 front 2023-12-16T21:39:05.622556+0000 (oldest deadline 2023-12-16T21:39:30.322546+0000) 2023-12-16T21:44:28.817 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:44:28 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:44:28.498+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:05.622506+0000 front 2023-12-16T21:39:05.622528+0000 (oldest deadline 2023-12-16T21:39:30.322546+0000) 2023-12-16T21:44:28.817 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:44:28 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:44:28.498+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:05.622479+0000 front 2023-12-16T21:39:05.622580+0000 (oldest deadline 2023-12-16T21:39:30.322546+0000) 2023-12-16T21:44:28.817 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:44:28 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:44:28.500+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:44:28.817 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:44:28 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:44:28.601+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:09.828911+0000 front 2023-12-16T21:39:09.828958+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:44:28.817 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:44:28 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:44:28.601+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:09.828868+0000 front 2023-12-16T21:39:09.828878+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:44:28.817 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:44:28 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:44:28.601+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:09.828947+0000 front 2023-12-16T21:39:09.828993+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:44:28.817 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:44:28 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:44:28.601+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:44:29.316 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:44:28 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:44:28.982+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:09.306583+0000 front 2023-12-16T21:39:09.306633+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:44:29.316 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:44:28 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:44:28.982+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:09.306551+0000 front 2023-12-16T21:39:09.306652+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:44:29.317 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:44:28 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:44:28.982+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:09.306605+0000 front 2023-12-16T21:39:09.306613+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:44:29.317 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:44:28 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:44:28.983+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:44:29.816 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:44:29 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:44:29.513+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:05.622397+0000 front 2023-12-16T21:39:05.622556+0000 (oldest deadline 2023-12-16T21:39:30.322546+0000) 2023-12-16T21:44:29.817 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:44:29 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:44:29.513+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:05.622506+0000 front 2023-12-16T21:39:05.622528+0000 (oldest deadline 2023-12-16T21:39:30.322546+0000) 2023-12-16T21:44:29.817 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:44:29 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:44:29.513+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:05.622479+0000 front 2023-12-16T21:39:05.622580+0000 (oldest deadline 2023-12-16T21:39:30.322546+0000) 2023-12-16T21:44:29.817 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:44:29 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:44:29.514+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:44:29.817 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:44:29 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:44:29.597+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:09.828911+0000 front 2023-12-16T21:39:09.828958+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:44:29.817 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:44:29 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:44:29.597+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:09.828868+0000 front 2023-12-16T21:39:09.828878+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:44:29.818 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:44:29 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:44:29.597+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:09.828947+0000 front 2023-12-16T21:39:09.828993+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:44:29.818 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:44:29 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:44:29.598+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:44:30.317 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:44:30 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:44:30.016+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:09.306583+0000 front 2023-12-16T21:39:09.306633+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:44:30.317 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:44:30 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:44:30.016+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:09.306551+0000 front 2023-12-16T21:39:09.306652+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:44:30.317 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:44:30 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:44:30.016+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:09.306605+0000 front 2023-12-16T21:39:09.306613+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:44:30.317 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:44:30 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:44:30.017+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:44:30.816 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:44:30 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:44:30.543+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:05.622397+0000 front 2023-12-16T21:39:05.622556+0000 (oldest deadline 2023-12-16T21:39:30.322546+0000) 2023-12-16T21:44:30.817 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:44:30 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:44:30.543+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:05.622506+0000 front 2023-12-16T21:39:05.622528+0000 (oldest deadline 2023-12-16T21:39:30.322546+0000) 2023-12-16T21:44:30.817 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:44:30 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:44:30.543+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:05.622479+0000 front 2023-12-16T21:39:05.622580+0000 (oldest deadline 2023-12-16T21:39:30.322546+0000) 2023-12-16T21:44:30.817 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:44:30 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:44:30.544+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:44:30.817 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:44:30 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:44:30.559+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:09.828911+0000 front 2023-12-16T21:39:09.828958+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:44:30.817 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:44:30 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:44:30.559+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:09.828868+0000 front 2023-12-16T21:39:09.828878+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:44:30.817 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:44:30 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:44:30.559+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:09.828947+0000 front 2023-12-16T21:39:09.828993+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:44:30.817 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:44:30 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:44:30.561+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:44:31.316 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:44:31 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:44:31.018+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:09.306583+0000 front 2023-12-16T21:39:09.306633+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:44:31.317 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:44:31 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:44:31.018+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:09.306551+0000 front 2023-12-16T21:39:09.306652+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:44:31.317 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:44:31 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:44:31.018+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:09.306605+0000 front 2023-12-16T21:39:09.306613+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:44:31.317 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:44:31 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:44:31.019+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:44:31.641 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Dec 16 21:44:31 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-mon-smithi045[80759]: 2023-12-16T21:44:31.350+0000 7f45a1f8f700 -1 mon.smithi045@0(probing) e2 get_health_metrics reporting 1904 slow ops, oldest is monmgrreport(gid 14214, 0 checks, 1 progress events) 2023-12-16T21:44:31.642 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:44:31 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:44:31.592+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:09.828911+0000 front 2023-12-16T21:39:09.828958+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:44:31.642 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:44:31 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:44:31.592+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:09.828868+0000 front 2023-12-16T21:39:09.828878+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:44:31.642 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:44:31 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:44:31.592+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:09.828947+0000 front 2023-12-16T21:39:09.828993+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:44:31.642 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:44:31 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:44:31.593+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:44:31.643 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:44:31 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:44:31.558+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:05.622397+0000 front 2023-12-16T21:39:05.622556+0000 (oldest deadline 2023-12-16T21:39:30.322546+0000) 2023-12-16T21:44:31.643 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:44:31 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:44:31.558+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:05.622506+0000 front 2023-12-16T21:39:05.622528+0000 (oldest deadline 2023-12-16T21:39:30.322546+0000) 2023-12-16T21:44:31.643 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:44:31 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:44:31.558+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:05.622479+0000 front 2023-12-16T21:39:05.622580+0000 (oldest deadline 2023-12-16T21:39:30.322546+0000) 2023-12-16T21:44:31.643 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:44:31 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:44:31.559+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:44:32.316 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:44:32 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:44:32.012+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:09.306583+0000 front 2023-12-16T21:39:09.306633+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:44:32.316 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:44:32 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:44:32.012+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:09.306551+0000 front 2023-12-16T21:39:09.306652+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:44:32.316 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:44:32 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:44:32.012+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:09.306605+0000 front 2023-12-16T21:39:09.306613+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:44:32.316 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:44:32 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:44:32.012+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:44:32.816 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:44:32 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:44:32.633+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:09.828911+0000 front 2023-12-16T21:39:09.828958+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:44:32.816 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:44:32 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:44:32.633+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:09.828868+0000 front 2023-12-16T21:39:09.828878+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:44:32.816 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:44:32 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:44:32.633+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:09.828947+0000 front 2023-12-16T21:39:09.828993+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:44:32.817 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:44:32 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:44:32.634+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:44:32.817 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:44:32 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:44:32.518+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:05.622397+0000 front 2023-12-16T21:39:05.622556+0000 (oldest deadline 2023-12-16T21:39:30.322546+0000) 2023-12-16T21:44:32.817 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:44:32 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:44:32.518+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:05.622506+0000 front 2023-12-16T21:39:05.622528+0000 (oldest deadline 2023-12-16T21:39:30.322546+0000) 2023-12-16T21:44:32.817 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:44:32 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:44:32.518+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:05.622479+0000 front 2023-12-16T21:39:05.622580+0000 (oldest deadline 2023-12-16T21:39:30.322546+0000) 2023-12-16T21:44:32.817 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:44:32 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:44:32.519+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:44:33.316 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:44:33 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:44:33.020+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:09.306583+0000 front 2023-12-16T21:39:09.306633+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:44:33.317 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:44:33 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:44:33.020+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:09.306551+0000 front 2023-12-16T21:39:09.306652+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:44:33.317 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:44:33 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:44:33.020+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:09.306605+0000 front 2023-12-16T21:39:09.306613+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:44:33.317 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:44:33 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:44:33.021+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:44:33.816 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:44:33 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:44:33.513+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:05.622397+0000 front 2023-12-16T21:39:05.622556+0000 (oldest deadline 2023-12-16T21:39:30.322546+0000) 2023-12-16T21:44:33.817 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:44:33 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:44:33.513+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:05.622506+0000 front 2023-12-16T21:39:05.622528+0000 (oldest deadline 2023-12-16T21:39:30.322546+0000) 2023-12-16T21:44:33.817 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:44:33 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:44:33.513+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:05.622479+0000 front 2023-12-16T21:39:05.622580+0000 (oldest deadline 2023-12-16T21:39:30.322546+0000) 2023-12-16T21:44:33.817 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:44:33 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:44:33.514+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:44:33.817 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:44:33 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:44:33.651+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:09.828911+0000 front 2023-12-16T21:39:09.828958+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:44:33.817 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:44:33 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:44:33.651+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:09.828868+0000 front 2023-12-16T21:39:09.828878+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:44:33.817 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:44:33 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:44:33.652+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:09.828947+0000 front 2023-12-16T21:39:09.828993+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:44:33.817 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:44:33 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:44:33.652+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:44:34.316 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:44:34 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:44:34.017+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:09.306583+0000 front 2023-12-16T21:39:09.306633+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:44:34.316 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:44:34 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:44:34.017+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:09.306551+0000 front 2023-12-16T21:39:09.306652+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:44:34.317 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:44:34 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:44:34.017+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:09.306605+0000 front 2023-12-16T21:39:09.306613+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:44:34.317 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:44:34 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:44:34.019+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:44:34.816 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:44:34 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:44:34.542+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:05.622397+0000 front 2023-12-16T21:39:05.622556+0000 (oldest deadline 2023-12-16T21:39:30.322546+0000) 2023-12-16T21:44:34.817 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:44:34 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:44:34.542+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:05.622506+0000 front 2023-12-16T21:39:05.622528+0000 (oldest deadline 2023-12-16T21:39:30.322546+0000) 2023-12-16T21:44:34.817 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:44:34 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:44:34.542+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:05.622479+0000 front 2023-12-16T21:39:05.622580+0000 (oldest deadline 2023-12-16T21:39:30.322546+0000) 2023-12-16T21:44:34.817 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:44:34 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:44:34.543+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:44:34.817 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:44:34 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:44:34.617+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:09.828911+0000 front 2023-12-16T21:39:09.828958+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:44:34.817 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:44:34 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:44:34.617+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:09.828868+0000 front 2023-12-16T21:39:09.828878+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:44:34.817 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:44:34 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:44:34.617+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:09.828947+0000 front 2023-12-16T21:39:09.828993+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:44:34.817 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:44:34 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:44:34.618+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:44:35.316 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:44:35 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:44:35.019+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:09.306583+0000 front 2023-12-16T21:39:09.306633+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:44:35.316 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:44:35 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:44:35.019+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:09.306551+0000 front 2023-12-16T21:39:09.306652+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:44:35.316 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:44:35 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:44:35.019+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:09.306605+0000 front 2023-12-16T21:39:09.306613+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:44:35.317 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:44:35 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:44:35.020+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:44:35.816 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:44:35 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:44:35.562+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:05.622397+0000 front 2023-12-16T21:39:05.622556+0000 (oldest deadline 2023-12-16T21:39:30.322546+0000) 2023-12-16T21:44:35.816 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:44:35 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:44:35.562+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:05.622506+0000 front 2023-12-16T21:39:05.622528+0000 (oldest deadline 2023-12-16T21:39:30.322546+0000) 2023-12-16T21:44:35.817 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:44:35 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:44:35.562+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:05.622479+0000 front 2023-12-16T21:39:05.622580+0000 (oldest deadline 2023-12-16T21:39:30.322546+0000) 2023-12-16T21:44:35.817 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:44:35 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:44:35.563+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:44:35.817 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:44:35 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:44:35.593+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:09.828911+0000 front 2023-12-16T21:39:09.828958+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:44:35.817 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:44:35 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:44:35.593+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:09.828868+0000 front 2023-12-16T21:39:09.828878+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:44:35.817 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:44:35 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:44:35.593+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:09.828947+0000 front 2023-12-16T21:39:09.828993+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:44:35.817 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:44:35 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:44:35.594+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:44:36.316 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:44:36 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:44:36.027+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:09.306583+0000 front 2023-12-16T21:39:09.306633+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:44:36.317 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:44:36 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:44:36.027+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:09.306551+0000 front 2023-12-16T21:39:09.306652+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:44:36.317 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:44:36 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:44:36.027+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:09.306605+0000 front 2023-12-16T21:39:09.306613+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:44:36.317 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:44:36 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:44:36.028+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:44:36.621 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:44:36 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:44:36.595+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:05.622397+0000 front 2023-12-16T21:39:05.622556+0000 (oldest deadline 2023-12-16T21:39:30.322546+0000) 2023-12-16T21:44:36.622 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:44:36 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:44:36.595+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:05.622506+0000 front 2023-12-16T21:39:05.622528+0000 (oldest deadline 2023-12-16T21:39:30.322546+0000) 2023-12-16T21:44:36.622 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:44:36 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:44:36.595+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:05.622479+0000 front 2023-12-16T21:39:05.622580+0000 (oldest deadline 2023-12-16T21:39:30.322546+0000) 2023-12-16T21:44:36.622 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:44:36 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:44:36.597+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:44:36.622 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Dec 16 21:44:36 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-mon-smithi045[80759]: 2023-12-16T21:44:36.350+0000 7f45a1f8f700 -1 mon.smithi045@0(probing) e2 get_health_metrics reporting 1999 slow ops, oldest is monmgrreport(gid 14214, 0 checks, 1 progress events) 2023-12-16T21:44:37.045 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:44:36 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:44:36.618+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:09.828911+0000 front 2023-12-16T21:39:09.828958+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:44:37.045 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:44:36 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:44:36.618+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:09.828868+0000 front 2023-12-16T21:39:09.828878+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:44:37.045 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:44:36 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:44:36.618+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:09.828947+0000 front 2023-12-16T21:39:09.828993+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:44:37.045 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:44:36 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:44:36.619+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:44:37.316 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:44:37 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:44:37.042+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:09.306583+0000 front 2023-12-16T21:39:09.306633+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:44:37.316 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:44:37 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:44:37.042+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:09.306551+0000 front 2023-12-16T21:39:09.306652+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:44:37.316 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:44:37 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:44:37.042+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:09.306605+0000 front 2023-12-16T21:39:09.306613+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:44:37.317 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:44:37 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:44:37.043+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:44:37.660 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:44:37 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:44:37.590+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:05.622397+0000 front 2023-12-16T21:39:05.622556+0000 (oldest deadline 2023-12-16T21:39:30.322546+0000) 2023-12-16T21:44:37.660 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:44:37 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:44:37.590+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:05.622506+0000 front 2023-12-16T21:39:05.622528+0000 (oldest deadline 2023-12-16T21:39:30.322546+0000) 2023-12-16T21:44:37.660 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:44:37 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:44:37.590+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:05.622479+0000 front 2023-12-16T21:39:05.622580+0000 (oldest deadline 2023-12-16T21:39:30.322546+0000) 2023-12-16T21:44:37.660 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:44:37 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:44:37.591+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:44:38.066 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:44:37 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:44:37.657+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:09.828911+0000 front 2023-12-16T21:39:09.828958+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:44:38.066 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:44:37 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:44:37.657+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:09.828868+0000 front 2023-12-16T21:39:09.828878+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:44:38.066 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:44:37 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:44:37.657+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:09.828947+0000 front 2023-12-16T21:39:09.828993+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:44:38.066 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:44:37 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:44:37.658+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:44:38.565 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:44:38 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:44:38.064+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:09.306583+0000 front 2023-12-16T21:39:09.306633+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:44:38.565 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:44:38 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:44:38.064+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:09.306551+0000 front 2023-12-16T21:39:09.306652+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:44:38.565 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:44:38 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:44:38.064+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:09.306605+0000 front 2023-12-16T21:39:09.306613+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:44:38.565 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:44:38 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:44:38.065+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:44:38.816 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:44:38 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:44:38.562+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:05.622397+0000 front 2023-12-16T21:39:05.622556+0000 (oldest deadline 2023-12-16T21:39:30.322546+0000) 2023-12-16T21:44:38.816 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:44:38 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:44:38.562+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:05.622506+0000 front 2023-12-16T21:39:05.622528+0000 (oldest deadline 2023-12-16T21:39:30.322546+0000) 2023-12-16T21:44:38.817 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:44:38 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:44:38.562+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:05.622479+0000 front 2023-12-16T21:39:05.622580+0000 (oldest deadline 2023-12-16T21:39:30.322546+0000) 2023-12-16T21:44:38.817 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:44:38 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:44:38.563+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:44:38.817 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:44:38 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:44:38.631+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:09.828911+0000 front 2023-12-16T21:39:09.828958+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:44:38.817 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:44:38 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:44:38.632+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:09.828868+0000 front 2023-12-16T21:39:09.828878+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:44:38.817 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:44:38 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:44:38.632+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:09.828947+0000 front 2023-12-16T21:39:09.828993+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:44:38.817 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:44:38 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:44:38.633+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:44:39.316 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:44:39 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:44:39.035+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:09.306583+0000 front 2023-12-16T21:39:09.306633+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:44:39.317 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:44:39 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:44:39.035+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:09.306551+0000 front 2023-12-16T21:39:09.306652+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:44:39.317 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:44:39 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:44:39.035+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:09.306605+0000 front 2023-12-16T21:39:09.306613+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:44:39.317 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:44:39 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:44:39.036+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:44:39.995 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:44:39 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:44:39.608+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:09.828911+0000 front 2023-12-16T21:39:09.828958+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:44:39.995 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:44:39 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:44:39.608+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:09.828868+0000 front 2023-12-16T21:39:09.828878+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:44:39.995 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:44:39 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:44:39.608+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:09.828947+0000 front 2023-12-16T21:39:09.828993+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:44:39.996 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:44:39 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:44:39.609+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:44:39.996 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:44:39 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:44:39.603+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:05.622397+0000 front 2023-12-16T21:39:05.622556+0000 (oldest deadline 2023-12-16T21:39:30.322546+0000) 2023-12-16T21:44:39.996 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:44:39 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:44:39.603+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:05.622506+0000 front 2023-12-16T21:39:05.622528+0000 (oldest deadline 2023-12-16T21:39:30.322546+0000) 2023-12-16T21:44:39.996 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:44:39 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:44:39.603+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:05.622479+0000 front 2023-12-16T21:39:05.622580+0000 (oldest deadline 2023-12-16T21:39:30.322546+0000) 2023-12-16T21:44:39.996 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:44:39 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:44:39.604+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:44:40.317 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:44:39 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:44:39.992+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:09.306583+0000 front 2023-12-16T21:39:09.306633+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:44:40.317 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:44:39 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:44:39.992+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:09.306551+0000 front 2023-12-16T21:39:09.306652+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:44:40.317 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:44:39 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:44:39.992+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:09.306605+0000 front 2023-12-16T21:39:09.306613+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:44:40.317 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:44:39 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:44:39.993+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:44:40.948 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:44:40 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:44:40.637+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:05.622397+0000 front 2023-12-16T21:39:05.622556+0000 (oldest deadline 2023-12-16T21:39:30.322546+0000) 2023-12-16T21:44:40.948 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:44:40 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:44:40.637+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:05.622506+0000 front 2023-12-16T21:39:05.622528+0000 (oldest deadline 2023-12-16T21:39:30.322546+0000) 2023-12-16T21:44:40.948 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:44:40 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:44:40.637+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:05.622479+0000 front 2023-12-16T21:39:05.622580+0000 (oldest deadline 2023-12-16T21:39:30.322546+0000) 2023-12-16T21:44:40.948 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:44:40 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:44:40.637+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:44:40.948 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:44:40 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:44:40.615+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:09.828911+0000 front 2023-12-16T21:39:09.828958+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:44:40.948 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:44:40 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:44:40.615+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:09.828868+0000 front 2023-12-16T21:39:09.828878+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:44:40.948 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:44:40 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:44:40.615+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:09.828947+0000 front 2023-12-16T21:39:09.828993+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:44:40.948 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:44:40 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:44:40.617+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:44:41.316 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:44:40 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:44:40.945+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:09.306583+0000 front 2023-12-16T21:39:09.306633+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:44:41.316 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:44:40 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:44:40.945+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:09.306551+0000 front 2023-12-16T21:39:09.306652+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:44:41.316 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:44:40 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:44:40.945+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:09.306605+0000 front 2023-12-16T21:39:09.306613+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:44:41.316 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:44:40 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:44:40.946+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:44:41.608 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Dec 16 21:44:41 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-mon-smithi045[80759]: 2023-12-16T21:44:41.351+0000 7f45a1f8f700 -1 mon.smithi045@0(probing) e2 get_health_metrics reporting 2098 slow ops, oldest is monmgrreport(gid 14214, 0 checks, 1 progress events) 2023-12-16T21:44:41.957 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:44:41 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:44:41.625+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:09.828911+0000 front 2023-12-16T21:39:09.828958+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:44:41.957 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:44:41 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:44:41.625+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:09.828868+0000 front 2023-12-16T21:39:09.828878+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:44:41.957 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:44:41 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:44:41.625+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:09.828947+0000 front 2023-12-16T21:39:09.828993+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:44:41.957 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:44:41 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:44:41.627+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:44:41.957 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:44:41 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:44:41.605+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:05.622397+0000 front 2023-12-16T21:39:05.622556+0000 (oldest deadline 2023-12-16T21:39:30.322546+0000) 2023-12-16T21:44:41.957 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:44:41 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:44:41.605+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:05.622506+0000 front 2023-12-16T21:39:05.622528+0000 (oldest deadline 2023-12-16T21:39:30.322546+0000) 2023-12-16T21:44:41.957 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:44:41 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:44:41.605+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:05.622479+0000 front 2023-12-16T21:39:05.622580+0000 (oldest deadline 2023-12-16T21:39:30.322546+0000) 2023-12-16T21:44:41.957 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:44:41 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:44:41.606+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:44:42.316 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:44:41 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:44:41.953+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:09.306583+0000 front 2023-12-16T21:39:09.306633+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:44:42.316 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:44:41 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:44:41.953+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:09.306551+0000 front 2023-12-16T21:39:09.306652+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:44:42.316 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:44:41 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:44:41.953+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:09.306605+0000 front 2023-12-16T21:39:09.306613+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:44:42.316 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:44:41 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:44:41.955+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:44:42.986 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:44:42 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:44:42.599+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:09.828911+0000 front 2023-12-16T21:39:09.828958+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:44:42.986 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:44:42 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:44:42.599+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:09.828868+0000 front 2023-12-16T21:39:09.828878+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:44:42.986 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:44:42 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:44:42.599+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:09.828947+0000 front 2023-12-16T21:39:09.828993+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:44:42.986 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:44:42 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:44:42.600+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:44:42.986 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:44:42 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:44:42.626+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:05.622397+0000 front 2023-12-16T21:39:05.622556+0000 (oldest deadline 2023-12-16T21:39:30.322546+0000) 2023-12-16T21:44:42.986 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:44:42 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:44:42.626+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:05.622506+0000 front 2023-12-16T21:39:05.622528+0000 (oldest deadline 2023-12-16T21:39:30.322546+0000) 2023-12-16T21:44:42.986 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:44:42 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:44:42.626+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:05.622479+0000 front 2023-12-16T21:39:05.622580+0000 (oldest deadline 2023-12-16T21:39:30.322546+0000) 2023-12-16T21:44:42.986 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:44:42 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:44:42.627+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:44:43.316 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:44:42 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:44:42.983+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:09.306583+0000 front 2023-12-16T21:39:09.306633+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:44:43.316 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:44:42 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:44:42.983+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:09.306551+0000 front 2023-12-16T21:39:09.306652+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:44:43.316 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:44:42 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:44:42.983+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:09.306605+0000 front 2023-12-16T21:39:09.306613+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:44:43.316 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:44:42 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:44:42.983+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:44:43.816 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:44:43 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:44:43.594+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:05.622397+0000 front 2023-12-16T21:39:05.622556+0000 (oldest deadline 2023-12-16T21:39:30.322546+0000) 2023-12-16T21:44:43.816 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:44:43 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:44:43.594+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:05.622506+0000 front 2023-12-16T21:39:05.622528+0000 (oldest deadline 2023-12-16T21:39:30.322546+0000) 2023-12-16T21:44:43.817 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:44:43 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:44:43.594+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:05.622479+0000 front 2023-12-16T21:39:05.622580+0000 (oldest deadline 2023-12-16T21:39:30.322546+0000) 2023-12-16T21:44:43.817 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:44:43 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:44:43.595+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:44:43.817 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:44:43 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:44:43.637+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:09.828911+0000 front 2023-12-16T21:39:09.828958+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:44:43.817 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:44:43 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:44:43.637+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:09.828868+0000 front 2023-12-16T21:39:09.828878+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:44:43.817 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:44:43 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:44:43.637+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:09.828947+0000 front 2023-12-16T21:39:09.828993+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:44:43.817 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:44:43 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:44:43.637+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:44:44.316 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:44:43 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:44:43.976+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:09.306583+0000 front 2023-12-16T21:39:09.306633+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:44:44.317 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:44:43 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:44:43.976+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:09.306551+0000 front 2023-12-16T21:39:09.306652+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:44:44.317 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:44:43 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:44:43.976+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:09.306605+0000 front 2023-12-16T21:39:09.306613+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:44:44.317 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:44:43 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:44:43.977+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:44:44.995 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:44:44 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:44:44.564+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:05.622397+0000 front 2023-12-16T21:39:05.622556+0000 (oldest deadline 2023-12-16T21:39:30.322546+0000) 2023-12-16T21:44:44.995 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:44:44 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:44:44.564+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:05.622506+0000 front 2023-12-16T21:39:05.622528+0000 (oldest deadline 2023-12-16T21:39:30.322546+0000) 2023-12-16T21:44:44.995 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:44:44 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:44:44.564+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:05.622479+0000 front 2023-12-16T21:39:05.622580+0000 (oldest deadline 2023-12-16T21:39:30.322546+0000) 2023-12-16T21:44:44.995 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:44:44 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:44:44.565+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:44:44.995 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:44:44 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:44:44.647+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:09.828911+0000 front 2023-12-16T21:39:09.828958+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:44:44.995 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:44:44 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:44:44.647+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:09.828868+0000 front 2023-12-16T21:39:09.828878+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:44:44.996 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:44:44 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:44:44.647+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:09.828947+0000 front 2023-12-16T21:39:09.828993+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:44:44.996 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:44:44 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:44:44.648+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:44:45.316 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:44:44 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:44:44.991+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:09.306583+0000 front 2023-12-16T21:39:09.306633+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:44:45.317 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:44:44 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:44:44.991+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:09.306551+0000 front 2023-12-16T21:39:09.306652+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:44:45.317 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:44:44 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:44:44.991+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:09.306605+0000 front 2023-12-16T21:39:09.306613+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:44:45.317 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:44:44 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:44:44.993+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:44:45.984 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:44:45 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:44:45.596+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:05.622397+0000 front 2023-12-16T21:39:05.622556+0000 (oldest deadline 2023-12-16T21:39:30.322546+0000) 2023-12-16T21:44:45.984 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:44:45 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:44:45.596+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:05.622506+0000 front 2023-12-16T21:39:05.622528+0000 (oldest deadline 2023-12-16T21:39:30.322546+0000) 2023-12-16T21:44:45.984 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:44:45 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:44:45.596+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:05.622479+0000 front 2023-12-16T21:39:05.622580+0000 (oldest deadline 2023-12-16T21:39:30.322546+0000) 2023-12-16T21:44:45.984 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:44:45 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:44:45.597+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:44:45.984 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:44:45 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:44:45.646+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:09.828911+0000 front 2023-12-16T21:39:09.828958+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:44:45.985 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:44:45 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:44:45.646+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:09.828868+0000 front 2023-12-16T21:39:09.828878+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:44:45.985 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:44:45 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:44:45.646+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:09.828947+0000 front 2023-12-16T21:39:09.828993+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:44:45.985 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:44:45 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:44:45.647+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:44:46.317 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:44:45 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:44:45.981+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:09.306583+0000 front 2023-12-16T21:39:09.306633+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:44:46.317 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:44:45 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:44:45.981+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:09.306551+0000 front 2023-12-16T21:39:09.306652+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:44:46.317 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:44:45 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:44:45.981+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:09.306605+0000 front 2023-12-16T21:39:09.306613+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:44:46.317 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:44:45 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:44:45.982+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:44:46.633 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Dec 16 21:44:46 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-mon-smithi045[80759]: 2023-12-16T21:44:46.351+0000 7f45a1f8f700 -1 mon.smithi045@0(probing) e2 get_health_metrics reporting 2192 slow ops, oldest is monmgrreport(gid 14214, 0 checks, 1 progress events) 2023-12-16T21:44:46.992 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:44:46 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:44:46.629+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:05.622397+0000 front 2023-12-16T21:39:05.622556+0000 (oldest deadline 2023-12-16T21:39:30.322546+0000) 2023-12-16T21:44:46.992 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:44:46 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:44:46.629+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:05.622506+0000 front 2023-12-16T21:39:05.622528+0000 (oldest deadline 2023-12-16T21:39:30.322546+0000) 2023-12-16T21:44:46.992 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:44:46 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:44:46.629+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:05.622479+0000 front 2023-12-16T21:39:05.622580+0000 (oldest deadline 2023-12-16T21:39:30.322546+0000) 2023-12-16T21:44:46.993 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:44:46 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:44:46.631+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:44:46.993 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:44:46 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:44:46.635+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:09.828911+0000 front 2023-12-16T21:39:09.828958+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:44:46.993 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:44:46 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:44:46.635+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:09.828868+0000 front 2023-12-16T21:39:09.828878+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:44:46.993 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:44:46 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:44:46.635+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:09.828947+0000 front 2023-12-16T21:39:09.828993+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:44:46.993 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:44:46 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:44:46.637+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:44:47.316 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:44:46 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:44:46.989+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:09.306583+0000 front 2023-12-16T21:39:09.306633+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:44:47.316 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:44:46 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:44:46.989+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:09.306551+0000 front 2023-12-16T21:39:09.306652+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:44:47.316 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:44:46 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:44:46.989+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:09.306605+0000 front 2023-12-16T21:39:09.306613+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:44:47.317 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:44:46 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:44:46.990+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:44:48.009 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:44:47 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:44:47.628+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:05.622397+0000 front 2023-12-16T21:39:05.622556+0000 (oldest deadline 2023-12-16T21:39:30.322546+0000) 2023-12-16T21:44:48.009 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:44:47 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:44:47.628+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:05.622506+0000 front 2023-12-16T21:39:05.622528+0000 (oldest deadline 2023-12-16T21:39:30.322546+0000) 2023-12-16T21:44:48.009 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:44:47 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:44:47.628+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:05.622479+0000 front 2023-12-16T21:39:05.622580+0000 (oldest deadline 2023-12-16T21:39:30.322546+0000) 2023-12-16T21:44:48.010 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:44:47 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:44:47.629+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:44:48.010 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:44:47 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:44:47.608+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:09.828911+0000 front 2023-12-16T21:39:09.828958+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:44:48.010 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:44:47 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:44:47.608+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:09.828868+0000 front 2023-12-16T21:39:09.828878+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:44:48.010 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:44:47 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:44:47.608+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:09.828947+0000 front 2023-12-16T21:39:09.828993+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:44:48.010 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:44:47 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:44:47.609+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:44:48.316 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:44:48 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:44:48.006+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:09.306583+0000 front 2023-12-16T21:39:09.306633+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:44:48.316 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:44:48 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:44:48.006+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:09.306551+0000 front 2023-12-16T21:39:09.306652+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:44:48.316 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:44:48 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:44:48.006+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:09.306605+0000 front 2023-12-16T21:39:09.306613+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:44:48.316 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:44:48 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:44:48.007+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:44:48.816 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:44:48 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:44:48.647+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:05.622397+0000 front 2023-12-16T21:39:05.622556+0000 (oldest deadline 2023-12-16T21:39:30.322546+0000) 2023-12-16T21:44:48.816 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:44:48 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:44:48.647+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:05.622506+0000 front 2023-12-16T21:39:05.622528+0000 (oldest deadline 2023-12-16T21:39:30.322546+0000) 2023-12-16T21:44:48.816 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:44:48 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:44:48.647+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:05.622479+0000 front 2023-12-16T21:39:05.622580+0000 (oldest deadline 2023-12-16T21:39:30.322546+0000) 2023-12-16T21:44:48.817 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:44:48 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:44:48.648+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:44:48.817 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:44:48 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:44:48.562+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:09.828911+0000 front 2023-12-16T21:39:09.828958+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:44:48.817 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:44:48 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:44:48.562+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:09.828868+0000 front 2023-12-16T21:39:09.828878+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:44:48.817 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:44:48 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:44:48.562+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:09.828947+0000 front 2023-12-16T21:39:09.828993+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:44:48.817 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:44:48 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:44:48.563+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:44:49.316 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:44:48 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:44:48.961+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:09.306583+0000 front 2023-12-16T21:39:09.306633+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:44:49.317 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:44:48 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:44:48.961+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:09.306551+0000 front 2023-12-16T21:39:09.306652+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:44:49.317 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:44:48 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:44:48.961+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:09.306605+0000 front 2023-12-16T21:39:09.306613+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:44:49.317 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:44:48 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:44:48.962+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:44:49.615 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:44:49 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:44:49.532+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:09.828911+0000 front 2023-12-16T21:39:09.828958+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:44:49.616 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:44:49 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:44:49.532+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:09.828868+0000 front 2023-12-16T21:39:09.828878+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:44:49.616 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:44:49 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:44:49.532+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:09.828947+0000 front 2023-12-16T21:39:09.828993+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:44:49.616 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:44:49 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:44:49.533+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:44:49.971 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:44:49 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:44:49.613+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:05.622397+0000 front 2023-12-16T21:39:05.622556+0000 (oldest deadline 2023-12-16T21:39:30.322546+0000) 2023-12-16T21:44:49.972 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:44:49 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:44:49.613+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:05.622506+0000 front 2023-12-16T21:39:05.622528+0000 (oldest deadline 2023-12-16T21:39:30.322546+0000) 2023-12-16T21:44:49.972 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:44:49 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:44:49.613+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:05.622479+0000 front 2023-12-16T21:39:05.622580+0000 (oldest deadline 2023-12-16T21:39:30.322546+0000) 2023-12-16T21:44:49.972 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:44:49 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:44:49.613+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:44:50.316 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:44:49 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:44:49.968+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:09.306583+0000 front 2023-12-16T21:39:09.306633+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:44:50.317 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:44:49 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:44:49.968+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:09.306551+0000 front 2023-12-16T21:39:09.306652+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:44:50.317 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:44:49 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:44:49.968+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:09.306605+0000 front 2023-12-16T21:39:09.306613+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:44:50.317 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:44:49 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:44:49.969+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:44:50.816 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:44:50 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:44:50.632+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:05.622397+0000 front 2023-12-16T21:39:05.622556+0000 (oldest deadline 2023-12-16T21:39:30.322546+0000) 2023-12-16T21:44:50.817 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:44:50 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:44:50.632+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:05.622506+0000 front 2023-12-16T21:39:05.622528+0000 (oldest deadline 2023-12-16T21:39:30.322546+0000) 2023-12-16T21:44:50.817 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:44:50 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:44:50.632+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:05.622479+0000 front 2023-12-16T21:39:05.622580+0000 (oldest deadline 2023-12-16T21:39:30.322546+0000) 2023-12-16T21:44:50.817 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:44:50 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:44:50.633+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:44:50.817 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:44:50 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:44:50.513+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:09.828911+0000 front 2023-12-16T21:39:09.828958+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:44:50.817 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:44:50 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:44:50.513+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:09.828868+0000 front 2023-12-16T21:39:09.828878+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:44:50.817 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:44:50 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:44:50.513+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:09.828947+0000 front 2023-12-16T21:39:09.828993+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:44:50.817 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:44:50 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:44:50.514+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:44:51.316 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:44:50 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:44:50.938+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:09.306583+0000 front 2023-12-16T21:39:09.306633+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:44:51.316 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:44:50 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:44:50.938+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:09.306551+0000 front 2023-12-16T21:39:09.306652+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:44:51.317 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:44:50 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:44:50.938+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:09.306605+0000 front 2023-12-16T21:39:09.306613+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:44:51.317 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:44:50 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:44:50.939+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:44:51.629 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Dec 16 21:44:51 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-mon-smithi045[80759]: 2023-12-16T21:44:51.352+0000 7f45a1f8f700 -1 mon.smithi045@0(probing) e2 get_health_metrics reporting 2287 slow ops, oldest is monmgrreport(gid 14214, 0 checks, 1 progress events) 2023-12-16T21:44:51.629 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:44:51 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:44:51.502+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:09.828911+0000 front 2023-12-16T21:39:09.828958+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:44:51.629 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:44:51 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:44:51.502+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:09.828868+0000 front 2023-12-16T21:39:09.828878+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:44:51.629 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:44:51 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:44:51.502+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:09.828947+0000 front 2023-12-16T21:39:09.828993+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:44:51.629 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:44:51 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:44:51.503+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:44:51.940 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:44:51 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:44:51.625+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:05.622397+0000 front 2023-12-16T21:39:05.622556+0000 (oldest deadline 2023-12-16T21:39:30.322546+0000) 2023-12-16T21:44:51.940 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:44:51 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:44:51.625+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:05.622506+0000 front 2023-12-16T21:39:05.622528+0000 (oldest deadline 2023-12-16T21:39:30.322546+0000) 2023-12-16T21:44:51.941 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:44:51 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:44:51.625+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:05.622479+0000 front 2023-12-16T21:39:05.622580+0000 (oldest deadline 2023-12-16T21:39:30.322546+0000) 2023-12-16T21:44:51.941 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:44:51 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:44:51.626+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:44:52.316 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:44:51 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:44:51.937+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:09.306583+0000 front 2023-12-16T21:39:09.306633+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:44:52.317 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:44:51 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:44:51.937+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:09.306551+0000 front 2023-12-16T21:39:09.306652+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:44:52.317 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:44:51 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:44:51.937+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:09.306605+0000 front 2023-12-16T21:39:09.306613+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:44:52.317 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:44:51 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:44:51.938+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:44:52.816 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:44:52 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:44:52.662+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:05.622397+0000 front 2023-12-16T21:39:05.622556+0000 (oldest deadline 2023-12-16T21:39:30.322546+0000) 2023-12-16T21:44:52.816 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:44:52 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:44:52.662+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:05.622506+0000 front 2023-12-16T21:39:05.622528+0000 (oldest deadline 2023-12-16T21:39:30.322546+0000) 2023-12-16T21:44:52.816 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:44:52 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:44:52.662+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:05.622479+0000 front 2023-12-16T21:39:05.622580+0000 (oldest deadline 2023-12-16T21:39:30.322546+0000) 2023-12-16T21:44:52.817 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:44:52 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:44:52.663+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:44:52.817 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:44:52 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:44:52.483+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:09.828911+0000 front 2023-12-16T21:39:09.828958+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:44:52.817 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:44:52 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:44:52.483+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:09.828868+0000 front 2023-12-16T21:39:09.828878+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:44:52.817 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:44:52 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:44:52.483+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:09.828947+0000 front 2023-12-16T21:39:09.828993+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:44:52.817 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:44:52 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:44:52.483+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:44:53.265 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:44:52 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:44:52.958+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:09.306583+0000 front 2023-12-16T21:39:09.306633+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:44:53.265 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:44:52 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:44:52.958+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:09.306551+0000 front 2023-12-16T21:39:09.306652+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:44:53.265 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:44:52 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:44:52.958+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:09.306605+0000 front 2023-12-16T21:39:09.306613+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:44:53.266 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:44:52 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:44:52.959+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:44:53.535 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:44:53 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:44:53.440+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:09.828911+0000 front 2023-12-16T21:39:09.828958+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:44:53.535 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:44:53 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:44:53.440+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:09.828868+0000 front 2023-12-16T21:39:09.828878+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:44:53.535 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:44:53 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:44:53.440+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:09.828947+0000 front 2023-12-16T21:39:09.828993+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:44:53.536 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:44:53 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:44:53.441+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:44:53.816 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:44:53 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:44:53.709+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:05.622397+0000 front 2023-12-16T21:39:05.622556+0000 (oldest deadline 2023-12-16T21:39:30.322546+0000) 2023-12-16T21:44:53.816 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:44:53 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:44:53.709+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:05.622506+0000 front 2023-12-16T21:39:05.622528+0000 (oldest deadline 2023-12-16T21:39:30.322546+0000) 2023-12-16T21:44:53.816 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:44:53 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:44:53.709+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:05.622479+0000 front 2023-12-16T21:39:05.622580+0000 (oldest deadline 2023-12-16T21:39:30.322546+0000) 2023-12-16T21:44:53.817 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:44:53 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:44:53.710+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:44:54.316 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:44:53 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:44:53.986+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:09.306583+0000 front 2023-12-16T21:39:09.306633+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:44:54.317 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:44:53 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:44:53.986+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:09.306551+0000 front 2023-12-16T21:39:09.306652+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:44:54.317 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:44:53 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:44:53.986+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:09.306605+0000 front 2023-12-16T21:39:09.306613+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:44:54.317 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:44:53 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:44:53.987+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:44:54.708 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:44:54 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:44:54.404+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:09.828911+0000 front 2023-12-16T21:39:09.828958+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:44:54.709 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:44:54 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:44:54.404+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:09.828868+0000 front 2023-12-16T21:39:09.828878+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:44:54.709 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:44:54 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:44:54.404+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:09.828947+0000 front 2023-12-16T21:39:09.828993+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:44:54.709 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:44:54 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:44:54.405+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:44:55.035 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:44:54 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:44:54.705+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:05.622397+0000 front 2023-12-16T21:39:05.622556+0000 (oldest deadline 2023-12-16T21:39:30.322546+0000) 2023-12-16T21:44:55.035 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:44:54 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:44:54.705+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:05.622506+0000 front 2023-12-16T21:39:05.622528+0000 (oldest deadline 2023-12-16T21:39:30.322546+0000) 2023-12-16T21:44:55.035 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:44:54 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:44:54.705+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:05.622479+0000 front 2023-12-16T21:39:05.622580+0000 (oldest deadline 2023-12-16T21:39:30.322546+0000) 2023-12-16T21:44:55.036 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:44:54 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:44:54.706+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:44:55.316 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:44:55 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:44:55.032+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:09.306583+0000 front 2023-12-16T21:39:09.306633+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:44:55.317 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:44:55 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:44:55.032+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:09.306551+0000 front 2023-12-16T21:39:09.306652+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:44:55.317 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:44:55 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:44:55.032+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:09.306605+0000 front 2023-12-16T21:39:09.306613+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:44:55.317 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:44:55 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:44:55.033+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:44:55.719 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:44:55 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:44:55.445+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:09.828911+0000 front 2023-12-16T21:39:09.828958+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:44:55.719 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:44:55 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:44:55.445+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:09.828868+0000 front 2023-12-16T21:39:09.828878+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:44:55.719 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:44:55 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:44:55.445+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:09.828947+0000 front 2023-12-16T21:39:09.828993+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:44:55.719 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:44:55 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:44:55.446+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:44:56.018 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:44:55 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:44:55.716+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:05.622397+0000 front 2023-12-16T21:39:05.622556+0000 (oldest deadline 2023-12-16T21:39:30.322546+0000) 2023-12-16T21:44:56.018 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:44:55 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:44:55.716+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:05.622506+0000 front 2023-12-16T21:39:05.622528+0000 (oldest deadline 2023-12-16T21:39:30.322546+0000) 2023-12-16T21:44:56.019 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:44:55 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:44:55.716+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:05.622479+0000 front 2023-12-16T21:39:05.622580+0000 (oldest deadline 2023-12-16T21:39:30.322546+0000) 2023-12-16T21:44:56.019 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:44:55 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:44:55.717+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:44:56.316 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:44:56 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:44:56.015+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:09.306583+0000 front 2023-12-16T21:39:09.306633+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:44:56.317 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:44:56 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:44:56.015+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:09.306551+0000 front 2023-12-16T21:39:09.306652+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:44:56.317 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:44:56 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:44:56.015+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:09.306605+0000 front 2023-12-16T21:39:09.306613+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:44:56.317 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:44:56 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:44:56.016+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:44:56.749 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Dec 16 21:44:56 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-mon-smithi045[80759]: 2023-12-16T21:44:56.353+0000 7f45a1f8f700 -1 mon.smithi045@0(probing) e2 get_health_metrics reporting 2382 slow ops, oldest is monmgrreport(gid 14214, 0 checks, 1 progress events) 2023-12-16T21:44:56.750 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:44:56 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:44:56.496+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:09.828911+0000 front 2023-12-16T21:39:09.828958+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:44:56.750 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:44:56 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:44:56.496+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:09.828868+0000 front 2023-12-16T21:39:09.828878+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:44:56.750 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:44:56 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:44:56.496+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:09.828947+0000 front 2023-12-16T21:39:09.828993+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:44:56.750 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:44:56 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:44:56.497+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:44:57.066 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:44:56 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:44:56.973+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:09.306583+0000 front 2023-12-16T21:39:09.306633+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:44:57.067 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:44:56 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:44:56.973+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:09.306551+0000 front 2023-12-16T21:39:09.306652+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:44:57.067 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:44:56 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:44:56.973+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:09.306605+0000 front 2023-12-16T21:39:09.306613+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:44:57.067 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:44:56 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:44:56.973+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:44:57.067 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:44:56 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:44:56.746+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:05.622397+0000 front 2023-12-16T21:39:05.622556+0000 (oldest deadline 2023-12-16T21:39:30.322546+0000) 2023-12-16T21:44:57.067 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:44:56 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:44:56.746+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:05.622506+0000 front 2023-12-16T21:39:05.622528+0000 (oldest deadline 2023-12-16T21:39:30.322546+0000) 2023-12-16T21:44:57.067 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:44:56 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:44:56.746+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:05.622479+0000 front 2023-12-16T21:39:05.622580+0000 (oldest deadline 2023-12-16T21:39:30.322546+0000) 2023-12-16T21:44:57.067 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:44:56 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:44:56.747+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:44:57.767 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:44:57 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:44:57.460+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:09.828911+0000 front 2023-12-16T21:39:09.828958+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:44:57.767 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:44:57 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:44:57.460+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:09.828868+0000 front 2023-12-16T21:39:09.828878+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:44:57.767 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:44:57 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:44:57.460+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:09.828947+0000 front 2023-12-16T21:39:09.828993+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:44:57.767 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:44:57 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:44:57.461+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:44:58.066 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:44:57 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:44:57.970+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:09.306583+0000 front 2023-12-16T21:39:09.306633+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:44:58.066 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:44:57 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:44:57.970+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:09.306551+0000 front 2023-12-16T21:39:09.306652+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:44:58.066 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:44:57 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:44:57.970+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:09.306605+0000 front 2023-12-16T21:39:09.306613+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:44:58.067 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:44:57 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:44:57.971+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:44:58.067 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:44:57 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:44:57.764+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:05.622397+0000 front 2023-12-16T21:39:05.622556+0000 (oldest deadline 2023-12-16T21:39:30.322546+0000) 2023-12-16T21:44:58.067 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:44:57 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:44:57.764+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:05.622506+0000 front 2023-12-16T21:39:05.622528+0000 (oldest deadline 2023-12-16T21:39:30.322546+0000) 2023-12-16T21:44:58.067 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:44:57 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:44:57.764+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:05.622479+0000 front 2023-12-16T21:39:05.622580+0000 (oldest deadline 2023-12-16T21:39:30.322546+0000) 2023-12-16T21:44:58.067 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:44:57 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:44:57.765+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:44:58.781 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:44:58 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:44:58.479+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:09.828911+0000 front 2023-12-16T21:39:09.828958+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:44:58.782 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:44:58 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:44:58.479+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:09.828868+0000 front 2023-12-16T21:39:09.828878+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:44:58.782 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:44:58 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:44:58.479+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:09.828947+0000 front 2023-12-16T21:39:09.828993+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:44:58.782 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:44:58 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:44:58.481+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:44:59.067 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:44:58 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:44:58.955+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:09.306583+0000 front 2023-12-16T21:39:09.306633+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:44:59.067 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:44:58 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:44:58.955+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:09.306551+0000 front 2023-12-16T21:39:09.306652+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:44:59.067 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:44:58 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:44:58.955+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:09.306605+0000 front 2023-12-16T21:39:09.306613+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:44:59.067 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:44:58 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:44:58.955+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:44:59.067 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:44:58 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:44:58.778+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:05.622397+0000 front 2023-12-16T21:39:05.622556+0000 (oldest deadline 2023-12-16T21:39:30.322546+0000) 2023-12-16T21:44:59.067 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:44:58 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:44:58.778+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:05.622506+0000 front 2023-12-16T21:39:05.622528+0000 (oldest deadline 2023-12-16T21:39:30.322546+0000) 2023-12-16T21:44:59.068 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:44:58 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:44:58.778+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:05.622479+0000 front 2023-12-16T21:39:05.622580+0000 (oldest deadline 2023-12-16T21:39:30.322546+0000) 2023-12-16T21:44:59.068 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:44:58 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:44:58.779+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:44:59.791 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:44:59 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:44:59.442+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:09.828911+0000 front 2023-12-16T21:39:09.828958+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:44:59.792 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:44:59 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:44:59.442+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:09.828868+0000 front 2023-12-16T21:39:09.828878+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:44:59.792 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:44:59 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:44:59.442+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:09.828947+0000 front 2023-12-16T21:39:09.828993+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:44:59.792 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:44:59 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:44:59.443+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:45:00.066 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:45:00 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:44:59.999+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:09.306583+0000 front 2023-12-16T21:39:09.306633+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:45:00.066 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:45:00 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:44:59.999+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:09.306551+0000 front 2023-12-16T21:39:09.306652+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:45:00.067 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:45:00 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:45:00.000+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:09.306605+0000 front 2023-12-16T21:39:09.306613+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:45:00.067 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:45:00 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:45:00.001+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:45:00.067 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:44:59 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:44:59.788+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:05.622397+0000 front 2023-12-16T21:39:05.622556+0000 (oldest deadline 2023-12-16T21:39:30.322546+0000) 2023-12-16T21:45:00.067 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:44:59 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:44:59.788+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:05.622506+0000 front 2023-12-16T21:39:05.622528+0000 (oldest deadline 2023-12-16T21:39:30.322546+0000) 2023-12-16T21:45:00.067 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:44:59 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:44:59.788+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:05.622479+0000 front 2023-12-16T21:39:05.622580+0000 (oldest deadline 2023-12-16T21:39:30.322546+0000) 2023-12-16T21:45:00.067 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:44:59 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:44:59.789+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:45:00.809 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:45:00 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:45:00.481+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:09.828911+0000 front 2023-12-16T21:39:09.828958+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:45:00.810 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:45:00 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:45:00.481+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:09.828868+0000 front 2023-12-16T21:39:09.828878+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:45:00.810 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:45:00 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:45:00.481+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:09.828947+0000 front 2023-12-16T21:39:09.828993+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:45:00.810 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:45:00 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:45:00.482+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:45:01.067 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:45:00 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:45:00.806+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:05.622397+0000 front 2023-12-16T21:39:05.622556+0000 (oldest deadline 2023-12-16T21:39:30.322546+0000) 2023-12-16T21:45:01.067 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:45:00 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:45:00.806+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:05.622506+0000 front 2023-12-16T21:39:05.622528+0000 (oldest deadline 2023-12-16T21:39:30.322546+0000) 2023-12-16T21:45:01.067 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:45:00 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:45:00.806+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:05.622479+0000 front 2023-12-16T21:39:05.622580+0000 (oldest deadline 2023-12-16T21:39:30.322546+0000) 2023-12-16T21:45:01.067 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:45:00 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:45:00.807+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:45:01.067 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:45:00 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:45:00.988+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:09.306583+0000 front 2023-12-16T21:39:09.306633+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:45:01.067 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:45:00 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:45:00.988+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:09.306551+0000 front 2023-12-16T21:39:09.306652+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:45:01.067 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:45:00 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:45:00.988+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:09.306605+0000 front 2023-12-16T21:39:09.306613+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:45:01.067 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:45:00 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:45:00.989+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:45:01.643 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Dec 16 21:45:01 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-mon-smithi045[80759]: 2023-12-16T21:45:01.353+0000 7f45a1f8f700 -1 mon.smithi045@0(probing) e2 get_health_metrics reporting 2479 slow ops, oldest is monmgrreport(gid 14214, 0 checks, 1 progress events) 2023-12-16T21:45:01.644 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:45:01 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:45:01.481+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:09.828911+0000 front 2023-12-16T21:39:09.828958+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:45:01.644 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:45:01 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:45:01.481+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:09.828868+0000 front 2023-12-16T21:39:09.828878+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:45:01.644 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:45:01 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:45:01.481+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:09.828947+0000 front 2023-12-16T21:39:09.828993+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:45:01.644 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:45:01 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:45:01.481+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:45:01.949 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:45:01 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:45:01.786+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:05.622397+0000 front 2023-12-16T21:39:05.622556+0000 (oldest deadline 2023-12-16T21:39:30.322546+0000) 2023-12-16T21:45:01.949 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:45:01 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:45:01.786+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:05.622506+0000 front 2023-12-16T21:39:05.622528+0000 (oldest deadline 2023-12-16T21:39:30.322546+0000) 2023-12-16T21:45:01.949 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:45:01 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:45:01.786+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:05.622479+0000 front 2023-12-16T21:39:05.622580+0000 (oldest deadline 2023-12-16T21:39:30.322546+0000) 2023-12-16T21:45:01.949 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:45:01 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:45:01.787+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:45:02.317 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:45:01 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:45:01.946+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:09.306583+0000 front 2023-12-16T21:39:09.306633+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:45:02.317 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:45:01 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:45:01.946+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:09.306551+0000 front 2023-12-16T21:39:09.306652+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:45:02.317 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:45:01 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:45:01.946+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:09.306605+0000 front 2023-12-16T21:39:09.306613+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:45:02.317 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:45:01 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:45:01.947+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:45:02.813 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:45:02 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:45:02.452+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:09.828911+0000 front 2023-12-16T21:39:09.828958+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:45:02.813 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:45:02 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:45:02.453+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:09.828868+0000 front 2023-12-16T21:39:09.828878+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:45:02.813 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:45:02 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:45:02.453+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:09.828947+0000 front 2023-12-16T21:39:09.828993+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:45:02.814 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:45:02 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:45:02.453+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:45:03.066 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:45:02 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:45:02.810+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:05.622397+0000 front 2023-12-16T21:39:05.622556+0000 (oldest deadline 2023-12-16T21:39:30.322546+0000) 2023-12-16T21:45:03.067 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:45:02 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:45:02.810+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:05.622506+0000 front 2023-12-16T21:39:05.622528+0000 (oldest deadline 2023-12-16T21:39:30.322546+0000) 2023-12-16T21:45:03.067 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:45:02 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:45:02.810+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:05.622479+0000 front 2023-12-16T21:39:05.622580+0000 (oldest deadline 2023-12-16T21:39:30.322546+0000) 2023-12-16T21:45:03.067 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:45:02 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:45:02.811+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:45:03.067 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:45:02 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:45:02.927+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:09.306583+0000 front 2023-12-16T21:39:09.306633+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:45:03.067 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:45:02 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:45:02.927+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:09.306551+0000 front 2023-12-16T21:39:09.306652+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:45:03.067 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:45:02 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:45:02.927+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:09.306605+0000 front 2023-12-16T21:39:09.306613+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:45:03.067 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:45:02 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:45:02.928+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:45:03.803 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:45:03 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:45:03.478+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:09.828911+0000 front 2023-12-16T21:39:09.828958+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:45:03.803 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:45:03 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:45:03.478+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:09.828868+0000 front 2023-12-16T21:39:09.828878+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:45:03.803 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:45:03 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:45:03.478+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:09.828947+0000 front 2023-12-16T21:39:09.828993+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:45:03.804 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:45:03 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:45:03.479+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:45:04.066 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:45:03 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:45:03.800+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:05.622397+0000 front 2023-12-16T21:39:05.622556+0000 (oldest deadline 2023-12-16T21:39:30.322546+0000) 2023-12-16T21:45:04.067 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:45:03 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:45:03.800+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:05.622506+0000 front 2023-12-16T21:39:05.622528+0000 (oldest deadline 2023-12-16T21:39:30.322546+0000) 2023-12-16T21:45:04.067 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:45:03 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:45:03.800+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:05.622479+0000 front 2023-12-16T21:39:05.622580+0000 (oldest deadline 2023-12-16T21:39:30.322546+0000) 2023-12-16T21:45:04.067 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:45:03 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:45:03.801+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:45:04.067 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:45:03 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:45:03.880+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:09.306583+0000 front 2023-12-16T21:39:09.306633+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:45:04.067 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:45:03 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:45:03.880+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:09.306551+0000 front 2023-12-16T21:39:09.306652+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:45:04.067 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:45:03 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:45:03.880+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:09.306605+0000 front 2023-12-16T21:39:09.306613+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:45:04.067 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:45:03 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:45:03.881+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:45:04.816 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:45:04 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:45:04.492+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:09.828911+0000 front 2023-12-16T21:39:09.828958+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:45:04.816 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:45:04 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:45:04.492+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:09.828868+0000 front 2023-12-16T21:39:09.828878+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:45:04.817 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:45:04 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:45:04.492+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:09.828947+0000 front 2023-12-16T21:39:09.828993+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:45:04.817 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:45:04 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:45:04.493+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:45:05.316 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:45:04 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:45:04.866+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:09.306583+0000 front 2023-12-16T21:39:09.306633+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:45:05.316 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:45:04 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:45:04.866+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:09.306551+0000 front 2023-12-16T21:39:09.306652+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:45:05.316 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:45:04 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:45:04.866+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:09.306605+0000 front 2023-12-16T21:39:09.306613+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:45:05.317 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:45:04 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:45:04.867+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:45:05.317 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:45:04 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:45:04.838+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:05.622397+0000 front 2023-12-16T21:39:05.622556+0000 (oldest deadline 2023-12-16T21:39:30.322546+0000) 2023-12-16T21:45:05.317 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:45:04 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:45:04.838+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:05.622506+0000 front 2023-12-16T21:39:05.622528+0000 (oldest deadline 2023-12-16T21:39:30.322546+0000) 2023-12-16T21:45:05.317 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:45:04 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:45:04.838+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:05.622479+0000 front 2023-12-16T21:39:05.622580+0000 (oldest deadline 2023-12-16T21:39:30.322546+0000) 2023-12-16T21:45:05.317 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:45:04 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:45:04.839+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:45:05.785 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:45:05 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:45:05.507+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:09.828911+0000 front 2023-12-16T21:39:09.828958+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:45:05.786 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:45:05 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:45:05.507+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:09.828868+0000 front 2023-12-16T21:39:09.828878+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:45:05.786 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:45:05 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:45:05.507+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:09.828947+0000 front 2023-12-16T21:39:09.828993+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:45:05.786 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:45:05 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:45:05.508+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:45:06.066 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:45:05 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:45:05.905+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:09.306583+0000 front 2023-12-16T21:39:09.306633+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:45:06.067 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:45:05 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:45:05.905+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:09.306551+0000 front 2023-12-16T21:39:09.306652+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:45:06.067 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:45:05 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:45:05.905+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:09.306605+0000 front 2023-12-16T21:39:09.306613+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:45:06.067 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:45:05 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:45:05.906+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:45:06.067 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:45:05 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:45:05.839+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:05.622397+0000 front 2023-12-16T21:39:05.622556+0000 (oldest deadline 2023-12-16T21:39:30.322546+0000) 2023-12-16T21:45:06.067 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:45:05 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:45:05.839+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:05.622506+0000 front 2023-12-16T21:39:05.622528+0000 (oldest deadline 2023-12-16T21:39:30.322546+0000) 2023-12-16T21:45:06.067 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:45:05 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:45:05.839+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:05.622479+0000 front 2023-12-16T21:39:05.622580+0000 (oldest deadline 2023-12-16T21:39:30.322546+0000) 2023-12-16T21:45:06.067 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:45:05 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:45:05.839+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:45:06.807 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Dec 16 21:45:06 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-mon-smithi045[80759]: 2023-12-16T21:45:06.354+0000 7f45a1f8f700 -1 mon.smithi045@0(probing) e2 get_health_metrics reporting 2573 slow ops, oldest is monmgrreport(gid 14214, 0 checks, 1 progress events) 2023-12-16T21:45:06.807 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:45:06 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:45:06.470+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:09.828911+0000 front 2023-12-16T21:39:09.828958+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:45:06.807 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:45:06 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:45:06.470+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:09.828868+0000 front 2023-12-16T21:39:09.828878+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:45:06.807 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:45:06 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:45:06.470+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:09.828947+0000 front 2023-12-16T21:39:09.828993+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:45:06.808 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:45:06 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:45:06.471+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:45:07.066 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:45:06 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:45:06.869+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:09.306583+0000 front 2023-12-16T21:39:09.306633+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:45:07.066 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:45:06 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:45:06.869+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:09.306551+0000 front 2023-12-16T21:39:09.306652+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:45:07.067 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:45:06 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:45:06.869+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:09.306605+0000 front 2023-12-16T21:39:09.306613+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:45:07.067 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:45:06 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:45:06.870+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:45:07.067 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:45:06 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:45:06.801+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:05.622397+0000 front 2023-12-16T21:39:05.622556+0000 (oldest deadline 2023-12-16T21:39:30.322546+0000) 2023-12-16T21:45:07.067 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:45:06 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:45:06.801+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:05.622506+0000 front 2023-12-16T21:39:05.622528+0000 (oldest deadline 2023-12-16T21:39:30.322546+0000) 2023-12-16T21:45:07.067 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:45:06 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:45:06.801+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:05.622479+0000 front 2023-12-16T21:39:05.622580+0000 (oldest deadline 2023-12-16T21:39:30.322546+0000) 2023-12-16T21:45:07.067 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:45:06 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:45:06.802+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:45:07.798 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:45:07 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:45:07.501+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:09.828911+0000 front 2023-12-16T21:39:09.828958+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:45:07.798 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:45:07 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:45:07.501+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:09.828868+0000 front 2023-12-16T21:39:09.828878+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:45:07.798 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:45:07 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:45:07.501+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:09.828947+0000 front 2023-12-16T21:39:09.828993+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:45:07.798 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:45:07 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:45:07.502+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:45:08.066 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:45:07 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:45:07.795+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:05.622397+0000 front 2023-12-16T21:39:05.622556+0000 (oldest deadline 2023-12-16T21:39:30.322546+0000) 2023-12-16T21:45:08.066 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:45:07 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:45:07.795+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:05.622506+0000 front 2023-12-16T21:39:05.622528+0000 (oldest deadline 2023-12-16T21:39:30.322546+0000) 2023-12-16T21:45:08.067 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:45:07 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:45:07.795+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:05.622479+0000 front 2023-12-16T21:39:05.622580+0000 (oldest deadline 2023-12-16T21:39:30.322546+0000) 2023-12-16T21:45:08.067 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:45:07 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:45:07.796+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:45:08.067 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:45:07 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:45:07.902+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:09.306583+0000 front 2023-12-16T21:39:09.306633+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:45:08.067 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:45:07 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:45:07.902+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:09.306551+0000 front 2023-12-16T21:39:09.306652+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:45:08.067 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:45:07 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:45:07.902+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:09.306605+0000 front 2023-12-16T21:39:09.306613+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:45:08.067 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:45:07 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:45:07.903+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:45:08.722 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:45:08 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:45:08.460+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:09.828911+0000 front 2023-12-16T21:39:09.828958+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:45:08.723 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:45:08 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:45:08.460+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:09.828868+0000 front 2023-12-16T21:39:09.828878+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:45:08.723 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:45:08 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:45:08.460+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:09.828947+0000 front 2023-12-16T21:39:09.828993+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:45:08.723 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:45:08 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:45:08.462+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:45:08.920 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:45:08 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:45:08.916+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:09.306583+0000 front 2023-12-16T21:39:09.306633+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:45:08.921 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:45:08 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:45:08.916+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:09.306551+0000 front 2023-12-16T21:39:09.306652+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:45:08.921 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:45:08 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:45:08.916+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:09.306605+0000 front 2023-12-16T21:39:09.306613+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:45:08.921 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:45:08 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:45:08.917+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:45:08.921 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:45:08 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:45:08.804+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:05.622397+0000 front 2023-12-16T21:39:05.622556+0000 (oldest deadline 2023-12-16T21:39:30.322546+0000) 2023-12-16T21:45:08.921 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:45:08 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:45:08.804+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:05.622506+0000 front 2023-12-16T21:39:05.622528+0000 (oldest deadline 2023-12-16T21:39:30.322546+0000) 2023-12-16T21:45:08.921 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:45:08 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:45:08.804+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:05.622479+0000 front 2023-12-16T21:39:05.622580+0000 (oldest deadline 2023-12-16T21:39:30.322546+0000) 2023-12-16T21:45:08.921 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:45:08 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:45:08.805+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:45:09.816 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:45:09 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:45:09.427+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:09.828911+0000 front 2023-12-16T21:39:09.828958+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:45:09.817 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:45:09 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:45:09.427+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:09.828868+0000 front 2023-12-16T21:39:09.828878+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:45:09.817 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:45:09 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:45:09.427+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:09.828947+0000 front 2023-12-16T21:39:09.828993+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:45:09.817 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:45:09 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:45:09.428+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:45:10.316 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:45:09 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:45:09.814+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:05.622397+0000 front 2023-12-16T21:39:05.622556+0000 (oldest deadline 2023-12-16T21:39:30.322546+0000) 2023-12-16T21:45:10.316 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:45:09 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:45:09.814+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:05.622506+0000 front 2023-12-16T21:39:05.622528+0000 (oldest deadline 2023-12-16T21:39:30.322546+0000) 2023-12-16T21:45:10.317 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:45:09 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:45:09.814+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:05.622479+0000 front 2023-12-16T21:39:05.622580+0000 (oldest deadline 2023-12-16T21:39:30.322546+0000) 2023-12-16T21:45:10.317 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:45:09 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:45:09.814+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:45:10.317 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:45:09 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:45:09.954+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:09.306583+0000 front 2023-12-16T21:39:09.306633+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:45:10.317 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:45:09 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:45:09.954+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:09.306551+0000 front 2023-12-16T21:39:09.306652+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:45:10.317 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:45:09 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:45:09.954+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:09.306605+0000 front 2023-12-16T21:39:09.306613+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:45:10.317 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:45:09 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:45:09.955+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:45:10.816 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:45:10 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:45:10.438+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:09.828911+0000 front 2023-12-16T21:39:09.828958+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:45:10.817 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:45:10 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:45:10.438+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:09.828868+0000 front 2023-12-16T21:39:09.828878+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:45:10.817 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:45:10 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:45:10.438+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:09.828947+0000 front 2023-12-16T21:39:09.828993+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:45:10.817 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:45:10 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:45:10.439+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:45:11.316 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:45:10 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:45:10.859+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:05.622397+0000 front 2023-12-16T21:39:05.622556+0000 (oldest deadline 2023-12-16T21:39:30.322546+0000) 2023-12-16T21:45:11.317 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:45:10 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:45:10.859+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:05.622506+0000 front 2023-12-16T21:39:05.622528+0000 (oldest deadline 2023-12-16T21:39:30.322546+0000) 2023-12-16T21:45:11.317 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:45:10 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:45:10.859+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:05.622479+0000 front 2023-12-16T21:39:05.622580+0000 (oldest deadline 2023-12-16T21:39:30.322546+0000) 2023-12-16T21:45:11.317 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:45:10 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:45:10.860+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:45:11.317 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:45:10 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:45:10.931+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:09.306583+0000 front 2023-12-16T21:39:09.306633+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:45:11.317 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:45:10 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:45:10.931+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:09.306551+0000 front 2023-12-16T21:39:09.306652+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:45:11.317 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:45:10 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:45:10.931+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:09.306605+0000 front 2023-12-16T21:39:09.306613+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:45:11.317 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:45:10 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:45:10.932+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:45:11.644 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Dec 16 21:45:11 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-mon-smithi045[80759]: 2023-12-16T21:45:11.354+0000 7f45a1f8f700 -1 mon.smithi045@0(probing) e2 get_health_metrics reporting 2669 slow ops, oldest is monmgrreport(gid 14214, 0 checks, 1 progress events) 2023-12-16T21:45:11.644 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:45:11 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:45:11.463+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:09.828911+0000 front 2023-12-16T21:39:09.828958+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:45:11.644 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:45:11 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:45:11.463+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:09.828868+0000 front 2023-12-16T21:39:09.828878+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:45:11.645 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:45:11 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:45:11.463+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:09.828947+0000 front 2023-12-16T21:39:09.828993+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:45:11.645 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:45:11 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:45:11.464+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:45:11.908 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:45:11 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:45:11.890+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:09.306583+0000 front 2023-12-16T21:39:09.306633+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:45:11.908 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:45:11 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:45:11.890+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:09.306551+0000 front 2023-12-16T21:39:09.306652+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:45:11.908 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:45:11 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:45:11.890+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:09.306605+0000 front 2023-12-16T21:39:09.306613+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:45:11.908 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:45:11 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:45:11.891+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:45:12.316 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:45:11 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:45:11.905+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:05.622397+0000 front 2023-12-16T21:39:05.622556+0000 (oldest deadline 2023-12-16T21:39:30.322546+0000) 2023-12-16T21:45:12.316 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:45:11 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:45:11.905+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:05.622506+0000 front 2023-12-16T21:39:05.622528+0000 (oldest deadline 2023-12-16T21:39:30.322546+0000) 2023-12-16T21:45:12.317 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:45:11 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:45:11.905+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:05.622479+0000 front 2023-12-16T21:39:05.622580+0000 (oldest deadline 2023-12-16T21:39:30.322546+0000) 2023-12-16T21:45:12.317 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:45:11 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:45:11.906+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:45:12.816 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:45:12 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:45:12.484+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:09.828911+0000 front 2023-12-16T21:39:09.828958+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:45:12.816 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:45:12 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:45:12.484+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:09.828868+0000 front 2023-12-16T21:39:09.828878+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:45:12.816 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:45:12 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:45:12.484+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:09.828947+0000 front 2023-12-16T21:39:09.828993+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:45:12.817 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:45:12 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:45:12.484+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:45:13.316 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:45:12 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:45:12.876+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:09.306583+0000 front 2023-12-16T21:39:09.306633+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:45:13.317 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:45:12 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:45:12.876+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:09.306551+0000 front 2023-12-16T21:39:09.306652+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:45:13.317 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:45:12 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:45:12.876+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:09.306605+0000 front 2023-12-16T21:39:09.306613+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:45:13.317 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:45:12 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:45:12.877+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:45:13.317 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:45:12 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:45:12.860+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:05.622397+0000 front 2023-12-16T21:39:05.622556+0000 (oldest deadline 2023-12-16T21:39:30.322546+0000) 2023-12-16T21:45:13.317 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:45:12 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:45:12.860+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:05.622506+0000 front 2023-12-16T21:39:05.622528+0000 (oldest deadline 2023-12-16T21:39:30.322546+0000) 2023-12-16T21:45:13.317 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:45:12 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:45:12.860+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:05.622479+0000 front 2023-12-16T21:39:05.622580+0000 (oldest deadline 2023-12-16T21:39:30.322546+0000) 2023-12-16T21:45:13.317 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:45:12 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:45:12.860+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:45:13.816 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:45:13 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:45:13.463+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:09.828911+0000 front 2023-12-16T21:39:09.828958+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:45:13.817 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:45:13 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:45:13.463+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:09.828868+0000 front 2023-12-16T21:39:09.828878+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:45:13.817 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:45:13 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:45:13.463+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:09.828947+0000 front 2023-12-16T21:39:09.828993+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:45:13.817 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:45:13 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:45:13.464+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:45:14.316 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:45:13 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:45:13.904+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:05.622397+0000 front 2023-12-16T21:39:05.622556+0000 (oldest deadline 2023-12-16T21:39:30.322546+0000) 2023-12-16T21:45:14.317 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:45:13 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:45:13.904+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:05.622506+0000 front 2023-12-16T21:39:05.622528+0000 (oldest deadline 2023-12-16T21:39:30.322546+0000) 2023-12-16T21:45:14.317 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:45:13 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:45:13.904+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:05.622479+0000 front 2023-12-16T21:39:05.622580+0000 (oldest deadline 2023-12-16T21:39:30.322546+0000) 2023-12-16T21:45:14.317 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:45:13 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:45:13.906+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:45:14.317 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:45:13 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:45:13.841+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:09.306583+0000 front 2023-12-16T21:39:09.306633+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:45:14.317 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:45:13 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:45:13.841+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:09.306551+0000 front 2023-12-16T21:39:09.306652+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:45:14.317 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:45:13 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:45:13.841+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:09.306605+0000 front 2023-12-16T21:39:09.306613+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:45:14.317 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:45:13 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:45:13.842+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:45:14.770 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:45:14 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:45:14.488+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:09.828911+0000 front 2023-12-16T21:39:09.828958+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:45:14.770 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:45:14 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:45:14.488+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:09.828868+0000 front 2023-12-16T21:39:09.828878+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:45:14.770 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:45:14 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:45:14.488+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:09.828947+0000 front 2023-12-16T21:39:09.828993+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:45:14.770 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:45:14 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:45:14.489+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:45:15.066 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:45:14 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:45:14.842+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:09.306583+0000 front 2023-12-16T21:39:09.306633+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:45:15.067 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:45:14 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:45:14.842+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:09.306551+0000 front 2023-12-16T21:39:09.306652+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:45:15.067 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:45:14 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:45:14.842+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:09.306605+0000 front 2023-12-16T21:39:09.306613+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:45:15.067 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:45:14 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:45:14.843+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:45:15.067 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:45:14 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:45:14.938+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:05.622397+0000 front 2023-12-16T21:39:05.622556+0000 (oldest deadline 2023-12-16T21:39:30.322546+0000) 2023-12-16T21:45:15.067 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:45:14 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:45:14.938+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:05.622506+0000 front 2023-12-16T21:39:05.622528+0000 (oldest deadline 2023-12-16T21:39:30.322546+0000) 2023-12-16T21:45:15.067 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:45:14 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:45:14.938+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:05.622479+0000 front 2023-12-16T21:39:05.622580+0000 (oldest deadline 2023-12-16T21:39:30.322546+0000) 2023-12-16T21:45:15.067 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:45:14 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:45:14.939+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:45:15.816 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:45:15 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:45:15.502+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:09.828911+0000 front 2023-12-16T21:39:09.828958+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:45:15.817 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:45:15 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:45:15.502+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:09.828868+0000 front 2023-12-16T21:39:09.828878+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:45:15.817 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:45:15 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:45:15.502+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:09.828947+0000 front 2023-12-16T21:39:09.828993+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:45:15.817 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:45:15 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:45:15.504+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:45:16.316 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:45:15 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:45:15.857+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:09.306583+0000 front 2023-12-16T21:39:09.306633+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:45:16.317 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:45:15 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:45:15.857+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:09.306551+0000 front 2023-12-16T21:39:09.306652+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:45:16.317 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:45:15 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:45:15.857+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:09.306605+0000 front 2023-12-16T21:39:09.306613+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:45:16.317 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:45:15 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:45:15.858+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:45:16.317 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:45:15 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:45:15.960+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:05.622397+0000 front 2023-12-16T21:39:05.622556+0000 (oldest deadline 2023-12-16T21:39:30.322546+0000) 2023-12-16T21:45:16.317 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:45:15 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:45:15.960+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:05.622506+0000 front 2023-12-16T21:39:05.622528+0000 (oldest deadline 2023-12-16T21:39:30.322546+0000) 2023-12-16T21:45:16.317 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:45:15 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:45:15.960+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:05.622479+0000 front 2023-12-16T21:39:05.622580+0000 (oldest deadline 2023-12-16T21:39:30.322546+0000) 2023-12-16T21:45:16.317 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:45:15 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:45:15.960+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:45:16.816 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Dec 16 21:45:16 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-mon-smithi045[80759]: 2023-12-16T21:45:16.355+0000 7f45a1f8f700 -1 mon.smithi045@0(probing) e2 get_health_metrics reporting 2763 slow ops, oldest is monmgrreport(gid 14214, 0 checks, 1 progress events) 2023-12-16T21:45:16.817 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:45:16 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:45:16.510+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:09.828911+0000 front 2023-12-16T21:39:09.828958+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:45:16.817 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:45:16 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:45:16.510+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:09.828868+0000 front 2023-12-16T21:39:09.828878+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:45:16.817 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:45:16 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:45:16.510+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:09.828947+0000 front 2023-12-16T21:39:09.828993+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:45:16.817 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:45:16 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:45:16.511+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:45:17.316 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:45:16 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:45:16.951+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:05.622397+0000 front 2023-12-16T21:39:05.622556+0000 (oldest deadline 2023-12-16T21:39:30.322546+0000) 2023-12-16T21:45:17.316 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:45:16 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:45:16.951+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:05.622506+0000 front 2023-12-16T21:39:05.622528+0000 (oldest deadline 2023-12-16T21:39:30.322546+0000) 2023-12-16T21:45:17.316 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:45:16 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:45:16.951+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:05.622479+0000 front 2023-12-16T21:39:05.622580+0000 (oldest deadline 2023-12-16T21:39:30.322546+0000) 2023-12-16T21:45:17.317 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:45:16 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:45:16.952+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:45:17.317 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:45:16 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:45:16.843+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:09.306583+0000 front 2023-12-16T21:39:09.306633+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:45:17.317 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:45:16 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:45:16.843+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:09.306551+0000 front 2023-12-16T21:39:09.306652+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:45:17.317 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:45:16 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:45:16.843+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:09.306605+0000 front 2023-12-16T21:39:09.306613+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:45:17.317 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:45:16 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:45:16.843+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:45:17.816 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:45:17 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:45:17.493+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:09.828911+0000 front 2023-12-16T21:39:09.828958+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:45:17.816 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:45:17 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:45:17.493+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:09.828868+0000 front 2023-12-16T21:39:09.828878+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:45:17.817 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:45:17 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:45:17.493+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:09.828947+0000 front 2023-12-16T21:39:09.828993+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:45:17.817 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:45:17 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:45:17.494+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:45:18.316 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:45:17 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:45:17.967+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:05.622397+0000 front 2023-12-16T21:39:05.622556+0000 (oldest deadline 2023-12-16T21:39:30.322546+0000) 2023-12-16T21:45:18.317 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:45:17 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:45:17.968+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:05.622506+0000 front 2023-12-16T21:39:05.622528+0000 (oldest deadline 2023-12-16T21:39:30.322546+0000) 2023-12-16T21:45:18.317 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:45:17 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:45:17.968+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:05.622479+0000 front 2023-12-16T21:39:05.622580+0000 (oldest deadline 2023-12-16T21:39:30.322546+0000) 2023-12-16T21:45:18.317 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:45:17 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:45:17.968+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:45:18.317 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:45:17 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:45:17.869+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:09.306583+0000 front 2023-12-16T21:39:09.306633+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:45:18.317 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:45:17 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:45:17.869+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:09.306551+0000 front 2023-12-16T21:39:09.306652+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:45:18.317 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:45:17 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:45:17.869+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:09.306605+0000 front 2023-12-16T21:39:09.306613+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:45:18.317 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:45:17 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:45:17.870+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:45:18.816 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:45:18 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:45:18.512+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:09.828911+0000 front 2023-12-16T21:39:09.828958+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:45:18.816 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:45:18 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:45:18.512+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:09.828868+0000 front 2023-12-16T21:39:09.828878+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:45:18.817 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:45:18 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:45:18.512+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:09.828947+0000 front 2023-12-16T21:39:09.828993+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:45:18.817 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:45:18 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:45:18.513+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:45:19.316 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:45:18 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:45:18.835+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:09.306583+0000 front 2023-12-16T21:39:09.306633+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:45:19.317 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:45:18 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:45:18.835+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:09.306551+0000 front 2023-12-16T21:39:09.306652+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:45:19.317 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:45:18 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:45:18.835+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:09.306605+0000 front 2023-12-16T21:39:09.306613+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:45:19.317 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:45:18 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:45:18.836+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:45:19.318 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:45:18 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:45:18.923+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:05.622397+0000 front 2023-12-16T21:39:05.622556+0000 (oldest deadline 2023-12-16T21:39:30.322546+0000) 2023-12-16T21:45:19.318 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:45:18 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:45:18.923+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:05.622506+0000 front 2023-12-16T21:39:05.622528+0000 (oldest deadline 2023-12-16T21:39:30.322546+0000) 2023-12-16T21:45:19.318 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:45:18 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:45:18.923+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:05.622479+0000 front 2023-12-16T21:39:05.622580+0000 (oldest deadline 2023-12-16T21:39:30.322546+0000) 2023-12-16T21:45:19.318 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:45:18 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:45:18.924+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:45:19.816 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:45:19 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:45:19.481+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:09.828911+0000 front 2023-12-16T21:39:09.828958+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:45:19.816 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:45:19 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:45:19.481+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:09.828868+0000 front 2023-12-16T21:39:09.828878+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:45:19.817 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:45:19 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:45:19.481+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:09.828947+0000 front 2023-12-16T21:39:09.828993+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:45:19.817 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:45:19 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:45:19.482+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:45:20.316 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:45:19 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:45:19.894+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:05.622397+0000 front 2023-12-16T21:39:05.622556+0000 (oldest deadline 2023-12-16T21:39:30.322546+0000) 2023-12-16T21:45:20.316 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:45:19 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:45:19.894+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:05.622506+0000 front 2023-12-16T21:39:05.622528+0000 (oldest deadline 2023-12-16T21:39:30.322546+0000) 2023-12-16T21:45:20.317 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:45:19 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:45:19.894+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:05.622479+0000 front 2023-12-16T21:39:05.622580+0000 (oldest deadline 2023-12-16T21:39:30.322546+0000) 2023-12-16T21:45:20.317 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:45:19 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:45:19.894+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:45:20.317 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:45:19 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:45:19.813+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:09.306583+0000 front 2023-12-16T21:39:09.306633+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:45:20.317 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:45:19 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:45:19.813+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:09.306551+0000 front 2023-12-16T21:39:09.306652+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:45:20.317 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:45:19 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:45:19.813+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:09.306605+0000 front 2023-12-16T21:39:09.306613+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:45:20.317 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:45:19 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:45:19.814+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:45:20.816 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:45:20 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:45:20.485+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:09.828911+0000 front 2023-12-16T21:39:09.828958+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:45:20.817 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:45:20 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:45:20.485+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:09.828868+0000 front 2023-12-16T21:39:09.828878+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:45:20.817 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:45:20 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:45:20.485+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:09.828947+0000 front 2023-12-16T21:39:09.828993+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:45:20.817 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:45:20 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:45:20.486+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:45:21.316 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:45:20 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:45:20.930+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:05.622397+0000 front 2023-12-16T21:39:05.622556+0000 (oldest deadline 2023-12-16T21:39:30.322546+0000) 2023-12-16T21:45:21.317 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:45:20 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:45:20.930+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:05.622506+0000 front 2023-12-16T21:39:05.622528+0000 (oldest deadline 2023-12-16T21:39:30.322546+0000) 2023-12-16T21:45:21.317 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:45:20 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:45:20.930+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:05.622479+0000 front 2023-12-16T21:39:05.622580+0000 (oldest deadline 2023-12-16T21:39:30.322546+0000) 2023-12-16T21:45:21.317 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:45:20 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:45:20.931+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:45:21.317 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:45:20 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:45:20.848+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:09.306583+0000 front 2023-12-16T21:39:09.306633+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:45:21.317 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:45:20 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:45:20.848+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:09.306551+0000 front 2023-12-16T21:39:09.306652+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:45:21.317 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:45:20 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:45:20.848+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:09.306605+0000 front 2023-12-16T21:39:09.306613+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:45:21.317 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:45:20 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:45:20.849+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:45:21.644 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Dec 16 21:45:21 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-mon-smithi045[80759]: 2023-12-16T21:45:21.356+0000 7f45a1f8f700 -1 mon.smithi045@0(probing) e2 get_health_metrics reporting 2860 slow ops, oldest is monmgrreport(gid 14214, 0 checks, 1 progress events) 2023-12-16T21:45:21.645 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:45:21 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:45:21.521+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:09.828911+0000 front 2023-12-16T21:39:09.828958+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:45:21.645 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:45:21 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:45:21.521+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:09.828868+0000 front 2023-12-16T21:39:09.828878+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:45:21.645 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:45:21 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:45:21.521+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:09.828947+0000 front 2023-12-16T21:39:09.828993+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:45:21.645 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:45:21 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:45:21.523+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:45:21.963 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:45:21 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:45:21.870+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:09.306583+0000 front 2023-12-16T21:39:09.306633+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:45:21.964 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:45:21 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:45:21.870+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:09.306551+0000 front 2023-12-16T21:39:09.306652+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:45:21.964 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:45:21 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:45:21.870+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:09.306605+0000 front 2023-12-16T21:39:09.306613+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:45:21.964 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:45:21 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:45:21.871+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:45:22.316 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:45:21 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:45:21.960+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:05.622397+0000 front 2023-12-16T21:39:05.622556+0000 (oldest deadline 2023-12-16T21:39:30.322546+0000) 2023-12-16T21:45:22.316 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:45:21 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:45:21.960+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:05.622506+0000 front 2023-12-16T21:39:05.622528+0000 (oldest deadline 2023-12-16T21:39:30.322546+0000) 2023-12-16T21:45:22.317 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:45:21 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:45:21.960+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:05.622479+0000 front 2023-12-16T21:39:05.622580+0000 (oldest deadline 2023-12-16T21:39:30.322546+0000) 2023-12-16T21:45:22.317 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:45:21 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:45:21.961+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:45:22.816 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:45:22 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:45:22.524+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:09.828911+0000 front 2023-12-16T21:39:09.828958+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:45:22.816 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:45:22 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:45:22.524+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:09.828868+0000 front 2023-12-16T21:39:09.828878+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:45:22.816 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:45:22 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:45:22.524+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:09.828947+0000 front 2023-12-16T21:39:09.828993+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:45:22.817 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:45:22 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:45:22.525+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:45:23.316 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:45:22 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:45:22.947+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:05.622397+0000 front 2023-12-16T21:39:05.622556+0000 (oldest deadline 2023-12-16T21:39:30.322546+0000) 2023-12-16T21:45:23.316 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:45:22 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:45:22.947+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:05.622506+0000 front 2023-12-16T21:39:05.622528+0000 (oldest deadline 2023-12-16T21:39:30.322546+0000) 2023-12-16T21:45:23.317 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:45:22 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:45:22.947+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:05.622479+0000 front 2023-12-16T21:39:05.622580+0000 (oldest deadline 2023-12-16T21:39:30.322546+0000) 2023-12-16T21:45:23.317 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:45:22 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:45:22.947+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:45:23.317 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:45:22 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:45:22.909+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:09.306583+0000 front 2023-12-16T21:39:09.306633+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:45:23.317 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:45:22 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:45:22.909+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:09.306551+0000 front 2023-12-16T21:39:09.306652+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:45:23.317 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:45:22 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:45:22.909+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:09.306605+0000 front 2023-12-16T21:39:09.306613+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:45:23.317 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:45:22 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:45:22.909+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:45:23.816 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:45:23 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:45:23.553+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:09.828911+0000 front 2023-12-16T21:39:09.828958+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:45:23.817 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:45:23 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:45:23.553+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:09.828868+0000 front 2023-12-16T21:39:09.828878+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:45:23.817 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:45:23 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:45:23.553+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:09.828947+0000 front 2023-12-16T21:39:09.828993+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:45:23.817 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:45:23 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:45:23.554+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:45:24.316 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:45:23 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:45:23.942+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:05.622397+0000 front 2023-12-16T21:39:05.622556+0000 (oldest deadline 2023-12-16T21:39:30.322546+0000) 2023-12-16T21:45:24.317 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:45:23 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:45:23.942+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:05.622506+0000 front 2023-12-16T21:39:05.622528+0000 (oldest deadline 2023-12-16T21:39:30.322546+0000) 2023-12-16T21:45:24.317 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:45:23 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:45:23.942+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:05.622479+0000 front 2023-12-16T21:39:05.622580+0000 (oldest deadline 2023-12-16T21:39:30.322546+0000) 2023-12-16T21:45:24.317 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:45:23 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:45:23.943+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:45:24.317 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:45:23 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:45:23.924+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:09.306583+0000 front 2023-12-16T21:39:09.306633+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:45:24.317 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:45:23 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:45:23.924+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:09.306551+0000 front 2023-12-16T21:39:09.306652+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:45:24.317 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:45:23 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:45:23.924+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:09.306605+0000 front 2023-12-16T21:39:09.306613+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:45:24.317 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:45:23 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:45:23.924+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:45:24.911 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:45:24 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:45:24.593+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:09.828911+0000 front 2023-12-16T21:39:09.828958+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:45:24.911 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:45:24 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:45:24.593+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:09.828868+0000 front 2023-12-16T21:39:09.828878+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:45:24.911 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:45:24 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:45:24.593+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:09.828947+0000 front 2023-12-16T21:39:09.828993+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:45:24.911 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:45:24 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:45:24.594+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:45:25.316 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:45:24 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:45:24.922+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:09.306583+0000 front 2023-12-16T21:39:09.306633+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:45:25.316 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:45:24 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:45:24.922+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:09.306551+0000 front 2023-12-16T21:39:09.306652+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:45:25.316 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:45:24 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:45:24.922+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:09.306605+0000 front 2023-12-16T21:39:09.306613+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:45:25.317 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:45:24 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:45:24.922+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:45:25.317 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:45:24 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:45:24.907+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:05.622397+0000 front 2023-12-16T21:39:05.622556+0000 (oldest deadline 2023-12-16T21:39:30.322546+0000) 2023-12-16T21:45:25.317 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:45:24 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:45:24.907+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:05.622506+0000 front 2023-12-16T21:39:05.622528+0000 (oldest deadline 2023-12-16T21:39:30.322546+0000) 2023-12-16T21:45:25.317 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:45:24 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:45:24.907+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:05.622479+0000 front 2023-12-16T21:39:05.622580+0000 (oldest deadline 2023-12-16T21:39:30.322546+0000) 2023-12-16T21:45:25.317 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:45:24 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:45:24.908+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:45:25.888 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:45:25 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:45:25.584+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:09.828911+0000 front 2023-12-16T21:39:09.828958+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:45:25.888 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:45:25 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:45:25.584+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:09.828868+0000 front 2023-12-16T21:39:09.828878+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:45:25.889 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:45:25 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:45:25.584+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:09.828947+0000 front 2023-12-16T21:39:09.828993+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:45:25.889 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:45:25 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:45:25.585+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:45:26.316 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:45:25 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:45:25.885+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:09.306583+0000 front 2023-12-16T21:39:09.306633+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:45:26.317 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:45:25 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:45:25.885+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:09.306551+0000 front 2023-12-16T21:39:09.306652+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:45:26.317 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:45:25 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:45:25.885+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:09.306605+0000 front 2023-12-16T21:39:09.306613+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:45:26.317 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:45:25 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:45:25.886+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:45:26.317 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:45:25 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:45:25.904+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:05.622397+0000 front 2023-12-16T21:39:05.622556+0000 (oldest deadline 2023-12-16T21:39:30.322546+0000) 2023-12-16T21:45:26.317 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:45:25 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:45:25.904+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:05.622506+0000 front 2023-12-16T21:39:05.622528+0000 (oldest deadline 2023-12-16T21:39:30.322546+0000) 2023-12-16T21:45:26.317 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:45:25 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:45:25.904+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:05.622479+0000 front 2023-12-16T21:39:05.622580+0000 (oldest deadline 2023-12-16T21:39:30.322546+0000) 2023-12-16T21:45:26.317 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:45:25 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:45:25.905+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:45:26.816 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Dec 16 21:45:26 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-mon-smithi045[80759]: 2023-12-16T21:45:26.357+0000 7f45a1f8f700 -1 mon.smithi045@0(probing) e2 get_health_metrics reporting 2953 slow ops, oldest is monmgrreport(gid 14214, 0 checks, 1 progress events) 2023-12-16T21:45:26.816 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:45:26 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:45:26.544+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:09.828911+0000 front 2023-12-16T21:39:09.828958+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:45:26.817 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:45:26 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:45:26.544+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:09.828868+0000 front 2023-12-16T21:39:09.828878+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:45:26.817 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:45:26 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:45:26.544+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:09.828947+0000 front 2023-12-16T21:39:09.828993+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:45:26.817 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:45:26 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:45:26.545+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:45:27.316 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:45:26 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:45:26.860+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:05.622397+0000 front 2023-12-16T21:39:05.622556+0000 (oldest deadline 2023-12-16T21:39:30.322546+0000) 2023-12-16T21:45:27.316 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:45:26 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:45:26.861+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:05.622506+0000 front 2023-12-16T21:39:05.622528+0000 (oldest deadline 2023-12-16T21:39:30.322546+0000) 2023-12-16T21:45:27.317 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:45:26 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:45:26.861+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:05.622479+0000 front 2023-12-16T21:39:05.622580+0000 (oldest deadline 2023-12-16T21:39:30.322546+0000) 2023-12-16T21:45:27.317 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:45:26 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:45:26.861+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:45:27.317 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:45:26 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:45:26.870+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:09.306583+0000 front 2023-12-16T21:39:09.306633+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:45:27.317 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:45:26 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:45:26.870+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:09.306551+0000 front 2023-12-16T21:39:09.306652+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:45:27.317 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:45:26 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:45:26.870+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:09.306605+0000 front 2023-12-16T21:39:09.306613+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:45:27.317 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:45:26 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:45:26.870+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:45:27.816 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:45:27 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:45:27.498+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:09.828911+0000 front 2023-12-16T21:39:09.828958+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:45:27.816 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:45:27 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:45:27.498+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:09.828868+0000 front 2023-12-16T21:39:09.828878+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:45:27.817 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:45:27 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:45:27.498+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:09.828947+0000 front 2023-12-16T21:39:09.828993+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:45:27.817 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:45:27 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:45:27.500+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:45:28.316 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:45:27 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:45:27.873+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:05.622397+0000 front 2023-12-16T21:39:05.622556+0000 (oldest deadline 2023-12-16T21:39:30.322546+0000) 2023-12-16T21:45:28.317 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:45:27 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:45:27.873+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:05.622506+0000 front 2023-12-16T21:39:05.622528+0000 (oldest deadline 2023-12-16T21:39:30.322546+0000) 2023-12-16T21:45:28.317 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:45:27 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:45:27.873+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:05.622479+0000 front 2023-12-16T21:39:05.622580+0000 (oldest deadline 2023-12-16T21:39:30.322546+0000) 2023-12-16T21:45:28.317 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:45:27 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:45:27.874+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:45:28.317 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:45:27 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:45:27.917+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:09.306583+0000 front 2023-12-16T21:39:09.306633+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:45:28.317 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:45:27 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:45:27.917+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:09.306551+0000 front 2023-12-16T21:39:09.306652+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:45:28.317 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:45:27 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:45:27.917+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:09.306605+0000 front 2023-12-16T21:39:09.306613+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:45:28.317 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:45:27 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:45:27.918+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:45:28.816 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:45:28 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:45:28.481+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:09.828911+0000 front 2023-12-16T21:39:09.828958+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:45:28.816 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:45:28 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:45:28.481+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:09.828868+0000 front 2023-12-16T21:39:09.828878+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:45:28.816 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:45:28 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:45:28.481+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:09.828947+0000 front 2023-12-16T21:39:09.828993+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:45:28.817 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:45:28 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:45:28.481+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:45:29.316 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:45:28 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:45:28.921+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:05.622397+0000 front 2023-12-16T21:39:05.622556+0000 (oldest deadline 2023-12-16T21:39:30.322546+0000) 2023-12-16T21:45:29.316 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:45:28 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:45:28.921+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:05.622506+0000 front 2023-12-16T21:39:05.622528+0000 (oldest deadline 2023-12-16T21:39:30.322546+0000) 2023-12-16T21:45:29.316 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:45:28 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:45:28.921+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:05.622479+0000 front 2023-12-16T21:39:05.622580+0000 (oldest deadline 2023-12-16T21:39:30.322546+0000) 2023-12-16T21:45:29.317 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:45:28 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:45:28.921+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:45:29.317 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:45:28 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:45:28.893+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:09.306583+0000 front 2023-12-16T21:39:09.306633+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:45:29.317 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:45:28 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:45:28.893+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:09.306551+0000 front 2023-12-16T21:39:09.306652+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:45:29.317 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:45:28 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:45:28.893+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:09.306605+0000 front 2023-12-16T21:39:09.306613+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:45:29.317 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:45:28 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:45:28.894+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:45:29.816 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:45:29 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:45:29.444+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:09.828911+0000 front 2023-12-16T21:39:09.828958+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:45:29.817 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:45:29 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:45:29.444+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:09.828868+0000 front 2023-12-16T21:39:09.828878+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:45:29.817 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:45:29 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:45:29.444+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:09.828947+0000 front 2023-12-16T21:39:09.828993+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:45:29.817 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:45:29 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:45:29.444+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:45:30.316 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:45:29 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:45:29.911+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:09.306583+0000 front 2023-12-16T21:39:09.306633+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:45:30.317 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:45:29 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:45:29.911+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:09.306551+0000 front 2023-12-16T21:39:09.306652+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:45:30.317 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:45:29 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:45:29.911+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:09.306605+0000 front 2023-12-16T21:39:09.306613+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:45:30.317 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:45:29 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:45:29.912+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:45:30.317 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:45:29 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:45:29.910+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:05.622397+0000 front 2023-12-16T21:39:05.622556+0000 (oldest deadline 2023-12-16T21:39:30.322546+0000) 2023-12-16T21:45:30.317 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:45:29 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:45:29.910+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:05.622506+0000 front 2023-12-16T21:39:05.622528+0000 (oldest deadline 2023-12-16T21:39:30.322546+0000) 2023-12-16T21:45:30.317 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:45:29 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:45:29.910+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:05.622479+0000 front 2023-12-16T21:39:05.622580+0000 (oldest deadline 2023-12-16T21:39:30.322546+0000) 2023-12-16T21:45:30.317 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:45:29 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:45:29.911+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:45:30.816 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:45:30 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:45:30.420+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:09.828911+0000 front 2023-12-16T21:39:09.828958+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:45:30.816 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:45:30 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:45:30.420+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:09.828868+0000 front 2023-12-16T21:39:09.828878+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:45:30.817 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:45:30 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:45:30.420+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:09.828947+0000 front 2023-12-16T21:39:09.828993+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:45:30.817 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:45:30 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:45:30.421+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:45:31.316 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:45:30 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:45:30.957+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:09.306583+0000 front 2023-12-16T21:39:09.306633+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:45:31.317 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:45:30 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:45:30.957+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:09.306551+0000 front 2023-12-16T21:39:09.306652+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:45:31.317 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:45:30 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:45:30.957+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:09.306605+0000 front 2023-12-16T21:39:09.306613+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:45:31.317 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:45:30 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:45:30.958+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:45:31.317 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:45:30 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:45:30.911+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:05.622397+0000 front 2023-12-16T21:39:05.622556+0000 (oldest deadline 2023-12-16T21:39:30.322546+0000) 2023-12-16T21:45:31.317 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:45:30 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:45:30.911+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:05.622506+0000 front 2023-12-16T21:39:05.622528+0000 (oldest deadline 2023-12-16T21:39:30.322546+0000) 2023-12-16T21:45:31.317 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:45:30 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:45:30.911+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:05.622479+0000 front 2023-12-16T21:39:05.622580+0000 (oldest deadline 2023-12-16T21:39:30.322546+0000) 2023-12-16T21:45:31.317 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:45:30 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:45:30.911+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:45:31.645 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Dec 16 21:45:31 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-mon-smithi045[80759]: 2023-12-16T21:45:31.356+0000 7f45a1f8f700 -1 mon.smithi045@0(probing) e2 get_health_metrics reporting 3048 slow ops, oldest is monmgrreport(gid 14214, 0 checks, 1 progress events) 2023-12-16T21:45:31.645 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:45:31 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:45:31.399+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:09.828911+0000 front 2023-12-16T21:39:09.828958+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:45:31.646 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:45:31 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:45:31.399+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:09.828868+0000 front 2023-12-16T21:39:09.828878+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:45:31.646 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:45:31 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:45:31.399+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:09.828947+0000 front 2023-12-16T21:39:09.828993+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:45:31.646 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:45:31 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:45:31.399+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:45:32.316 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:45:31 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:45:31.973+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:09.306583+0000 front 2023-12-16T21:39:09.306633+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:45:32.316 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:45:31 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:45:31.973+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:09.306551+0000 front 2023-12-16T21:39:09.306652+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:45:32.317 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:45:31 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:45:31.973+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:09.306605+0000 front 2023-12-16T21:39:09.306613+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:45:32.317 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:45:31 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:45:31.974+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:45:32.317 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:45:31 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:45:31.912+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:05.622397+0000 front 2023-12-16T21:39:05.622556+0000 (oldest deadline 2023-12-16T21:39:30.322546+0000) 2023-12-16T21:45:32.317 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:45:31 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:45:31.912+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:05.622506+0000 front 2023-12-16T21:39:05.622528+0000 (oldest deadline 2023-12-16T21:39:30.322546+0000) 2023-12-16T21:45:32.317 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:45:31 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:45:31.912+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:05.622479+0000 front 2023-12-16T21:39:05.622580+0000 (oldest deadline 2023-12-16T21:39:30.322546+0000) 2023-12-16T21:45:32.317 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:45:31 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:45:31.913+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:45:32.816 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:45:32 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:45:32.433+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:09.828911+0000 front 2023-12-16T21:39:09.828958+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:45:32.816 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:45:32 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:45:32.433+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:09.828868+0000 front 2023-12-16T21:39:09.828878+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:45:32.817 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:45:32 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:45:32.433+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:09.828947+0000 front 2023-12-16T21:39:09.828993+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:45:32.817 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:45:32 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:45:32.434+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:45:33.201 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:45:32 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:45:32.937+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:05.622397+0000 front 2023-12-16T21:39:05.622556+0000 (oldest deadline 2023-12-16T21:39:30.322546+0000) 2023-12-16T21:45:33.201 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:45:32 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:45:32.937+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:05.622506+0000 front 2023-12-16T21:39:05.622528+0000 (oldest deadline 2023-12-16T21:39:30.322546+0000) 2023-12-16T21:45:33.201 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:45:32 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:45:32.937+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:05.622479+0000 front 2023-12-16T21:39:05.622580+0000 (oldest deadline 2023-12-16T21:39:30.322546+0000) 2023-12-16T21:45:33.201 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:45:32 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:45:32.938+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:45:33.202 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:45:32 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:45:32.997+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:09.306583+0000 front 2023-12-16T21:39:09.306633+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:45:33.202 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:45:32 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:45:32.997+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:09.306551+0000 front 2023-12-16T21:39:09.306652+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:45:33.202 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:45:32 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:45:32.997+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:09.306605+0000 front 2023-12-16T21:39:09.306613+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:45:33.202 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:45:32 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:45:32.997+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:45:33.537 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:45:33 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:45:33.443+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:09.828911+0000 front 2023-12-16T21:39:09.828958+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:45:33.537 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:45:33 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:45:33.443+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:09.828868+0000 front 2023-12-16T21:39:09.828878+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:45:33.537 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:45:33 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:45:33.443+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:09.828947+0000 front 2023-12-16T21:39:09.828993+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:45:33.537 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:45:33 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:45:33.443+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:45:34.316 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:45:33 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:45:33.996+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:09.306583+0000 front 2023-12-16T21:39:09.306633+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:45:34.317 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:45:33 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:45:33.996+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:09.306551+0000 front 2023-12-16T21:39:09.306652+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:45:34.317 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:45:33 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:45:33.996+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:09.306605+0000 front 2023-12-16T21:39:09.306613+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:45:34.317 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:45:33 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:45:33.998+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:45:34.317 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:45:33 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:45:33.907+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:05.622397+0000 front 2023-12-16T21:39:05.622556+0000 (oldest deadline 2023-12-16T21:39:30.322546+0000) 2023-12-16T21:45:34.317 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:45:33 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:45:33.907+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:05.622506+0000 front 2023-12-16T21:39:05.622528+0000 (oldest deadline 2023-12-16T21:39:30.322546+0000) 2023-12-16T21:45:34.317 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:45:33 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:45:33.907+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:05.622479+0000 front 2023-12-16T21:39:05.622580+0000 (oldest deadline 2023-12-16T21:39:30.322546+0000) 2023-12-16T21:45:34.317 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:45:33 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:45:33.907+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:45:34.816 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:45:34 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:45:34.399+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:09.828911+0000 front 2023-12-16T21:39:09.828958+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:45:34.816 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:45:34 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:45:34.399+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:09.828868+0000 front 2023-12-16T21:39:09.828878+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:45:34.817 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:45:34 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:45:34.399+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:09.828947+0000 front 2023-12-16T21:39:09.828993+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:45:34.817 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:45:34 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:45:34.400+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:45:35.316 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:45:35 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:45:35.035+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:09.306583+0000 front 2023-12-16T21:39:09.306633+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:45:35.317 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:45:35 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:45:35.035+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:09.306551+0000 front 2023-12-16T21:39:09.306652+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:45:35.317 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:45:35 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:45:35.035+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:09.306605+0000 front 2023-12-16T21:39:09.306613+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:45:35.317 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:45:35 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:45:35.035+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:45:35.317 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:45:34 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:45:34.937+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:05.622397+0000 front 2023-12-16T21:39:05.622556+0000 (oldest deadline 2023-12-16T21:39:30.322546+0000) 2023-12-16T21:45:35.317 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:45:34 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:45:34.937+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:05.622506+0000 front 2023-12-16T21:39:05.622528+0000 (oldest deadline 2023-12-16T21:39:30.322546+0000) 2023-12-16T21:45:35.318 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:45:34 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:45:34.937+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:05.622479+0000 front 2023-12-16T21:39:05.622580+0000 (oldest deadline 2023-12-16T21:39:30.322546+0000) 2023-12-16T21:45:35.318 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:45:34 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:45:34.938+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:45:35.816 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:45:35 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:45:35.413+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:09.828911+0000 front 2023-12-16T21:39:09.828958+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:45:35.817 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:45:35 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:45:35.413+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:09.828868+0000 front 2023-12-16T21:39:09.828878+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:45:35.817 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:45:35 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:45:35.413+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:09.828947+0000 front 2023-12-16T21:39:09.828993+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:45:35.817 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:45:35 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:45:35.414+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:45:36.273 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:45:36 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:45:36.055+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:09.306583+0000 front 2023-12-16T21:39:09.306633+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:45:36.273 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:45:36 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:45:36.055+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:09.306551+0000 front 2023-12-16T21:39:09.306652+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:45:36.273 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:45:36 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:45:36.055+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:09.306605+0000 front 2023-12-16T21:39:09.306613+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:45:36.273 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:45:36 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:45:36.056+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:45:36.273 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:45:35 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:45:35.944+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:05.622397+0000 front 2023-12-16T21:39:05.622556+0000 (oldest deadline 2023-12-16T21:39:30.322546+0000) 2023-12-16T21:45:36.274 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:45:35 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:45:35.944+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:05.622506+0000 front 2023-12-16T21:39:05.622528+0000 (oldest deadline 2023-12-16T21:39:30.322546+0000) 2023-12-16T21:45:36.274 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:45:35 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:45:35.944+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:05.622479+0000 front 2023-12-16T21:39:05.622580+0000 (oldest deadline 2023-12-16T21:39:30.322546+0000) 2023-12-16T21:45:36.274 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:45:35 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:45:35.944+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:45:36.566 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Dec 16 21:45:36 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-mon-smithi045[80759]: 2023-12-16T21:45:36.358+0000 7f45a1f8f700 -1 mon.smithi045@0(probing) e2 get_health_metrics reporting 3143 slow ops, oldest is monmgrreport(gid 14214, 0 checks, 1 progress events) 2023-12-16T21:45:36.567 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:45:36 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:45:36.463+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:09.828911+0000 front 2023-12-16T21:39:09.828958+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:45:36.567 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:45:36 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:45:36.463+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:09.828868+0000 front 2023-12-16T21:39:09.828878+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:45:36.567 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:45:36 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:45:36.463+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:09.828947+0000 front 2023-12-16T21:39:09.828993+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:45:36.567 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:45:36 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:45:36.464+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:45:37.316 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:45:36 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:45:36.955+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:05.622397+0000 front 2023-12-16T21:39:05.622556+0000 (oldest deadline 2023-12-16T21:39:30.322546+0000) 2023-12-16T21:45:37.317 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:45:36 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:45:36.955+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:05.622506+0000 front 2023-12-16T21:39:05.622528+0000 (oldest deadline 2023-12-16T21:39:30.322546+0000) 2023-12-16T21:45:37.317 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:45:36 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:45:36.955+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:05.622479+0000 front 2023-12-16T21:39:05.622580+0000 (oldest deadline 2023-12-16T21:39:30.322546+0000) 2023-12-16T21:45:37.317 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:45:36 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:45:36.956+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:45:37.317 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:45:37 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:45:37.019+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:09.306583+0000 front 2023-12-16T21:39:09.306633+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:45:37.317 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:45:37 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:45:37.019+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:09.306551+0000 front 2023-12-16T21:39:09.306652+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:45:37.317 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:45:37 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:45:37.019+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:09.306605+0000 front 2023-12-16T21:39:09.306613+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:45:37.317 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:45:37 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:45:37.020+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:45:37.816 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:45:37 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:45:37.474+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:09.828911+0000 front 2023-12-16T21:39:09.828958+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:45:37.817 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:45:37 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:45:37.474+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:09.828868+0000 front 2023-12-16T21:39:09.828878+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:45:37.817 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:45:37 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:45:37.474+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:09.828947+0000 front 2023-12-16T21:39:09.828993+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:45:37.817 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:45:37 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:45:37.474+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:45:38.316 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:45:38 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:45:38.023+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:09.306583+0000 front 2023-12-16T21:39:09.306633+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:45:38.317 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:45:38 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:45:38.023+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:09.306551+0000 front 2023-12-16T21:39:09.306652+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:45:38.317 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:45:38 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:45:38.023+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:09.306605+0000 front 2023-12-16T21:39:09.306613+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:45:38.317 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:45:38 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:45:38.024+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:45:38.317 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:45:37 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:45:37.909+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:05.622397+0000 front 2023-12-16T21:39:05.622556+0000 (oldest deadline 2023-12-16T21:39:30.322546+0000) 2023-12-16T21:45:38.317 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:45:37 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:45:37.909+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:05.622506+0000 front 2023-12-16T21:39:05.622528+0000 (oldest deadline 2023-12-16T21:39:30.322546+0000) 2023-12-16T21:45:38.317 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:45:37 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:45:37.909+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:05.622479+0000 front 2023-12-16T21:39:05.622580+0000 (oldest deadline 2023-12-16T21:39:30.322546+0000) 2023-12-16T21:45:38.317 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:45:37 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:45:37.910+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:45:38.723 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:45:38 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:45:38.425+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:09.828911+0000 front 2023-12-16T21:39:09.828958+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:45:38.723 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:45:38 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:45:38.425+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:09.828868+0000 front 2023-12-16T21:39:09.828878+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:45:38.723 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:45:38 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:45:38.425+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:09.828947+0000 front 2023-12-16T21:39:09.828993+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:45:38.723 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:45:38 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:45:38.426+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:45:39.038 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:45:38 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:45:38.909+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:05.622397+0000 front 2023-12-16T21:39:05.622556+0000 (oldest deadline 2023-12-16T21:39:30.322546+0000) 2023-12-16T21:45:39.038 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:45:38 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:45:38.909+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:05.622506+0000 front 2023-12-16T21:39:05.622528+0000 (oldest deadline 2023-12-16T21:39:30.322546+0000) 2023-12-16T21:45:39.038 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:45:38 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:45:38.909+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:05.622479+0000 front 2023-12-16T21:39:05.622580+0000 (oldest deadline 2023-12-16T21:39:30.322546+0000) 2023-12-16T21:45:39.038 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:45:38 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:45:38.910+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:45:39.316 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:45:39 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:45:39.034+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:09.306583+0000 front 2023-12-16T21:39:09.306633+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:45:39.316 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:45:39 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:45:39.034+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:09.306551+0000 front 2023-12-16T21:39:09.306652+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:45:39.317 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:45:39 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:45:39.034+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:09.306605+0000 front 2023-12-16T21:39:09.306613+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:45:39.317 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:45:39 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:45:39.036+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:45:39.816 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:45:39 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:45:39.450+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:09.828911+0000 front 2023-12-16T21:39:09.828958+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:45:39.816 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:45:39 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:45:39.450+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:09.828868+0000 front 2023-12-16T21:39:09.828878+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:45:39.817 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:45:39 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:45:39.450+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:09.828947+0000 front 2023-12-16T21:39:09.828993+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:45:39.817 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:45:39 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:45:39.451+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:45:40.316 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:45:39 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:45:39.909+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:05.622397+0000 front 2023-12-16T21:39:05.622556+0000 (oldest deadline 2023-12-16T21:39:30.322546+0000) 2023-12-16T21:45:40.316 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:45:39 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:45:39.909+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:05.622506+0000 front 2023-12-16T21:39:05.622528+0000 (oldest deadline 2023-12-16T21:39:30.322546+0000) 2023-12-16T21:45:40.317 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:45:39 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:45:39.909+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:05.622479+0000 front 2023-12-16T21:39:05.622580+0000 (oldest deadline 2023-12-16T21:39:30.322546+0000) 2023-12-16T21:45:40.317 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:45:39 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:45:39.910+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:45:40.317 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:45:40 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:45:40.029+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:09.306583+0000 front 2023-12-16T21:39:09.306633+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:45:40.317 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:45:40 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:45:40.029+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:09.306551+0000 front 2023-12-16T21:39:09.306652+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:45:40.317 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:45:40 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:45:40.029+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:09.306605+0000 front 2023-12-16T21:39:09.306613+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:45:40.317 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:45:40 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:45:40.030+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:45:40.816 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:45:40 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:45:40.449+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:09.828911+0000 front 2023-12-16T21:39:09.828958+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:45:40.817 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:45:40 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:45:40.449+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:09.828868+0000 front 2023-12-16T21:39:09.828878+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:45:40.817 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:45:40 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:45:40.449+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:09.828947+0000 front 2023-12-16T21:39:09.828993+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:45:40.817 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:45:40 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:45:40.450+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:45:41.316 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:45:41 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:45:41.030+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:09.306583+0000 front 2023-12-16T21:39:09.306633+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:45:41.317 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:45:41 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:45:41.030+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:09.306551+0000 front 2023-12-16T21:39:09.306652+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:45:41.317 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:45:41 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:45:41.030+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:09.306605+0000 front 2023-12-16T21:39:09.306613+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:45:41.317 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:45:41 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:45:41.031+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:45:41.317 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:45:40 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:45:40.885+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:05.622397+0000 front 2023-12-16T21:39:05.622556+0000 (oldest deadline 2023-12-16T21:39:30.322546+0000) 2023-12-16T21:45:41.317 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:45:40 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:45:40.885+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:05.622506+0000 front 2023-12-16T21:39:05.622528+0000 (oldest deadline 2023-12-16T21:39:30.322546+0000) 2023-12-16T21:45:41.317 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:45:40 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:45:40.885+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:05.622479+0000 front 2023-12-16T21:39:05.622580+0000 (oldest deadline 2023-12-16T21:39:30.322546+0000) 2023-12-16T21:45:41.317 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:45:40 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:45:40.886+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:45:41.646 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Dec 16 21:45:41 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-mon-smithi045[80759]: 2023-12-16T21:45:41.359+0000 7f45a1f8f700 -1 mon.smithi045@0(probing) e2 get_health_metrics reporting 3239 slow ops, oldest is monmgrreport(gid 14214, 0 checks, 1 progress events) 2023-12-16T21:45:41.646 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:45:41 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:45:41.463+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:09.828911+0000 front 2023-12-16T21:39:09.828958+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:45:41.646 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:45:41 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:45:41.463+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:09.828868+0000 front 2023-12-16T21:39:09.828878+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:45:41.647 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:45:41 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:45:41.463+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:09.828947+0000 front 2023-12-16T21:39:09.828993+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:45:41.647 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:45:41 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:45:41.464+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:45:42.066 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:45:41 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:45:41.854+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:05.622397+0000 front 2023-12-16T21:39:05.622556+0000 (oldest deadline 2023-12-16T21:39:30.322546+0000) 2023-12-16T21:45:42.066 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:45:41 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:45:41.854+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:05.622506+0000 front 2023-12-16T21:39:05.622528+0000 (oldest deadline 2023-12-16T21:39:30.322546+0000) 2023-12-16T21:45:42.066 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:45:41 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:45:41.854+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:05.622479+0000 front 2023-12-16T21:39:05.622580+0000 (oldest deadline 2023-12-16T21:39:30.322546+0000) 2023-12-16T21:45:42.067 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:45:41 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:45:41.855+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:45:42.417 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:45:42 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:45:42.065+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:09.306583+0000 front 2023-12-16T21:39:09.306633+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:45:42.417 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:45:42 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:45:42.065+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:09.306551+0000 front 2023-12-16T21:39:09.306652+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:45:42.417 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:45:42 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:45:42.065+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:09.306605+0000 front 2023-12-16T21:39:09.306613+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:45:42.417 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:45:42 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:45:42.066+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:45:42.816 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:45:42 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:45:42.505+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:09.828911+0000 front 2023-12-16T21:39:09.828958+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:45:42.816 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:45:42 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:45:42.505+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:09.828868+0000 front 2023-12-16T21:39:09.828878+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:45:42.817 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:45:42 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:45:42.505+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:09.828947+0000 front 2023-12-16T21:39:09.828993+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:45:42.817 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:45:42 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:45:42.506+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:45:43.316 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:45:42 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:45:42.885+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:05.622397+0000 front 2023-12-16T21:39:05.622556+0000 (oldest deadline 2023-12-16T21:39:30.322546+0000) 2023-12-16T21:45:43.317 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:45:42 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:45:42.885+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:05.622506+0000 front 2023-12-16T21:39:05.622528+0000 (oldest deadline 2023-12-16T21:39:30.322546+0000) 2023-12-16T21:45:43.317 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:45:42 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:45:42.885+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:05.622479+0000 front 2023-12-16T21:39:05.622580+0000 (oldest deadline 2023-12-16T21:39:30.322546+0000) 2023-12-16T21:45:43.317 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:45:42 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:45:42.886+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:45:43.317 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:45:43 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:45:43.070+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:09.306583+0000 front 2023-12-16T21:39:09.306633+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:45:43.317 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:45:43 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:45:43.070+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:09.306551+0000 front 2023-12-16T21:39:09.306652+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:45:43.317 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:45:43 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:45:43.070+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:09.306605+0000 front 2023-12-16T21:39:09.306613+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:45:43.317 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:45:43 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:45:43.071+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:45:43.816 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:45:43 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:45:43.549+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:09.828911+0000 front 2023-12-16T21:39:09.828958+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:45:43.817 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:45:43 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:45:43.549+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:09.828868+0000 front 2023-12-16T21:39:09.828878+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:45:43.817 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:45:43 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:45:43.549+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:09.828947+0000 front 2023-12-16T21:39:09.828993+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:45:43.817 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:45:43 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:45:43.550+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:45:44.316 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:45:44 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:45:44.112+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:09.306583+0000 front 2023-12-16T21:39:09.306633+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:45:44.317 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:45:44 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:45:44.112+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:09.306551+0000 front 2023-12-16T21:39:09.306652+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:45:44.317 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:45:44 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:45:44.112+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:09.306605+0000 front 2023-12-16T21:39:09.306613+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:45:44.317 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:45:44 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:45:44.113+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:45:44.317 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:45:43 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:45:43.930+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:05.622397+0000 front 2023-12-16T21:39:05.622556+0000 (oldest deadline 2023-12-16T21:39:30.322546+0000) 2023-12-16T21:45:44.317 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:45:43 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:45:43.930+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:05.622506+0000 front 2023-12-16T21:39:05.622528+0000 (oldest deadline 2023-12-16T21:39:30.322546+0000) 2023-12-16T21:45:44.317 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:45:43 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:45:43.930+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:05.622479+0000 front 2023-12-16T21:39:05.622580+0000 (oldest deadline 2023-12-16T21:39:30.322546+0000) 2023-12-16T21:45:44.317 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:45:43 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:45:43.931+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:45:44.816 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:45:44 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:45:44.512+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:09.828911+0000 front 2023-12-16T21:39:09.828958+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:45:44.817 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:45:44 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:45:44.512+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:09.828868+0000 front 2023-12-16T21:39:09.828878+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:45:44.817 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:45:44 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:45:44.512+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:09.828947+0000 front 2023-12-16T21:39:09.828993+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:45:44.817 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:45:44 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:45:44.514+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:45:45.316 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:45:44 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:45:44.908+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:05.622397+0000 front 2023-12-16T21:39:05.622556+0000 (oldest deadline 2023-12-16T21:39:30.322546+0000) 2023-12-16T21:45:45.316 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:45:44 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:45:44.908+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:05.622506+0000 front 2023-12-16T21:39:05.622528+0000 (oldest deadline 2023-12-16T21:39:30.322546+0000) 2023-12-16T21:45:45.317 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:45:44 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:45:44.908+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:05.622479+0000 front 2023-12-16T21:39:05.622580+0000 (oldest deadline 2023-12-16T21:39:30.322546+0000) 2023-12-16T21:45:45.317 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:45:44 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:45:44.909+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:45:45.317 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:45:45 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:45:45.075+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:09.306583+0000 front 2023-12-16T21:39:09.306633+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:45:45.317 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:45:45 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:45:45.075+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:09.306551+0000 front 2023-12-16T21:39:09.306652+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:45:45.317 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:45:45 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:45:45.075+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:09.306605+0000 front 2023-12-16T21:39:09.306613+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:45:45.317 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:45:45 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:45:45.076+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:45:45.816 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:45:45 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:45:45.477+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:09.828911+0000 front 2023-12-16T21:39:09.828958+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:45:45.817 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:45:45 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:45:45.477+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:09.828868+0000 front 2023-12-16T21:39:09.828878+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:45:45.817 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:45:45 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:45:45.477+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:09.828947+0000 front 2023-12-16T21:39:09.828993+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:45:45.817 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:45:45 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:45:45.478+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:45:46.317 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:45:46 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:45:46.064+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:09.306583+0000 front 2023-12-16T21:39:09.306633+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:45:46.317 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:45:46 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:45:46.064+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:09.306551+0000 front 2023-12-16T21:39:09.306652+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:45:46.317 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:45:46 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:45:46.064+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:09.306605+0000 front 2023-12-16T21:39:09.306613+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:45:46.317 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:45:46 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:45:46.065+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:45:46.318 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:45:45 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:45:45.889+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:05.622397+0000 front 2023-12-16T21:39:05.622556+0000 (oldest deadline 2023-12-16T21:39:30.322546+0000) 2023-12-16T21:45:46.318 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:45:45 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:45:45.889+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:05.622506+0000 front 2023-12-16T21:39:05.622528+0000 (oldest deadline 2023-12-16T21:39:30.322546+0000) 2023-12-16T21:45:46.318 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:45:45 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:45:45.889+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:05.622479+0000 front 2023-12-16T21:39:05.622580+0000 (oldest deadline 2023-12-16T21:39:30.322546+0000) 2023-12-16T21:45:46.318 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:45:45 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:45:45.889+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:45:46.816 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Dec 16 21:45:46 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-mon-smithi045[80759]: 2023-12-16T21:45:46.359+0000 7f45a1f8f700 -1 mon.smithi045@0(probing) e2 get_health_metrics reporting 3333 slow ops, oldest is monmgrreport(gid 14214, 0 checks, 1 progress events) 2023-12-16T21:45:46.817 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:45:46 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:45:46.438+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:09.828911+0000 front 2023-12-16T21:39:09.828958+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:45:46.817 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:45:46 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:45:46.438+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:09.828868+0000 front 2023-12-16T21:39:09.828878+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:45:46.817 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:45:46 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:45:46.438+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:09.828947+0000 front 2023-12-16T21:39:09.828993+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:45:46.817 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:45:46 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:45:46.439+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:45:47.316 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:45:47 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:45:47.054+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:09.306583+0000 front 2023-12-16T21:39:09.306633+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:45:47.317 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:45:47 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:45:47.054+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:09.306551+0000 front 2023-12-16T21:39:09.306652+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:45:47.317 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:45:47 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:45:47.054+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:09.306605+0000 front 2023-12-16T21:39:09.306613+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:45:47.317 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:45:47 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:45:47.055+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:45:47.317 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:45:46 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:45:46.931+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:05.622397+0000 front 2023-12-16T21:39:05.622556+0000 (oldest deadline 2023-12-16T21:39:30.322546+0000) 2023-12-16T21:45:47.317 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:45:46 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:45:46.931+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:05.622506+0000 front 2023-12-16T21:39:05.622528+0000 (oldest deadline 2023-12-16T21:39:30.322546+0000) 2023-12-16T21:45:47.317 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:45:46 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:45:46.931+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:05.622479+0000 front 2023-12-16T21:39:05.622580+0000 (oldest deadline 2023-12-16T21:39:30.322546+0000) 2023-12-16T21:45:47.317 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:45:46 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:45:46.931+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:45:47.817 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:45:47 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:45:47.420+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:09.828911+0000 front 2023-12-16T21:39:09.828958+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:45:47.817 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:45:47 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:45:47.420+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:09.828868+0000 front 2023-12-16T21:39:09.828878+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:45:47.817 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:45:47 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:45:47.420+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:09.828947+0000 front 2023-12-16T21:39:09.828993+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:45:47.817 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:45:47 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:45:47.421+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:45:48.317 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:45:48 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:45:48.031+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:09.306583+0000 front 2023-12-16T21:39:09.306633+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:45:48.317 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:45:48 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:45:48.031+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:09.306551+0000 front 2023-12-16T21:39:09.306652+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:45:48.317 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:45:48 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:45:48.031+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:09.306605+0000 front 2023-12-16T21:39:09.306613+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:45:48.318 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:45:48 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:45:48.032+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:45:48.318 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:45:47 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:45:47.904+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:05.622397+0000 front 2023-12-16T21:39:05.622556+0000 (oldest deadline 2023-12-16T21:39:30.322546+0000) 2023-12-16T21:45:48.318 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:45:47 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:45:47.904+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:05.622506+0000 front 2023-12-16T21:39:05.622528+0000 (oldest deadline 2023-12-16T21:39:30.322546+0000) 2023-12-16T21:45:48.318 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:45:47 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:45:47.904+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:05.622479+0000 front 2023-12-16T21:39:05.622580+0000 (oldest deadline 2023-12-16T21:39:30.322546+0000) 2023-12-16T21:45:48.318 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:45:47 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:45:47.905+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:45:48.723 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:45:48 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:45:48.403+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:09.828911+0000 front 2023-12-16T21:39:09.828958+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:45:48.723 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:45:48 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:45:48.403+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:09.828868+0000 front 2023-12-16T21:39:09.828878+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:45:48.723 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:45:48 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:45:48.403+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:09.828947+0000 front 2023-12-16T21:39:09.828993+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:45:48.723 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:45:48 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:45:48.404+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:45:48.993 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:45:48 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:45:48.857+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:05.622397+0000 front 2023-12-16T21:39:05.622556+0000 (oldest deadline 2023-12-16T21:39:30.322546+0000) 2023-12-16T21:45:48.993 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:45:48 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:45:48.857+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:05.622506+0000 front 2023-12-16T21:39:05.622528+0000 (oldest deadline 2023-12-16T21:39:30.322546+0000) 2023-12-16T21:45:48.993 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:45:48 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:45:48.857+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:05.622479+0000 front 2023-12-16T21:39:05.622580+0000 (oldest deadline 2023-12-16T21:39:30.322546+0000) 2023-12-16T21:45:48.993 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:45:48 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:45:48.858+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:45:49.316 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:45:48 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:45:48.989+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:09.306583+0000 front 2023-12-16T21:39:09.306633+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:45:49.316 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:45:48 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:45:48.989+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:09.306551+0000 front 2023-12-16T21:39:09.306652+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:45:49.317 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:45:48 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:45:48.989+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:09.306605+0000 front 2023-12-16T21:39:09.306613+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:45:49.317 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:45:48 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:45:48.990+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:45:49.816 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:45:49 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:45:49.421+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:09.828911+0000 front 2023-12-16T21:39:09.828958+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:45:49.816 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:45:49 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:45:49.421+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:09.828868+0000 front 2023-12-16T21:39:09.828878+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:45:49.816 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:45:49 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:45:49.421+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:09.828947+0000 front 2023-12-16T21:39:09.828993+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:45:49.817 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:45:49 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:45:49.422+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:45:50.316 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:45:49 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:45:49.961+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:09.306583+0000 front 2023-12-16T21:39:09.306633+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:45:50.316 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:45:49 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:45:49.961+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:09.306551+0000 front 2023-12-16T21:39:09.306652+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:45:50.317 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:45:49 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:45:49.961+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:09.306605+0000 front 2023-12-16T21:39:09.306613+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:45:50.317 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:45:49 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:45:49.962+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:45:50.317 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:45:49 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:45:49.863+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:05.622397+0000 front 2023-12-16T21:39:05.622556+0000 (oldest deadline 2023-12-16T21:39:30.322546+0000) 2023-12-16T21:45:50.317 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:45:49 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:45:49.863+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:05.622506+0000 front 2023-12-16T21:39:05.622528+0000 (oldest deadline 2023-12-16T21:39:30.322546+0000) 2023-12-16T21:45:50.317 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:45:49 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:45:49.863+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:05.622479+0000 front 2023-12-16T21:39:05.622580+0000 (oldest deadline 2023-12-16T21:39:30.322546+0000) 2023-12-16T21:45:50.317 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:45:49 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:45:49.864+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:45:50.816 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:45:50 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:45:50.419+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:09.828911+0000 front 2023-12-16T21:39:09.828958+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:45:50.817 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:45:50 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:45:50.419+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:09.828868+0000 front 2023-12-16T21:39:09.828878+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:45:50.817 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:45:50 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:45:50.419+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:09.828947+0000 front 2023-12-16T21:39:09.828993+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:45:50.817 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:45:50 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:45:50.420+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:45:51.317 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:45:50 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:45:50.889+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:05.622397+0000 front 2023-12-16T21:39:05.622556+0000 (oldest deadline 2023-12-16T21:39:30.322546+0000) 2023-12-16T21:45:51.317 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:45:50 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:45:50.889+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:05.622506+0000 front 2023-12-16T21:39:05.622528+0000 (oldest deadline 2023-12-16T21:39:30.322546+0000) 2023-12-16T21:45:51.317 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:45:50 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:45:50.889+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:05.622479+0000 front 2023-12-16T21:39:05.622580+0000 (oldest deadline 2023-12-16T21:39:30.322546+0000) 2023-12-16T21:45:51.317 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:45:50 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:45:50.890+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:45:51.317 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:45:50 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:45:50.934+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:09.306583+0000 front 2023-12-16T21:39:09.306633+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:45:51.318 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:45:50 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:45:50.934+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:09.306551+0000 front 2023-12-16T21:39:09.306652+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:45:51.318 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:45:50 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:45:50.934+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:09.306605+0000 front 2023-12-16T21:39:09.306613+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:45:51.318 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:45:50 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:45:50.934+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:45:51.647 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Dec 16 21:45:51 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-mon-smithi045[80759]: 2023-12-16T21:45:51.360+0000 7f45a1f8f700 -1 mon.smithi045@0(probing) e2 get_health_metrics reporting 3429 slow ops, oldest is monmgrreport(gid 14214, 0 checks, 1 progress events) 2023-12-16T21:45:51.647 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:45:51 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:45:51.466+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:09.828911+0000 front 2023-12-16T21:39:09.828958+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:45:51.647 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:45:51 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:45:51.466+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:09.828868+0000 front 2023-12-16T21:39:09.828878+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:45:51.647 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:45:51 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:45:51.466+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:09.828947+0000 front 2023-12-16T21:39:09.828993+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:45:51.647 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:45:51 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:45:51.466+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:45:52.316 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:45:51 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:45:51.983+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:09.306583+0000 front 2023-12-16T21:39:09.306633+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:45:52.316 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:45:51 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:45:51.983+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:09.306551+0000 front 2023-12-16T21:39:09.306652+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:45:52.317 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:45:51 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:45:51.983+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:09.306605+0000 front 2023-12-16T21:39:09.306613+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:45:52.317 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:45:51 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:45:51.984+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:45:52.317 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:45:51 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:45:51.901+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:05.622397+0000 front 2023-12-16T21:39:05.622556+0000 (oldest deadline 2023-12-16T21:39:30.322546+0000) 2023-12-16T21:45:52.317 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:45:51 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:45:51.901+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:05.622506+0000 front 2023-12-16T21:39:05.622528+0000 (oldest deadline 2023-12-16T21:39:30.322546+0000) 2023-12-16T21:45:52.317 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:45:51 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:45:51.901+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:05.622479+0000 front 2023-12-16T21:39:05.622580+0000 (oldest deadline 2023-12-16T21:39:30.322546+0000) 2023-12-16T21:45:52.317 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:45:51 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:45:51.902+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:45:52.816 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:45:52 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:45:52.504+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:09.828911+0000 front 2023-12-16T21:39:09.828958+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:45:52.816 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:45:52 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:45:52.504+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:09.828868+0000 front 2023-12-16T21:39:09.828878+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:45:52.817 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:45:52 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:45:52.504+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:09.828947+0000 front 2023-12-16T21:39:09.828993+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:45:52.817 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:45:52 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:45:52.505+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:45:53.316 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:45:52 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:45:52.936+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:09.306583+0000 front 2023-12-16T21:39:09.306633+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:45:53.316 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:45:52 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:45:52.936+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:09.306551+0000 front 2023-12-16T21:39:09.306652+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:45:53.317 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:45:52 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:45:52.936+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:09.306605+0000 front 2023-12-16T21:39:09.306613+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:45:53.317 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:45:52 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:45:52.937+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:45:53.317 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:45:52 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:45:52.898+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:05.622397+0000 front 2023-12-16T21:39:05.622556+0000 (oldest deadline 2023-12-16T21:39:30.322546+0000) 2023-12-16T21:45:53.317 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:45:52 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:45:52.898+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:05.622506+0000 front 2023-12-16T21:39:05.622528+0000 (oldest deadline 2023-12-16T21:39:30.322546+0000) 2023-12-16T21:45:53.317 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:45:52 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:45:52.898+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:05.622479+0000 front 2023-12-16T21:39:05.622580+0000 (oldest deadline 2023-12-16T21:39:30.322546+0000) 2023-12-16T21:45:53.317 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:45:52 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:45:52.899+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:45:53.816 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:45:53 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:45:53.532+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:09.828911+0000 front 2023-12-16T21:39:09.828958+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:45:53.817 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:45:53 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:45:53.532+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:09.828868+0000 front 2023-12-16T21:39:09.828878+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:45:53.817 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:45:53 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:45:53.532+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:09.828947+0000 front 2023-12-16T21:39:09.828993+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:45:53.817 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:45:53 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:45:53.533+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:45:54.316 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:45:53 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:45:53.896+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:09.306583+0000 front 2023-12-16T21:39:09.306633+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:45:54.316 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:45:53 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:45:53.896+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:09.306551+0000 front 2023-12-16T21:39:09.306652+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:45:54.317 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:45:53 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:45:53.896+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:09.306605+0000 front 2023-12-16T21:39:09.306613+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:45:54.317 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:45:53 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:45:53.897+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:45:54.317 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:45:53 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:45:53.946+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:05.622397+0000 front 2023-12-16T21:39:05.622556+0000 (oldest deadline 2023-12-16T21:39:30.322546+0000) 2023-12-16T21:45:54.317 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:45:53 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:45:53.946+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:05.622506+0000 front 2023-12-16T21:39:05.622528+0000 (oldest deadline 2023-12-16T21:39:30.322546+0000) 2023-12-16T21:45:54.317 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:45:53 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:45:53.946+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:05.622479+0000 front 2023-12-16T21:39:05.622580+0000 (oldest deadline 2023-12-16T21:39:30.322546+0000) 2023-12-16T21:45:54.317 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:45:53 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:45:53.947+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:45:54.898 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:45:54 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:45:54.576+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:09.828911+0000 front 2023-12-16T21:39:09.828958+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:45:54.899 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:45:54 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:45:54.576+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:09.828868+0000 front 2023-12-16T21:39:09.828878+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:45:54.899 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:45:54 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:45:54.576+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:09.828947+0000 front 2023-12-16T21:39:09.828993+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:45:54.899 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:45:54 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:45:54.577+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:45:55.316 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:45:54 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:45:54.935+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:05.622397+0000 front 2023-12-16T21:39:05.622556+0000 (oldest deadline 2023-12-16T21:39:30.322546+0000) 2023-12-16T21:45:55.317 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:45:54 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:45:54.935+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:05.622506+0000 front 2023-12-16T21:39:05.622528+0000 (oldest deadline 2023-12-16T21:39:30.322546+0000) 2023-12-16T21:45:55.317 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:45:54 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:45:54.935+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:05.622479+0000 front 2023-12-16T21:39:05.622580+0000 (oldest deadline 2023-12-16T21:39:30.322546+0000) 2023-12-16T21:45:55.317 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:45:54 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:45:54.936+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:45:55.317 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:45:54 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:45:54.895+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:09.306583+0000 front 2023-12-16T21:39:09.306633+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:45:55.317 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:45:54 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:45:54.895+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:09.306551+0000 front 2023-12-16T21:39:09.306652+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:45:55.317 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:45:54 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:45:54.895+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:09.306605+0000 front 2023-12-16T21:39:09.306613+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:45:55.317 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:45:54 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:45:54.896+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:45:55.872 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:45:55 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:45:55.604+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:09.828911+0000 front 2023-12-16T21:39:09.828958+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:45:55.872 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:45:55 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:45:55.604+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:09.828868+0000 front 2023-12-16T21:39:09.828878+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:45:55.872 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:45:55 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:45:55.604+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:09.828947+0000 front 2023-12-16T21:39:09.828993+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:45:55.872 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:45:55 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:45:55.606+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:45:56.317 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:45:55 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:45:55.972+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:05.622397+0000 front 2023-12-16T21:39:05.622556+0000 (oldest deadline 2023-12-16T21:39:30.322546+0000) 2023-12-16T21:45:56.317 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:45:55 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:45:55.972+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:05.622506+0000 front 2023-12-16T21:39:05.622528+0000 (oldest deadline 2023-12-16T21:39:30.322546+0000) 2023-12-16T21:45:56.317 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:45:55 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:45:55.972+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:05.622479+0000 front 2023-12-16T21:39:05.622580+0000 (oldest deadline 2023-12-16T21:39:30.322546+0000) 2023-12-16T21:45:56.317 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:45:55 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:45:55.973+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:45:56.317 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:45:55 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:45:55.869+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:09.306583+0000 front 2023-12-16T21:39:09.306633+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:45:56.317 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:45:55 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:45:55.869+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:09.306551+0000 front 2023-12-16T21:39:09.306652+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:45:56.317 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:45:55 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:45:55.869+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:09.306605+0000 front 2023-12-16T21:39:09.306613+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:45:56.317 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:45:55 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:45:55.870+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:45:56.816 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Dec 16 21:45:56 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-mon-smithi045[80759]: 2023-12-16T21:45:56.361+0000 7f45a1f8f700 -1 mon.smithi045@0(probing) e2 get_health_metrics reporting 3524 slow ops, oldest is monmgrreport(gid 14214, 0 checks, 1 progress events) 2023-12-16T21:45:56.817 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:45:56 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:45:56.589+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:09.828911+0000 front 2023-12-16T21:39:09.828958+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:45:56.817 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:45:56 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:45:56.589+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:09.828868+0000 front 2023-12-16T21:39:09.828878+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:45:56.817 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:45:56 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:45:56.589+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:09.828947+0000 front 2023-12-16T21:39:09.828993+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:45:56.817 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:45:56 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:45:56.590+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:45:57.316 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:45:56 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:45:56.865+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:09.306583+0000 front 2023-12-16T21:39:09.306633+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:45:57.317 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:45:56 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:45:56.865+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:09.306551+0000 front 2023-12-16T21:39:09.306652+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:45:57.317 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:45:56 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:45:56.865+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:09.306605+0000 front 2023-12-16T21:39:09.306613+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:45:57.317 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:45:56 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:45:56.866+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:45:57.317 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:45:56 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:45:56.931+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:05.622397+0000 front 2023-12-16T21:39:05.622556+0000 (oldest deadline 2023-12-16T21:39:30.322546+0000) 2023-12-16T21:45:57.317 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:45:56 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:45:56.931+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:05.622506+0000 front 2023-12-16T21:39:05.622528+0000 (oldest deadline 2023-12-16T21:39:30.322546+0000) 2023-12-16T21:45:57.317 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:45:56 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:45:56.931+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:05.622479+0000 front 2023-12-16T21:39:05.622580+0000 (oldest deadline 2023-12-16T21:39:30.322546+0000) 2023-12-16T21:45:57.317 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:45:56 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:45:56.932+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:45:57.891 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:45:57 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:45:57.579+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:09.828911+0000 front 2023-12-16T21:39:09.828958+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:45:57.891 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:45:57 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:45:57.579+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:09.828868+0000 front 2023-12-16T21:39:09.828878+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:45:57.891 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:45:57 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:45:57.579+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:09.828947+0000 front 2023-12-16T21:39:09.828993+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:45:57.891 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:45:57 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:45:57.580+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:45:58.316 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:45:57 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:45:57.905+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:05.622397+0000 front 2023-12-16T21:39:05.622556+0000 (oldest deadline 2023-12-16T21:39:30.322546+0000) 2023-12-16T21:45:58.316 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:45:57 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:45:57.905+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:05.622506+0000 front 2023-12-16T21:39:05.622528+0000 (oldest deadline 2023-12-16T21:39:30.322546+0000) 2023-12-16T21:45:58.317 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:45:57 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:45:57.905+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:05.622479+0000 front 2023-12-16T21:39:05.622580+0000 (oldest deadline 2023-12-16T21:39:30.322546+0000) 2023-12-16T21:45:58.317 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:45:57 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:45:57.906+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:45:58.317 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:45:57 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:45:57.888+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:09.306583+0000 front 2023-12-16T21:39:09.306633+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:45:58.317 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:45:57 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:45:57.888+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:09.306551+0000 front 2023-12-16T21:39:09.306652+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:45:58.317 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:45:57 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:45:57.888+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:09.306605+0000 front 2023-12-16T21:39:09.306613+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:45:58.317 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:45:57 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:45:57.889+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:45:58.883 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:45:58 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:45:58.581+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:09.828911+0000 front 2023-12-16T21:39:09.828958+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:45:58.884 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:45:58 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:45:58.581+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:09.828868+0000 front 2023-12-16T21:39:09.828878+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:45:58.884 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:45:58 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:45:58.581+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:09.828947+0000 front 2023-12-16T21:39:09.828993+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:45:58.884 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:45:58 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:45:58.582+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:45:59.316 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:45:58 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:45:58.897+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:05.622397+0000 front 2023-12-16T21:39:05.622556+0000 (oldest deadline 2023-12-16T21:39:30.322546+0000) 2023-12-16T21:45:59.316 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:45:58 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:45:58.897+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:05.622506+0000 front 2023-12-16T21:39:05.622528+0000 (oldest deadline 2023-12-16T21:39:30.322546+0000) 2023-12-16T21:45:59.317 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:45:58 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:45:58.897+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:05.622479+0000 front 2023-12-16T21:39:05.622580+0000 (oldest deadline 2023-12-16T21:39:30.322546+0000) 2023-12-16T21:45:59.317 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:45:58 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:45:58.898+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:45:59.317 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:45:58 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:45:58.880+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:09.306583+0000 front 2023-12-16T21:39:09.306633+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:45:59.317 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:45:58 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:45:58.880+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:09.306551+0000 front 2023-12-16T21:39:09.306652+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:45:59.317 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:45:58 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:45:58.880+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:09.306605+0000 front 2023-12-16T21:39:09.306613+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:45:59.317 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:45:58 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:45:58.881+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:45:59.816 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:45:59 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:45:59.534+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:09.828911+0000 front 2023-12-16T21:39:09.828958+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:45:59.816 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:45:59 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:45:59.534+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:09.828868+0000 front 2023-12-16T21:39:09.828878+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:45:59.817 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:45:59 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:45:59.534+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:09.828947+0000 front 2023-12-16T21:39:09.828993+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:45:59.817 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:45:59 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:45:59.534+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:46:00.316 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:45:59 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:45:59.870+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:05.622397+0000 front 2023-12-16T21:39:05.622556+0000 (oldest deadline 2023-12-16T21:39:30.322546+0000) 2023-12-16T21:46:00.316 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:45:59 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:45:59.870+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:05.622506+0000 front 2023-12-16T21:39:05.622528+0000 (oldest deadline 2023-12-16T21:39:30.322546+0000) 2023-12-16T21:46:00.317 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:45:59 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:45:59.870+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:05.622479+0000 front 2023-12-16T21:39:05.622580+0000 (oldest deadline 2023-12-16T21:39:30.322546+0000) 2023-12-16T21:46:00.317 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:45:59 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:45:59.871+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:46:00.317 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:45:59 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:45:59.883+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:09.306583+0000 front 2023-12-16T21:39:09.306633+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:46:00.317 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:45:59 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:45:59.883+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:09.306551+0000 front 2023-12-16T21:39:09.306652+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:46:00.317 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:45:59 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:45:59.883+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:09.306605+0000 front 2023-12-16T21:39:09.306613+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:46:00.317 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:45:59 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:45:59.884+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:46:00.816 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:46:00 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:46:00.513+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:09.828911+0000 front 2023-12-16T21:39:09.828958+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:46:00.817 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:46:00 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:46:00.513+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:09.828868+0000 front 2023-12-16T21:39:09.828878+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:46:00.817 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:46:00 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:46:00.513+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:09.828947+0000 front 2023-12-16T21:39:09.828993+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:46:00.817 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:46:00 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:46:00.514+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:46:01.316 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:46:00 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:46:00.844+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:09.306583+0000 front 2023-12-16T21:39:09.306633+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:46:01.317 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:46:00 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:46:00.844+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:09.306551+0000 front 2023-12-16T21:39:09.306652+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:46:01.317 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:46:00 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:46:00.844+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:09.306605+0000 front 2023-12-16T21:39:09.306613+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:46:01.317 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:46:00 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:46:00.845+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:46:01.317 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:46:00 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:46:00.906+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:05.622397+0000 front 2023-12-16T21:39:05.622556+0000 (oldest deadline 2023-12-16T21:39:30.322546+0000) 2023-12-16T21:46:01.317 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:46:00 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:46:00.906+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:05.622506+0000 front 2023-12-16T21:39:05.622528+0000 (oldest deadline 2023-12-16T21:39:30.322546+0000) 2023-12-16T21:46:01.317 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:46:00 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:46:00.906+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:05.622479+0000 front 2023-12-16T21:39:05.622580+0000 (oldest deadline 2023-12-16T21:39:30.322546+0000) 2023-12-16T21:46:01.317 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:46:00 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:46:00.908+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:46:01.648 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Dec 16 21:46:01 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-mon-smithi045[80759]: 2023-12-16T21:46:01.362+0000 7f45a1f8f700 -1 mon.smithi045@0(probing) e2 get_health_metrics reporting 3622 slow ops, oldest is monmgrreport(gid 14214, 0 checks, 1 progress events) 2023-12-16T21:46:01.648 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:46:01 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:46:01.480+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:09.828911+0000 front 2023-12-16T21:39:09.828958+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:46:01.648 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:46:01 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:46:01.480+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:09.828868+0000 front 2023-12-16T21:39:09.828878+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:46:01.648 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:46:01 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:46:01.480+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:09.828947+0000 front 2023-12-16T21:39:09.828993+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:46:01.648 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:46:01 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:46:01.482+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:46:02.066 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:46:01 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:46:01.891+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:09.306583+0000 front 2023-12-16T21:39:09.306633+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:46:02.067 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:46:01 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:46:01.891+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:09.306551+0000 front 2023-12-16T21:39:09.306652+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:46:02.067 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:46:01 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:46:01.891+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:09.306605+0000 front 2023-12-16T21:39:09.306613+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:46:02.067 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:46:01 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:46:01.893+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:46:02.067 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:46:01 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:46:01.862+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:05.622397+0000 front 2023-12-16T21:39:05.622556+0000 (oldest deadline 2023-12-16T21:39:30.322546+0000) 2023-12-16T21:46:02.067 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:46:01 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:46:01.862+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:05.622506+0000 front 2023-12-16T21:39:05.622528+0000 (oldest deadline 2023-12-16T21:39:30.322546+0000) 2023-12-16T21:46:02.067 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:46:01 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:46:01.862+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:05.622479+0000 front 2023-12-16T21:39:05.622580+0000 (oldest deadline 2023-12-16T21:39:30.322546+0000) 2023-12-16T21:46:02.067 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:46:01 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:46:01.862+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:46:02.816 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:46:02 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:46:02.498+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:09.828911+0000 front 2023-12-16T21:39:09.828958+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:46:02.816 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:46:02 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:46:02.498+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:09.828868+0000 front 2023-12-16T21:39:09.828878+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:46:02.817 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:46:02 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:46:02.498+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:09.828947+0000 front 2023-12-16T21:39:09.828993+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:46:02.817 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:46:02 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:46:02.498+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:46:03.316 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:46:02 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:46:02.817+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:05.622397+0000 front 2023-12-16T21:39:05.622556+0000 (oldest deadline 2023-12-16T21:39:30.322546+0000) 2023-12-16T21:46:03.317 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:46:02 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:46:02.817+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:05.622506+0000 front 2023-12-16T21:39:05.622528+0000 (oldest deadline 2023-12-16T21:39:30.322546+0000) 2023-12-16T21:46:03.317 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:46:02 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:46:02.817+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:05.622479+0000 front 2023-12-16T21:39:05.622580+0000 (oldest deadline 2023-12-16T21:39:30.322546+0000) 2023-12-16T21:46:03.317 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:46:02 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:46:02.818+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:46:03.317 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:46:02 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:46:02.940+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:09.306583+0000 front 2023-12-16T21:39:09.306633+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:46:03.317 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:46:02 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:46:02.940+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:09.306551+0000 front 2023-12-16T21:39:09.306652+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:46:03.317 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:46:02 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:46:02.940+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:09.306605+0000 front 2023-12-16T21:39:09.306613+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:46:03.317 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:46:02 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:46:02.941+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:46:03.816 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:46:03 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:46:03.481+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:09.828911+0000 front 2023-12-16T21:39:09.828958+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:46:03.816 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:46:03 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:46:03.481+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:09.828868+0000 front 2023-12-16T21:39:09.828878+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:46:03.816 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:46:03 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:46:03.481+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:09.828947+0000 front 2023-12-16T21:39:09.828993+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:46:03.816 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:46:03 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:46:03.482+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:46:04.066 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:46:03 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:46:03.914+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:09.306583+0000 front 2023-12-16T21:39:09.306633+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:46:04.066 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:46:03 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:46:03.914+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:09.306551+0000 front 2023-12-16T21:39:09.306652+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:46:04.067 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:46:03 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:46:03.914+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:09.306605+0000 front 2023-12-16T21:39:09.306613+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:46:04.067 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:46:03 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:46:03.915+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:46:04.067 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:46:03 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:46:03.813+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:05.622397+0000 front 2023-12-16T21:39:05.622556+0000 (oldest deadline 2023-12-16T21:39:30.322546+0000) 2023-12-16T21:46:04.067 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:46:03 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:46:03.813+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:05.622506+0000 front 2023-12-16T21:39:05.622528+0000 (oldest deadline 2023-12-16T21:39:30.322546+0000) 2023-12-16T21:46:04.067 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:46:03 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:46:03.813+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:05.622479+0000 front 2023-12-16T21:39:05.622580+0000 (oldest deadline 2023-12-16T21:39:30.322546+0000) 2023-12-16T21:46:04.067 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:46:03 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:46:03.814+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:46:04.803 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:46:04 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:46:04.526+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:09.828911+0000 front 2023-12-16T21:39:09.828958+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:46:04.803 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:46:04 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:46:04.526+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:09.828868+0000 front 2023-12-16T21:39:09.828878+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:46:04.803 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:46:04 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:46:04.526+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:09.828947+0000 front 2023-12-16T21:39:09.828993+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:46:04.803 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:46:04 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:46:04.527+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:46:05.066 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:46:04 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:46:04.887+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:09.306583+0000 front 2023-12-16T21:39:09.306633+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:46:05.066 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:46:04 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:46:04.887+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:09.306551+0000 front 2023-12-16T21:39:09.306652+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:46:05.067 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:46:04 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:46:04.887+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:09.306605+0000 front 2023-12-16T21:39:09.306613+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:46:05.067 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:46:04 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:46:04.888+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:46:05.067 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:46:04 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:46:04.800+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:05.622397+0000 front 2023-12-16T21:39:05.622556+0000 (oldest deadline 2023-12-16T21:39:30.322546+0000) 2023-12-16T21:46:05.067 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:46:04 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:46:04.800+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:05.622506+0000 front 2023-12-16T21:39:05.622528+0000 (oldest deadline 2023-12-16T21:39:30.322546+0000) 2023-12-16T21:46:05.067 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:46:04 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:46:04.800+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:05.622479+0000 front 2023-12-16T21:39:05.622580+0000 (oldest deadline 2023-12-16T21:39:30.322546+0000) 2023-12-16T21:46:05.067 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:46:04 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:46:04.801+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:46:05.816 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:46:05 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:46:05.505+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:09.828911+0000 front 2023-12-16T21:39:09.828958+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:46:05.817 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:46:05 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:46:05.505+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:09.828868+0000 front 2023-12-16T21:39:09.828878+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:46:05.817 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:46:05 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:46:05.505+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:09.828947+0000 front 2023-12-16T21:39:09.828993+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:46:05.817 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:46:05 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:46:05.507+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:46:06.316 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:46:05 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:46:05.875+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:09.306583+0000 front 2023-12-16T21:39:09.306633+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:46:06.317 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:46:05 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:46:05.875+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:09.306551+0000 front 2023-12-16T21:39:09.306652+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:46:06.317 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:46:05 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:46:05.875+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:09.306605+0000 front 2023-12-16T21:39:09.306613+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:46:06.317 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:46:05 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:46:05.876+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:46:06.317 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:46:05 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:46:05.837+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:05.622397+0000 front 2023-12-16T21:39:05.622556+0000 (oldest deadline 2023-12-16T21:39:30.322546+0000) 2023-12-16T21:46:06.317 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:46:05 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:46:05.837+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:05.622506+0000 front 2023-12-16T21:39:05.622528+0000 (oldest deadline 2023-12-16T21:39:30.322546+0000) 2023-12-16T21:46:06.317 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:46:05 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:46:05.837+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:05.622479+0000 front 2023-12-16T21:39:05.622580+0000 (oldest deadline 2023-12-16T21:39:30.322546+0000) 2023-12-16T21:46:06.317 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:46:05 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:46:05.838+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:46:06.816 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Dec 16 21:46:06 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-mon-smithi045[80759]: 2023-12-16T21:46:06.362+0000 7f45a1f8f700 -1 mon.smithi045@0(probing) e2 get_health_metrics reporting 3719 slow ops, oldest is monmgrreport(gid 14214, 0 checks, 1 progress events) 2023-12-16T21:46:06.817 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:46:06 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:46:06.508+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:09.828911+0000 front 2023-12-16T21:39:09.828958+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:46:06.817 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:46:06 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:46:06.508+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:09.828868+0000 front 2023-12-16T21:39:09.828878+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:46:06.817 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:46:06 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:46:06.508+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:09.828947+0000 front 2023-12-16T21:39:09.828993+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:46:06.817 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:46:06 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:46:06.510+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:46:07.316 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:46:06 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:46:06.874+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:09.306583+0000 front 2023-12-16T21:39:09.306633+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:46:07.317 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:46:06 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:46:06.874+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:09.306551+0000 front 2023-12-16T21:39:09.306652+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:46:07.317 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:46:06 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:46:06.874+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:09.306605+0000 front 2023-12-16T21:39:09.306613+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:46:07.317 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:46:06 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:46:06.875+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:46:07.317 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:46:06 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:46:06.833+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:05.622397+0000 front 2023-12-16T21:39:05.622556+0000 (oldest deadline 2023-12-16T21:39:30.322546+0000) 2023-12-16T21:46:07.317 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:46:06 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:46:06.833+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:05.622506+0000 front 2023-12-16T21:39:05.622528+0000 (oldest deadline 2023-12-16T21:39:30.322546+0000) 2023-12-16T21:46:07.317 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:46:06 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:46:06.833+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:05.622479+0000 front 2023-12-16T21:39:05.622580+0000 (oldest deadline 2023-12-16T21:39:30.322546+0000) 2023-12-16T21:46:07.317 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:46:06 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:46:06.835+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:46:07.816 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:46:07 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:46:07.538+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:09.828911+0000 front 2023-12-16T21:39:09.828958+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:46:07.816 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:46:07 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:46:07.538+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:09.828868+0000 front 2023-12-16T21:39:09.828878+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:46:07.817 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:46:07 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:46:07.538+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:09.828947+0000 front 2023-12-16T21:39:09.828993+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:46:07.817 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:46:07 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:46:07.539+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:46:08.316 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:46:07 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:46:07.852+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:09.306583+0000 front 2023-12-16T21:39:09.306633+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:46:08.316 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:46:07 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:46:07.852+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:09.306551+0000 front 2023-12-16T21:39:09.306652+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:46:08.316 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:46:07 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:46:07.852+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:09.306605+0000 front 2023-12-16T21:39:09.306613+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:46:08.317 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:46:07 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:46:07.853+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:46:08.317 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:46:07 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:46:07.825+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:05.622397+0000 front 2023-12-16T21:39:05.622556+0000 (oldest deadline 2023-12-16T21:39:30.322546+0000) 2023-12-16T21:46:08.317 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:46:07 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:46:07.825+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:05.622506+0000 front 2023-12-16T21:39:05.622528+0000 (oldest deadline 2023-12-16T21:39:30.322546+0000) 2023-12-16T21:46:08.317 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:46:07 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:46:07.825+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:05.622479+0000 front 2023-12-16T21:39:05.622580+0000 (oldest deadline 2023-12-16T21:39:30.322546+0000) 2023-12-16T21:46:08.317 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:46:07 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:46:07.825+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:46:08.816 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:46:08 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:46:08.551+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:09.828911+0000 front 2023-12-16T21:39:09.828958+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:46:08.839 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:46:08 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:46:08.551+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:09.828868+0000 front 2023-12-16T21:39:09.828878+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:46:08.839 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:46:08 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:46:08.551+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:09.828947+0000 front 2023-12-16T21:39:09.828993+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:46:08.840 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:46:08 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:46:08.551+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:46:09.316 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:46:08 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:46:08.851+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:09.306583+0000 front 2023-12-16T21:39:09.306633+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:46:09.317 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:46:08 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:46:08.851+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:09.306551+0000 front 2023-12-16T21:39:09.306652+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:46:09.317 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:46:08 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:46:08.851+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:09.306605+0000 front 2023-12-16T21:39:09.306613+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:46:09.317 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:46:08 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:46:08.852+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:46:09.317 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:46:08 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:46:08.851+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:05.622397+0000 front 2023-12-16T21:39:05.622556+0000 (oldest deadline 2023-12-16T21:39:30.322546+0000) 2023-12-16T21:46:09.317 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:46:08 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:46:08.851+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:05.622506+0000 front 2023-12-16T21:39:05.622528+0000 (oldest deadline 2023-12-16T21:39:30.322546+0000) 2023-12-16T21:46:09.317 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:46:08 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:46:08.851+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:05.622479+0000 front 2023-12-16T21:39:05.622580+0000 (oldest deadline 2023-12-16T21:39:30.322546+0000) 2023-12-16T21:46:09.317 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:46:08 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:46:08.852+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:46:09.816 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:46:09 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:46:09.543+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:09.828911+0000 front 2023-12-16T21:39:09.828958+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:46:09.816 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:46:09 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:46:09.543+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:09.828868+0000 front 2023-12-16T21:39:09.828878+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:46:09.817 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:46:09 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:46:09.543+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:09.828947+0000 front 2023-12-16T21:39:09.828993+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:46:09.817 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:46:09 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:46:09.544+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:46:10.316 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:46:09 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:46:09.855+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:09.306583+0000 front 2023-12-16T21:39:09.306633+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:46:10.317 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:46:09 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:46:09.855+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:09.306551+0000 front 2023-12-16T21:39:09.306652+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:46:10.317 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:46:09 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:46:09.855+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:09.306605+0000 front 2023-12-16T21:39:09.306613+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:46:10.317 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:46:09 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:46:09.855+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:46:10.317 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:46:09 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:46:09.817+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:05.622397+0000 front 2023-12-16T21:39:05.622556+0000 (oldest deadline 2023-12-16T21:39:30.322546+0000) 2023-12-16T21:46:10.317 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:46:09 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:46:09.817+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:05.622506+0000 front 2023-12-16T21:39:05.622528+0000 (oldest deadline 2023-12-16T21:39:30.322546+0000) 2023-12-16T21:46:10.317 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:46:09 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:46:09.817+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:05.622479+0000 front 2023-12-16T21:39:05.622580+0000 (oldest deadline 2023-12-16T21:39:30.322546+0000) 2023-12-16T21:46:10.317 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:46:09 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:46:09.818+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:46:10.854 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:46:10 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:46:10.785+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:05.622397+0000 front 2023-12-16T21:39:05.622556+0000 (oldest deadline 2023-12-16T21:39:30.322546+0000) 2023-12-16T21:46:10.855 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:46:10 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:46:10.785+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:05.622506+0000 front 2023-12-16T21:39:05.622528+0000 (oldest deadline 2023-12-16T21:39:30.322546+0000) 2023-12-16T21:46:10.855 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:46:10 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:46:10.785+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:05.622479+0000 front 2023-12-16T21:39:05.622580+0000 (oldest deadline 2023-12-16T21:39:30.322546+0000) 2023-12-16T21:46:10.855 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:46:10 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:46:10.786+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:46:10.855 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:46:10 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:46:10.564+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:09.828911+0000 front 2023-12-16T21:39:09.828958+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:46:10.855 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:46:10 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:46:10.564+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:09.828868+0000 front 2023-12-16T21:39:09.828878+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:46:10.855 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:46:10 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:46:10.564+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:09.828947+0000 front 2023-12-16T21:39:09.828993+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:46:10.855 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:46:10 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:46:10.565+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:46:11.316 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:46:10 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:46:10.851+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:09.306583+0000 front 2023-12-16T21:39:09.306633+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:46:11.316 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:46:10 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:46:10.851+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:09.306551+0000 front 2023-12-16T21:39:09.306652+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:46:11.316 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:46:10 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:46:10.851+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:09.306605+0000 front 2023-12-16T21:39:09.306613+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:46:11.317 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:46:10 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:46:10.852+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:46:11.648 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Dec 16 21:46:11 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-mon-smithi045[80759]: 2023-12-16T21:46:11.363+0000 7f45a1f8f700 -1 mon.smithi045@0(probing) e2 get_health_metrics reporting 3817 slow ops, oldest is monmgrreport(gid 14214, 0 checks, 1 progress events) 2023-12-16T21:46:11.649 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:46:11 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:46:11.552+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:09.828911+0000 front 2023-12-16T21:39:09.828958+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:46:11.649 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:46:11 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:46:11.552+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:09.828868+0000 front 2023-12-16T21:39:09.828878+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:46:11.649 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:46:11 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:46:11.552+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:09.828947+0000 front 2023-12-16T21:39:09.828993+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:46:11.649 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:46:11 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:46:11.553+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:46:12.066 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:46:11 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:46:11.770+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:05.622397+0000 front 2023-12-16T21:39:05.622556+0000 (oldest deadline 2023-12-16T21:39:30.322546+0000) 2023-12-16T21:46:12.066 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:46:11 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:46:11.770+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:05.622506+0000 front 2023-12-16T21:39:05.622528+0000 (oldest deadline 2023-12-16T21:39:30.322546+0000) 2023-12-16T21:46:12.067 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:46:11 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:46:11.770+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:05.622479+0000 front 2023-12-16T21:39:05.622580+0000 (oldest deadline 2023-12-16T21:39:30.322546+0000) 2023-12-16T21:46:12.067 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:46:11 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:46:11.771+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:46:12.067 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:46:11 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:46:11.888+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:09.306583+0000 front 2023-12-16T21:39:09.306633+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:46:12.067 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:46:11 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:46:11.888+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:09.306551+0000 front 2023-12-16T21:39:09.306652+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:46:12.067 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:46:11 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:46:11.888+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:09.306605+0000 front 2023-12-16T21:39:09.306613+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:46:12.067 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:46:11 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:46:11.889+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:46:12.816 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:46:12 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:46:12.723+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:05.622397+0000 front 2023-12-16T21:39:05.622556+0000 (oldest deadline 2023-12-16T21:39:30.322546+0000) 2023-12-16T21:46:12.816 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:46:12 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:46:12.723+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:05.622506+0000 front 2023-12-16T21:39:05.622528+0000 (oldest deadline 2023-12-16T21:39:30.322546+0000) 2023-12-16T21:46:12.817 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:46:12 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:46:12.723+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:05.622479+0000 front 2023-12-16T21:39:05.622580+0000 (oldest deadline 2023-12-16T21:39:30.322546+0000) 2023-12-16T21:46:12.817 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:46:12 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:46:12.724+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:46:12.817 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:46:12 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:46:12.528+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:09.828911+0000 front 2023-12-16T21:39:09.828958+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:46:12.817 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:46:12 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:46:12.528+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:09.828868+0000 front 2023-12-16T21:39:09.828878+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:46:12.817 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:46:12 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:46:12.528+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:09.828947+0000 front 2023-12-16T21:39:09.828993+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:46:12.817 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:46:12 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:46:12.529+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:46:13.137 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:46:12 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:46:12.854+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:09.306583+0000 front 2023-12-16T21:39:09.306633+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:46:13.137 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:46:12 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:46:12.854+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:09.306551+0000 front 2023-12-16T21:39:09.306652+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:46:13.137 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:46:12 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:46:12.854+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:09.306605+0000 front 2023-12-16T21:39:09.306613+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:46:13.137 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:46:12 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:46:12.855+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:46:13.811 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:46:13 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:46:13.748+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:05.622397+0000 front 2023-12-16T21:39:05.622556+0000 (oldest deadline 2023-12-16T21:39:30.322546+0000) 2023-12-16T21:46:13.811 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:46:13 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:46:13.748+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:05.622506+0000 front 2023-12-16T21:39:05.622528+0000 (oldest deadline 2023-12-16T21:39:30.322546+0000) 2023-12-16T21:46:13.811 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:46:13 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:46:13.748+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:05.622479+0000 front 2023-12-16T21:39:05.622580+0000 (oldest deadline 2023-12-16T21:39:30.322546+0000) 2023-12-16T21:46:13.812 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:46:13 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:46:13.750+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:46:13.812 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:46:13 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:46:13.524+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:09.828911+0000 front 2023-12-16T21:39:09.828958+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:46:13.812 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:46:13 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:46:13.524+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:09.828868+0000 front 2023-12-16T21:39:09.828878+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:46:13.812 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:46:13 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:46:13.524+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:09.828947+0000 front 2023-12-16T21:39:09.828993+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:46:13.812 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:46:13 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:46:13.525+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:46:14.066 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:46:13 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:46:13.808+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:09.306583+0000 front 2023-12-16T21:39:09.306633+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:46:14.066 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:46:13 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:46:13.808+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:09.306551+0000 front 2023-12-16T21:39:09.306652+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:46:14.066 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:46:13 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:46:13.808+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:09.306605+0000 front 2023-12-16T21:39:09.306613+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:46:14.067 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:46:13 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:46:13.810+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:46:14.767 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:46:14 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:46:14.763+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:09.306583+0000 front 2023-12-16T21:39:09.306633+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:46:14.768 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:46:14 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:46:14.763+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:09.306551+0000 front 2023-12-16T21:39:09.306652+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:46:14.768 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:46:14 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:46:14.763+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:09.306605+0000 front 2023-12-16T21:39:09.306613+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:46:14.768 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:46:14 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:46:14.764+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:46:14.768 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:46:14 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:46:14.514+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:09.828911+0000 front 2023-12-16T21:39:09.828958+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:46:14.768 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:46:14 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:46:14.514+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:09.828868+0000 front 2023-12-16T21:39:09.828878+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:46:14.768 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:46:14 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:46:14.514+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:09.828947+0000 front 2023-12-16T21:39:09.828993+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:46:14.768 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:46:14 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:46:14.515+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:46:14.768 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:46:14 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:46:14.729+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:05.622397+0000 front 2023-12-16T21:39:05.622556+0000 (oldest deadline 2023-12-16T21:39:30.322546+0000) 2023-12-16T21:46:14.769 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:46:14 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:46:14.729+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:05.622506+0000 front 2023-12-16T21:39:05.622528+0000 (oldest deadline 2023-12-16T21:39:30.322546+0000) 2023-12-16T21:46:14.769 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:46:14 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:46:14.729+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:05.622479+0000 front 2023-12-16T21:39:05.622580+0000 (oldest deadline 2023-12-16T21:39:30.322546+0000) 2023-12-16T21:46:14.769 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:46:14 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:46:14.730+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:46:15.782 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:46:15 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:46:15.752+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:05.622397+0000 front 2023-12-16T21:39:05.622556+0000 (oldest deadline 2023-12-16T21:39:30.322546+0000) 2023-12-16T21:46:15.782 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:46:15 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:46:15.752+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:05.622506+0000 front 2023-12-16T21:39:05.622528+0000 (oldest deadline 2023-12-16T21:39:30.322546+0000) 2023-12-16T21:46:15.782 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:46:15 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:46:15.752+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:05.622479+0000 front 2023-12-16T21:39:05.622580+0000 (oldest deadline 2023-12-16T21:39:30.322546+0000) 2023-12-16T21:46:15.782 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:46:15 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:46:15.753+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:46:15.782 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:46:15 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:46:15.510+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:09.828911+0000 front 2023-12-16T21:39:09.828958+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:46:15.782 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:46:15 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:46:15.510+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:09.828868+0000 front 2023-12-16T21:39:09.828878+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:46:15.782 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:46:15 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:46:15.510+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:09.828947+0000 front 2023-12-16T21:39:09.828993+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:46:15.782 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:46:15 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:46:15.511+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:46:16.066 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:46:15 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:46:15.779+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:09.306583+0000 front 2023-12-16T21:39:09.306633+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:46:16.066 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:46:15 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:46:15.779+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:09.306551+0000 front 2023-12-16T21:39:09.306652+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:46:16.066 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:46:15 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:46:15.779+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:09.306605+0000 front 2023-12-16T21:39:09.306613+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:46:16.067 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:46:15 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:46:15.780+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:46:16.489 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Dec 16 21:46:16 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-mon-smithi045[80759]: 2023-12-16T21:46:16.364+0000 7f45a1f8f700 -1 mon.smithi045@0(probing) e2 get_health_metrics reporting 3912 slow ops, oldest is monmgrreport(gid 14214, 0 checks, 1 progress events) 2023-12-16T21:46:16.746 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:46:16 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:46:16.543+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:09.828911+0000 front 2023-12-16T21:39:09.828958+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:46:16.747 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:46:16 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:46:16.543+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:09.828868+0000 front 2023-12-16T21:39:09.828878+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:46:16.747 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:46:16 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:46:16.543+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:09.828947+0000 front 2023-12-16T21:39:09.828993+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:46:16.747 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:46:16 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:46:16.544+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:46:17.067 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:46:16 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:46:16.743+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:05.622397+0000 front 2023-12-16T21:39:05.622556+0000 (oldest deadline 2023-12-16T21:39:30.322546+0000) 2023-12-16T21:46:17.067 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:46:16 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:46:16.743+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:05.622506+0000 front 2023-12-16T21:39:05.622528+0000 (oldest deadline 2023-12-16T21:39:30.322546+0000) 2023-12-16T21:46:17.067 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:46:16 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:46:16.743+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:05.622479+0000 front 2023-12-16T21:39:05.622580+0000 (oldest deadline 2023-12-16T21:39:30.322546+0000) 2023-12-16T21:46:17.067 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:46:16 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:46:16.745+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:46:17.067 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:46:16 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:46:16.759+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:09.306583+0000 front 2023-12-16T21:39:09.306633+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:46:17.068 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:46:16 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:46:16.759+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:09.306551+0000 front 2023-12-16T21:39:09.306652+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:46:17.068 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:46:16 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:46:16.759+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:09.306605+0000 front 2023-12-16T21:39:09.306613+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:46:17.068 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:46:16 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:46:16.759+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:46:17.816 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:46:17 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:46:17.700+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:05.622397+0000 front 2023-12-16T21:39:05.622556+0000 (oldest deadline 2023-12-16T21:39:30.322546+0000) 2023-12-16T21:46:17.816 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:46:17 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:46:17.700+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:05.622506+0000 front 2023-12-16T21:39:05.622528+0000 (oldest deadline 2023-12-16T21:39:30.322546+0000) 2023-12-16T21:46:17.817 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:46:17 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:46:17.700+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:05.622479+0000 front 2023-12-16T21:39:05.622580+0000 (oldest deadline 2023-12-16T21:39:30.322546+0000) 2023-12-16T21:46:17.817 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:46:17 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:46:17.701+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:46:17.817 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:46:17 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:46:17.725+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:09.306583+0000 front 2023-12-16T21:39:09.306633+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:46:17.817 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:46:17 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:46:17.725+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:09.306551+0000 front 2023-12-16T21:39:09.306652+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:46:17.817 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:46:17 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:46:17.725+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:09.306605+0000 front 2023-12-16T21:39:09.306613+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:46:17.817 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:46:17 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:46:17.727+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:46:17.817 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:46:17 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:46:17.525+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:09.828911+0000 front 2023-12-16T21:39:09.828958+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:46:17.817 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:46:17 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:46:17.525+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:09.828868+0000 front 2023-12-16T21:39:09.828878+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:46:17.818 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:46:17 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:46:17.525+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:09.828947+0000 front 2023-12-16T21:39:09.828993+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:46:17.818 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:46:17 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:46:17.526+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:46:18.816 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:46:18 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:46:18.717+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:09.306583+0000 front 2023-12-16T21:39:09.306633+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:46:18.817 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:46:18 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:46:18.717+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:09.306551+0000 front 2023-12-16T21:39:09.306652+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:46:18.817 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:46:18 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:46:18.717+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:09.306605+0000 front 2023-12-16T21:39:09.306613+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:46:18.817 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:46:18 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:46:18.718+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:46:18.817 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:46:18 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:46:18.559+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:09.828911+0000 front 2023-12-16T21:39:09.828958+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:46:18.817 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:46:18 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:46:18.559+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:09.828868+0000 front 2023-12-16T21:39:09.828878+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:46:18.817 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:46:18 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:46:18.559+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:09.828947+0000 front 2023-12-16T21:39:09.828993+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:46:18.817 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:46:18 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:46:18.560+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:46:18.817 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:46:18 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:46:18.699+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:05.622397+0000 front 2023-12-16T21:39:05.622556+0000 (oldest deadline 2023-12-16T21:39:30.322546+0000) 2023-12-16T21:46:18.818 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:46:18 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:46:18.699+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:05.622506+0000 front 2023-12-16T21:39:05.622528+0000 (oldest deadline 2023-12-16T21:39:30.322546+0000) 2023-12-16T21:46:18.818 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:46:18 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:46:18.699+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:05.622479+0000 front 2023-12-16T21:39:05.622580+0000 (oldest deadline 2023-12-16T21:39:30.322546+0000) 2023-12-16T21:46:18.818 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:46:18 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:46:18.700+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:46:20.066 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:46:19 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:46:19.732+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:09.306583+0000 front 2023-12-16T21:39:09.306633+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:46:20.067 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:46:19 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:46:19.732+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:09.306551+0000 front 2023-12-16T21:39:09.306652+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:46:20.067 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:46:19 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:46:19.732+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:09.306605+0000 front 2023-12-16T21:39:09.306613+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:46:20.067 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:46:19 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:46:19.732+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:46:20.067 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:46:19 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:46:19.607+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:09.828911+0000 front 2023-12-16T21:39:09.828958+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:46:20.067 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:46:19 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:46:19.607+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:09.828868+0000 front 2023-12-16T21:39:09.828878+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:46:20.067 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:46:19 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:46:19.607+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:09.828947+0000 front 2023-12-16T21:39:09.828993+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:46:20.067 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:46:19 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:46:19.608+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:46:20.067 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:46:19 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:46:19.730+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:05.622397+0000 front 2023-12-16T21:39:05.622556+0000 (oldest deadline 2023-12-16T21:39:30.322546+0000) 2023-12-16T21:46:20.068 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:46:19 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:46:19.730+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:05.622506+0000 front 2023-12-16T21:39:05.622528+0000 (oldest deadline 2023-12-16T21:39:30.322546+0000) 2023-12-16T21:46:20.068 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:46:19 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:46:19.731+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:05.622479+0000 front 2023-12-16T21:39:05.622580+0000 (oldest deadline 2023-12-16T21:39:30.322546+0000) 2023-12-16T21:46:20.068 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:46:19 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:46:19.732+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:46:21.066 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:46:20 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:46:20.762+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:09.306583+0000 front 2023-12-16T21:39:09.306633+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:46:21.067 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:46:20 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:46:20.762+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:09.306551+0000 front 2023-12-16T21:39:09.306652+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:46:21.067 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:46:20 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:46:20.762+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:09.306605+0000 front 2023-12-16T21:39:09.306613+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:46:21.067 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:46:20 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:46:20.762+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:46:21.067 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:46:20 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:46:20.588+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:09.828911+0000 front 2023-12-16T21:39:09.828958+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:46:21.067 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:46:20 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:46:20.588+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:09.828868+0000 front 2023-12-16T21:39:09.828878+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:46:21.067 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:46:20 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:46:20.588+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:09.828947+0000 front 2023-12-16T21:39:09.828993+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:46:21.067 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:46:20 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:46:20.588+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:46:21.067 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:46:20 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:46:20.749+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:05.622397+0000 front 2023-12-16T21:39:05.622556+0000 (oldest deadline 2023-12-16T21:39:30.322546+0000) 2023-12-16T21:46:21.068 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:46:20 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:46:20.749+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:05.622506+0000 front 2023-12-16T21:39:05.622528+0000 (oldest deadline 2023-12-16T21:39:30.322546+0000) 2023-12-16T21:46:21.068 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:46:20 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:46:20.749+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:05.622479+0000 front 2023-12-16T21:39:05.622580+0000 (oldest deadline 2023-12-16T21:39:30.322546+0000) 2023-12-16T21:46:21.068 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:46:20 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:46:20.750+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:46:21.649 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Dec 16 21:46:21 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-mon-smithi045[80759]: 2023-12-16T21:46:21.364+0000 7f45a1f8f700 -1 mon.smithi045@0(probing) e2 get_health_metrics reporting 4010 slow ops, oldest is monmgrreport(gid 14214, 0 checks, 1 progress events) 2023-12-16T21:46:21.649 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:46:21 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:46:21.596+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:09.828911+0000 front 2023-12-16T21:39:09.828958+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:46:21.649 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:46:21 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:46:21.596+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:09.828868+0000 front 2023-12-16T21:39:09.828878+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:46:21.649 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:46:21 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:46:21.596+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:09.828947+0000 front 2023-12-16T21:39:09.828993+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:46:21.649 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:46:21 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:46:21.597+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:46:22.067 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:46:21 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:46:21.800+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:05.622397+0000 front 2023-12-16T21:39:05.622556+0000 (oldest deadline 2023-12-16T21:39:30.322546+0000) 2023-12-16T21:46:22.067 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:46:21 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:46:21.800+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:05.622506+0000 front 2023-12-16T21:39:05.622528+0000 (oldest deadline 2023-12-16T21:39:30.322546+0000) 2023-12-16T21:46:22.067 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:46:21 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:46:21.800+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:05.622479+0000 front 2023-12-16T21:39:05.622580+0000 (oldest deadline 2023-12-16T21:39:30.322546+0000) 2023-12-16T21:46:22.067 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:46:21 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:46:21.800+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:46:22.067 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:46:21 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:46:21.803+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:09.306583+0000 front 2023-12-16T21:39:09.306633+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:46:22.068 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:46:21 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:46:21.803+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:09.306551+0000 front 2023-12-16T21:39:09.306652+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:46:22.068 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:46:21 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:46:21.803+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:09.306605+0000 front 2023-12-16T21:39:09.306613+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:46:22.068 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:46:21 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:46:21.804+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:46:22.764 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:46:22 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:46:22.553+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:09.828911+0000 front 2023-12-16T21:39:09.828958+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:46:22.764 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:46:22 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:46:22.553+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:09.828868+0000 front 2023-12-16T21:39:09.828878+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:46:22.764 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:46:22 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:46:22.553+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:09.828947+0000 front 2023-12-16T21:39:09.828993+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:46:22.764 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:46:22 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:46:22.554+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:46:23.066 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:46:22 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:46:22.760+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:05.622397+0000 front 2023-12-16T21:39:05.622556+0000 (oldest deadline 2023-12-16T21:39:30.322546+0000) 2023-12-16T21:46:23.066 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:46:22 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:46:22.760+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:05.622506+0000 front 2023-12-16T21:39:05.622528+0000 (oldest deadline 2023-12-16T21:39:30.322546+0000) 2023-12-16T21:46:23.067 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:46:22 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:46:22.760+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:05.622479+0000 front 2023-12-16T21:39:05.622580+0000 (oldest deadline 2023-12-16T21:39:30.322546+0000) 2023-12-16T21:46:23.067 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:46:22 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:46:22.761+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:46:23.067 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:46:22 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:46:22.789+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:09.306583+0000 front 2023-12-16T21:39:09.306633+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:46:23.067 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:46:22 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:46:22.789+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:09.306551+0000 front 2023-12-16T21:39:09.306652+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:46:23.067 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:46:22 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:46:22.789+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:09.306605+0000 front 2023-12-16T21:39:09.306613+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:46:23.067 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:46:22 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:46:22.790+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:46:23.817 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:46:23 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:46:23.766+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:05.622397+0000 front 2023-12-16T21:39:05.622556+0000 (oldest deadline 2023-12-16T21:39:30.322546+0000) 2023-12-16T21:46:23.817 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:46:23 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:46:23.766+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:05.622506+0000 front 2023-12-16T21:39:05.622528+0000 (oldest deadline 2023-12-16T21:39:30.322546+0000) 2023-12-16T21:46:23.817 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:46:23 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:46:23.766+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:05.622479+0000 front 2023-12-16T21:39:05.622580+0000 (oldest deadline 2023-12-16T21:39:30.322546+0000) 2023-12-16T21:46:23.817 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:46:23 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:46:23.767+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:46:23.817 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:46:23 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:46:23.546+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:09.828911+0000 front 2023-12-16T21:39:09.828958+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:46:23.817 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:46:23 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:46:23.546+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:09.828868+0000 front 2023-12-16T21:39:09.828878+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:46:23.817 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:46:23 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:46:23.546+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:09.828947+0000 front 2023-12-16T21:39:09.828993+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:46:23.817 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:46:23 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:46:23.547+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:46:24.316 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:46:23 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:46:23.821+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:09.306583+0000 front 2023-12-16T21:39:09.306633+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:46:24.317 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:46:23 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:46:23.821+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:09.306551+0000 front 2023-12-16T21:39:09.306652+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:46:24.317 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:46:23 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:46:23.821+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:09.306605+0000 front 2023-12-16T21:39:09.306613+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:46:24.317 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:46:23 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:46:23.821+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:46:24.816 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:46:24 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:46:24.765+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:05.622397+0000 front 2023-12-16T21:39:05.622556+0000 (oldest deadline 2023-12-16T21:39:30.322546+0000) 2023-12-16T21:46:24.817 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:46:24 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:46:24.765+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:05.622506+0000 front 2023-12-16T21:39:05.622528+0000 (oldest deadline 2023-12-16T21:39:30.322546+0000) 2023-12-16T21:46:24.817 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:46:24 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:46:24.765+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:05.622479+0000 front 2023-12-16T21:39:05.622580+0000 (oldest deadline 2023-12-16T21:39:30.322546+0000) 2023-12-16T21:46:24.817 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:46:24 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:46:24.766+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:46:24.817 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:46:24 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:46:24.531+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:09.828911+0000 front 2023-12-16T21:39:09.828958+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:46:24.817 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:46:24 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:46:24.531+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:09.828868+0000 front 2023-12-16T21:39:09.828878+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:46:24.817 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:46:24 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:46:24.531+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:09.828947+0000 front 2023-12-16T21:39:09.828993+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:46:24.817 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:46:24 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:46:24.532+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:46:25.316 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:46:24 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:46:24.847+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:09.306583+0000 front 2023-12-16T21:39:09.306633+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:46:25.317 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:46:24 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:46:24.847+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:09.306551+0000 front 2023-12-16T21:39:09.306652+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:46:25.317 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:46:24 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:46:24.847+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:09.306605+0000 front 2023-12-16T21:39:09.306613+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:46:25.317 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:46:24 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:46:24.848+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:46:25.723 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:46:25 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:46:25.535+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:09.828911+0000 front 2023-12-16T21:39:09.828958+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:46:25.724 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:46:25 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:46:25.535+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:09.828868+0000 front 2023-12-16T21:39:09.828878+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:46:25.724 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:46:25 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:46:25.535+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:09.828947+0000 front 2023-12-16T21:39:09.828993+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:46:25.724 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:46:25 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:46:25.535+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:46:26.066 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:46:25 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:46:25.891+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:09.306583+0000 front 2023-12-16T21:39:09.306633+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:46:26.067 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:46:25 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:46:25.891+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:09.306551+0000 front 2023-12-16T21:39:09.306652+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:46:26.067 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:46:25 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:46:25.891+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:09.306605+0000 front 2023-12-16T21:39:09.306613+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:46:26.067 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:46:25 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:46:25.892+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:46:26.067 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:46:25 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:46:25.720+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:05.622397+0000 front 2023-12-16T21:39:05.622556+0000 (oldest deadline 2023-12-16T21:39:30.322546+0000) 2023-12-16T21:46:26.067 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:46:25 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:46:25.720+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:05.622506+0000 front 2023-12-16T21:39:05.622528+0000 (oldest deadline 2023-12-16T21:39:30.322546+0000) 2023-12-16T21:46:26.067 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:46:25 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:46:25.720+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:05.622479+0000 front 2023-12-16T21:39:05.622580+0000 (oldest deadline 2023-12-16T21:39:30.322546+0000) 2023-12-16T21:46:26.067 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:46:25 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:46:25.721+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:46:26.675 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Dec 16 21:46:26 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-mon-smithi045[80759]: 2023-12-16T21:46:26.366+0000 7f45a1f8f700 -1 mon.smithi045@0(probing) e2 get_health_metrics reporting 4106 slow ops, oldest is monmgrreport(gid 14214, 0 checks, 1 progress events) 2023-12-16T21:46:26.675 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:46:26 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:46:26.546+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:09.828911+0000 front 2023-12-16T21:39:09.828958+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:46:26.675 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:46:26 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:46:26.546+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:09.828868+0000 front 2023-12-16T21:39:09.828878+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:46:26.675 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:46:26 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:46:26.546+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:09.828947+0000 front 2023-12-16T21:39:09.828993+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:46:26.675 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:46:26 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:46:26.547+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:46:26.935 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:46:26 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:46:26.672+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:05.622397+0000 front 2023-12-16T21:39:05.622556+0000 (oldest deadline 2023-12-16T21:39:30.322546+0000) 2023-12-16T21:46:26.935 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:46:26 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:46:26.672+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:05.622506+0000 front 2023-12-16T21:39:05.622528+0000 (oldest deadline 2023-12-16T21:39:30.322546+0000) 2023-12-16T21:46:26.935 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:46:26 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:46:26.672+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:05.622479+0000 front 2023-12-16T21:39:05.622580+0000 (oldest deadline 2023-12-16T21:39:30.322546+0000) 2023-12-16T21:46:26.935 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:46:26 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:46:26.672+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:46:27.316 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:46:26 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:46:26.932+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:09.306583+0000 front 2023-12-16T21:39:09.306633+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:46:27.316 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:46:26 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:46:26.932+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:09.306551+0000 front 2023-12-16T21:39:09.306652+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:46:27.317 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:46:26 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:46:26.932+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:09.306605+0000 front 2023-12-16T21:39:09.306613+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:46:27.317 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:46:26 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:46:26.933+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:46:27.816 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:46:27 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:46:27.540+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:09.828911+0000 front 2023-12-16T21:39:09.828958+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:46:27.817 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:46:27 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:46:27.540+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:09.828868+0000 front 2023-12-16T21:39:09.828878+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:46:27.817 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:46:27 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:46:27.540+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:09.828947+0000 front 2023-12-16T21:39:09.828993+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:46:27.817 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:46:27 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:46:27.540+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:46:27.817 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:46:27 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:46:27.689+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:05.622397+0000 front 2023-12-16T21:39:05.622556+0000 (oldest deadline 2023-12-16T21:39:30.322546+0000) 2023-12-16T21:46:27.817 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:46:27 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:46:27.689+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:05.622506+0000 front 2023-12-16T21:39:05.622528+0000 (oldest deadline 2023-12-16T21:39:30.322546+0000) 2023-12-16T21:46:27.817 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:46:27 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:46:27.689+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:05.622479+0000 front 2023-12-16T21:39:05.622580+0000 (oldest deadline 2023-12-16T21:39:30.322546+0000) 2023-12-16T21:46:27.817 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:46:27 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:46:27.690+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:46:28.316 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:46:27 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:46:27.959+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:09.306583+0000 front 2023-12-16T21:39:09.306633+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:46:28.317 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:46:27 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:46:27.959+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:09.306551+0000 front 2023-12-16T21:39:09.306652+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:46:28.317 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:46:27 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:46:27.959+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:09.306605+0000 front 2023-12-16T21:39:09.306613+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:46:28.317 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:46:27 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:46:27.960+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:46:28.816 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:46:28 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:46:28.518+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:09.828911+0000 front 2023-12-16T21:39:09.828958+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:46:28.817 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:46:28 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:46:28.518+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:09.828868+0000 front 2023-12-16T21:39:09.828878+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:46:28.817 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:46:28 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:46:28.518+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:09.828947+0000 front 2023-12-16T21:39:09.828993+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:46:28.817 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:46:28 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:46:28.518+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:46:28.817 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:46:28 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:46:28.729+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:05.622397+0000 front 2023-12-16T21:39:05.622556+0000 (oldest deadline 2023-12-16T21:39:30.322546+0000) 2023-12-16T21:46:28.817 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:46:28 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:46:28.729+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:05.622506+0000 front 2023-12-16T21:39:05.622528+0000 (oldest deadline 2023-12-16T21:39:30.322546+0000) 2023-12-16T21:46:28.817 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:46:28 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:46:28.729+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:05.622479+0000 front 2023-12-16T21:39:05.622580+0000 (oldest deadline 2023-12-16T21:39:30.322546+0000) 2023-12-16T21:46:28.817 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:46:28 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:46:28.730+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:46:29.316 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:46:28 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:46:28.967+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:09.306583+0000 front 2023-12-16T21:39:09.306633+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:46:29.317 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:46:28 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:46:28.967+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:09.306551+0000 front 2023-12-16T21:39:09.306652+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:46:29.317 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:46:28 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:46:28.967+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:09.306605+0000 front 2023-12-16T21:39:09.306613+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:46:29.317 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:46:28 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:46:28.968+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:46:29.742 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:46:29 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:46:29.473+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:09.828911+0000 front 2023-12-16T21:39:09.828958+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:46:29.742 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:46:29 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:46:29.473+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:09.828868+0000 front 2023-12-16T21:39:09.828878+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:46:29.743 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:46:29 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:46:29.473+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:09.828947+0000 front 2023-12-16T21:39:09.828993+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:46:29.743 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:46:29 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:46:29.474+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:46:30.021 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:46:29 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:46:29.739+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:05.622397+0000 front 2023-12-16T21:39:05.622556+0000 (oldest deadline 2023-12-16T21:39:30.322546+0000) 2023-12-16T21:46:30.021 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:46:29 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:46:29.739+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:05.622506+0000 front 2023-12-16T21:39:05.622528+0000 (oldest deadline 2023-12-16T21:39:30.322546+0000) 2023-12-16T21:46:30.021 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:46:29 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:46:29.739+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:05.622479+0000 front 2023-12-16T21:39:05.622580+0000 (oldest deadline 2023-12-16T21:39:30.322546+0000) 2023-12-16T21:46:30.022 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:46:29 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:46:29.740+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:46:30.317 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:46:30 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:46:30.018+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:09.306583+0000 front 2023-12-16T21:39:09.306633+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:46:30.317 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:46:30 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:46:30.018+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:09.306551+0000 front 2023-12-16T21:39:09.306652+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:46:30.317 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:46:30 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:46:30.018+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:09.306605+0000 front 2023-12-16T21:39:09.306613+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:46:30.317 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:46:30 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:46:30.019+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:46:30.745 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:46:30 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:46:30.444+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:09.828911+0000 front 2023-12-16T21:39:09.828958+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:46:30.745 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:46:30 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:46:30.444+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:09.828868+0000 front 2023-12-16T21:39:09.828878+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:46:30.745 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:46:30 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:46:30.444+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:09.828947+0000 front 2023-12-16T21:39:09.828993+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:46:30.745 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:46:30 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:46:30.445+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:46:31.067 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:46:30 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:46:30.986+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:09.306583+0000 front 2023-12-16T21:39:09.306633+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:46:31.067 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:46:30 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:46:30.986+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:09.306551+0000 front 2023-12-16T21:39:09.306652+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:46:31.067 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:46:30 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:46:30.986+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:09.306605+0000 front 2023-12-16T21:39:09.306613+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:46:31.067 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:46:30 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:46:30.986+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:46:31.067 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:46:30 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:46:30.742+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:05.622397+0000 front 2023-12-16T21:39:05.622556+0000 (oldest deadline 2023-12-16T21:39:30.322546+0000) 2023-12-16T21:46:31.067 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:46:30 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:46:30.742+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:05.622506+0000 front 2023-12-16T21:39:05.622528+0000 (oldest deadline 2023-12-16T21:39:30.322546+0000) 2023-12-16T21:46:31.067 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:46:30 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:46:30.742+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:05.622479+0000 front 2023-12-16T21:39:05.622580+0000 (oldest deadline 2023-12-16T21:39:30.322546+0000) 2023-12-16T21:46:31.067 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:46:30 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:46:30.743+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:46:31.650 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Dec 16 21:46:31 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-mon-smithi045[80759]: 2023-12-16T21:46:31.366+0000 7f45a1f8f700 -1 mon.smithi045@0(probing) e2 get_health_metrics reporting 4218 slow ops, oldest is monmgrreport(gid 14214, 0 checks, 1 progress events) 2023-12-16T21:46:31.650 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:46:31 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:46:31.494+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:09.828911+0000 front 2023-12-16T21:39:09.828958+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:46:31.650 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:46:31 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:46:31.494+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:09.828868+0000 front 2023-12-16T21:39:09.828878+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:46:31.650 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:46:31 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:46:31.494+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:09.828947+0000 front 2023-12-16T21:39:09.828993+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:46:31.650 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:46:31 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:46:31.495+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:46:31.990 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:46:31 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:46:31.760+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:05.622397+0000 front 2023-12-16T21:39:05.622556+0000 (oldest deadline 2023-12-16T21:39:30.322546+0000) 2023-12-16T21:46:31.991 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:46:31 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:46:31.760+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:05.622506+0000 front 2023-12-16T21:39:05.622528+0000 (oldest deadline 2023-12-16T21:39:30.322546+0000) 2023-12-16T21:46:31.991 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:46:31 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:46:31.760+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:05.622479+0000 front 2023-12-16T21:39:05.622580+0000 (oldest deadline 2023-12-16T21:39:30.322546+0000) 2023-12-16T21:46:31.991 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:46:31 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:46:31.761+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:46:32.316 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:46:31 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:46:31.987+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:09.306583+0000 front 2023-12-16T21:39:09.306633+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:46:32.316 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:46:31 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:46:31.987+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:09.306551+0000 front 2023-12-16T21:39:09.306652+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:46:32.317 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:46:31 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:46:31.987+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:09.306605+0000 front 2023-12-16T21:39:09.306613+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:46:32.317 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:46:31 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:46:31.988+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:46:32.813 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:46:32 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:46:32.483+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:09.828911+0000 front 2023-12-16T21:39:09.828958+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:46:32.813 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:46:32 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:46:32.483+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:09.828868+0000 front 2023-12-16T21:39:09.828878+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:46:32.813 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:46:32 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:46:32.483+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:09.828947+0000 front 2023-12-16T21:39:09.828993+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:46:32.813 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:46:32 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:46:32.484+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:46:33.066 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:46:32 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:46:32.989+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:09.306583+0000 front 2023-12-16T21:39:09.306633+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:46:33.067 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:46:32 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:46:32.989+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:09.306551+0000 front 2023-12-16T21:39:09.306652+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:46:33.067 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:46:32 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:46:32.989+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:09.306605+0000 front 2023-12-16T21:39:09.306613+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:46:33.067 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:46:32 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:46:32.990+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:46:33.067 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:46:32 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:46:32.810+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:05.622397+0000 front 2023-12-16T21:39:05.622556+0000 (oldest deadline 2023-12-16T21:39:30.322546+0000) 2023-12-16T21:46:33.067 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:46:32 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:46:32.810+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:05.622506+0000 front 2023-12-16T21:39:05.622528+0000 (oldest deadline 2023-12-16T21:39:30.322546+0000) 2023-12-16T21:46:33.067 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:46:32 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:46:32.810+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:05.622479+0000 front 2023-12-16T21:39:05.622580+0000 (oldest deadline 2023-12-16T21:39:30.322546+0000) 2023-12-16T21:46:33.067 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:46:32 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:46:32.810+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:46:33.815 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:46:33 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:46:33.444+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:09.828911+0000 front 2023-12-16T21:39:09.828958+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:46:33.815 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:46:33 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:46:33.444+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:09.828868+0000 front 2023-12-16T21:39:09.828878+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:46:33.815 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:46:33 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:46:33.444+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:09.828947+0000 front 2023-12-16T21:39:09.828993+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:46:33.815 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:46:33 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:46:33.446+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:46:34.067 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:46:33 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:46:33.810+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:05.622397+0000 front 2023-12-16T21:39:05.622556+0000 (oldest deadline 2023-12-16T21:39:30.322546+0000) 2023-12-16T21:46:34.067 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:46:33 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:46:33.810+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:05.622506+0000 front 2023-12-16T21:39:05.622528+0000 (oldest deadline 2023-12-16T21:39:30.322546+0000) 2023-12-16T21:46:34.067 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:46:33 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:46:33.810+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:05.622479+0000 front 2023-12-16T21:39:05.622580+0000 (oldest deadline 2023-12-16T21:39:30.322546+0000) 2023-12-16T21:46:34.067 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:46:33 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:46:33.811+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:46:34.068 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:46:33 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:46:33.972+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:09.306583+0000 front 2023-12-16T21:39:09.306633+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:46:34.068 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:46:33 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:46:33.972+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:09.306551+0000 front 2023-12-16T21:39:09.306652+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:46:34.068 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:46:33 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:46:33.972+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:09.306605+0000 front 2023-12-16T21:39:09.306613+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:46:34.068 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:46:33 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:46:33.972+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:46:34.769 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:46:34 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:46:34.764+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:05.622397+0000 front 2023-12-16T21:39:05.622556+0000 (oldest deadline 2023-12-16T21:39:30.322546+0000) 2023-12-16T21:46:34.769 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:46:34 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:46:34.764+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:05.622506+0000 front 2023-12-16T21:39:05.622528+0000 (oldest deadline 2023-12-16T21:39:30.322546+0000) 2023-12-16T21:46:34.769 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:46:34 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:46:34.764+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:05.622479+0000 front 2023-12-16T21:39:05.622580+0000 (oldest deadline 2023-12-16T21:39:30.322546+0000) 2023-12-16T21:46:34.769 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:46:34 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:46:34.471+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:09.828911+0000 front 2023-12-16T21:39:09.828958+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:46:34.769 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:46:34 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:46:34.471+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:09.828868+0000 front 2023-12-16T21:39:09.828878+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:46:34.769 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:46:34 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:46:34.472+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:09.828947+0000 front 2023-12-16T21:39:09.828993+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:46:34.770 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:46:34 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:46:34.472+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:46:35.066 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:46:34 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:46:34.766+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:46:35.067 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:46:34 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:46:34.959+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:09.306583+0000 front 2023-12-16T21:39:09.306633+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:46:35.067 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:46:34 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:46:34.959+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:09.306551+0000 front 2023-12-16T21:39:09.306652+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:46:35.067 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:46:34 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:46:34.959+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:09.306605+0000 front 2023-12-16T21:39:09.306613+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:46:35.067 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:46:34 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:46:34.960+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:46:35.787 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:46:35 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:46:35.504+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:09.828911+0000 front 2023-12-16T21:39:09.828958+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:46:35.788 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:46:35 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:46:35.504+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:09.828868+0000 front 2023-12-16T21:39:09.828878+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:46:35.788 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:46:35 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:46:35.504+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:09.828947+0000 front 2023-12-16T21:39:09.828993+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:46:35.788 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:46:35 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:46:35.504+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:46:36.066 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:46:35 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:46:35.783+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:05.622397+0000 front 2023-12-16T21:39:05.622556+0000 (oldest deadline 2023-12-16T21:39:30.322546+0000) 2023-12-16T21:46:36.067 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:46:35 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:46:35.783+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:05.622506+0000 front 2023-12-16T21:39:05.622528+0000 (oldest deadline 2023-12-16T21:39:30.322546+0000) 2023-12-16T21:46:36.067 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:46:35 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:46:35.783+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:05.622479+0000 front 2023-12-16T21:39:05.622580+0000 (oldest deadline 2023-12-16T21:39:30.322546+0000) 2023-12-16T21:46:36.067 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:46:35 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:46:35.784+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:46:36.067 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:46:35 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:46:35.962+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:09.306583+0000 front 2023-12-16T21:39:09.306633+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:46:36.067 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:46:35 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:46:35.962+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:09.306551+0000 front 2023-12-16T21:39:09.306652+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:46:36.067 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:46:35 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:46:35.962+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:09.306605+0000 front 2023-12-16T21:39:09.306613+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:46:36.067 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:46:35 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:46:35.963+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:46:36.813 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Dec 16 21:46:36 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-mon-smithi045[80759]: 2023-12-16T21:46:36.366+0000 7f45a1f8f700 -1 mon.smithi045@0(probing) e2 get_health_metrics reporting 4315 slow ops, oldest is monmgrreport(gid 14214, 0 checks, 1 progress events) 2023-12-16T21:46:36.813 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:46:36 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:46:36.526+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:09.828911+0000 front 2023-12-16T21:39:09.828958+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:46:36.813 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:46:36 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:46:36.526+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:09.828868+0000 front 2023-12-16T21:39:09.828878+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:46:36.814 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:46:36 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:46:36.526+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:09.828947+0000 front 2023-12-16T21:39:09.828993+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:46:36.814 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:46:36 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:46:36.527+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:46:37.066 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:46:36 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:46:36.973+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:09.306583+0000 front 2023-12-16T21:39:09.306633+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:46:37.067 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:46:36 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:46:36.973+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:09.306551+0000 front 2023-12-16T21:39:09.306652+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:46:37.067 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:46:36 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:46:36.973+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:09.306605+0000 front 2023-12-16T21:39:09.306613+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:46:37.067 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:46:36 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:46:36.973+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:46:37.067 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:46:36 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:46:36.808+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:05.622397+0000 front 2023-12-16T21:39:05.622556+0000 (oldest deadline 2023-12-16T21:39:30.322546+0000) 2023-12-16T21:46:37.067 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:46:36 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:46:36.808+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:05.622506+0000 front 2023-12-16T21:39:05.622528+0000 (oldest deadline 2023-12-16T21:39:30.322546+0000) 2023-12-16T21:46:37.067 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:46:36 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:46:36.808+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:05.622479+0000 front 2023-12-16T21:39:05.622580+0000 (oldest deadline 2023-12-16T21:39:30.322546+0000) 2023-12-16T21:46:37.067 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:46:36 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:46:36.810+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:46:37.816 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:46:37 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:46:37.504+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:09.828911+0000 front 2023-12-16T21:39:09.828958+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:46:37.816 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:46:37 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:46:37.504+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:09.828868+0000 front 2023-12-16T21:39:09.828878+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:46:37.817 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:46:37 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:46:37.504+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:09.828947+0000 front 2023-12-16T21:39:09.828993+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:46:37.817 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:46:37 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:46:37.504+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:46:38.316 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:46:37 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:46:37.984+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:09.306583+0000 front 2023-12-16T21:39:09.306633+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:46:38.316 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:46:37 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:46:37.984+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:09.306551+0000 front 2023-12-16T21:39:09.306652+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:46:38.317 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:46:37 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:46:37.984+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:09.306605+0000 front 2023-12-16T21:39:09.306613+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:46:38.317 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:46:37 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:46:37.986+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:46:38.317 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:46:37 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:46:37.841+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:05.622397+0000 front 2023-12-16T21:39:05.622556+0000 (oldest deadline 2023-12-16T21:39:30.322546+0000) 2023-12-16T21:46:38.317 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:46:37 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:46:37.841+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:05.622506+0000 front 2023-12-16T21:39:05.622528+0000 (oldest deadline 2023-12-16T21:39:30.322546+0000) 2023-12-16T21:46:38.317 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:46:37 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:46:37.841+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:05.622479+0000 front 2023-12-16T21:39:05.622580+0000 (oldest deadline 2023-12-16T21:39:30.322546+0000) 2023-12-16T21:46:38.317 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:46:37 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:46:37.841+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:46:38.816 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:46:38 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:46:38.507+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:09.828911+0000 front 2023-12-16T21:39:09.828958+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:46:38.816 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:46:38 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:46:38.507+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:09.828868+0000 front 2023-12-16T21:39:09.828878+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:46:38.817 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:46:38 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:46:38.507+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:09.828947+0000 front 2023-12-16T21:39:09.828993+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:46:38.817 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:46:38 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:46:38.508+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:46:39.316 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:46:38 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:46:38.864+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:05.622397+0000 front 2023-12-16T21:39:05.622556+0000 (oldest deadline 2023-12-16T21:39:30.322546+0000) 2023-12-16T21:46:39.316 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:46:38 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:46:38.864+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:05.622506+0000 front 2023-12-16T21:39:05.622528+0000 (oldest deadline 2023-12-16T21:39:30.322546+0000) 2023-12-16T21:46:39.317 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:46:38 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:46:38.864+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:05.622479+0000 front 2023-12-16T21:39:05.622580+0000 (oldest deadline 2023-12-16T21:39:30.322546+0000) 2023-12-16T21:46:39.317 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:46:38 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:46:38.865+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:46:39.317 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:46:38 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:46:38.995+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:09.306583+0000 front 2023-12-16T21:39:09.306633+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:46:39.317 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:46:38 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:46:38.995+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:09.306551+0000 front 2023-12-16T21:39:09.306652+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:46:39.317 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:46:38 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:46:38.995+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:09.306605+0000 front 2023-12-16T21:39:09.306613+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:46:39.317 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:46:38 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:46:38.996+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:46:39.816 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:46:39 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:46:39.555+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:09.828911+0000 front 2023-12-16T21:39:09.828958+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:46:39.817 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:46:39 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:46:39.555+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:09.828868+0000 front 2023-12-16T21:39:09.828878+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:46:39.817 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:46:39 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:46:39.555+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:09.828947+0000 front 2023-12-16T21:39:09.828993+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:46:39.817 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:46:39 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:46:39.556+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:46:40.317 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:46:39 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:46:39.982+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:09.306583+0000 front 2023-12-16T21:39:09.306633+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:46:40.317 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:46:39 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:46:39.982+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:09.306551+0000 front 2023-12-16T21:39:09.306652+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:46:40.317 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:46:39 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:46:39.982+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:09.306605+0000 front 2023-12-16T21:39:09.306613+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:46:40.317 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:46:39 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:46:39.983+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:46:40.318 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:46:39 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:46:39.818+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:05.622397+0000 front 2023-12-16T21:39:05.622556+0000 (oldest deadline 2023-12-16T21:39:30.322546+0000) 2023-12-16T21:46:40.318 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:46:39 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:46:39.818+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:05.622506+0000 front 2023-12-16T21:39:05.622528+0000 (oldest deadline 2023-12-16T21:39:30.322546+0000) 2023-12-16T21:46:40.318 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:46:39 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:46:39.818+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:05.622479+0000 front 2023-12-16T21:39:05.622580+0000 (oldest deadline 2023-12-16T21:39:30.322546+0000) 2023-12-16T21:46:40.318 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:46:39 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:46:39.818+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:46:40.816 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:46:40 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:46:40.525+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:09.828911+0000 front 2023-12-16T21:39:09.828958+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:46:40.816 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:46:40 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:46:40.525+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:09.828868+0000 front 2023-12-16T21:39:09.828878+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:46:40.817 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:46:40 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:46:40.525+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:09.828947+0000 front 2023-12-16T21:39:09.828993+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:46:40.817 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:46:40 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:46:40.525+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:46:41.316 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:46:40 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:46:40.840+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:05.622397+0000 front 2023-12-16T21:39:05.622556+0000 (oldest deadline 2023-12-16T21:39:30.322546+0000) 2023-12-16T21:46:41.317 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:46:40 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:46:40.840+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:05.622506+0000 front 2023-12-16T21:39:05.622528+0000 (oldest deadline 2023-12-16T21:39:30.322546+0000) 2023-12-16T21:46:41.317 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:46:40 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:46:40.840+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:05.622479+0000 front 2023-12-16T21:39:05.622580+0000 (oldest deadline 2023-12-16T21:39:30.322546+0000) 2023-12-16T21:46:41.317 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:46:40 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:46:40.840+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:46:41.317 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:46:40 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:46:40.951+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:09.306583+0000 front 2023-12-16T21:39:09.306633+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:46:41.317 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:46:40 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:46:40.951+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:09.306551+0000 front 2023-12-16T21:39:09.306652+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:46:41.317 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:46:40 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:46:40.951+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:09.306605+0000 front 2023-12-16T21:39:09.306613+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:46:41.317 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:46:40 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:46:40.952+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:46:41.650 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Dec 16 21:46:41 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-mon-smithi045[80759]: 2023-12-16T21:46:41.368+0000 7f45a1f8f700 -1 mon.smithi045@0(probing) e2 get_health_metrics reporting 4422 slow ops, oldest is monmgrreport(gid 14214, 0 checks, 1 progress events) 2023-12-16T21:46:41.651 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:46:41 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:46:41.565+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:09.828911+0000 front 2023-12-16T21:39:09.828958+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:46:41.651 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:46:41 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:46:41.565+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:09.828868+0000 front 2023-12-16T21:39:09.828878+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:46:41.651 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:46:41 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:46:41.565+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:09.828947+0000 front 2023-12-16T21:39:09.828993+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:46:41.651 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:46:41 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:46:41.566+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:46:41.980 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:46:41 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:46:41.835+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:05.622397+0000 front 2023-12-16T21:39:05.622556+0000 (oldest deadline 2023-12-16T21:39:30.322546+0000) 2023-12-16T21:46:41.980 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:46:41 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:46:41.835+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:05.622506+0000 front 2023-12-16T21:39:05.622528+0000 (oldest deadline 2023-12-16T21:39:30.322546+0000) 2023-12-16T21:46:41.980 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:46:41 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:46:41.835+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:05.622479+0000 front 2023-12-16T21:39:05.622580+0000 (oldest deadline 2023-12-16T21:39:30.322546+0000) 2023-12-16T21:46:41.980 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:46:41 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:46:41.836+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:46:42.316 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:46:41 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:46:41.977+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:09.306583+0000 front 2023-12-16T21:39:09.306633+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:46:42.316 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:46:41 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:46:41.977+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:09.306551+0000 front 2023-12-16T21:39:09.306652+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:46:42.317 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:46:41 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:46:41.977+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:09.306605+0000 front 2023-12-16T21:39:09.306613+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:46:42.317 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:46:41 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:46:41.978+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:46:42.832 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:46:42 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:46:42.573+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:09.828911+0000 front 2023-12-16T21:39:09.828958+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:46:42.832 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:46:42 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:46:42.573+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:09.828868+0000 front 2023-12-16T21:39:09.828878+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:46:42.832 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:46:42 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:46:42.573+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:09.828947+0000 front 2023-12-16T21:39:09.828993+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:46:42.832 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:46:42 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:46:42.574+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:46:43.316 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:46:43 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:46:43.023+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:09.306583+0000 front 2023-12-16T21:39:09.306633+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:46:43.316 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:46:43 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:46:43.023+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:09.306551+0000 front 2023-12-16T21:39:09.306652+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:46:43.317 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:46:43 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:46:43.023+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:09.306605+0000 front 2023-12-16T21:39:09.306613+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:46:43.317 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:46:43 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:46:43.024+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:46:43.317 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:46:42 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:46:42.829+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:05.622397+0000 front 2023-12-16T21:39:05.622556+0000 (oldest deadline 2023-12-16T21:39:30.322546+0000) 2023-12-16T21:46:43.317 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:46:42 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:46:42.829+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:05.622506+0000 front 2023-12-16T21:39:05.622528+0000 (oldest deadline 2023-12-16T21:39:30.322546+0000) 2023-12-16T21:46:43.317 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:46:42 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:46:42.829+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:05.622479+0000 front 2023-12-16T21:39:05.622580+0000 (oldest deadline 2023-12-16T21:39:30.322546+0000) 2023-12-16T21:46:43.317 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:46:42 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:46:42.830+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:46:43.814 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:46:43 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:46:43.595+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:09.828911+0000 front 2023-12-16T21:39:09.828958+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:46:43.815 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:46:43 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:46:43.595+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:09.828868+0000 front 2023-12-16T21:39:09.828878+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:46:43.815 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:46:43 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:46:43.595+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:09.828947+0000 front 2023-12-16T21:39:09.828993+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:46:43.815 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:46:43 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:46:43.595+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:46:44.066 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:46:43 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:46:43.811+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:05.622397+0000 front 2023-12-16T21:39:05.622556+0000 (oldest deadline 2023-12-16T21:39:30.322546+0000) 2023-12-16T21:46:44.067 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:46:43 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:46:43.811+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:05.622506+0000 front 2023-12-16T21:39:05.622528+0000 (oldest deadline 2023-12-16T21:39:30.322546+0000) 2023-12-16T21:46:44.067 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:46:43 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:46:43.811+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:05.622479+0000 front 2023-12-16T21:39:05.622580+0000 (oldest deadline 2023-12-16T21:39:30.322546+0000) 2023-12-16T21:46:44.067 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:46:43 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:46:43.812+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:46:44.067 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:46:43 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:46:43.978+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:09.306583+0000 front 2023-12-16T21:39:09.306633+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:46:44.067 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:46:43 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:46:43.978+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:09.306551+0000 front 2023-12-16T21:39:09.306652+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:46:44.067 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:46:43 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:46:43.978+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:09.306605+0000 front 2023-12-16T21:39:09.306613+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:46:44.067 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:46:43 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:46:43.978+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:46:44.958 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:46:44 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:46:44.638+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:09.828911+0000 front 2023-12-16T21:39:09.828958+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:46:44.958 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:46:44 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:46:44.638+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:09.828868+0000 front 2023-12-16T21:39:09.828878+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:46:44.959 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:46:44 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:46:44.638+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:09.828947+0000 front 2023-12-16T21:39:09.828993+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:46:44.959 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:46:44 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:46:44.640+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:46:44.959 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:46:44 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:46:44.848+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:05.622397+0000 front 2023-12-16T21:39:05.622556+0000 (oldest deadline 2023-12-16T21:39:30.322546+0000) 2023-12-16T21:46:44.959 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:46:44 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:46:44.848+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:05.622506+0000 front 2023-12-16T21:39:05.622528+0000 (oldest deadline 2023-12-16T21:39:30.322546+0000) 2023-12-16T21:46:44.959 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:46:44 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:46:44.849+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:05.622479+0000 front 2023-12-16T21:39:05.622580+0000 (oldest deadline 2023-12-16T21:39:30.322546+0000) 2023-12-16T21:46:44.959 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:46:44 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:46:44.850+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:46:45.317 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:46:44 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:46:44.955+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:09.306583+0000 front 2023-12-16T21:39:09.306633+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:46:45.317 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:46:44 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:46:44.955+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:09.306551+0000 front 2023-12-16T21:39:09.306652+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:46:45.317 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:46:44 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:46:44.955+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:09.306605+0000 front 2023-12-16T21:39:09.306613+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:46:45.317 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:46:44 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:46:44.956+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:46:45.938 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:46:45 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:46:45.834+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:05.622397+0000 front 2023-12-16T21:39:05.622556+0000 (oldest deadline 2023-12-16T21:39:30.322546+0000) 2023-12-16T21:46:45.938 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:46:45 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:46:45.834+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:05.622506+0000 front 2023-12-16T21:39:05.622528+0000 (oldest deadline 2023-12-16T21:39:30.322546+0000) 2023-12-16T21:46:45.938 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:46:45 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:46:45.834+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:05.622479+0000 front 2023-12-16T21:39:05.622580+0000 (oldest deadline 2023-12-16T21:39:30.322546+0000) 2023-12-16T21:46:45.938 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:46:45 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:46:45.835+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:46:45.939 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:46:45 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:46:45.638+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:09.828911+0000 front 2023-12-16T21:39:09.828958+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:46:45.939 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:46:45 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:46:45.638+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:09.828868+0000 front 2023-12-16T21:39:09.828878+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:46:45.939 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:46:45 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:46:45.638+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:09.828947+0000 front 2023-12-16T21:39:09.828993+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:46:45.939 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:46:45 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:46:45.638+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:46:46.316 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:46:45 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:46:45.935+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:09.306583+0000 front 2023-12-16T21:39:09.306633+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:46:46.317 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:46:45 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:46:45.935+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:09.306551+0000 front 2023-12-16T21:39:09.306652+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:46:46.317 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:46:45 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:46:45.935+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:09.306605+0000 front 2023-12-16T21:39:09.306613+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:46:46.317 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:46:45 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:46:45.936+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:46:46.667 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Dec 16 21:46:46 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-mon-smithi045[80759]: 2023-12-16T21:46:46.369+0000 7f45a1f8f700 -1 mon.smithi045@0(probing) e2 get_health_metrics reporting 4529 slow ops, oldest is monmgrreport(gid 14214, 0 checks, 1 progress events) 2023-12-16T21:46:47.066 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:46:46 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:46:46.906+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:09.306583+0000 front 2023-12-16T21:39:09.306633+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:46:47.067 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:46:46 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:46:46.906+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:09.306551+0000 front 2023-12-16T21:39:09.306652+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:46:47.067 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:46:46 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:46:46.906+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:09.306605+0000 front 2023-12-16T21:39:09.306613+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:46:47.067 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:46:46 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:46:46.908+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:46:47.067 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:46:46 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:46:46.817+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:05.622397+0000 front 2023-12-16T21:39:05.622556+0000 (oldest deadline 2023-12-16T21:39:30.322546+0000) 2023-12-16T21:46:47.067 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:46:46 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:46:46.817+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:05.622506+0000 front 2023-12-16T21:39:05.622528+0000 (oldest deadline 2023-12-16T21:39:30.322546+0000) 2023-12-16T21:46:47.067 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:46:46 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:46:46.817+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:05.622479+0000 front 2023-12-16T21:39:05.622580+0000 (oldest deadline 2023-12-16T21:39:30.322546+0000) 2023-12-16T21:46:47.067 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:46:46 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:46:46.817+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:46:47.067 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:46:46 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:46:46.664+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:09.828911+0000 front 2023-12-16T21:39:09.828958+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:46:47.068 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:46:46 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:46:46.664+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:09.828868+0000 front 2023-12-16T21:39:09.828878+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:46:47.068 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:46:46 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:46:46.664+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:09.828947+0000 front 2023-12-16T21:39:09.828993+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:46:47.068 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:46:46 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:46:46.664+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:46:48.066 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:46:47 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:46:47.924+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:09.306583+0000 front 2023-12-16T21:39:09.306633+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:46:48.067 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:46:47 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:46:47.924+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:09.306551+0000 front 2023-12-16T21:39:09.306652+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:46:48.067 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:46:47 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:46:47.924+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:09.306605+0000 front 2023-12-16T21:39:09.306613+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:46:48.067 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:46:47 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:46:47.925+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:46:48.067 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:46:47 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:46:47.702+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:09.828911+0000 front 2023-12-16T21:39:09.828958+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:46:48.067 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:46:47 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:46:47.702+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:09.828868+0000 front 2023-12-16T21:39:09.828878+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:46:48.067 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:46:47 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:46:47.702+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:09.828947+0000 front 2023-12-16T21:39:09.828993+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:46:48.067 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:46:47 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:46:47.702+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:46:48.067 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:46:47 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:46:47.786+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:05.622397+0000 front 2023-12-16T21:39:05.622556+0000 (oldest deadline 2023-12-16T21:39:30.322546+0000) 2023-12-16T21:46:48.067 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:46:47 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:46:47.786+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:05.622506+0000 front 2023-12-16T21:39:05.622528+0000 (oldest deadline 2023-12-16T21:39:30.322546+0000) 2023-12-16T21:46:48.068 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:46:47 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:46:47.786+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:05.622479+0000 front 2023-12-16T21:39:05.622580+0000 (oldest deadline 2023-12-16T21:39:30.322546+0000) 2023-12-16T21:46:48.068 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:46:47 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:46:47.786+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:46:49.066 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:46:48 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:46:48.733+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:09.828911+0000 front 2023-12-16T21:39:09.828958+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:46:49.067 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:46:48 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:46:48.733+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:09.828868+0000 front 2023-12-16T21:39:09.828878+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:46:49.067 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:46:48 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:46:48.733+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:09.828947+0000 front 2023-12-16T21:39:09.828993+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:46:49.067 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:46:48 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:46:48.734+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:46:49.067 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:46:48 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:46:48.828+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:05.622397+0000 front 2023-12-16T21:39:05.622556+0000 (oldest deadline 2023-12-16T21:39:30.322546+0000) 2023-12-16T21:46:49.067 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:46:48 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:46:48.828+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:05.622506+0000 front 2023-12-16T21:39:05.622528+0000 (oldest deadline 2023-12-16T21:39:30.322546+0000) 2023-12-16T21:46:49.067 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:46:48 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:46:48.828+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:05.622479+0000 front 2023-12-16T21:39:05.622580+0000 (oldest deadline 2023-12-16T21:39:30.322546+0000) 2023-12-16T21:46:49.067 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:46:48 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:46:48.830+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:46:49.068 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:46:48 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:46:48.965+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:09.306583+0000 front 2023-12-16T21:39:09.306633+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:46:49.068 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:46:48 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:46:48.965+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:09.306551+0000 front 2023-12-16T21:39:09.306652+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:46:49.068 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:46:48 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:46:48.965+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:09.306605+0000 front 2023-12-16T21:39:09.306613+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:46:49.068 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:46:48 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:46:48.966+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:46:50.067 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:46:49 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:46:49.878+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:05.622397+0000 front 2023-12-16T21:39:05.622556+0000 (oldest deadline 2023-12-16T21:39:30.322546+0000) 2023-12-16T21:46:50.067 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:46:49 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:46:49.878+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:05.622506+0000 front 2023-12-16T21:39:05.622528+0000 (oldest deadline 2023-12-16T21:39:30.322546+0000) 2023-12-16T21:46:50.067 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:46:49 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:46:49.878+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:05.622479+0000 front 2023-12-16T21:39:05.622580+0000 (oldest deadline 2023-12-16T21:39:30.322546+0000) 2023-12-16T21:46:50.067 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:46:49 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:46:49.879+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:46:50.067 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:46:49 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:46:49.945+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:09.306583+0000 front 2023-12-16T21:39:09.306633+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:46:50.067 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:46:49 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:46:49.945+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:09.306551+0000 front 2023-12-16T21:39:09.306652+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:46:50.067 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:46:49 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:46:49.945+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:09.306605+0000 front 2023-12-16T21:39:09.306613+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:46:50.067 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:46:49 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:46:49.946+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:46:50.068 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:46:49 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:46:49.764+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:09.828911+0000 front 2023-12-16T21:39:09.828958+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:46:50.068 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:46:49 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:46:49.764+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:09.828868+0000 front 2023-12-16T21:39:09.828878+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:46:50.068 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:46:49 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:46:49.764+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:09.828947+0000 front 2023-12-16T21:39:09.828993+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:46:50.068 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:46:49 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:46:49.764+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:46:51.066 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:46:50 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:46:50.891+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:05.622397+0000 front 2023-12-16T21:39:05.622556+0000 (oldest deadline 2023-12-16T21:39:30.322546+0000) 2023-12-16T21:46:51.067 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:46:50 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:46:50.891+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:05.622506+0000 front 2023-12-16T21:39:05.622528+0000 (oldest deadline 2023-12-16T21:39:30.322546+0000) 2023-12-16T21:46:51.067 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:46:50 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:46:50.891+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:05.622479+0000 front 2023-12-16T21:39:05.622580+0000 (oldest deadline 2023-12-16T21:39:30.322546+0000) 2023-12-16T21:46:51.067 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:46:50 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:46:50.892+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:46:51.067 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:46:50 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:46:50.775+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:09.828911+0000 front 2023-12-16T21:39:09.828958+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:46:51.067 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:46:50 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:46:50.775+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:09.828868+0000 front 2023-12-16T21:39:09.828878+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:46:51.067 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:46:50 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:46:50.775+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:09.828947+0000 front 2023-12-16T21:39:09.828993+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:46:51.067 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:46:50 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:46:50.776+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:46:51.067 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:46:50 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:46:50.926+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:09.306583+0000 front 2023-12-16T21:39:09.306633+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:46:51.068 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:46:50 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:46:50.926+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:09.306551+0000 front 2023-12-16T21:39:09.306652+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:46:51.068 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:46:50 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:46:50.926+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:09.306605+0000 front 2023-12-16T21:39:09.306613+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:46:51.068 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:46:50 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:46:50.927+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:46:51.651 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Dec 16 21:46:51 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-mon-smithi045[80759]: 2023-12-16T21:46:51.370+0000 7f45a1f8f700 -1 mon.smithi045@0(probing) e2 get_health_metrics reporting 4662 slow ops, oldest is monmgrreport(gid 14214, 0 checks, 1 progress events) 2023-12-16T21:46:51.948 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:46:51 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:46:51.751+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:09.828911+0000 front 2023-12-16T21:39:09.828958+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:46:51.949 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:46:51 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:46:51.751+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:09.828868+0000 front 2023-12-16T21:39:09.828878+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:46:51.949 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:46:51 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:46:51.751+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:09.828947+0000 front 2023-12-16T21:39:09.828993+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:46:51.949 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:46:51 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:46:51.752+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:46:51.949 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:46:51 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:46:51.884+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:05.622397+0000 front 2023-12-16T21:39:05.622556+0000 (oldest deadline 2023-12-16T21:39:30.322546+0000) 2023-12-16T21:46:51.949 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:46:51 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:46:51.884+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:05.622506+0000 front 2023-12-16T21:39:05.622528+0000 (oldest deadline 2023-12-16T21:39:30.322546+0000) 2023-12-16T21:46:51.949 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:46:51 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:46:51.884+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:05.622479+0000 front 2023-12-16T21:39:05.622580+0000 (oldest deadline 2023-12-16T21:39:30.322546+0000) 2023-12-16T21:46:51.949 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:46:51 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:46:51.885+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:46:52.316 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:46:51 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:46:51.945+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:09.306583+0000 front 2023-12-16T21:39:09.306633+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:46:52.316 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:46:51 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:46:51.945+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:09.306551+0000 front 2023-12-16T21:39:09.306652+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:46:52.317 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:46:51 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:46:51.945+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:09.306605+0000 front 2023-12-16T21:39:09.306613+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:46:52.317 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:46:51 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:46:51.946+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:46:53.066 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:46:52 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:46:52.932+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:05.622397+0000 front 2023-12-16T21:39:05.622556+0000 (oldest deadline 2023-12-16T21:39:30.322546+0000) 2023-12-16T21:46:53.067 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:46:52 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:46:52.932+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:05.622506+0000 front 2023-12-16T21:39:05.622528+0000 (oldest deadline 2023-12-16T21:39:30.322546+0000) 2023-12-16T21:46:53.067 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:46:52 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:46:52.932+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:05.622479+0000 front 2023-12-16T21:39:05.622580+0000 (oldest deadline 2023-12-16T21:39:30.322546+0000) 2023-12-16T21:46:53.067 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:46:52 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:46:52.933+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:46:53.067 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:46:52 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:46:52.954+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:09.306583+0000 front 2023-12-16T21:39:09.306633+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:46:53.067 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:46:52 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:46:52.954+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:09.306551+0000 front 2023-12-16T21:39:09.306652+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:46:53.067 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:46:52 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:46:52.954+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:09.306605+0000 front 2023-12-16T21:39:09.306613+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:46:53.068 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:46:52 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:46:52.954+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:46:53.068 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:46:52 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:46:52.733+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:09.828911+0000 front 2023-12-16T21:39:09.828958+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:46:53.068 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:46:52 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:46:52.733+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:09.828868+0000 front 2023-12-16T21:39:09.828878+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:46:53.068 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:46:52 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:46:52.733+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:09.828947+0000 front 2023-12-16T21:39:09.828993+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:46:53.068 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:46:52 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:46:52.734+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:46:53.816 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:46:53 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:46:53.717+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:09.828911+0000 front 2023-12-16T21:39:09.828958+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:46:53.816 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:46:53 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:46:53.717+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:09.828868+0000 front 2023-12-16T21:39:09.828878+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:46:53.817 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:46:53 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:46:53.717+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:09.828947+0000 front 2023-12-16T21:39:09.828993+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:46:53.817 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:46:53 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:46:53.718+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:46:54.316 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:46:53 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:46:53.972+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:05.622397+0000 front 2023-12-16T21:39:05.622556+0000 (oldest deadline 2023-12-16T21:39:30.322546+0000) 2023-12-16T21:46:54.317 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:46:53 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:46:53.972+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:05.622506+0000 front 2023-12-16T21:39:05.622528+0000 (oldest deadline 2023-12-16T21:39:30.322546+0000) 2023-12-16T21:46:54.317 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:46:53 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:46:53.972+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:05.622479+0000 front 2023-12-16T21:39:05.622580+0000 (oldest deadline 2023-12-16T21:39:30.322546+0000) 2023-12-16T21:46:54.317 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:46:53 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:46:53.974+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:46:54.317 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:46:53 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:46:53.997+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:09.306583+0000 front 2023-12-16T21:39:09.306633+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:46:54.317 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:46:53 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:46:53.997+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:09.306551+0000 front 2023-12-16T21:39:09.306652+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:46:54.318 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:46:53 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:46:53.997+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:09.306605+0000 front 2023-12-16T21:39:09.306613+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:46:54.318 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:46:53 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:46:53.998+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:46:54.986 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:46:54 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:46:54.709+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:09.828911+0000 front 2023-12-16T21:39:09.828958+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:46:54.987 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:46:54 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:46:54.709+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:09.828868+0000 front 2023-12-16T21:39:09.828878+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:46:54.987 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:46:54 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:46:54.709+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:09.828947+0000 front 2023-12-16T21:39:09.828993+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:46:54.987 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:46:54 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:46:54.710+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:46:55.316 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:46:54 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:46:54.983+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:05.622397+0000 front 2023-12-16T21:39:05.622556+0000 (oldest deadline 2023-12-16T21:39:30.322546+0000) 2023-12-16T21:46:55.317 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:46:54 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:46:54.983+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:05.622506+0000 front 2023-12-16T21:39:05.622528+0000 (oldest deadline 2023-12-16T21:39:30.322546+0000) 2023-12-16T21:46:55.317 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:46:54 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:46:54.983+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:05.622479+0000 front 2023-12-16T21:39:05.622580+0000 (oldest deadline 2023-12-16T21:39:30.322546+0000) 2023-12-16T21:46:55.317 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:46:54 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:46:54.984+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:46:55.317 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:46:55 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:46:55.013+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:09.306583+0000 front 2023-12-16T21:39:09.306633+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:46:55.317 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:46:55 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:46:55.013+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:09.306551+0000 front 2023-12-16T21:39:09.306652+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:46:55.317 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:46:55 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:46:55.013+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:09.306605+0000 front 2023-12-16T21:39:09.306613+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:46:55.317 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:46:55 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:46:55.013+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:46:55.984 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:46:55 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:46:55.697+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:09.828911+0000 front 2023-12-16T21:39:09.828958+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:46:55.984 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:46:55 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:46:55.697+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:09.828868+0000 front 2023-12-16T21:39:09.828878+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:46:55.984 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:46:55 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:46:55.697+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:09.828947+0000 front 2023-12-16T21:39:09.828993+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:46:55.984 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:46:55 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:46:55.698+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:46:56.316 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:46:56 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:46:55.999+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:09.306583+0000 front 2023-12-16T21:39:09.306633+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:46:56.317 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:46:56 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:46:55.999+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:09.306551+0000 front 2023-12-16T21:39:09.306652+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:46:56.317 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:46:56 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:46:55.999+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:09.306605+0000 front 2023-12-16T21:39:09.306613+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:46:56.317 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:46:56 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:46:56.000+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:46:56.317 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:46:55 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:46:55.980+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:05.622397+0000 front 2023-12-16T21:39:05.622556+0000 (oldest deadline 2023-12-16T21:39:30.322546+0000) 2023-12-16T21:46:56.317 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:46:55 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:46:55.980+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:05.622506+0000 front 2023-12-16T21:39:05.622528+0000 (oldest deadline 2023-12-16T21:39:30.322546+0000) 2023-12-16T21:46:56.317 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:46:55 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:46:55.980+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:05.622479+0000 front 2023-12-16T21:39:05.622580+0000 (oldest deadline 2023-12-16T21:39:30.322546+0000) 2023-12-16T21:46:56.317 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:46:55 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:46:55.982+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:46:56.748 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Dec 16 21:46:56 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-mon-smithi045[80759]: 2023-12-16T21:46:56.371+0000 7f45a1f8f700 -1 mon.smithi045@0(probing) e2 get_health_metrics reporting 4757 slow ops, oldest is monmgrreport(gid 14214, 0 checks, 1 progress events) 2023-12-16T21:46:57.066 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:46:56 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:46:56.979+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:05.622397+0000 front 2023-12-16T21:39:05.622556+0000 (oldest deadline 2023-12-16T21:39:30.322546+0000) 2023-12-16T21:46:57.067 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:46:56 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:46:56.979+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:05.622506+0000 front 2023-12-16T21:39:05.622528+0000 (oldest deadline 2023-12-16T21:39:30.322546+0000) 2023-12-16T21:46:57.067 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:46:56 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:46:56.979+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:05.622479+0000 front 2023-12-16T21:39:05.622580+0000 (oldest deadline 2023-12-16T21:39:30.322546+0000) 2023-12-16T21:46:57.067 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:46:56 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:46:56.980+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:46:57.067 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:46:56 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:46:56.745+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:09.828911+0000 front 2023-12-16T21:39:09.828958+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:46:57.067 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:46:56 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:46:56.745+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:09.828868+0000 front 2023-12-16T21:39:09.828878+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:46:57.067 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:46:56 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:46:56.745+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:09.828947+0000 front 2023-12-16T21:39:09.828993+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:46:57.067 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:46:56 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:46:56.746+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:46:57.067 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:46:56 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:46:56.987+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:09.306583+0000 front 2023-12-16T21:39:09.306633+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:46:57.067 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:46:56 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:46:56.987+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:09.306551+0000 front 2023-12-16T21:39:09.306652+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:46:57.068 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:46:56 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:46:56.987+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:09.306605+0000 front 2023-12-16T21:39:09.306613+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:46:57.068 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:46:56 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:46:56.988+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:46:57.985 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:46:57 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:46:57.729+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:09.828911+0000 front 2023-12-16T21:39:09.828958+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:46:57.985 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:46:57 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:46:57.730+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:09.828868+0000 front 2023-12-16T21:39:09.828878+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:46:57.985 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:46:57 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:46:57.730+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:09.828947+0000 front 2023-12-16T21:39:09.828993+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:46:57.985 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:46:57 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:46:57.730+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:46:58.316 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:46:57 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:46:57.982+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:09.306583+0000 front 2023-12-16T21:39:09.306633+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:46:58.317 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:46:57 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:46:57.982+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:09.306551+0000 front 2023-12-16T21:39:09.306652+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:46:58.317 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:46:57 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:46:57.982+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:09.306605+0000 front 2023-12-16T21:39:09.306613+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:46:58.317 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:46:57 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:46:57.983+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:46:58.317 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:46:58 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:46:58.027+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:05.622397+0000 front 2023-12-16T21:39:05.622556+0000 (oldest deadline 2023-12-16T21:39:30.322546+0000) 2023-12-16T21:46:58.317 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:46:58 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:46:58.027+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:05.622506+0000 front 2023-12-16T21:39:05.622528+0000 (oldest deadline 2023-12-16T21:39:30.322546+0000) 2023-12-16T21:46:58.317 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:46:58 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:46:58.027+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:05.622479+0000 front 2023-12-16T21:39:05.622580+0000 (oldest deadline 2023-12-16T21:39:30.322546+0000) 2023-12-16T21:46:58.317 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:46:58 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:46:58.028+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:46:58.972 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:46:58 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:46:58.712+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:09.828911+0000 front 2023-12-16T21:39:09.828958+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:46:58.972 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:46:58 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:46:58.712+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:09.828868+0000 front 2023-12-16T21:39:09.828878+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:46:58.972 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:46:58 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:46:58.712+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:09.828947+0000 front 2023-12-16T21:39:09.828993+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:46:58.972 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:46:58 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:46:58.713+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:46:59.316 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:46:58 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:46:58.969+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:09.306583+0000 front 2023-12-16T21:39:09.306633+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:46:59.316 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:46:58 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:46:58.969+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:09.306551+0000 front 2023-12-16T21:39:09.306652+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:46:59.317 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:46:58 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:46:58.969+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:09.306605+0000 front 2023-12-16T21:39:09.306613+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:46:59.317 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:46:58 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:46:58.970+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:46:59.317 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:46:59 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:46:59.048+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:05.622397+0000 front 2023-12-16T21:39:05.622556+0000 (oldest deadline 2023-12-16T21:39:30.322546+0000) 2023-12-16T21:46:59.317 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:46:59 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:46:59.048+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:05.622506+0000 front 2023-12-16T21:39:05.622528+0000 (oldest deadline 2023-12-16T21:39:30.322546+0000) 2023-12-16T21:46:59.317 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:46:59 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:46:59.048+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:05.622479+0000 front 2023-12-16T21:39:05.622580+0000 (oldest deadline 2023-12-16T21:39:30.322546+0000) 2023-12-16T21:46:59.317 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:46:59 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:46:59.049+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:47:00.019 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:46:59 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:46:59.722+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:09.828911+0000 front 2023-12-16T21:39:09.828958+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:47:00.019 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:46:59 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:46:59.722+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:09.828868+0000 front 2023-12-16T21:39:09.828878+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:47:00.019 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:46:59 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:46:59.722+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:09.828947+0000 front 2023-12-16T21:39:09.828993+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:47:00.019 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:46:59 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:46:59.723+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:47:00.316 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:47:00 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:47:00.028+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:05.622397+0000 front 2023-12-16T21:39:05.622556+0000 (oldest deadline 2023-12-16T21:39:30.322546+0000) 2023-12-16T21:47:00.316 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:47:00 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:47:00.028+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:05.622506+0000 front 2023-12-16T21:39:05.622528+0000 (oldest deadline 2023-12-16T21:39:30.322546+0000) 2023-12-16T21:47:00.317 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:47:00 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:47:00.028+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:05.622479+0000 front 2023-12-16T21:39:05.622580+0000 (oldest deadline 2023-12-16T21:39:30.322546+0000) 2023-12-16T21:47:00.317 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:47:00 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:47:00.028+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:47:00.317 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:47:00 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:47:00.016+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:09.306583+0000 front 2023-12-16T21:39:09.306633+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:47:00.317 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:47:00 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:47:00.016+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:09.306551+0000 front 2023-12-16T21:39:09.306652+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:47:00.317 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:47:00 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:47:00.016+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:09.306605+0000 front 2023-12-16T21:39:09.306613+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:47:00.317 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:47:00 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:47:00.016+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:47:01.024 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:47:00 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:47:00.753+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:09.828911+0000 front 2023-12-16T21:39:09.828958+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:47:01.024 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:47:00 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:47:00.753+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:09.828868+0000 front 2023-12-16T21:39:09.828878+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:47:01.024 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:47:00 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:47:00.753+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:09.828947+0000 front 2023-12-16T21:39:09.828993+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:47:01.024 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:47:00 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:47:00.754+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:47:01.317 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:47:01 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:47:01.020+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:09.306583+0000 front 2023-12-16T21:39:09.306633+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:47:01.317 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:47:01 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:47:01.020+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:09.306551+0000 front 2023-12-16T21:39:09.306652+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:47:01.317 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:47:01 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:47:01.021+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:09.306605+0000 front 2023-12-16T21:39:09.306613+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:47:01.317 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:47:01 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:47:01.021+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:47:01.317 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:47:01 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:47:01.064+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:05.622397+0000 front 2023-12-16T21:39:05.622556+0000 (oldest deadline 2023-12-16T21:39:30.322546+0000) 2023-12-16T21:47:01.318 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:47:01 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:47:01.064+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:05.622506+0000 front 2023-12-16T21:39:05.622528+0000 (oldest deadline 2023-12-16T21:39:30.322546+0000) 2023-12-16T21:47:01.318 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:47:01 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:47:01.064+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:05.622479+0000 front 2023-12-16T21:39:05.622580+0000 (oldest deadline 2023-12-16T21:39:30.322546+0000) 2023-12-16T21:47:01.318 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:47:01 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:47:01.066+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:47:01.651 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Dec 16 21:47:01 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-mon-smithi045[80759]: 2023-12-16T21:47:01.371+0000 7f45a1f8f700 -1 mon.smithi045@0(probing) e2 get_health_metrics reporting 4854 slow ops, oldest is monmgrreport(gid 14214, 0 checks, 1 progress events) 2023-12-16T21:47:01.994 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:47:01 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:47:01.771+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:09.828911+0000 front 2023-12-16T21:39:09.828958+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:47:01.994 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:47:01 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:47:01.771+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:09.828868+0000 front 2023-12-16T21:39:09.828878+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:47:01.994 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:47:01 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:47:01.771+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:09.828947+0000 front 2023-12-16T21:39:09.828993+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:47:01.994 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:47:01 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:47:01.772+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:47:02.290 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:47:02 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:47:02.058+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:05.622397+0000 front 2023-12-16T21:39:05.622556+0000 (oldest deadline 2023-12-16T21:39:30.322546+0000) 2023-12-16T21:47:02.290 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:47:02 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:47:02.058+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:05.622506+0000 front 2023-12-16T21:39:05.622528+0000 (oldest deadline 2023-12-16T21:39:30.322546+0000) 2023-12-16T21:47:02.290 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:47:02 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:47:02.058+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:05.622479+0000 front 2023-12-16T21:39:05.622580+0000 (oldest deadline 2023-12-16T21:39:30.322546+0000) 2023-12-16T21:47:02.291 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:47:02 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:47:02.060+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:47:02.291 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:47:01 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:47:01.991+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:09.306583+0000 front 2023-12-16T21:39:09.306633+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:47:02.291 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:47:01 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:47:01.991+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:09.306551+0000 front 2023-12-16T21:39:09.306652+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:47:02.291 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:47:01 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:47:01.991+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:09.306605+0000 front 2023-12-16T21:39:09.306613+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:47:02.291 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:47:01 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:47:01.992+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:47:03.013 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:47:02 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:47:02.745+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:09.828911+0000 front 2023-12-16T21:39:09.828958+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:47:03.013 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:47:02 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:47:02.745+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:09.828868+0000 front 2023-12-16T21:39:09.828878+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:47:03.013 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:47:02 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:47:02.745+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:09.828947+0000 front 2023-12-16T21:39:09.828993+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:47:03.014 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:47:02 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:47:02.746+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:47:03.014 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:47:02 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:47:02.986+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:09.306583+0000 front 2023-12-16T21:39:09.306633+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:47:03.014 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:47:02 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:47:02.986+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:09.306551+0000 front 2023-12-16T21:39:09.306652+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:47:03.014 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:47:02 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:47:02.986+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:09.306605+0000 front 2023-12-16T21:39:09.306613+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:47:03.014 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:47:02 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:47:02.987+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:47:03.316 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:47:03 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:47:03.010+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:05.622397+0000 front 2023-12-16T21:39:05.622556+0000 (oldest deadline 2023-12-16T21:39:30.322546+0000) 2023-12-16T21:47:03.316 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:47:03 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:47:03.010+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:05.622506+0000 front 2023-12-16T21:39:05.622528+0000 (oldest deadline 2023-12-16T21:39:30.322546+0000) 2023-12-16T21:47:03.316 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:47:03 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:47:03.010+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:05.622479+0000 front 2023-12-16T21:39:05.622580+0000 (oldest deadline 2023-12-16T21:39:30.322546+0000) 2023-12-16T21:47:03.317 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:47:03 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:47:03.011+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:47:03.817 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:47:03 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:47:03.778+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:09.828911+0000 front 2023-12-16T21:39:09.828958+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:47:03.817 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:47:03 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:47:03.778+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:09.828868+0000 front 2023-12-16T21:39:09.828878+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:47:03.817 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:47:03 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:47:03.778+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:09.828947+0000 front 2023-12-16T21:39:09.828993+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:47:03.817 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:47:03 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:47:03.779+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:47:04.316 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:47:04 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:47:04.005+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:09.306583+0000 front 2023-12-16T21:39:09.306633+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:47:04.317 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:47:04 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:47:04.005+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:09.306551+0000 front 2023-12-16T21:39:09.306652+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:47:04.317 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:47:04 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:47:04.005+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:09.306605+0000 front 2023-12-16T21:39:09.306613+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:47:04.317 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:47:04 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:47:04.007+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:47:04.317 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:47:03 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:47:03.995+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:05.622397+0000 front 2023-12-16T21:39:05.622556+0000 (oldest deadline 2023-12-16T21:39:30.322546+0000) 2023-12-16T21:47:04.317 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:47:03 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:47:03.995+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:05.622506+0000 front 2023-12-16T21:39:05.622528+0000 (oldest deadline 2023-12-16T21:39:30.322546+0000) 2023-12-16T21:47:04.317 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:47:03 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:47:03.995+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:05.622479+0000 front 2023-12-16T21:39:05.622580+0000 (oldest deadline 2023-12-16T21:39:30.322546+0000) 2023-12-16T21:47:04.317 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:47:03 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:47:03.996+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:47:05.044 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:47:04 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:47:04.730+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:09.828911+0000 front 2023-12-16T21:39:09.828958+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:47:05.044 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:47:04 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:47:04.730+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:09.828868+0000 front 2023-12-16T21:39:09.828878+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:47:05.044 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:47:04 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:47:04.730+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:09.828947+0000 front 2023-12-16T21:39:09.828993+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:47:05.044 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:47:04 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:47:04.731+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:47:05.045 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:47:04 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:47:04.958+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:05.622397+0000 front 2023-12-16T21:39:05.622556+0000 (oldest deadline 2023-12-16T21:39:30.322546+0000) 2023-12-16T21:47:05.045 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:47:04 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:47:04.958+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:05.622506+0000 front 2023-12-16T21:39:05.622528+0000 (oldest deadline 2023-12-16T21:39:30.322546+0000) 2023-12-16T21:47:05.045 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:47:04 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:47:04.958+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:05.622479+0000 front 2023-12-16T21:39:05.622580+0000 (oldest deadline 2023-12-16T21:39:30.322546+0000) 2023-12-16T21:47:05.045 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:47:04 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:47:04.959+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:47:05.316 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:47:05 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:47:05.041+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:09.306583+0000 front 2023-12-16T21:39:09.306633+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:47:05.316 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:47:05 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:47:05.041+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:09.306551+0000 front 2023-12-16T21:39:09.306652+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:47:05.317 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:47:05 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:47:05.041+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:09.306605+0000 front 2023-12-16T21:39:09.306613+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:47:05.317 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:47:05 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:47:05.042+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:47:05.969 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:47:05 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:47:05.685+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:09.828911+0000 front 2023-12-16T21:39:09.828958+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:47:05.970 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:47:05 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:47:05.685+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:09.828868+0000 front 2023-12-16T21:39:09.828878+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:47:05.970 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:47:05 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:47:05.685+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:09.828947+0000 front 2023-12-16T21:39:09.828993+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:47:05.970 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:47:05 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:47:05.686+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:47:06.316 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:47:06 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:47:06.049+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:09.306583+0000 front 2023-12-16T21:39:09.306633+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:47:06.317 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:47:06 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:47:06.049+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:09.306551+0000 front 2023-12-16T21:39:09.306652+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:47:06.317 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:47:06 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:47:06.049+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:09.306605+0000 front 2023-12-16T21:39:09.306613+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:47:06.317 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:47:06 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:47:06.049+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:47:06.317 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:47:05 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:47:05.966+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:05.622397+0000 front 2023-12-16T21:39:05.622556+0000 (oldest deadline 2023-12-16T21:39:30.322546+0000) 2023-12-16T21:47:06.317 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:47:05 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:47:05.966+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:05.622506+0000 front 2023-12-16T21:39:05.622528+0000 (oldest deadline 2023-12-16T21:39:30.322546+0000) 2023-12-16T21:47:06.317 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:47:05 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:47:05.966+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:05.622479+0000 front 2023-12-16T21:39:05.622580+0000 (oldest deadline 2023-12-16T21:39:30.322546+0000) 2023-12-16T21:47:06.317 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:47:05 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:47:05.967+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:47:06.686 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Dec 16 21:47:06 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-mon-smithi045[80759]: 2023-12-16T21:47:06.372+0000 7f45a1f8f700 -1 mon.smithi045@0(probing) e2 get_health_metrics reporting 4949 slow ops, oldest is monmgrreport(gid 14214, 0 checks, 1 progress events) 2023-12-16T21:47:07.004 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:47:06 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:47:06.683+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:09.828911+0000 front 2023-12-16T21:39:09.828958+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:47:07.005 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:47:06 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:47:06.683+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:09.828868+0000 front 2023-12-16T21:39:09.828878+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:47:07.005 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:47:06 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:47:06.683+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:09.828947+0000 front 2023-12-16T21:39:09.828993+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:47:07.005 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:47:06 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:47:06.684+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:47:07.317 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:47:07 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:47:07.002+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:05.622397+0000 front 2023-12-16T21:39:05.622556+0000 (oldest deadline 2023-12-16T21:39:30.322546+0000) 2023-12-16T21:47:07.317 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:47:07 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:47:07.002+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:05.622506+0000 front 2023-12-16T21:39:05.622528+0000 (oldest deadline 2023-12-16T21:39:30.322546+0000) 2023-12-16T21:47:07.317 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:47:07 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:47:07.002+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:05.622479+0000 front 2023-12-16T21:39:05.622580+0000 (oldest deadline 2023-12-16T21:39:30.322546+0000) 2023-12-16T21:47:07.317 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:47:07 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:47:07.003+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:47:07.317 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:47:07 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:47:07.000+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:09.306583+0000 front 2023-12-16T21:39:09.306633+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:47:07.317 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:47:07 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:47:07.000+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:09.306551+0000 front 2023-12-16T21:39:09.306652+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:47:07.317 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:47:07 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:47:07.000+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:09.306605+0000 front 2023-12-16T21:39:09.306613+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:47:07.317 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:47:07 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:47:07.001+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:47:07.956 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:47:07 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:47:07.658+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:09.828911+0000 front 2023-12-16T21:39:09.828958+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:47:07.956 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:47:07 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:47:07.658+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:09.828868+0000 front 2023-12-16T21:39:09.828878+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:47:07.956 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:47:07 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:47:07.658+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:09.828947+0000 front 2023-12-16T21:39:09.828993+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:47:07.956 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:47:07 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:47:07.659+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:47:08.316 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:47:07 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:47:07.953+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:09.306583+0000 front 2023-12-16T21:39:09.306633+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:47:08.317 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:47:07 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:47:07.953+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:09.306551+0000 front 2023-12-16T21:39:09.306652+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:47:08.317 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:47:07 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:47:07.953+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:09.306605+0000 front 2023-12-16T21:39:09.306613+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:47:08.317 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:47:07 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:47:07.954+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:47:08.318 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:47:07 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:47:07.999+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:05.622397+0000 front 2023-12-16T21:39:05.622556+0000 (oldest deadline 2023-12-16T21:39:30.322546+0000) 2023-12-16T21:47:08.318 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:47:07 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:47:07.999+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:05.622506+0000 front 2023-12-16T21:39:05.622528+0000 (oldest deadline 2023-12-16T21:39:30.322546+0000) 2023-12-16T21:47:08.318 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:47:07 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:47:07.999+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:05.622479+0000 front 2023-12-16T21:39:05.622580+0000 (oldest deadline 2023-12-16T21:39:30.322546+0000) 2023-12-16T21:47:08.318 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:47:08 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:47:07.999+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:47:08.946 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:47:08 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:47:08.676+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:09.828911+0000 front 2023-12-16T21:39:09.828958+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:47:08.946 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:47:08 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:47:08.676+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:09.828868+0000 front 2023-12-16T21:39:09.828878+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:47:08.946 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:47:08 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:47:08.676+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:09.828947+0000 front 2023-12-16T21:39:09.828993+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:47:08.946 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:47:08 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:47:08.677+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:47:09.316 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:47:08 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:47:08.966+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:05.622397+0000 front 2023-12-16T21:39:05.622556+0000 (oldest deadline 2023-12-16T21:39:30.322546+0000) 2023-12-16T21:47:09.317 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:47:08 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:47:08.966+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:05.622506+0000 front 2023-12-16T21:39:05.622528+0000 (oldest deadline 2023-12-16T21:39:30.322546+0000) 2023-12-16T21:47:09.317 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:47:08 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:47:08.966+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:05.622479+0000 front 2023-12-16T21:39:05.622580+0000 (oldest deadline 2023-12-16T21:39:30.322546+0000) 2023-12-16T21:47:09.317 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:47:08 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:47:08.967+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:47:09.317 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:47:08 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:47:08.942+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:09.306583+0000 front 2023-12-16T21:39:09.306633+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:47:09.317 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:47:08 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:47:08.943+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:09.306551+0000 front 2023-12-16T21:39:09.306652+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:47:09.317 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:47:08 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:47:08.943+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:09.306605+0000 front 2023-12-16T21:39:09.306613+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:47:09.317 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:47:08 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:47:08.943+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:47:09.972 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:47:09 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:47:09.657+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:09.828911+0000 front 2023-12-16T21:39:09.828958+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:47:09.972 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:47:09 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:47:09.657+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:09.828868+0000 front 2023-12-16T21:39:09.828878+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:47:09.972 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:47:09 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:47:09.657+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:09.828947+0000 front 2023-12-16T21:39:09.828993+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:47:09.972 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:47:09 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:47:09.658+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:47:10.317 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:47:09 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:47:09.969+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:09.306583+0000 front 2023-12-16T21:39:09.306633+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:47:10.317 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:47:09 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:47:09.969+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:09.306551+0000 front 2023-12-16T21:39:09.306652+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:47:10.317 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:47:09 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:47:09.969+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:09.306605+0000 front 2023-12-16T21:39:09.306613+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:47:10.317 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:47:09 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:47:09.969+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:47:10.317 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:47:10 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:47:10.004+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:05.622397+0000 front 2023-12-16T21:39:05.622556+0000 (oldest deadline 2023-12-16T21:39:30.322546+0000) 2023-12-16T21:47:10.317 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:47:10 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:47:10.004+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:05.622506+0000 front 2023-12-16T21:39:05.622528+0000 (oldest deadline 2023-12-16T21:39:30.322546+0000) 2023-12-16T21:47:10.317 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:47:10 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:47:10.005+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:05.622479+0000 front 2023-12-16T21:39:05.622580+0000 (oldest deadline 2023-12-16T21:39:30.322546+0000) 2023-12-16T21:47:10.317 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:47:10 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:47:10.005+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:47:10.967 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:47:10 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:47:10.694+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:09.828911+0000 front 2023-12-16T21:39:09.828958+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:47:10.967 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:47:10 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:47:10.694+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:09.828868+0000 front 2023-12-16T21:39:09.828878+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:47:10.967 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:47:10 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:47:10.694+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:09.828947+0000 front 2023-12-16T21:39:09.828993+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:47:10.967 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:47:10 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:47:10.695+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:47:11.317 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:47:10 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:47:10.991+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:09.306583+0000 front 2023-12-16T21:39:09.306633+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:47:11.317 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:47:10 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:47:10.991+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:09.306551+0000 front 2023-12-16T21:39:09.306652+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:47:11.317 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:47:10 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:47:10.991+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:09.306605+0000 front 2023-12-16T21:39:09.306613+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:47:11.317 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:47:10 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:47:10.993+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:47:11.317 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:47:10 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:47:10.964+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:05.622397+0000 front 2023-12-16T21:39:05.622556+0000 (oldest deadline 2023-12-16T21:39:30.322546+0000) 2023-12-16T21:47:11.317 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:47:10 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:47:10.964+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:05.622506+0000 front 2023-12-16T21:39:05.622528+0000 (oldest deadline 2023-12-16T21:39:30.322546+0000) 2023-12-16T21:47:11.317 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:47:10 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:47:10.964+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:05.622479+0000 front 2023-12-16T21:39:05.622580+0000 (oldest deadline 2023-12-16T21:39:30.322546+0000) 2023-12-16T21:47:11.318 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:47:10 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:47:10.965+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:47:11.652 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Dec 16 21:47:11 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-mon-smithi045[80759]: 2023-12-16T21:47:11.373+0000 7f45a1f8f700 -1 mon.smithi045@0(probing) e2 get_health_metrics reporting 5046 slow ops, oldest is monmgrreport(gid 14214, 0 checks, 1 progress events) 2023-12-16T21:47:11.938 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:47:11 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:47:11.730+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:09.828911+0000 front 2023-12-16T21:39:09.828958+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:47:11.938 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:47:11 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:47:11.730+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:09.828868+0000 front 2023-12-16T21:39:09.828878+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:47:11.938 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:47:11 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:47:11.730+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:09.828947+0000 front 2023-12-16T21:39:09.828993+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:47:11.938 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:47:11 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:47:11.731+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:47:12.316 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:47:11 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:47:11.935+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:05.622397+0000 front 2023-12-16T21:39:05.622556+0000 (oldest deadline 2023-12-16T21:39:30.322546+0000) 2023-12-16T21:47:12.317 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:47:11 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:47:11.935+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:05.622506+0000 front 2023-12-16T21:39:05.622528+0000 (oldest deadline 2023-12-16T21:39:30.322546+0000) 2023-12-16T21:47:12.317 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:47:11 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:47:11.935+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:05.622479+0000 front 2023-12-16T21:39:05.622580+0000 (oldest deadline 2023-12-16T21:39:30.322546+0000) 2023-12-16T21:47:12.317 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:47:11 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:47:11.936+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:47:12.317 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:47:12 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:47:12.016+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:09.306583+0000 front 2023-12-16T21:39:09.306633+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:47:12.317 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:47:12 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:47:12.016+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:09.306551+0000 front 2023-12-16T21:39:09.306652+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:47:12.317 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:47:12 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:47:12.016+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:09.306605+0000 front 2023-12-16T21:39:09.306613+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:47:12.317 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:47:12 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:47:12.017+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:47:13.062 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:47:12 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:47:12.962+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:05.622397+0000 front 2023-12-16T21:39:05.622556+0000 (oldest deadline 2023-12-16T21:39:30.322546+0000) 2023-12-16T21:47:13.062 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:47:12 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:47:12.962+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:05.622506+0000 front 2023-12-16T21:39:05.622528+0000 (oldest deadline 2023-12-16T21:39:30.322546+0000) 2023-12-16T21:47:13.062 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:47:12 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:47:12.962+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:05.622479+0000 front 2023-12-16T21:39:05.622580+0000 (oldest deadline 2023-12-16T21:39:30.322546+0000) 2023-12-16T21:47:13.062 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:47:12 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:47:12.963+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:47:13.062 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:47:12 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:47:12.764+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:09.828911+0000 front 2023-12-16T21:39:09.828958+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:47:13.062 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:47:12 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:47:12.764+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:09.828868+0000 front 2023-12-16T21:39:09.828878+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:47:13.062 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:47:12 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:47:12.764+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:09.828947+0000 front 2023-12-16T21:39:09.828993+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:47:13.062 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:47:12 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:47:12.765+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:47:13.316 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:47:13 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:47:13.058+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:09.306583+0000 front 2023-12-16T21:39:09.306633+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:47:13.316 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:47:13 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:47:13.058+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:09.306551+0000 front 2023-12-16T21:39:09.306652+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:47:13.317 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:47:13 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:47:13.058+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:09.306605+0000 front 2023-12-16T21:39:09.306613+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:47:13.317 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:47:13 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:47:13.060+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:47:13.816 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:47:13 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:47:13.781+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:09.828911+0000 front 2023-12-16T21:39:09.828958+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:47:13.817 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:47:13 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:47:13.781+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:09.828868+0000 front 2023-12-16T21:39:09.828878+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:47:13.817 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:47:13 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:47:13.781+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:09.828947+0000 front 2023-12-16T21:39:09.828993+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:47:13.817 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:47:13 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:47:13.781+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:47:14.317 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:47:14 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:47:14.095+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:09.306583+0000 front 2023-12-16T21:39:09.306633+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:47:14.317 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:47:14 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:47:14.095+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:09.306551+0000 front 2023-12-16T21:39:09.306652+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:47:14.318 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:47:14 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:47:14.095+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:09.306605+0000 front 2023-12-16T21:39:09.306613+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:47:14.318 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:47:14 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:47:14.096+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:47:14.318 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:47:13 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:47:13.952+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:05.622397+0000 front 2023-12-16T21:39:05.622556+0000 (oldest deadline 2023-12-16T21:39:30.322546+0000) 2023-12-16T21:47:14.318 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:47:13 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:47:13.952+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:05.622506+0000 front 2023-12-16T21:39:05.622528+0000 (oldest deadline 2023-12-16T21:39:30.322546+0000) 2023-12-16T21:47:14.318 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:47:13 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:47:13.952+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:05.622479+0000 front 2023-12-16T21:39:05.622580+0000 (oldest deadline 2023-12-16T21:39:30.322546+0000) 2023-12-16T21:47:14.318 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:47:13 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:47:13.953+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:47:14.965 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:47:14 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:47:14.739+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:09.828911+0000 front 2023-12-16T21:39:09.828958+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:47:14.965 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:47:14 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:47:14.739+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:09.828868+0000 front 2023-12-16T21:39:09.828878+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:47:14.965 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:47:14 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:47:14.739+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:09.828947+0000 front 2023-12-16T21:39:09.828993+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:47:14.965 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:47:14 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:47:14.740+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:47:15.317 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:47:15 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:47:15.122+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:09.306583+0000 front 2023-12-16T21:39:09.306633+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:47:15.317 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:47:15 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:47:15.122+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:09.306551+0000 front 2023-12-16T21:39:09.306652+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:47:15.317 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:47:15 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:47:15.122+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:09.306605+0000 front 2023-12-16T21:39:09.306613+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:47:15.317 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:47:15 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:47:15.122+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:47:15.317 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:47:14 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:47:14.961+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:05.622397+0000 front 2023-12-16T21:39:05.622556+0000 (oldest deadline 2023-12-16T21:39:30.322546+0000) 2023-12-16T21:47:15.317 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:47:14 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:47:14.961+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:05.622506+0000 front 2023-12-16T21:39:05.622528+0000 (oldest deadline 2023-12-16T21:39:30.322546+0000) 2023-12-16T21:47:15.317 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:47:14 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:47:14.961+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:05.622479+0000 front 2023-12-16T21:39:05.622580+0000 (oldest deadline 2023-12-16T21:39:30.322546+0000) 2023-12-16T21:47:15.317 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:47:14 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:47:14.962+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:47:16.066 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:47:15 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:47:15.773+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:09.828911+0000 front 2023-12-16T21:39:09.828958+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:47:16.067 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:47:15 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:47:15.773+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:09.828868+0000 front 2023-12-16T21:39:09.828878+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:47:16.067 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:47:15 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:47:15.773+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:09.828947+0000 front 2023-12-16T21:39:09.828993+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:47:16.067 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:47:15 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:47:15.774+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:47:16.067 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:47:15 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:47:15.940+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:05.622397+0000 front 2023-12-16T21:39:05.622556+0000 (oldest deadline 2023-12-16T21:39:30.322546+0000) 2023-12-16T21:47:16.067 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:47:15 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:47:15.940+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:05.622506+0000 front 2023-12-16T21:39:05.622528+0000 (oldest deadline 2023-12-16T21:39:30.322546+0000) 2023-12-16T21:47:16.067 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:47:15 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:47:15.940+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:05.622479+0000 front 2023-12-16T21:39:05.622580+0000 (oldest deadline 2023-12-16T21:39:30.322546+0000) 2023-12-16T21:47:16.067 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:47:15 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:47:15.941+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:47:16.377 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:47:16 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:47:16.080+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:09.306583+0000 front 2023-12-16T21:39:09.306633+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:47:16.377 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:47:16 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:47:16.080+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:09.306551+0000 front 2023-12-16T21:39:09.306652+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:47:16.378 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:47:16 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:47:16.080+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:09.306605+0000 front 2023-12-16T21:39:09.306613+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:47:16.378 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:47:16 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:47:16.081+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:47:16.767 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Dec 16 21:47:16 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-mon-smithi045[80759]: 2023-12-16T21:47:16.374+0000 7f45a1f8f700 -1 mon.smithi045@0(probing) e2 get_health_metrics reporting 5141 slow ops, oldest is monmgrreport(gid 14214, 0 checks, 1 progress events) 2023-12-16T21:47:17.064 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:47:16 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:47:16.929+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:05.622397+0000 front 2023-12-16T21:39:05.622556+0000 (oldest deadline 2023-12-16T21:39:30.322546+0000) 2023-12-16T21:47:17.064 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:47:16 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:47:16.929+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:05.622506+0000 front 2023-12-16T21:39:05.622528+0000 (oldest deadline 2023-12-16T21:39:30.322546+0000) 2023-12-16T21:47:17.065 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:47:16 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:47:16.929+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:05.622479+0000 front 2023-12-16T21:39:05.622580+0000 (oldest deadline 2023-12-16T21:39:30.322546+0000) 2023-12-16T21:47:17.065 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:47:16 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:47:16.930+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:47:17.065 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:47:16 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:47:16.764+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:09.828911+0000 front 2023-12-16T21:39:09.828958+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:47:17.065 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:47:16 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:47:16.764+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:09.828868+0000 front 2023-12-16T21:39:09.828878+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:47:17.065 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:47:16 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:47:16.764+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:09.828947+0000 front 2023-12-16T21:39:09.828993+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:47:17.066 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:47:16 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:47:16.765+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:47:17.316 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:47:17 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:47:17.061+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:09.306583+0000 front 2023-12-16T21:39:09.306633+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:47:17.317 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:47:17 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:47:17.061+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:09.306551+0000 front 2023-12-16T21:39:09.306652+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:47:17.317 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:47:17 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:47:17.061+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:09.306605+0000 front 2023-12-16T21:39:09.306613+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:47:17.317 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:47:17 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:47:17.062+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:47:18.038 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:47:17 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:47:17.940+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:05.622397+0000 front 2023-12-16T21:39:05.622556+0000 (oldest deadline 2023-12-16T21:39:30.322546+0000) 2023-12-16T21:47:18.038 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:47:17 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:47:17.940+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:05.622506+0000 front 2023-12-16T21:39:05.622528+0000 (oldest deadline 2023-12-16T21:39:30.322546+0000) 2023-12-16T21:47:18.039 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:47:17 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:47:17.940+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:05.622479+0000 front 2023-12-16T21:39:05.622580+0000 (oldest deadline 2023-12-16T21:39:30.322546+0000) 2023-12-16T21:47:18.039 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:47:17 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:47:17.940+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:47:18.039 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:47:17 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:47:17.763+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:09.828911+0000 front 2023-12-16T21:39:09.828958+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:47:18.039 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:47:17 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:47:17.763+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:09.828868+0000 front 2023-12-16T21:39:09.828878+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:47:18.039 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:47:17 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:47:17.763+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:09.828947+0000 front 2023-12-16T21:39:09.828993+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:47:18.039 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:47:17 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:47:17.764+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:47:18.316 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:47:18 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:47:18.035+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:09.306583+0000 front 2023-12-16T21:39:09.306633+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:47:18.317 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:47:18 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:47:18.035+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:09.306551+0000 front 2023-12-16T21:39:09.306652+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:47:18.317 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:47:18 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:47:18.035+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:09.306605+0000 front 2023-12-16T21:39:09.306613+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:47:18.317 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:47:18 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:47:18.036+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:47:18.973 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:47:18 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:47:18.969+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:05.622397+0000 front 2023-12-16T21:39:05.622556+0000 (oldest deadline 2023-12-16T21:39:30.322546+0000) 2023-12-16T21:47:18.973 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:47:18 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:47:18.969+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:05.622506+0000 front 2023-12-16T21:39:05.622528+0000 (oldest deadline 2023-12-16T21:39:30.322546+0000) 2023-12-16T21:47:18.973 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:47:18 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:47:18.969+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:05.622479+0000 front 2023-12-16T21:39:05.622580+0000 (oldest deadline 2023-12-16T21:39:30.322546+0000) 2023-12-16T21:47:18.973 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:47:18 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:47:18.740+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:09.828911+0000 front 2023-12-16T21:39:09.828958+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:47:18.973 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:47:18 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:47:18.740+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:09.828868+0000 front 2023-12-16T21:39:09.828878+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:47:18.973 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:47:18 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:47:18.740+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:09.828947+0000 front 2023-12-16T21:39:09.828993+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:47:18.973 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:47:18 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:47:18.741+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:47:19.316 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:47:18 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:47:18.970+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:47:19.317 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:47:19 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:47:19.054+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:09.306583+0000 front 2023-12-16T21:39:09.306633+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:47:19.317 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:47:19 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:47:19.054+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:09.306551+0000 front 2023-12-16T21:39:09.306652+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:47:19.317 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:47:19 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:47:19.054+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:09.306605+0000 front 2023-12-16T21:39:09.306613+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:47:19.317 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:47:19 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:47:19.055+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:47:20.066 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:47:19 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:47:19.944+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:05.622397+0000 front 2023-12-16T21:39:05.622556+0000 (oldest deadline 2023-12-16T21:39:30.322546+0000) 2023-12-16T21:47:20.067 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:47:19 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:47:19.944+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:05.622506+0000 front 2023-12-16T21:39:05.622528+0000 (oldest deadline 2023-12-16T21:39:30.322546+0000) 2023-12-16T21:47:20.067 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:47:19 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:47:19.944+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:05.622479+0000 front 2023-12-16T21:39:05.622580+0000 (oldest deadline 2023-12-16T21:39:30.322546+0000) 2023-12-16T21:47:20.067 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:47:19 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:47:19.945+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:47:20.067 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:47:19 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:47:19.775+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:09.828911+0000 front 2023-12-16T21:39:09.828958+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:47:20.067 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:47:19 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:47:19.775+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:09.828868+0000 front 2023-12-16T21:39:09.828878+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:47:20.067 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:47:19 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:47:19.775+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:09.828947+0000 front 2023-12-16T21:39:09.828993+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:47:20.067 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:47:19 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:47:19.776+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:47:20.566 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:47:20 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:47:20.078+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:09.306583+0000 front 2023-12-16T21:39:09.306633+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:47:20.567 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:47:20 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:47:20.078+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:09.306551+0000 front 2023-12-16T21:39:09.306652+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:47:20.567 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:47:20 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:47:20.078+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:09.306605+0000 front 2023-12-16T21:39:09.306613+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:47:20.567 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:47:20 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:47:20.079+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:47:20.978 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:47:20 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:47:20.767+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:09.828911+0000 front 2023-12-16T21:39:09.828958+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:47:20.978 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:47:20 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:47:20.767+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:09.828868+0000 front 2023-12-16T21:39:09.828878+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:47:20.978 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:47:20 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:47:20.767+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:09.828947+0000 front 2023-12-16T21:39:09.828993+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:47:20.978 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:47:20 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:47:20.768+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:47:21.316 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:47:21 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:47:21.042+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:09.306583+0000 front 2023-12-16T21:39:09.306633+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:47:21.317 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:47:21 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:47:21.042+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:09.306551+0000 front 2023-12-16T21:39:09.306652+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:47:21.317 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:47:21 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:47:21.042+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:09.306605+0000 front 2023-12-16T21:39:09.306613+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:47:21.317 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:47:21 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:47:21.043+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:47:21.317 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:47:20 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:47:20.975+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:05.622397+0000 front 2023-12-16T21:39:05.622556+0000 (oldest deadline 2023-12-16T21:39:30.322546+0000) 2023-12-16T21:47:21.317 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:47:20 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:47:20.975+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:05.622506+0000 front 2023-12-16T21:39:05.622528+0000 (oldest deadline 2023-12-16T21:39:30.322546+0000) 2023-12-16T21:47:21.317 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:47:20 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:47:20.975+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:05.622479+0000 front 2023-12-16T21:39:05.622580+0000 (oldest deadline 2023-12-16T21:39:30.322546+0000) 2023-12-16T21:47:21.317 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:47:20 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:47:20.976+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:47:21.653 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Dec 16 21:47:21 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-mon-smithi045[80759]: 2023-12-16T21:47:21.375+0000 7f45a1f8f700 -1 mon.smithi045@0(probing) e2 get_health_metrics reporting 5238 slow ops, oldest is monmgrreport(gid 14214, 0 checks, 1 progress events) 2023-12-16T21:47:22.006 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:47:21 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:47:21.750+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:09.828911+0000 front 2023-12-16T21:39:09.828958+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:47:22.007 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:47:21 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:47:21.750+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:09.828868+0000 front 2023-12-16T21:39:09.828878+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:47:22.007 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:47:21 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:47:21.750+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:09.828947+0000 front 2023-12-16T21:39:09.828993+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:47:22.007 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:47:21 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:47:21.751+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:47:22.316 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:47:22 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:47:22.003+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:05.622397+0000 front 2023-12-16T21:39:05.622556+0000 (oldest deadline 2023-12-16T21:39:30.322546+0000) 2023-12-16T21:47:22.316 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:47:22 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:47:22.003+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:05.622506+0000 front 2023-12-16T21:39:05.622528+0000 (oldest deadline 2023-12-16T21:39:30.322546+0000) 2023-12-16T21:47:22.317 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:47:22 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:47:22.003+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:05.622479+0000 front 2023-12-16T21:39:05.622580+0000 (oldest deadline 2023-12-16T21:39:30.322546+0000) 2023-12-16T21:47:22.317 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:47:22 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:47:22.004+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:47:22.317 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:47:22 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:47:22.016+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:09.306583+0000 front 2023-12-16T21:39:09.306633+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:47:22.317 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:47:22 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:47:22.016+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:09.306551+0000 front 2023-12-16T21:39:09.306652+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:47:22.317 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:47:22 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:47:22.016+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:09.306605+0000 front 2023-12-16T21:39:09.306613+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:47:22.317 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:47:22 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:47:22.017+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:47:23.066 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:47:22 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:47:22.993+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:09.306583+0000 front 2023-12-16T21:39:09.306633+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:47:23.067 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:47:22 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:47:22.993+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:09.306551+0000 front 2023-12-16T21:39:09.306652+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:47:23.067 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:47:22 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:47:22.993+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:09.306605+0000 front 2023-12-16T21:39:09.306613+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:47:23.067 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:47:22 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:47:22.993+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:47:23.067 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:47:22 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:47:22.972+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:05.622397+0000 front 2023-12-16T21:39:05.622556+0000 (oldest deadline 2023-12-16T21:39:30.322546+0000) 2023-12-16T21:47:23.067 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:47:22 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:47:22.972+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:05.622506+0000 front 2023-12-16T21:39:05.622528+0000 (oldest deadline 2023-12-16T21:39:30.322546+0000) 2023-12-16T21:47:23.067 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:47:22 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:47:22.972+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:05.622479+0000 front 2023-12-16T21:39:05.622580+0000 (oldest deadline 2023-12-16T21:39:30.322546+0000) 2023-12-16T21:47:23.067 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:47:22 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:47:22.972+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:47:23.068 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:47:22 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:47:22.769+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:09.828911+0000 front 2023-12-16T21:39:09.828958+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:47:23.068 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:47:22 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:47:22.769+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:09.828868+0000 front 2023-12-16T21:39:09.828878+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:47:23.068 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:47:22 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:47:22.769+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:09.828947+0000 front 2023-12-16T21:39:09.828993+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:47:23.068 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:47:22 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:47:22.770+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:47:24.066 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:47:23 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:47:23.994+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:05.622397+0000 front 2023-12-16T21:39:05.622556+0000 (oldest deadline 2023-12-16T21:39:30.322546+0000) 2023-12-16T21:47:24.067 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:47:23 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:47:23.994+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:05.622506+0000 front 2023-12-16T21:39:05.622528+0000 (oldest deadline 2023-12-16T21:39:30.322546+0000) 2023-12-16T21:47:24.067 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:47:23 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:47:23.994+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:05.622479+0000 front 2023-12-16T21:39:05.622580+0000 (oldest deadline 2023-12-16T21:39:30.322546+0000) 2023-12-16T21:47:24.067 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:47:23 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:47:23.994+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:47:24.067 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:47:24 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:47:24.028+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:09.306583+0000 front 2023-12-16T21:39:09.306633+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:47:24.067 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:47:24 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:47:24.028+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:09.306551+0000 front 2023-12-16T21:39:09.306652+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:47:24.067 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:47:24 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:47:24.028+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:09.306605+0000 front 2023-12-16T21:39:09.306613+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:47:24.067 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:47:24 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:47:24.030+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:47:24.067 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:47:23 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:47:23.817+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:09.828911+0000 front 2023-12-16T21:39:09.828958+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:47:24.068 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:47:23 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:47:23.817+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:09.828868+0000 front 2023-12-16T21:39:09.828878+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:47:24.068 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:47:23 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:47:23.817+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:09.828947+0000 front 2023-12-16T21:39:09.828993+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:47:24.068 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:47:23 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:47:23.818+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:47:25.069 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:47:25 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:47:25.046+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:09.306583+0000 front 2023-12-16T21:39:09.306633+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:47:25.069 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:47:25 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:47:25.046+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:09.306551+0000 front 2023-12-16T21:39:09.306652+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:47:25.069 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:47:25 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:47:25.046+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:09.306605+0000 front 2023-12-16T21:39:09.306613+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:47:25.069 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:47:25 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:47:25.047+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:47:25.070 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:47:25 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:47:25.019+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:05.622397+0000 front 2023-12-16T21:39:05.622556+0000 (oldest deadline 2023-12-16T21:39:30.322546+0000) 2023-12-16T21:47:25.070 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:47:25 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:47:25.019+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:05.622506+0000 front 2023-12-16T21:39:05.622528+0000 (oldest deadline 2023-12-16T21:39:30.322546+0000) 2023-12-16T21:47:25.070 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:47:25 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:47:25.019+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:05.622479+0000 front 2023-12-16T21:39:05.622580+0000 (oldest deadline 2023-12-16T21:39:30.322546+0000) 2023-12-16T21:47:25.070 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:47:25 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:47:25.020+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:47:25.070 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:47:24 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:47:24.802+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:09.828911+0000 front 2023-12-16T21:39:09.828958+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:47:25.070 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:47:24 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:47:24.802+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:09.828868+0000 front 2023-12-16T21:39:09.828878+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:47:25.070 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:47:24 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:47:24.802+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:09.828947+0000 front 2023-12-16T21:39:09.828993+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:47:25.070 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:47:24 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:47:24.804+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:47:26.060 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:47:25 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:47:25.769+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:09.828911+0000 front 2023-12-16T21:39:09.828958+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:47:26.060 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:47:25 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:47:25.769+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:09.828868+0000 front 2023-12-16T21:39:09.828878+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:47:26.060 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:47:25 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:47:25.770+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:09.828947+0000 front 2023-12-16T21:39:09.828993+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:47:26.060 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:47:25 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:47:25.770+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:47:26.318 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:47:26 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:47:26.056+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:05.622397+0000 front 2023-12-16T21:39:05.622556+0000 (oldest deadline 2023-12-16T21:39:30.322546+0000) 2023-12-16T21:47:26.318 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:47:26 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:47:26.056+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:05.622506+0000 front 2023-12-16T21:39:05.622528+0000 (oldest deadline 2023-12-16T21:39:30.322546+0000) 2023-12-16T21:47:26.318 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:47:26 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:47:26.056+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:05.622479+0000 front 2023-12-16T21:39:05.622580+0000 (oldest deadline 2023-12-16T21:39:30.322546+0000) 2023-12-16T21:47:26.318 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:47:26 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:47:26.057+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:47:26.318 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:47:26 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:47:26.087+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:09.306583+0000 front 2023-12-16T21:39:09.306633+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:47:26.319 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:47:26 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:47:26.087+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:09.306551+0000 front 2023-12-16T21:39:09.306652+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:47:26.319 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:47:26 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:47:26.087+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:09.306605+0000 front 2023-12-16T21:39:09.306613+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:47:26.319 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:47:26 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:47:26.088+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:47:26.730 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Dec 16 21:47:26 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-mon-smithi045[80759]: 2023-12-16T21:47:26.376+0000 7f45a1f8f700 -1 mon.smithi045@0(probing) e2 get_health_metrics reporting 5333 slow ops, oldest is monmgrreport(gid 14214, 0 checks, 1 progress events) 2023-12-16T21:47:27.051 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:47:26 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:47:26.726+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:09.828911+0000 front 2023-12-16T21:39:09.828958+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:47:27.051 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:47:26 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:47:26.726+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:09.828868+0000 front 2023-12-16T21:39:09.828878+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:47:27.051 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:47:26 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:47:26.727+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:09.828947+0000 front 2023-12-16T21:39:09.828993+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:47:27.052 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:47:26 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:47:26.728+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:47:27.316 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:47:27 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:47:27.048+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:09.306583+0000 front 2023-12-16T21:39:09.306633+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:47:27.317 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:47:27 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:47:27.048+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:09.306551+0000 front 2023-12-16T21:39:09.306652+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:47:27.317 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:47:27 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:47:27.048+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:09.306605+0000 front 2023-12-16T21:39:09.306613+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:47:27.317 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:47:27 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:47:27.049+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:47:27.317 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:47:27 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:47:27.076+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:05.622397+0000 front 2023-12-16T21:39:05.622556+0000 (oldest deadline 2023-12-16T21:39:30.322546+0000) 2023-12-16T21:47:27.317 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:47:27 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:47:27.076+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:05.622506+0000 front 2023-12-16T21:39:05.622528+0000 (oldest deadline 2023-12-16T21:39:30.322546+0000) 2023-12-16T21:47:27.317 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:47:27 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:47:27.076+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:05.622479+0000 front 2023-12-16T21:39:05.622580+0000 (oldest deadline 2023-12-16T21:39:30.322546+0000) 2023-12-16T21:47:27.317 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:47:27 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:47:27.078+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:47:28.052 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:47:27 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:47:27.770+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:09.828911+0000 front 2023-12-16T21:39:09.828958+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:47:28.052 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:47:27 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:47:27.770+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:09.828868+0000 front 2023-12-16T21:39:09.828878+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:47:28.052 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:47:27 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:47:27.770+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:09.828947+0000 front 2023-12-16T21:39:09.828993+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:47:28.053 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:47:27 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:47:27.771+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:47:28.316 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:47:28 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:47:28.069+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:09.306583+0000 front 2023-12-16T21:39:09.306633+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:47:28.316 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:47:28 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:47:28.069+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:09.306551+0000 front 2023-12-16T21:39:09.306652+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:47:28.317 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:47:28 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:47:28.069+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:09.306605+0000 front 2023-12-16T21:39:09.306613+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:47:28.317 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:47:28 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:47:28.070+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:47:28.317 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:47:28 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:47:28.049+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:05.622397+0000 front 2023-12-16T21:39:05.622556+0000 (oldest deadline 2023-12-16T21:39:30.322546+0000) 2023-12-16T21:47:28.317 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:47:28 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:47:28.049+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:05.622506+0000 front 2023-12-16T21:39:05.622528+0000 (oldest deadline 2023-12-16T21:39:30.322546+0000) 2023-12-16T21:47:28.317 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:47:28 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:47:28.049+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:05.622479+0000 front 2023-12-16T21:39:05.622580+0000 (oldest deadline 2023-12-16T21:39:30.322546+0000) 2023-12-16T21:47:28.317 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:47:28 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:47:28.050+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:47:29.066 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:47:28 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:47:28.792+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:09.828911+0000 front 2023-12-16T21:39:09.828958+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:47:29.067 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:47:28 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:47:28.792+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:09.828868+0000 front 2023-12-16T21:39:09.828878+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:47:29.067 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:47:28 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:47:28.792+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:09.828947+0000 front 2023-12-16T21:39:09.828993+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:47:29.067 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:47:28 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:47:28.792+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:47:29.567 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:47:29 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:47:29.103+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:09.306583+0000 front 2023-12-16T21:39:09.306633+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:47:29.567 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:47:29 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:47:29.103+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:09.306551+0000 front 2023-12-16T21:39:09.306652+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:47:29.567 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:47:29 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:47:29.103+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:09.306605+0000 front 2023-12-16T21:39:09.306613+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:47:29.567 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:47:29 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:47:29.104+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:47:29.567 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:47:29 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:47:29.071+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:05.622397+0000 front 2023-12-16T21:39:05.622556+0000 (oldest deadline 2023-12-16T21:39:30.322546+0000) 2023-12-16T21:47:29.568 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:47:29 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:47:29.071+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:05.622506+0000 front 2023-12-16T21:39:05.622528+0000 (oldest deadline 2023-12-16T21:39:30.322546+0000) 2023-12-16T21:47:29.568 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:47:29 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:47:29.071+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:05.622479+0000 front 2023-12-16T21:39:05.622580+0000 (oldest deadline 2023-12-16T21:39:30.322546+0000) 2023-12-16T21:47:29.568 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:47:29 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:47:29.072+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:47:30.048 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:47:29 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:47:29.764+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:09.828911+0000 front 2023-12-16T21:39:09.828958+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:47:30.049 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:47:29 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:47:29.764+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:09.828868+0000 front 2023-12-16T21:39:09.828878+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:47:30.049 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:47:29 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:47:29.764+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:09.828947+0000 front 2023-12-16T21:39:09.828993+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:47:30.049 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:47:29 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:47:29.765+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:47:30.316 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:47:30 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:47:30.092+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:09.306583+0000 front 2023-12-16T21:39:09.306633+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:47:30.317 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:47:30 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:47:30.092+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:09.306551+0000 front 2023-12-16T21:39:09.306652+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:47:30.317 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:47:30 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:47:30.092+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:09.306605+0000 front 2023-12-16T21:39:09.306613+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:47:30.317 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:47:30 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:47:30.093+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:47:30.317 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:47:30 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:47:30.045+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:05.622397+0000 front 2023-12-16T21:39:05.622556+0000 (oldest deadline 2023-12-16T21:39:30.322546+0000) 2023-12-16T21:47:30.317 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:47:30 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:47:30.045+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:05.622506+0000 front 2023-12-16T21:39:05.622528+0000 (oldest deadline 2023-12-16T21:39:30.322546+0000) 2023-12-16T21:47:30.317 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:47:30 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:47:30.045+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:05.622479+0000 front 2023-12-16T21:39:05.622580+0000 (oldest deadline 2023-12-16T21:39:30.322546+0000) 2023-12-16T21:47:30.317 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:47:30 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:47:30.046+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:47:31.066 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:47:30 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:47:30.812+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:09.828911+0000 front 2023-12-16T21:39:09.828958+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:47:31.067 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:47:30 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:47:30.812+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:09.828868+0000 front 2023-12-16T21:39:09.828878+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:47:31.067 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:47:30 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:47:30.812+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:09.828947+0000 front 2023-12-16T21:39:09.828993+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:47:31.067 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:47:30 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:47:30.812+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:47:31.380 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:47:31 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:47:31.096+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:05.622397+0000 front 2023-12-16T21:39:05.622556+0000 (oldest deadline 2023-12-16T21:39:30.322546+0000) 2023-12-16T21:47:31.381 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:47:31 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:47:31.096+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:05.622506+0000 front 2023-12-16T21:39:05.622528+0000 (oldest deadline 2023-12-16T21:39:30.322546+0000) 2023-12-16T21:47:31.381 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:47:31 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:47:31.096+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:05.622479+0000 front 2023-12-16T21:39:05.622580+0000 (oldest deadline 2023-12-16T21:39:30.322546+0000) 2023-12-16T21:47:31.381 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:47:31 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:47:31.096+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:47:31.381 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:47:31 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:47:31.092+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:09.306583+0000 front 2023-12-16T21:39:09.306633+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:47:31.381 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:47:31 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:47:31.092+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:09.306551+0000 front 2023-12-16T21:39:09.306652+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:47:31.381 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:47:31 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:47:31.092+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:09.306605+0000 front 2023-12-16T21:39:09.306613+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:47:31.381 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:47:31 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:47:31.092+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:47:31.654 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Dec 16 21:47:31 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-mon-smithi045[80759]: 2023-12-16T21:47:31.377+0000 7f45a1f8f700 -1 mon.smithi045@0(probing) e2 get_health_metrics reporting 5430 slow ops, oldest is monmgrreport(gid 14214, 0 checks, 1 progress events) 2023-12-16T21:47:32.064 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:47:31 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:47:31.846+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:09.828911+0000 front 2023-12-16T21:39:09.828958+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:47:32.064 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:47:31 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:47:31.846+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:09.828868+0000 front 2023-12-16T21:39:09.828878+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:47:32.064 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:47:31 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:47:31.846+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:09.828947+0000 front 2023-12-16T21:39:09.828993+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:47:32.064 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:47:31 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:47:31.847+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:47:32.316 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:47:32 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:47:32.080+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:09.306583+0000 front 2023-12-16T21:39:09.306633+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:47:32.317 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:47:32 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:47:32.080+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:09.306551+0000 front 2023-12-16T21:39:09.306652+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:47:32.317 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:47:32 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:47:32.080+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:09.306605+0000 front 2023-12-16T21:39:09.306613+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:47:32.317 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:47:32 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:47:32.081+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:47:32.317 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:47:32 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:47:32.061+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:05.622397+0000 front 2023-12-16T21:39:05.622556+0000 (oldest deadline 2023-12-16T21:39:30.322546+0000) 2023-12-16T21:47:32.317 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:47:32 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:47:32.061+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:05.622506+0000 front 2023-12-16T21:39:05.622528+0000 (oldest deadline 2023-12-16T21:39:30.322546+0000) 2023-12-16T21:47:32.317 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:47:32 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:47:32.061+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:05.622479+0000 front 2023-12-16T21:39:05.622580+0000 (oldest deadline 2023-12-16T21:39:30.322546+0000) 2023-12-16T21:47:32.317 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:47:32 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:47:32.062+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:47:33.060 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:47:32 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:47:32.803+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:09.828911+0000 front 2023-12-16T21:39:09.828958+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:47:33.060 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:47:32 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:47:32.803+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:09.828868+0000 front 2023-12-16T21:39:09.828878+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:47:33.060 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:47:32 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:47:32.803+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:09.828947+0000 front 2023-12-16T21:39:09.828993+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:47:33.060 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:47:32 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:47:32.804+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:47:33.316 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:47:33 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:47:33.056+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:05.622397+0000 front 2023-12-16T21:39:05.622556+0000 (oldest deadline 2023-12-16T21:39:30.322546+0000) 2023-12-16T21:47:33.316 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:47:33 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:47:33.056+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:05.622506+0000 front 2023-12-16T21:39:05.622528+0000 (oldest deadline 2023-12-16T21:39:30.322546+0000) 2023-12-16T21:47:33.317 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:47:33 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:47:33.056+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:05.622479+0000 front 2023-12-16T21:39:05.622580+0000 (oldest deadline 2023-12-16T21:39:30.322546+0000) 2023-12-16T21:47:33.317 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:47:33 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:47:33.058+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:47:33.317 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:47:33 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:47:33.129+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:09.306583+0000 front 2023-12-16T21:39:09.306633+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:47:33.317 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:47:33 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:47:33.129+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:09.306551+0000 front 2023-12-16T21:39:09.306652+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:47:33.317 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:47:33 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:47:33.129+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:09.306605+0000 front 2023-12-16T21:39:09.306613+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:47:33.317 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:47:33 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:47:33.130+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:47:34.101 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:47:33 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:47:33.824+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:09.828911+0000 front 2023-12-16T21:39:09.828958+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:47:34.101 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:47:33 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:47:33.824+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:09.828868+0000 front 2023-12-16T21:39:09.828878+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:47:34.101 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:47:33 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:47:33.824+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:09.828947+0000 front 2023-12-16T21:39:09.828993+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:47:34.101 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:47:33 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:47:33.825+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:47:34.566 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:47:34 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:47:34.154+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:09.306583+0000 front 2023-12-16T21:39:09.306633+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:47:34.566 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:47:34 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:47:34.154+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:09.306551+0000 front 2023-12-16T21:39:09.306652+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:47:34.567 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:47:34 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:47:34.154+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:09.306605+0000 front 2023-12-16T21:39:09.306613+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:47:34.567 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:47:34 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:47:34.155+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:47:34.567 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:47:34 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:47:34.095+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:05.622397+0000 front 2023-12-16T21:39:05.622556+0000 (oldest deadline 2023-12-16T21:39:30.322546+0000) 2023-12-16T21:47:34.567 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:47:34 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:47:34.095+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:05.622506+0000 front 2023-12-16T21:39:05.622528+0000 (oldest deadline 2023-12-16T21:39:30.322546+0000) 2023-12-16T21:47:34.567 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:47:34 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:47:34.095+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:05.622479+0000 front 2023-12-16T21:39:05.622580+0000 (oldest deadline 2023-12-16T21:39:30.322546+0000) 2023-12-16T21:47:34.567 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:47:34 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:47:34.096+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:47:35.146 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:47:34 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:47:34.871+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:09.828911+0000 front 2023-12-16T21:39:09.828958+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:47:35.146 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:47:34 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:47:34.871+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:09.828868+0000 front 2023-12-16T21:39:09.828878+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:47:35.146 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:47:34 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:47:34.871+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:09.828947+0000 front 2023-12-16T21:39:09.828993+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:47:35.146 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:47:34 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:47:34.872+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:47:35.146 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:47:35 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:47:35.051+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:05.622397+0000 front 2023-12-16T21:39:05.622556+0000 (oldest deadline 2023-12-16T21:39:30.322546+0000) 2023-12-16T21:47:35.147 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:47:35 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:47:35.051+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:05.622506+0000 front 2023-12-16T21:39:05.622528+0000 (oldest deadline 2023-12-16T21:39:30.322546+0000) 2023-12-16T21:47:35.147 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:47:35 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:47:35.051+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:05.622479+0000 front 2023-12-16T21:39:05.622580+0000 (oldest deadline 2023-12-16T21:39:30.322546+0000) 2023-12-16T21:47:35.147 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:47:35 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:47:35.052+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:47:35.566 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:47:35 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:47:35.143+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:09.306583+0000 front 2023-12-16T21:39:09.306633+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:47:35.566 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:47:35 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:47:35.143+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:09.306551+0000 front 2023-12-16T21:39:09.306652+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:47:35.567 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:47:35 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:47:35.143+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:09.306605+0000 front 2023-12-16T21:39:09.306613+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:47:35.567 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:47:35 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:47:35.144+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:47:36.163 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:47:36 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:47:36.078+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:05.622397+0000 front 2023-12-16T21:39:05.622556+0000 (oldest deadline 2023-12-16T21:39:30.322546+0000) 2023-12-16T21:47:36.163 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:47:36 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:47:36.078+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:05.622506+0000 front 2023-12-16T21:39:05.622528+0000 (oldest deadline 2023-12-16T21:39:30.322546+0000) 2023-12-16T21:47:36.163 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:47:36 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:47:36.078+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:05.622479+0000 front 2023-12-16T21:39:05.622580+0000 (oldest deadline 2023-12-16T21:39:30.322546+0000) 2023-12-16T21:47:36.163 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:47:36 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:47:36.078+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:47:36.164 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:47:35 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:47:35.845+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:09.828911+0000 front 2023-12-16T21:39:09.828958+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:47:36.164 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:47:35 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:47:35.845+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:09.828868+0000 front 2023-12-16T21:39:09.828878+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:47:36.164 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:47:35 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:47:35.845+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:09.828947+0000 front 2023-12-16T21:39:09.828993+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:47:36.164 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:47:35 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:47:35.846+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:47:36.504 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:47:36 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:47:36.160+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:09.306583+0000 front 2023-12-16T21:39:09.306633+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:47:36.505 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:47:36 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:47:36.160+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:09.306551+0000 front 2023-12-16T21:39:09.306652+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:47:36.505 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:47:36 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:47:36.160+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:09.306605+0000 front 2023-12-16T21:39:09.306613+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:47:36.505 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:47:36 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:47:36.161+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:47:36.505 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Dec 16 21:47:36 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-mon-smithi045[80759]: 2023-12-16T21:47:36.378+0000 7f45a1f8f700 -1 mon.smithi045@0(probing) e2 get_health_metrics reporting 5525 slow ops, oldest is monmgrreport(gid 14214, 0 checks, 1 progress events) 2023-12-16T21:47:37.185 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:47:37 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:47:37.123+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:05.622397+0000 front 2023-12-16T21:39:05.622556+0000 (oldest deadline 2023-12-16T21:39:30.322546+0000) 2023-12-16T21:47:37.185 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:47:37 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:47:37.123+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:05.622506+0000 front 2023-12-16T21:39:05.622528+0000 (oldest deadline 2023-12-16T21:39:30.322546+0000) 2023-12-16T21:47:37.185 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:47:37 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:47:37.123+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:05.622479+0000 front 2023-12-16T21:39:05.622580+0000 (oldest deadline 2023-12-16T21:39:30.322546+0000) 2023-12-16T21:47:37.186 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:47:37 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:47:37.124+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:47:37.186 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:47:36 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:47:36.884+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:09.828911+0000 front 2023-12-16T21:39:09.828958+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:47:37.186 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:47:36 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:47:36.884+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:09.828868+0000 front 2023-12-16T21:39:09.828878+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:47:37.186 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:47:36 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:47:36.884+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:09.828947+0000 front 2023-12-16T21:39:09.828993+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:47:37.186 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:47:36 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:47:36.886+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:47:37.566 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:47:37 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:47:37.182+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:09.306583+0000 front 2023-12-16T21:39:09.306633+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:47:37.567 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:47:37 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:47:37.182+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:09.306551+0000 front 2023-12-16T21:39:09.306652+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:47:37.567 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:47:37 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:47:37.182+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:09.306605+0000 front 2023-12-16T21:39:09.306613+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:47:37.567 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:47:37 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:47:37.183+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:47:38.214 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:47:38 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:47:38.082+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:05.622397+0000 front 2023-12-16T21:39:05.622556+0000 (oldest deadline 2023-12-16T21:39:30.322546+0000) 2023-12-16T21:47:38.214 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:47:38 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:47:38.082+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:05.622506+0000 front 2023-12-16T21:39:05.622528+0000 (oldest deadline 2023-12-16T21:39:30.322546+0000) 2023-12-16T21:47:38.214 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:47:38 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:47:38.082+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:05.622479+0000 front 2023-12-16T21:39:05.622580+0000 (oldest deadline 2023-12-16T21:39:30.322546+0000) 2023-12-16T21:47:38.214 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:47:38 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:47:38.083+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:47:38.214 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:47:37 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:47:37.913+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:09.828911+0000 front 2023-12-16T21:39:09.828958+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:47:38.214 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:47:37 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:47:37.913+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:09.828868+0000 front 2023-12-16T21:39:09.828878+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:47:38.214 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:47:37 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:47:37.913+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:09.828947+0000 front 2023-12-16T21:39:09.828993+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:47:38.215 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:47:37 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:47:37.914+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:47:38.566 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:47:38 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:47:38.209+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:09.306583+0000 front 2023-12-16T21:39:09.306633+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:47:38.566 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:47:38 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:47:38.209+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:09.306551+0000 front 2023-12-16T21:39:09.306652+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:47:38.567 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:47:38 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:47:38.209+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:09.306605+0000 front 2023-12-16T21:39:09.306613+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:47:38.567 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:47:38 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:47:38.211+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:47:39.063 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:47:38 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:47:38.939+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:09.828911+0000 front 2023-12-16T21:39:09.828958+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:47:39.063 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:47:38 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:47:38.939+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:09.828868+0000 front 2023-12-16T21:39:09.828878+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:47:39.063 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:47:38 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:47:38.939+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:09.828947+0000 front 2023-12-16T21:39:09.828993+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:47:39.064 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:47:38 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:47:38.940+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:47:39.316 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:47:39 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:47:39.236+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:09.306583+0000 front 2023-12-16T21:39:09.306633+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:47:39.317 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:47:39 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:47:39.236+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:09.306551+0000 front 2023-12-16T21:39:09.306652+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:47:39.317 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:47:39 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:47:39.236+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:09.306605+0000 front 2023-12-16T21:39:09.306613+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:47:39.317 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:47:39 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:47:39.237+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:47:39.317 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:47:39 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:47:39.059+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:05.622397+0000 front 2023-12-16T21:39:05.622556+0000 (oldest deadline 2023-12-16T21:39:30.322546+0000) 2023-12-16T21:47:39.317 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:47:39 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:47:39.059+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:05.622506+0000 front 2023-12-16T21:39:05.622528+0000 (oldest deadline 2023-12-16T21:39:30.322546+0000) 2023-12-16T21:47:39.317 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:47:39 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:47:39.059+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:05.622479+0000 front 2023-12-16T21:39:05.622580+0000 (oldest deadline 2023-12-16T21:39:30.322546+0000) 2023-12-16T21:47:39.317 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:47:39 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:47:39.060+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:47:40.243 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:47:39 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:47:39.946+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:09.828911+0000 front 2023-12-16T21:39:09.828958+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:47:40.244 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:47:39 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:47:39.946+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:09.828868+0000 front 2023-12-16T21:39:09.828878+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:47:40.244 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:47:39 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:47:39.946+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:09.828947+0000 front 2023-12-16T21:39:09.828993+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:47:40.244 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:47:39 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:47:39.947+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:47:40.244 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:47:40 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:47:40.092+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:05.622397+0000 front 2023-12-16T21:39:05.622556+0000 (oldest deadline 2023-12-16T21:39:30.322546+0000) 2023-12-16T21:47:40.244 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:47:40 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:47:40.092+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:05.622506+0000 front 2023-12-16T21:39:05.622528+0000 (oldest deadline 2023-12-16T21:39:30.322546+0000) 2023-12-16T21:47:40.244 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:47:40 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:47:40.092+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:05.622479+0000 front 2023-12-16T21:39:05.622580+0000 (oldest deadline 2023-12-16T21:39:30.322546+0000) 2023-12-16T21:47:40.244 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:47:40 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:47:40.093+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:47:40.566 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:47:40 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:47:40.239+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:09.306583+0000 front 2023-12-16T21:39:09.306633+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:47:40.567 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:47:40 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:47:40.239+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:09.306551+0000 front 2023-12-16T21:39:09.306652+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:47:40.567 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:47:40 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:47:40.239+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:09.306605+0000 front 2023-12-16T21:39:09.306613+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:47:40.567 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:47:40 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:47:40.240+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:47:41.255 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:47:40 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:47:40.980+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:09.828911+0000 front 2023-12-16T21:39:09.828958+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:47:41.255 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:47:40 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:47:40.980+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:09.828868+0000 front 2023-12-16T21:39:09.828878+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:47:41.255 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:47:40 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:47:40.980+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:09.828947+0000 front 2023-12-16T21:39:09.828993+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:47:41.255 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:47:40 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:47:40.981+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:47:41.255 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:47:41 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:47:41.076+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:05.622397+0000 front 2023-12-16T21:39:05.622556+0000 (oldest deadline 2023-12-16T21:39:30.322546+0000) 2023-12-16T21:47:41.255 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:47:41 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:47:41.077+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:05.622506+0000 front 2023-12-16T21:39:05.622528+0000 (oldest deadline 2023-12-16T21:39:30.322546+0000) 2023-12-16T21:47:41.255 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:47:41 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:47:41.077+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:05.622479+0000 front 2023-12-16T21:39:05.622580+0000 (oldest deadline 2023-12-16T21:39:30.322546+0000) 2023-12-16T21:47:41.255 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:47:41 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:47:41.077+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:47:41.506 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:47:41 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:47:41.251+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:09.306583+0000 front 2023-12-16T21:39:09.306633+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:47:41.507 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:47:41 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:47:41.251+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:09.306551+0000 front 2023-12-16T21:39:09.306652+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:47:41.507 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:47:41 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:47:41.251+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:09.306605+0000 front 2023-12-16T21:39:09.306613+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:47:41.507 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:47:41 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:47:41.251+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:47:41.507 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Dec 16 21:47:41 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-mon-smithi045[80759]: 2023-12-16T21:47:41.378+0000 7f45a1f8f700 -1 mon.smithi045@0(probing) e2 get_health_metrics reporting 5622 slow ops, oldest is monmgrreport(gid 14214, 0 checks, 1 progress events) 2023-12-16T21:47:42.288 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:47:42 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:47:42.031+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:09.828911+0000 front 2023-12-16T21:39:09.828958+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:47:42.288 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:47:42 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:47:42.031+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:09.828868+0000 front 2023-12-16T21:39:09.828878+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:47:42.288 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:47:42 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:47:42.031+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:09.828947+0000 front 2023-12-16T21:39:09.828993+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:47:42.289 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:47:42 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:47:42.031+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:47:42.289 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:47:42 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:47:42.077+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:05.622397+0000 front 2023-12-16T21:39:05.622556+0000 (oldest deadline 2023-12-16T21:39:30.322546+0000) 2023-12-16T21:47:42.289 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:47:42 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:47:42.077+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:05.622506+0000 front 2023-12-16T21:39:05.622528+0000 (oldest deadline 2023-12-16T21:39:30.322546+0000) 2023-12-16T21:47:42.289 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:47:42 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:47:42.077+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:05.622479+0000 front 2023-12-16T21:39:05.622580+0000 (oldest deadline 2023-12-16T21:39:30.322546+0000) 2023-12-16T21:47:42.289 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:47:42 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:47:42.079+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:47:42.566 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:47:42 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:47:42.284+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:09.306583+0000 front 2023-12-16T21:39:09.306633+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:47:42.566 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:47:42 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:47:42.284+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:09.306551+0000 front 2023-12-16T21:39:09.306652+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:47:42.567 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:47:42 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:47:42.284+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:09.306605+0000 front 2023-12-16T21:39:09.306613+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:47:42.567 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:47:42 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:47:42.285+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:47:43.544 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:47:43 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:47:43.273+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:09.306583+0000 front 2023-12-16T21:39:09.306633+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:47:43.544 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:47:43 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:47:43.273+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:09.306551+0000 front 2023-12-16T21:39:09.306652+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:47:43.544 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:47:43 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:47:43.273+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:09.306605+0000 front 2023-12-16T21:39:09.306613+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:47:43.544 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:47:43 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:47:43.273+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:47:43.545 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:47:43 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:47:43.071+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:09.828911+0000 front 2023-12-16T21:39:09.828958+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:47:43.545 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:47:43 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:47:43.071+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:09.828868+0000 front 2023-12-16T21:39:09.828878+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:47:43.545 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:47:43 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:47:43.071+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:09.828947+0000 front 2023-12-16T21:39:09.828993+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:47:43.545 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:47:43 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:47:43.072+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:47:43.545 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:47:43 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:47:43.114+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:05.622397+0000 front 2023-12-16T21:39:05.622556+0000 (oldest deadline 2023-12-16T21:39:30.322546+0000) 2023-12-16T21:47:43.545 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:47:43 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:47:43.114+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:05.622506+0000 front 2023-12-16T21:39:05.622528+0000 (oldest deadline 2023-12-16T21:39:30.322546+0000) 2023-12-16T21:47:43.545 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:47:43 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:47:43.114+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:05.622479+0000 front 2023-12-16T21:39:05.622580+0000 (oldest deadline 2023-12-16T21:39:30.322546+0000) 2023-12-16T21:47:43.545 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:47:43 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:47:43.114+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:47:44.566 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:47:44 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:47:44.143+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:05.622397+0000 front 2023-12-16T21:39:05.622556+0000 (oldest deadline 2023-12-16T21:39:30.322546+0000) 2023-12-16T21:47:44.566 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:47:44 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:47:44.143+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:05.622506+0000 front 2023-12-16T21:39:05.622528+0000 (oldest deadline 2023-12-16T21:39:30.322546+0000) 2023-12-16T21:47:44.567 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:47:44 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:47:44.143+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:05.622479+0000 front 2023-12-16T21:39:05.622580+0000 (oldest deadline 2023-12-16T21:39:30.322546+0000) 2023-12-16T21:47:44.567 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:47:44 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:47:44.144+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:47:44.567 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:47:44 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:47:44.309+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:09.306583+0000 front 2023-12-16T21:39:09.306633+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:47:44.567 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:47:44 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:47:44.309+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:09.306551+0000 front 2023-12-16T21:39:09.306652+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:47:44.567 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:47:44 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:47:44.309+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:09.306605+0000 front 2023-12-16T21:39:09.306613+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:47:44.567 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:47:44 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:47:44.310+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:47:44.567 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:47:44 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:47:44.089+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:09.828911+0000 front 2023-12-16T21:39:09.828958+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:47:44.568 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:47:44 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:47:44.089+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:09.828868+0000 front 2023-12-16T21:39:09.828878+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:47:44.568 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:47:44 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:47:44.089+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:09.828947+0000 front 2023-12-16T21:39:09.828993+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:47:44.568 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:47:44 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:47:44.090+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:47:45.567 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:47:45 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:47:45.306+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:09.306583+0000 front 2023-12-16T21:39:09.306633+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:47:45.567 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:47:45 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:47:45.306+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:09.306551+0000 front 2023-12-16T21:39:09.306652+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:47:45.567 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:47:45 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:47:45.306+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:09.306605+0000 front 2023-12-16T21:39:09.306613+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:47:45.567 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:47:45 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:47:45.307+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:47:45.567 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:47:45 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:47:45.094+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:05.622397+0000 front 2023-12-16T21:39:05.622556+0000 (oldest deadline 2023-12-16T21:39:30.322546+0000) 2023-12-16T21:47:45.567 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:47:45 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:47:45.094+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:05.622506+0000 front 2023-12-16T21:39:05.622528+0000 (oldest deadline 2023-12-16T21:39:30.322546+0000) 2023-12-16T21:47:45.567 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:47:45 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:47:45.094+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:05.622479+0000 front 2023-12-16T21:39:05.622580+0000 (oldest deadline 2023-12-16T21:39:30.322546+0000) 2023-12-16T21:47:45.568 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:47:45 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:47:45.095+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:47:45.568 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:47:45 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:47:45.123+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:09.828911+0000 front 2023-12-16T21:39:09.828958+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:47:45.569 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:47:45 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:47:45.123+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:09.828868+0000 front 2023-12-16T21:39:09.828878+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:47:45.569 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:47:45 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:47:45.123+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:09.828947+0000 front 2023-12-16T21:39:09.828993+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:47:45.569 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:47:45 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:47:45.123+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:47:46.384 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:47:46 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:47:46.082+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:09.828911+0000 front 2023-12-16T21:39:09.828958+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:47:46.384 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:47:46 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:47:46.082+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:09.828868+0000 front 2023-12-16T21:39:09.828878+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:47:46.384 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:47:46 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:47:46.082+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:09.828947+0000 front 2023-12-16T21:39:09.828993+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:47:46.384 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:47:46 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:47:46.083+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:47:46.384 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:47:46 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:47:46.331+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:09.306583+0000 front 2023-12-16T21:39:09.306633+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:47:46.384 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:47:46 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:47:46.331+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:09.306551+0000 front 2023-12-16T21:39:09.306652+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:47:46.384 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:47:46 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:47:46.331+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:09.306605+0000 front 2023-12-16T21:39:09.306613+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:47:46.384 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:47:46 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:47:46.332+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:47:46.385 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:47:46 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:47:46.140+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:05.622397+0000 front 2023-12-16T21:39:05.622556+0000 (oldest deadline 2023-12-16T21:39:30.322546+0000) 2023-12-16T21:47:46.385 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:47:46 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:47:46.140+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:05.622506+0000 front 2023-12-16T21:39:05.622528+0000 (oldest deadline 2023-12-16T21:39:30.322546+0000) 2023-12-16T21:47:46.385 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:47:46 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:47:46.140+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:05.622479+0000 front 2023-12-16T21:39:05.622580+0000 (oldest deadline 2023-12-16T21:39:30.322546+0000) 2023-12-16T21:47:46.385 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:47:46 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:47:46.141+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:47:46.816 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Dec 16 21:47:46 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-mon-smithi045[80759]: 2023-12-16T21:47:46.380+0000 7f45a1f8f700 -1 mon.smithi045@0(probing) e2 get_health_metrics reporting 5716 slow ops, oldest is monmgrreport(gid 14214, 0 checks, 1 progress events) 2023-12-16T21:47:47.566 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:47:47 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:47:47.132+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:09.828911+0000 front 2023-12-16T21:39:09.828958+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:47:47.567 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:47:47 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:47:47.132+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:09.828868+0000 front 2023-12-16T21:39:09.828878+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:47:47.567 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:47:47 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:47:47.132+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:09.828947+0000 front 2023-12-16T21:39:09.828993+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:47:47.567 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:47:47 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:47:47.133+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:47:47.567 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:47:47 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:47:47.284+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:09.306583+0000 front 2023-12-16T21:39:09.306633+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:47:47.567 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:47:47 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:47:47.284+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:09.306551+0000 front 2023-12-16T21:39:09.306652+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:47:47.567 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:47:47 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:47:47.284+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:09.306605+0000 front 2023-12-16T21:39:09.306613+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:47:47.567 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:47:47 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:47:47.285+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:47:47.568 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:47:47 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:47:47.113+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:05.622397+0000 front 2023-12-16T21:39:05.622556+0000 (oldest deadline 2023-12-16T21:39:30.322546+0000) 2023-12-16T21:47:47.568 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:47:47 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:47:47.113+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:05.622506+0000 front 2023-12-16T21:39:05.622528+0000 (oldest deadline 2023-12-16T21:39:30.322546+0000) 2023-12-16T21:47:47.568 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:47:47 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:47:47.113+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:05.622479+0000 front 2023-12-16T21:39:05.622580+0000 (oldest deadline 2023-12-16T21:39:30.322546+0000) 2023-12-16T21:47:47.568 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:47:47 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:47:47.114+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:47:48.567 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:47:48 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:47:48.160+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:05.622397+0000 front 2023-12-16T21:39:05.622556+0000 (oldest deadline 2023-12-16T21:39:30.322546+0000) 2023-12-16T21:47:48.567 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:47:48 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:47:48.160+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:05.622506+0000 front 2023-12-16T21:39:05.622528+0000 (oldest deadline 2023-12-16T21:39:30.322546+0000) 2023-12-16T21:47:48.567 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:47:48 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:47:48.160+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:05.622479+0000 front 2023-12-16T21:39:05.622580+0000 (oldest deadline 2023-12-16T21:39:30.322546+0000) 2023-12-16T21:47:48.567 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:47:48 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:47:48.161+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:47:48.567 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:47:48 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:47:48.305+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:09.306583+0000 front 2023-12-16T21:39:09.306633+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:47:48.567 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:47:48 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:47:48.305+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:09.306551+0000 front 2023-12-16T21:39:09.306652+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:47:48.567 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:47:48 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:47:48.305+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:09.306605+0000 front 2023-12-16T21:39:09.306613+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:47:48.568 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:47:48 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:47:48.306+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:47:48.568 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:47:48 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:47:48.177+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:09.828911+0000 front 2023-12-16T21:39:09.828958+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:47:48.568 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:47:48 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:47:48.177+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:09.828868+0000 front 2023-12-16T21:39:09.828878+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:47:48.568 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:47:48 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:47:48.177+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:09.828947+0000 front 2023-12-16T21:39:09.828993+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:47:48.568 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:47:48 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:47:48.178+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:47:49.567 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:47:49 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:47:49.273+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:09.306583+0000 front 2023-12-16T21:39:09.306633+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:47:49.567 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:47:49 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:47:49.273+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:09.306551+0000 front 2023-12-16T21:39:09.306652+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:47:49.567 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:47:49 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:47:49.273+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:09.306605+0000 front 2023-12-16T21:39:09.306613+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:47:49.567 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:47:49 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:47:49.274+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:47:49.567 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:47:49 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:47:49.175+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:05.622397+0000 front 2023-12-16T21:39:05.622556+0000 (oldest deadline 2023-12-16T21:39:30.322546+0000) 2023-12-16T21:47:49.568 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:47:49 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:47:49.175+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:05.622506+0000 front 2023-12-16T21:39:05.622528+0000 (oldest deadline 2023-12-16T21:39:30.322546+0000) 2023-12-16T21:47:49.568 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:47:49 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:47:49.175+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:05.622479+0000 front 2023-12-16T21:39:05.622580+0000 (oldest deadline 2023-12-16T21:39:30.322546+0000) 2023-12-16T21:47:49.568 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:47:49 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:47:49.175+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:47:49.568 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:47:49 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:47:49.205+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:09.828911+0000 front 2023-12-16T21:39:09.828958+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:47:49.568 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:47:49 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:47:49.205+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:09.828868+0000 front 2023-12-16T21:39:09.828878+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:47:49.568 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:47:49 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:47:49.205+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:09.828947+0000 front 2023-12-16T21:39:09.828993+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:47:49.568 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:47:49 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:47:49.205+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:47:50.567 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:47:50 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:47:50.165+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:05.622397+0000 front 2023-12-16T21:39:05.622556+0000 (oldest deadline 2023-12-16T21:39:30.322546+0000) 2023-12-16T21:47:50.567 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:47:50 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:47:50.165+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:05.622506+0000 front 2023-12-16T21:39:05.622528+0000 (oldest deadline 2023-12-16T21:39:30.322546+0000) 2023-12-16T21:47:50.567 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:47:50 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:47:50.165+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:05.622479+0000 front 2023-12-16T21:39:05.622580+0000 (oldest deadline 2023-12-16T21:39:30.322546+0000) 2023-12-16T21:47:50.567 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:47:50 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:47:50.166+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:47:50.567 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:47:50 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:47:50.253+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:09.306583+0000 front 2023-12-16T21:39:09.306633+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:47:50.568 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:47:50 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:47:50.253+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:09.306551+0000 front 2023-12-16T21:39:09.306652+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:47:50.568 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:47:50 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:47:50.253+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:09.306605+0000 front 2023-12-16T21:39:09.306613+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:47:50.568 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:47:50 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:47:50.254+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:47:50.568 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:47:50 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:47:50.158+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:09.828911+0000 front 2023-12-16T21:39:09.828958+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:47:50.568 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:47:50 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:47:50.158+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:09.828868+0000 front 2023-12-16T21:39:09.828878+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:47:50.568 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:47:50 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:47:50.158+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:09.828947+0000 front 2023-12-16T21:39:09.828993+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:47:50.569 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:47:50 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:47:50.159+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:47:51.405 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:47:51 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:47:51.302+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:09.306583+0000 front 2023-12-16T21:39:09.306633+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:47:51.406 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:47:51 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:47:51.302+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:09.306551+0000 front 2023-12-16T21:39:09.306652+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:47:51.406 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:47:51 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:47:51.302+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:09.306605+0000 front 2023-12-16T21:39:09.306613+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:47:51.406 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:47:51 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:47:51.303+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:47:51.406 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:47:51 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:47:51.138+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:09.828911+0000 front 2023-12-16T21:39:09.828958+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:47:51.407 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:47:51 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:47:51.138+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:09.828868+0000 front 2023-12-16T21:39:09.828878+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:47:51.407 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:47:51 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:47:51.138+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:09.828947+0000 front 2023-12-16T21:39:09.828993+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:47:51.407 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:47:51 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:47:51.139+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:47:51.407 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:47:51 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:47:51.142+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:05.622397+0000 front 2023-12-16T21:39:05.622556+0000 (oldest deadline 2023-12-16T21:39:30.322546+0000) 2023-12-16T21:47:51.408 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:47:51 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:47:51.142+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:05.622506+0000 front 2023-12-16T21:39:05.622528+0000 (oldest deadline 2023-12-16T21:39:30.322546+0000) 2023-12-16T21:47:51.408 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:47:51 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:47:51.142+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:05.622479+0000 front 2023-12-16T21:39:05.622580+0000 (oldest deadline 2023-12-16T21:39:30.322546+0000) 2023-12-16T21:47:51.408 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:47:51 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:47:51.143+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:47:51.408 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Dec 16 21:47:51 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-mon-smithi045[80759]: 2023-12-16T21:47:51.381+0000 7f45a1f8f700 -1 mon.smithi045@0(probing) e2 get_health_metrics reporting 5813 slow ops, oldest is monmgrreport(gid 14214, 0 checks, 1 progress events) 2023-12-16T21:47:52.567 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:47:52 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:47:52.283+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:09.306583+0000 front 2023-12-16T21:39:09.306633+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:47:52.567 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:47:52 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:47:52.283+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:09.306551+0000 front 2023-12-16T21:39:09.306652+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:47:52.567 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:47:52 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:47:52.283+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:09.306605+0000 front 2023-12-16T21:39:09.306613+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:47:52.567 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:47:52 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:47:52.283+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:47:52.567 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:47:52 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:47:52.127+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:09.828911+0000 front 2023-12-16T21:39:09.828958+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:47:52.568 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:47:52 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:47:52.127+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:09.828868+0000 front 2023-12-16T21:39:09.828878+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:47:52.568 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:47:52 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:47:52.127+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:09.828947+0000 front 2023-12-16T21:39:09.828993+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:47:52.568 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:47:52 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:47:52.129+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:47:52.568 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:47:52 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:47:52.151+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:05.622397+0000 front 2023-12-16T21:39:05.622556+0000 (oldest deadline 2023-12-16T21:39:30.322546+0000) 2023-12-16T21:47:52.568 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:47:52 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:47:52.151+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:05.622506+0000 front 2023-12-16T21:39:05.622528+0000 (oldest deadline 2023-12-16T21:39:30.322546+0000) 2023-12-16T21:47:52.568 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:47:52 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:47:52.151+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:05.622479+0000 front 2023-12-16T21:39:05.622580+0000 (oldest deadline 2023-12-16T21:39:30.322546+0000) 2023-12-16T21:47:52.568 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:47:52 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:47:52.152+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:47:53.544 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:47:53 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:47:53.120+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:05.622397+0000 front 2023-12-16T21:39:05.622556+0000 (oldest deadline 2023-12-16T21:39:30.322546+0000) 2023-12-16T21:47:53.545 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:47:53 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:47:53.120+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:05.622506+0000 front 2023-12-16T21:39:05.622528+0000 (oldest deadline 2023-12-16T21:39:30.322546+0000) 2023-12-16T21:47:53.545 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:47:53 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:47:53.120+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:05.622479+0000 front 2023-12-16T21:39:05.622580+0000 (oldest deadline 2023-12-16T21:39:30.322546+0000) 2023-12-16T21:47:53.545 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:47:53 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:47:53.121+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:47:53.545 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:47:53 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:47:53.317+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:09.306583+0000 front 2023-12-16T21:39:09.306633+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:47:53.545 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:47:53 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:47:53.317+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:09.306551+0000 front 2023-12-16T21:39:09.306652+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:47:53.545 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:47:53 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:47:53.317+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:09.306605+0000 front 2023-12-16T21:39:09.306613+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:47:53.545 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:47:53 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:47:53.319+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:47:53.546 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:47:53 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:47:53.178+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:09.828911+0000 front 2023-12-16T21:39:09.828958+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:47:53.546 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:47:53 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:47:53.178+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:09.828868+0000 front 2023-12-16T21:39:09.828878+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:47:53.546 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:47:53 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:47:53.178+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:09.828947+0000 front 2023-12-16T21:39:09.828993+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:47:53.546 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:47:53 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:47:53.179+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:47:54.513 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:47:54 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:47:54.130+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:09.828911+0000 front 2023-12-16T21:39:09.828958+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:47:54.513 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:47:54 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:47:54.130+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:09.828868+0000 front 2023-12-16T21:39:09.828878+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:47:54.514 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:47:54 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:47:54.130+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:09.828947+0000 front 2023-12-16T21:39:09.828993+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:47:54.514 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:47:54 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:47:54.131+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:47:54.514 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:47:54 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:47:54.328+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:09.306583+0000 front 2023-12-16T21:39:09.306633+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:47:54.514 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:47:54 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:47:54.328+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:09.306551+0000 front 2023-12-16T21:39:09.306652+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:47:54.514 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:47:54 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:47:54.328+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:09.306605+0000 front 2023-12-16T21:39:09.306613+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:47:54.514 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:47:54 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:47:54.329+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:47:54.514 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:47:54 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:47:54.127+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:05.622397+0000 front 2023-12-16T21:39:05.622556+0000 (oldest deadline 2023-12-16T21:39:30.322546+0000) 2023-12-16T21:47:54.515 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:47:54 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:47:54.127+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:05.622506+0000 front 2023-12-16T21:39:05.622528+0000 (oldest deadline 2023-12-16T21:39:30.322546+0000) 2023-12-16T21:47:54.515 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:47:54 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:47:54.127+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:05.622479+0000 front 2023-12-16T21:39:05.622580+0000 (oldest deadline 2023-12-16T21:39:30.322546+0000) 2023-12-16T21:47:54.515 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:47:54 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:47:54.128+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:47:55.566 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:47:55 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:47:55.310+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:09.306583+0000 front 2023-12-16T21:39:09.306633+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:47:55.567 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:47:55 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:47:55.310+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:09.306551+0000 front 2023-12-16T21:39:09.306652+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:47:55.567 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:47:55 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:47:55.310+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:09.306605+0000 front 2023-12-16T21:39:09.306613+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:47:55.567 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:47:55 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:47:55.311+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:47:55.567 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:47:55 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:47:55.127+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:05.622397+0000 front 2023-12-16T21:39:05.622556+0000 (oldest deadline 2023-12-16T21:39:30.322546+0000) 2023-12-16T21:47:55.567 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:47:55 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:47:55.127+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:05.622506+0000 front 2023-12-16T21:39:05.622528+0000 (oldest deadline 2023-12-16T21:39:30.322546+0000) 2023-12-16T21:47:55.567 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:47:55 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:47:55.127+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:05.622479+0000 front 2023-12-16T21:39:05.622580+0000 (oldest deadline 2023-12-16T21:39:30.322546+0000) 2023-12-16T21:47:55.567 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:47:55 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:47:55.128+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:47:55.568 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:47:55 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:47:55.179+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:09.828911+0000 front 2023-12-16T21:39:09.828958+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:47:55.568 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:47:55 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:47:55.179+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:09.828868+0000 front 2023-12-16T21:39:09.828878+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:47:55.568 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:47:55 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:47:55.179+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:09.828947+0000 front 2023-12-16T21:39:09.828993+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:47:55.568 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:47:55 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:47:55.179+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:47:56.507 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:47:56 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:47:56.340+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:09.306583+0000 front 2023-12-16T21:39:09.306633+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:47:56.508 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:47:56 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:47:56.340+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:09.306551+0000 front 2023-12-16T21:39:09.306652+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:47:56.508 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:47:56 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:47:56.340+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:09.306605+0000 front 2023-12-16T21:39:09.306613+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:47:56.508 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:47:56 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:47:56.341+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:47:56.508 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:47:56 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:47:56.159+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:05.622397+0000 front 2023-12-16T21:39:05.622556+0000 (oldest deadline 2023-12-16T21:39:30.322546+0000) 2023-12-16T21:47:56.508 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:47:56 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:47:56.159+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:05.622506+0000 front 2023-12-16T21:39:05.622528+0000 (oldest deadline 2023-12-16T21:39:30.322546+0000) 2023-12-16T21:47:56.508 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:47:56 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:47:56.159+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:05.622479+0000 front 2023-12-16T21:39:05.622580+0000 (oldest deadline 2023-12-16T21:39:30.322546+0000) 2023-12-16T21:47:56.508 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:47:56 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:47:56.160+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:47:56.509 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:47:56 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:47:56.181+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:09.828911+0000 front 2023-12-16T21:39:09.828958+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:47:56.509 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:47:56 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:47:56.181+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:09.828868+0000 front 2023-12-16T21:39:09.828878+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:47:56.509 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:47:56 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:47:56.181+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:09.828947+0000 front 2023-12-16T21:39:09.828993+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:47:56.509 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:47:56 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:47:56.183+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:47:56.510 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Dec 16 21:47:56 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-mon-smithi045[80759]: 2023-12-16T21:47:56.382+0000 7f45a1f8f700 -1 mon.smithi045@0(probing) e2 get_health_metrics reporting 5908 slow ops, oldest is monmgrreport(gid 14214, 0 checks, 1 progress events) 2023-12-16T21:47:57.566 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:47:57 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:47:57.348+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:09.306583+0000 front 2023-12-16T21:39:09.306633+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:47:57.567 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:47:57 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:47:57.348+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:09.306551+0000 front 2023-12-16T21:39:09.306652+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:47:57.567 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:47:57 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:47:57.348+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:09.306605+0000 front 2023-12-16T21:39:09.306613+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:47:57.567 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:47:57 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:47:57.348+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:47:57.567 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:47:57 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:47:57.119+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:05.622397+0000 front 2023-12-16T21:39:05.622556+0000 (oldest deadline 2023-12-16T21:39:30.322546+0000) 2023-12-16T21:47:57.567 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:47:57 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:47:57.119+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:05.622506+0000 front 2023-12-16T21:39:05.622528+0000 (oldest deadline 2023-12-16T21:39:30.322546+0000) 2023-12-16T21:47:57.567 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:47:57 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:47:57.119+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:05.622479+0000 front 2023-12-16T21:39:05.622580+0000 (oldest deadline 2023-12-16T21:39:30.322546+0000) 2023-12-16T21:47:57.567 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:47:57 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:47:57.119+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:47:57.568 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:47:57 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:47:57.141+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:09.828911+0000 front 2023-12-16T21:39:09.828958+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:47:57.568 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:47:57 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:47:57.142+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:09.828868+0000 front 2023-12-16T21:39:09.828878+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:47:57.568 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:47:57 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:47:57.142+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:09.828947+0000 front 2023-12-16T21:39:09.828993+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:47:57.568 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:47:57 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:47:57.143+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:47:58.369 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:47:58 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:47:58.168+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:05.622397+0000 front 2023-12-16T21:39:05.622556+0000 (oldest deadline 2023-12-16T21:39:30.322546+0000) 2023-12-16T21:47:58.369 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:47:58 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:47:58.168+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:05.622506+0000 front 2023-12-16T21:39:05.622528+0000 (oldest deadline 2023-12-16T21:39:30.322546+0000) 2023-12-16T21:47:58.369 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:47:58 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:47:58.168+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:05.622479+0000 front 2023-12-16T21:39:05.622580+0000 (oldest deadline 2023-12-16T21:39:30.322546+0000) 2023-12-16T21:47:58.369 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:47:58 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:47:58.169+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:47:58.369 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:47:58 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:47:58.152+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:09.828911+0000 front 2023-12-16T21:39:09.828958+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:47:58.369 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:47:58 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:47:58.152+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:09.828868+0000 front 2023-12-16T21:39:09.828878+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:47:58.369 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:47:58 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:47:58.152+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:09.828947+0000 front 2023-12-16T21:39:09.828993+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:47:58.370 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:47:58 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:47:58.153+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:47:58.722 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:47:58 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:47:58.365+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:09.306583+0000 front 2023-12-16T21:39:09.306633+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:47:58.722 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:47:58 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:47:58.365+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:09.306551+0000 front 2023-12-16T21:39:09.306652+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:47:58.723 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:47:58 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:47:58.365+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:09.306605+0000 front 2023-12-16T21:39:09.306613+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:47:58.723 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:47:58 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:47:58.366+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:47:59.417 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:47:59 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:47:59.161+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:09.828911+0000 front 2023-12-16T21:39:09.828958+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:47:59.418 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:47:59 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:47:59.161+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:09.828868+0000 front 2023-12-16T21:39:09.828878+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:47:59.418 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:47:59 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:47:59.161+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:09.828947+0000 front 2023-12-16T21:39:09.828993+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:47:59.418 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:47:59 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:47:59.162+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:47:59.418 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:47:59 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:47:59.190+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:05.622397+0000 front 2023-12-16T21:39:05.622556+0000 (oldest deadline 2023-12-16T21:39:30.322546+0000) 2023-12-16T21:47:59.418 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:47:59 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:47:59.190+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:05.622506+0000 front 2023-12-16T21:39:05.622528+0000 (oldest deadline 2023-12-16T21:39:30.322546+0000) 2023-12-16T21:47:59.418 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:47:59 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:47:59.190+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:05.622479+0000 front 2023-12-16T21:39:05.622580+0000 (oldest deadline 2023-12-16T21:39:30.322546+0000) 2023-12-16T21:47:59.418 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:47:59 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:47:59.191+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:47:59.816 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:47:59 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:47:59.414+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:09.306583+0000 front 2023-12-16T21:39:09.306633+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:47:59.816 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:47:59 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:47:59.414+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:09.306551+0000 front 2023-12-16T21:39:09.306652+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:47:59.816 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:47:59 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:47:59.414+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:09.306605+0000 front 2023-12-16T21:39:09.306613+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:47:59.817 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:47:59 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:47:59.415+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:48:00.566 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:48:00 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:48:00.175+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:09.828911+0000 front 2023-12-16T21:39:09.828958+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:48:00.567 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:48:00 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:48:00.175+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:09.828868+0000 front 2023-12-16T21:39:09.828878+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:48:00.567 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:48:00 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:48:00.175+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:09.828947+0000 front 2023-12-16T21:39:09.828993+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:48:00.567 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:48:00 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:48:00.176+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:48:00.567 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:48:00 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:48:00.387+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:09.306583+0000 front 2023-12-16T21:39:09.306633+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:48:00.567 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:48:00 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:48:00.387+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:09.306551+0000 front 2023-12-16T21:39:09.306652+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:48:00.567 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:48:00 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:48:00.387+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:09.306605+0000 front 2023-12-16T21:39:09.306613+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:48:00.567 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:48:00 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:48:00.388+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:48:00.567 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:48:00 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:48:00.188+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:05.622397+0000 front 2023-12-16T21:39:05.622556+0000 (oldest deadline 2023-12-16T21:39:30.322546+0000) 2023-12-16T21:48:00.568 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:48:00 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:48:00.188+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:05.622506+0000 front 2023-12-16T21:39:05.622528+0000 (oldest deadline 2023-12-16T21:39:30.322546+0000) 2023-12-16T21:48:00.568 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:48:00 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:48:00.188+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:05.622479+0000 front 2023-12-16T21:39:05.622580+0000 (oldest deadline 2023-12-16T21:39:30.322546+0000) 2023-12-16T21:48:00.568 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:48:00 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:48:00.189+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:48:01.513 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Dec 16 21:48:01 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-mon-smithi045[80759]: 2023-12-16T21:48:01.383+0000 7f45a1f8f700 -1 mon.smithi045@0(probing) e2 get_health_metrics reporting 6005 slow ops, oldest is monmgrreport(gid 14214, 0 checks, 1 progress events) 2023-12-16T21:48:01.513 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:48:01 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:48:01.399+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:09.306583+0000 front 2023-12-16T21:39:09.306633+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:48:01.514 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:48:01 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:48:01.399+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:09.306551+0000 front 2023-12-16T21:39:09.306652+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:48:01.514 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:48:01 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:48:01.399+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:09.306605+0000 front 2023-12-16T21:39:09.306613+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:48:01.514 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:48:01 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:48:01.400+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:48:01.514 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:48:01 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:48:01.182+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:09.828911+0000 front 2023-12-16T21:39:09.828958+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:48:01.514 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:48:01 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:48:01.182+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:09.828868+0000 front 2023-12-16T21:39:09.828878+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:48:01.514 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:48:01 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:48:01.182+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:09.828947+0000 front 2023-12-16T21:39:09.828993+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:48:01.514 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:48:01 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:48:01.183+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:48:01.514 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:48:01 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:48:01.173+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:05.622397+0000 front 2023-12-16T21:39:05.622556+0000 (oldest deadline 2023-12-16T21:39:30.322546+0000) 2023-12-16T21:48:01.515 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:48:01 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:48:01.173+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:05.622506+0000 front 2023-12-16T21:39:05.622528+0000 (oldest deadline 2023-12-16T21:39:30.322546+0000) 2023-12-16T21:48:01.515 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:48:01 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:48:01.173+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:05.622479+0000 front 2023-12-16T21:39:05.622580+0000 (oldest deadline 2023-12-16T21:39:30.322546+0000) 2023-12-16T21:48:01.515 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:48:01 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:48:01.174+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:48:02.567 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:48:02 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:48:02.210+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:05.622397+0000 front 2023-12-16T21:39:05.622556+0000 (oldest deadline 2023-12-16T21:39:30.322546+0000) 2023-12-16T21:48:02.567 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:48:02 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:48:02.210+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:05.622506+0000 front 2023-12-16T21:39:05.622528+0000 (oldest deadline 2023-12-16T21:39:30.322546+0000) 2023-12-16T21:48:02.567 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:48:02 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:48:02.210+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:05.622479+0000 front 2023-12-16T21:39:05.622580+0000 (oldest deadline 2023-12-16T21:39:30.322546+0000) 2023-12-16T21:48:02.567 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:48:02 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:48:02.211+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:48:02.567 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:48:02 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:48:02.382+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:09.306583+0000 front 2023-12-16T21:39:09.306633+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:48:02.567 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:48:02 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:48:02.382+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:09.306551+0000 front 2023-12-16T21:39:09.306652+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:48:02.567 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:48:02 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:48:02.382+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:09.306605+0000 front 2023-12-16T21:39:09.306613+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:48:02.567 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:48:02 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:48:02.383+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:48:02.568 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:48:02 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:48:02.145+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:09.828911+0000 front 2023-12-16T21:39:09.828958+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:48:02.568 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:48:02 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:48:02.145+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:09.828868+0000 front 2023-12-16T21:39:09.828878+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:48:02.568 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:48:02 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:48:02.145+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:09.828947+0000 front 2023-12-16T21:39:09.828993+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:48:02.568 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:48:02 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:48:02.145+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:48:03.393 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:48:03 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:48:03.259+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:05.622397+0000 front 2023-12-16T21:39:05.622556+0000 (oldest deadline 2023-12-16T21:39:30.322546+0000) 2023-12-16T21:48:03.393 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:48:03 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:48:03.259+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:05.622506+0000 front 2023-12-16T21:39:05.622528+0000 (oldest deadline 2023-12-16T21:39:30.322546+0000) 2023-12-16T21:48:03.393 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:48:03 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:48:03.259+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:05.622479+0000 front 2023-12-16T21:39:05.622580+0000 (oldest deadline 2023-12-16T21:39:30.322546+0000) 2023-12-16T21:48:03.393 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:48:03 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:48:03.260+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:48:03.393 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:48:03 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:48:03.098+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:09.828911+0000 front 2023-12-16T21:39:09.828958+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:48:03.393 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:48:03 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:48:03.098+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:09.828868+0000 front 2023-12-16T21:39:09.828878+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:48:03.393 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:48:03 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:48:03.098+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:09.828947+0000 front 2023-12-16T21:39:09.828993+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:48:03.393 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:48:03 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:48:03.099+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:48:03.729 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:48:03 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:48:03.389+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:09.306583+0000 front 2023-12-16T21:39:09.306633+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:48:03.729 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:48:03 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:48:03.389+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:09.306551+0000 front 2023-12-16T21:39:09.306652+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:48:03.729 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:48:03 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:48:03.389+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:09.306605+0000 front 2023-12-16T21:39:09.306613+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:48:03.729 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:48:03 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:48:03.390+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:48:04.407 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:48:04 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:48:04.276+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:05.622397+0000 front 2023-12-16T21:39:05.622556+0000 (oldest deadline 2023-12-16T21:39:30.322546+0000) 2023-12-16T21:48:04.407 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:48:04 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:48:04.276+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:05.622506+0000 front 2023-12-16T21:39:05.622528+0000 (oldest deadline 2023-12-16T21:39:30.322546+0000) 2023-12-16T21:48:04.407 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:48:04 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:48:04.276+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:05.622479+0000 front 2023-12-16T21:39:05.622580+0000 (oldest deadline 2023-12-16T21:39:30.322546+0000) 2023-12-16T21:48:04.408 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:48:04 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:48:04.277+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:48:04.408 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:48:04 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:48:04.112+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:09.828911+0000 front 2023-12-16T21:39:09.828958+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:48:04.408 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:48:04 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:48:04.112+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:09.828868+0000 front 2023-12-16T21:39:09.828878+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:48:04.408 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:48:04 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:48:04.112+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:09.828947+0000 front 2023-12-16T21:39:09.828993+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:48:04.408 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:48:04 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:48:04.112+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:48:04.817 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:48:04 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:48:04.404+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:09.306583+0000 front 2023-12-16T21:39:09.306633+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:48:04.817 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:48:04 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:48:04.404+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:09.306551+0000 front 2023-12-16T21:39:09.306652+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:48:04.817 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:48:04 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:48:04.404+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:09.306605+0000 front 2023-12-16T21:39:09.306613+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:48:04.817 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:48:04 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:48:04.405+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:48:05.368 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:48:05 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:48:05.254+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:05.622397+0000 front 2023-12-16T21:39:05.622556+0000 (oldest deadline 2023-12-16T21:39:30.322546+0000) 2023-12-16T21:48:05.368 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:48:05 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:48:05.254+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:05.622506+0000 front 2023-12-16T21:39:05.622528+0000 (oldest deadline 2023-12-16T21:39:30.322546+0000) 2023-12-16T21:48:05.368 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:48:05 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:48:05.254+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:05.622479+0000 front 2023-12-16T21:39:05.622580+0000 (oldest deadline 2023-12-16T21:39:30.322546+0000) 2023-12-16T21:48:05.368 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:48:05 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:48:05.255+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:48:05.368 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:48:05 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:48:05.064+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:09.828911+0000 front 2023-12-16T21:39:09.828958+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:48:05.369 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:48:05 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:48:05.064+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:09.828868+0000 front 2023-12-16T21:39:09.828878+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:48:05.369 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:48:05 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:48:05.065+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:09.828947+0000 front 2023-12-16T21:39:09.828993+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:48:05.369 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:48:05 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:48:05.066+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:48:05.816 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:48:05 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:48:05.365+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:09.306583+0000 front 2023-12-16T21:39:09.306633+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:48:05.816 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:48:05 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:48:05.365+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:09.306551+0000 front 2023-12-16T21:39:09.306652+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:48:05.817 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:48:05 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:48:05.365+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:09.306605+0000 front 2023-12-16T21:39:09.306613+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:48:05.817 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:48:05 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:48:05.366+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:48:06.387 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:48:06 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:48:06.332+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:09.306583+0000 front 2023-12-16T21:39:09.306633+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:48:06.387 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:48:06 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:48:06.332+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:09.306551+0000 front 2023-12-16T21:39:09.306652+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:48:06.387 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:48:06 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:48:06.332+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:09.306605+0000 front 2023-12-16T21:39:09.306613+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:48:06.388 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:48:06 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:48:06.333+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:48:06.388 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:48:06 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:48:06.281+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:05.622397+0000 front 2023-12-16T21:39:05.622556+0000 (oldest deadline 2023-12-16T21:39:30.322546+0000) 2023-12-16T21:48:06.388 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:48:06 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:48:06.281+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:05.622506+0000 front 2023-12-16T21:39:05.622528+0000 (oldest deadline 2023-12-16T21:39:30.322546+0000) 2023-12-16T21:48:06.388 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:48:06 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:48:06.281+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:05.622479+0000 front 2023-12-16T21:39:05.622580+0000 (oldest deadline 2023-12-16T21:39:30.322546+0000) 2023-12-16T21:48:06.388 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:48:06 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:48:06.282+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:48:06.388 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:48:06 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:48:06.095+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:09.828911+0000 front 2023-12-16T21:39:09.828958+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:48:06.388 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:48:06 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:48:06.095+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:09.828868+0000 front 2023-12-16T21:39:09.828878+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:48:06.389 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:48:06 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:48:06.095+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:09.828947+0000 front 2023-12-16T21:39:09.828993+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:48:06.389 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:48:06 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:48:06.096+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:48:06.802 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Dec 16 21:48:06 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-mon-smithi045[80759]: 2023-12-16T21:48:06.384+0000 7f45a1f8f700 -1 mon.smithi045@0(probing) e2 get_health_metrics reporting 6100 slow ops, oldest is monmgrreport(gid 14214, 0 checks, 1 progress events) 2023-12-16T21:48:07.316 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:48:07 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:48:07.296+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:05.622397+0000 front 2023-12-16T21:39:05.622556+0000 (oldest deadline 2023-12-16T21:39:30.322546+0000) 2023-12-16T21:48:07.317 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:48:07 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:48:07.296+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:05.622506+0000 front 2023-12-16T21:39:05.622528+0000 (oldest deadline 2023-12-16T21:39:30.322546+0000) 2023-12-16T21:48:07.317 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:48:07 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:48:07.296+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:05.622479+0000 front 2023-12-16T21:39:05.622580+0000 (oldest deadline 2023-12-16T21:39:30.322546+0000) 2023-12-16T21:48:07.317 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:48:07 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:48:07.296+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:48:07.317 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:48:07 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:48:07.057+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:09.828911+0000 front 2023-12-16T21:39:09.828958+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:48:07.317 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:48:07 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:48:07.057+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:09.828868+0000 front 2023-12-16T21:39:09.828878+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:48:07.317 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:48:07 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:48:07.057+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:09.828947+0000 front 2023-12-16T21:39:09.828993+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:48:07.317 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:48:07 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:48:07.058+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:48:07.816 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:48:07 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:48:07.326+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:09.306583+0000 front 2023-12-16T21:39:09.306633+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:48:07.817 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:48:07 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:48:07.326+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:09.306551+0000 front 2023-12-16T21:39:09.306652+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:48:07.817 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:48:07 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:48:07.326+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:09.306605+0000 front 2023-12-16T21:39:09.306613+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:48:07.817 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:48:07 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:48:07.327+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:48:08.316 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:48:08 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:48:08.255+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:05.622397+0000 front 2023-12-16T21:39:05.622556+0000 (oldest deadline 2023-12-16T21:39:30.322546+0000) 2023-12-16T21:48:08.317 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:48:08 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:48:08.255+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:05.622506+0000 front 2023-12-16T21:39:05.622528+0000 (oldest deadline 2023-12-16T21:39:30.322546+0000) 2023-12-16T21:48:08.317 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:48:08 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:48:08.255+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:05.622479+0000 front 2023-12-16T21:39:05.622580+0000 (oldest deadline 2023-12-16T21:39:30.322546+0000) 2023-12-16T21:48:08.317 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:48:08 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:48:08.256+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:48:08.317 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:48:08 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:48:08.013+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:09.828911+0000 front 2023-12-16T21:39:09.828958+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:48:08.317 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:48:08 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:48:08.013+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:09.828868+0000 front 2023-12-16T21:39:09.828878+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:48:08.317 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:48:08 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:48:08.013+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:09.828947+0000 front 2023-12-16T21:39:09.828993+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:48:08.317 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:48:08 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:48:08.014+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:48:08.723 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:48:08 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:48:08.365+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:09.306583+0000 front 2023-12-16T21:39:09.306633+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:48:08.724 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:48:08 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:48:08.365+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:09.306551+0000 front 2023-12-16T21:39:09.306652+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:48:08.724 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:48:08 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:48:08.365+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:09.306605+0000 front 2023-12-16T21:39:09.306613+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:48:08.724 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:48:08 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:48:08.366+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:48:09.269 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:48:08 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:48:08.985+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:09.828911+0000 front 2023-12-16T21:39:09.828958+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:48:09.270 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:48:08 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:48:08.985+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:09.828868+0000 front 2023-12-16T21:39:09.828878+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:48:09.270 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:48:08 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:48:08.985+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:09.828947+0000 front 2023-12-16T21:39:09.828993+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:48:09.270 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:48:08 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:48:08.986+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:48:09.566 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:48:09 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:48:09.266+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:05.622397+0000 front 2023-12-16T21:39:05.622556+0000 (oldest deadline 2023-12-16T21:39:30.322546+0000) 2023-12-16T21:48:09.567 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:48:09 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:48:09.266+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:05.622506+0000 front 2023-12-16T21:39:05.622528+0000 (oldest deadline 2023-12-16T21:39:30.322546+0000) 2023-12-16T21:48:09.567 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:48:09 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:48:09.266+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:05.622479+0000 front 2023-12-16T21:39:05.622580+0000 (oldest deadline 2023-12-16T21:39:30.322546+0000) 2023-12-16T21:48:09.567 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:48:09 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:48:09.267+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:48:09.567 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:48:09 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:48:09.404+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:09.306583+0000 front 2023-12-16T21:39:09.306633+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:48:09.567 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:48:09 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:48:09.404+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:09.306551+0000 front 2023-12-16T21:39:09.306652+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:48:09.567 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:48:09 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:48:09.404+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:09.306605+0000 front 2023-12-16T21:39:09.306613+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:48:09.567 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:48:09 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:48:09.405+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:48:10.316 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:48:09 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:48:09.990+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:09.828911+0000 front 2023-12-16T21:39:09.828958+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:48:10.317 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:48:09 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:48:09.990+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:09.828868+0000 front 2023-12-16T21:39:09.828878+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:48:10.317 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:48:09 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:48:09.990+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:09.828947+0000 front 2023-12-16T21:39:09.828993+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:48:10.317 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:48:09 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:48:09.991+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:48:10.317 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:48:10 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:48:10.221+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:05.622397+0000 front 2023-12-16T21:39:05.622556+0000 (oldest deadline 2023-12-16T21:39:30.322546+0000) 2023-12-16T21:48:10.317 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:48:10 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:48:10.221+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:05.622506+0000 front 2023-12-16T21:39:05.622528+0000 (oldest deadline 2023-12-16T21:39:30.322546+0000) 2023-12-16T21:48:10.317 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:48:10 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:48:10.221+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:05.622479+0000 front 2023-12-16T21:39:05.622580+0000 (oldest deadline 2023-12-16T21:39:30.322546+0000) 2023-12-16T21:48:10.317 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:48:10 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:48:10.221+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:48:10.817 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:48:10 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:48:10.448+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:09.306583+0000 front 2023-12-16T21:39:09.306633+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:48:10.817 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:48:10 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:48:10.448+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:09.306551+0000 front 2023-12-16T21:39:09.306652+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:48:10.817 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:48:10 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:48:10.448+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:09.306605+0000 front 2023-12-16T21:39:09.306613+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:48:10.817 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:48:10 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:48:10.449+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:48:11.317 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:48:11 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:48:11.035+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:09.828911+0000 front 2023-12-16T21:39:09.828958+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:48:11.317 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:48:11 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:48:11.035+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:09.828868+0000 front 2023-12-16T21:39:09.828878+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:48:11.317 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:48:11 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:48:11.035+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:09.828947+0000 front 2023-12-16T21:39:09.828993+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:48:11.317 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:48:11 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:48:11.036+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:48:11.320 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:48:11 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:48:11.213+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:05.622397+0000 front 2023-12-16T21:39:05.622556+0000 (oldest deadline 2023-12-16T21:39:30.322546+0000) 2023-12-16T21:48:11.320 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:48:11 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:48:11.213+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:05.622506+0000 front 2023-12-16T21:39:05.622528+0000 (oldest deadline 2023-12-16T21:39:30.322546+0000) 2023-12-16T21:48:11.321 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:48:11 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:48:11.213+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:05.622479+0000 front 2023-12-16T21:39:05.622580+0000 (oldest deadline 2023-12-16T21:39:30.322546+0000) 2023-12-16T21:48:11.321 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:48:11 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:48:11.213+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:48:11.657 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Dec 16 21:48:11 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-mon-smithi045[80759]: 2023-12-16T21:48:11.385+0000 7f45a1f8f700 -1 mon.smithi045@0(probing) e2 get_health_metrics reporting 6197 slow ops, oldest is monmgrreport(gid 14214, 0 checks, 1 progress events) 2023-12-16T21:48:11.658 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:48:11 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:48:11.412+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:09.306583+0000 front 2023-12-16T21:39:09.306633+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:48:11.658 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:48:11 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:48:11.412+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:09.306551+0000 front 2023-12-16T21:39:09.306652+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:48:11.658 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:48:11 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:48:11.412+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:09.306605+0000 front 2023-12-16T21:39:09.306613+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:48:11.658 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:48:11 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:48:11.413+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:48:12.410 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:48:12 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:48:12.074+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:09.828911+0000 front 2023-12-16T21:39:09.828958+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:48:12.411 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:48:12 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:48:12.074+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:09.828868+0000 front 2023-12-16T21:39:09.828878+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:48:12.411 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:48:12 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:48:12.074+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:09.828947+0000 front 2023-12-16T21:39:09.828993+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:48:12.411 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:48:12 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:48:12.076+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:48:12.411 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:48:12 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:48:12.251+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:05.622397+0000 front 2023-12-16T21:39:05.622556+0000 (oldest deadline 2023-12-16T21:39:30.322546+0000) 2023-12-16T21:48:12.411 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:48:12 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:48:12.251+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:05.622506+0000 front 2023-12-16T21:39:05.622528+0000 (oldest deadline 2023-12-16T21:39:30.322546+0000) 2023-12-16T21:48:12.412 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:48:12 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:48:12.251+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:05.622479+0000 front 2023-12-16T21:39:05.622580+0000 (oldest deadline 2023-12-16T21:39:30.322546+0000) 2023-12-16T21:48:12.412 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:48:12 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:48:12.252+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:48:12.816 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:48:12 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:48:12.407+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:09.306583+0000 front 2023-12-16T21:39:09.306633+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:48:12.816 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:48:12 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:48:12.407+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:09.306551+0000 front 2023-12-16T21:39:09.306652+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:48:12.816 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:48:12 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:48:12.407+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:09.306605+0000 front 2023-12-16T21:39:09.306613+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:48:12.817 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:48:12 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:48:12.408+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:48:13.256 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:48:13 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:48:13.042+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:09.828911+0000 front 2023-12-16T21:39:09.828958+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:48:13.256 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:48:13 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:48:13.042+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:09.828868+0000 front 2023-12-16T21:39:09.828878+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:48:13.256 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:48:13 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:48:13.042+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:09.828947+0000 front 2023-12-16T21:39:09.828993+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:48:13.257 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:48:13 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:48:13.044+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:48:13.547 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:48:13 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:48:13.253+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:05.622397+0000 front 2023-12-16T21:39:05.622556+0000 (oldest deadline 2023-12-16T21:39:30.322546+0000) 2023-12-16T21:48:13.547 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:48:13 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:48:13.253+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:05.622506+0000 front 2023-12-16T21:39:05.622528+0000 (oldest deadline 2023-12-16T21:39:30.322546+0000) 2023-12-16T21:48:13.547 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:48:13 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:48:13.253+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:05.622479+0000 front 2023-12-16T21:39:05.622580+0000 (oldest deadline 2023-12-16T21:39:30.322546+0000) 2023-12-16T21:48:13.547 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:48:13 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:48:13.253+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:48:13.548 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:48:13 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:48:13.364+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:09.306583+0000 front 2023-12-16T21:39:09.306633+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:48:13.548 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:48:13 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:48:13.364+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:09.306551+0000 front 2023-12-16T21:39:09.306652+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:48:13.548 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:48:13 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:48:13.364+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:09.306605+0000 front 2023-12-16T21:39:09.306613+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:48:13.548 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:48:13 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:48:13.364+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:48:14.371 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:48:14 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:48:14.276+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:05.622397+0000 front 2023-12-16T21:39:05.622556+0000 (oldest deadline 2023-12-16T21:39:30.322546+0000) 2023-12-16T21:48:14.371 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:48:14 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:48:14.276+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:05.622506+0000 front 2023-12-16T21:39:05.622528+0000 (oldest deadline 2023-12-16T21:39:30.322546+0000) 2023-12-16T21:48:14.371 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:48:14 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:48:14.276+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:05.622479+0000 front 2023-12-16T21:39:05.622580+0000 (oldest deadline 2023-12-16T21:39:30.322546+0000) 2023-12-16T21:48:14.371 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:48:14 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:48:14.277+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:48:14.371 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:48:14 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:48:14.084+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:09.828911+0000 front 2023-12-16T21:39:09.828958+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:48:14.371 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:48:14 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:48:14.084+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:09.828868+0000 front 2023-12-16T21:39:09.828878+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:48:14.371 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:48:14 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:48:14.084+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:09.828947+0000 front 2023-12-16T21:39:09.828993+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:48:14.371 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:48:14 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:48:14.084+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:48:14.816 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:48:14 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:48:14.367+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:09.306583+0000 front 2023-12-16T21:39:09.306633+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:48:14.817 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:48:14 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:48:14.367+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:09.306551+0000 front 2023-12-16T21:39:09.306652+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:48:14.817 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:48:14 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:48:14.367+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:09.306605+0000 front 2023-12-16T21:39:09.306613+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:48:14.817 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:48:14 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:48:14.368+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:48:15.566 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:48:15 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:48:15.338+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:09.306583+0000 front 2023-12-16T21:39:09.306633+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:48:15.567 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:48:15 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:48:15.338+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:09.306551+0000 front 2023-12-16T21:39:09.306652+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:48:15.567 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:48:15 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:48:15.338+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:09.306605+0000 front 2023-12-16T21:39:09.306613+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:48:15.567 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:48:15 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:48:15.339+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:48:15.567 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:48:15 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:48:15.246+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:05.622397+0000 front 2023-12-16T21:39:05.622556+0000 (oldest deadline 2023-12-16T21:39:30.322546+0000) 2023-12-16T21:48:15.567 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:48:15 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:48:15.246+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:05.622506+0000 front 2023-12-16T21:39:05.622528+0000 (oldest deadline 2023-12-16T21:39:30.322546+0000) 2023-12-16T21:48:15.567 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:48:15 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:48:15.246+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:05.622479+0000 front 2023-12-16T21:39:05.622580+0000 (oldest deadline 2023-12-16T21:39:30.322546+0000) 2023-12-16T21:48:15.567 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:48:15 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:48:15.248+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:48:15.568 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:48:15 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:48:15.100+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:09.828911+0000 front 2023-12-16T21:39:09.828958+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:48:15.568 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:48:15 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:48:15.100+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:09.828868+0000 front 2023-12-16T21:39:09.828878+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:48:15.568 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:48:15 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:48:15.100+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:09.828947+0000 front 2023-12-16T21:39:09.828993+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:48:15.568 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:48:15 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:48:15.101+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:48:16.296 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:48:16 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:48:16.080+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:09.828911+0000 front 2023-12-16T21:39:09.828958+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:48:16.297 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:48:16 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:48:16.080+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:09.828868+0000 front 2023-12-16T21:39:09.828878+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:48:16.297 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:48:16 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:48:16.080+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:09.828947+0000 front 2023-12-16T21:39:09.828993+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:48:16.297 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:48:16 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:48:16.081+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:48:16.566 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Dec 16 21:48:16 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-mon-smithi045[80759]: 2023-12-16T21:48:16.386+0000 7f45a1f8f700 -1 mon.smithi045@0(probing) e2 get_health_metrics reporting 6291 slow ops, oldest is monmgrreport(gid 14214, 0 checks, 1 progress events) 2023-12-16T21:48:16.567 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:48:16 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:48:16.293+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:05.622397+0000 front 2023-12-16T21:39:05.622556+0000 (oldest deadline 2023-12-16T21:39:30.322546+0000) 2023-12-16T21:48:16.567 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:48:16 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:48:16.293+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:05.622506+0000 front 2023-12-16T21:39:05.622528+0000 (oldest deadline 2023-12-16T21:39:30.322546+0000) 2023-12-16T21:48:16.567 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:48:16 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:48:16.293+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:05.622479+0000 front 2023-12-16T21:39:05.622580+0000 (oldest deadline 2023-12-16T21:39:30.322546+0000) 2023-12-16T21:48:16.567 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:48:16 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:48:16.294+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:48:16.568 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:48:16 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:48:16.365+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:09.306583+0000 front 2023-12-16T21:39:09.306633+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:48:16.568 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:48:16 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:48:16.365+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:09.306551+0000 front 2023-12-16T21:39:09.306652+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:48:16.568 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:48:16 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:48:16.365+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:09.306605+0000 front 2023-12-16T21:39:09.306613+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:48:16.568 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:48:16 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:48:16.366+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:48:17.302 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:48:17 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:48:17.035+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:09.828911+0000 front 2023-12-16T21:39:09.828958+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:48:17.302 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:48:17 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:48:17.035+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:09.828868+0000 front 2023-12-16T21:39:09.828878+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:48:17.302 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:48:17 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:48:17.035+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:09.828947+0000 front 2023-12-16T21:39:09.828993+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:48:17.302 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:48:17 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:48:17.036+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:48:17.566 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:48:17 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:48:17.339+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:09.306583+0000 front 2023-12-16T21:39:09.306633+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:48:17.567 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:48:17 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:48:17.339+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:09.306551+0000 front 2023-12-16T21:39:09.306652+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:48:17.567 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:48:17 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:48:17.339+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:09.306605+0000 front 2023-12-16T21:39:09.306613+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:48:17.567 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:48:17 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:48:17.339+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:48:17.567 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:48:17 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:48:17.299+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:05.622397+0000 front 2023-12-16T21:39:05.622556+0000 (oldest deadline 2023-12-16T21:39:30.322546+0000) 2023-12-16T21:48:17.567 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:48:17 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:48:17.299+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:05.622506+0000 front 2023-12-16T21:39:05.622528+0000 (oldest deadline 2023-12-16T21:39:30.322546+0000) 2023-12-16T21:48:17.567 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:48:17 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:48:17.299+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:05.622479+0000 front 2023-12-16T21:39:05.622580+0000 (oldest deadline 2023-12-16T21:39:30.322546+0000) 2023-12-16T21:48:17.567 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:48:17 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:48:17.300+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:48:18.350 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:48:18 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:48:18.084+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:09.828911+0000 front 2023-12-16T21:39:09.828958+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:48:18.350 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:48:18 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:48:18.084+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:09.828868+0000 front 2023-12-16T21:39:09.828878+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:48:18.350 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:48:18 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:48:18.084+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:09.828947+0000 front 2023-12-16T21:39:09.828993+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:48:18.350 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:48:18 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:48:18.085+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:48:18.722 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:48:18 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:48:18.382+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:09.306583+0000 front 2023-12-16T21:39:09.306633+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:48:18.722 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:48:18 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:48:18.382+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:09.306551+0000 front 2023-12-16T21:39:09.306652+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:48:18.722 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:48:18 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:48:18.382+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:09.306605+0000 front 2023-12-16T21:39:09.306613+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:48:18.723 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:48:18 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:48:18.382+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:48:18.723 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:48:18 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:48:18.347+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:05.622397+0000 front 2023-12-16T21:39:05.622556+0000 (oldest deadline 2023-12-16T21:39:30.322546+0000) 2023-12-16T21:48:18.723 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:48:18 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:48:18.347+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:05.622506+0000 front 2023-12-16T21:39:05.622528+0000 (oldest deadline 2023-12-16T21:39:30.322546+0000) 2023-12-16T21:48:18.723 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:48:18 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:48:18.347+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:05.622479+0000 front 2023-12-16T21:39:05.622580+0000 (oldest deadline 2023-12-16T21:39:30.322546+0000) 2023-12-16T21:48:18.723 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:48:18 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:48:18.348+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:48:19.393 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:48:19 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:48:19.370+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:05.622397+0000 front 2023-12-16T21:39:05.622556+0000 (oldest deadline 2023-12-16T21:39:30.322546+0000) 2023-12-16T21:48:19.394 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:48:19 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:48:19.370+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:05.622506+0000 front 2023-12-16T21:39:05.622528+0000 (oldest deadline 2023-12-16T21:39:30.322546+0000) 2023-12-16T21:48:19.394 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:48:19 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:48:19.370+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:05.622479+0000 front 2023-12-16T21:39:05.622580+0000 (oldest deadline 2023-12-16T21:39:30.322546+0000) 2023-12-16T21:48:19.394 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:48:19 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:48:19.371+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:48:19.394 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:48:19 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:48:19.135+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:09.828911+0000 front 2023-12-16T21:39:09.828958+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:48:19.394 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:48:19 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:48:19.135+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:09.828868+0000 front 2023-12-16T21:39:09.828878+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:48:19.394 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:48:19 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:48:19.135+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:09.828947+0000 front 2023-12-16T21:39:09.828993+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:48:19.394 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:48:19 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:48:19.136+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:48:19.817 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:48:19 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:48:19.390+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:09.306583+0000 front 2023-12-16T21:39:09.306633+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:48:19.817 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:48:19 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:48:19.390+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:09.306551+0000 front 2023-12-16T21:39:09.306652+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:48:19.817 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:48:19 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:48:19.390+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:09.306605+0000 front 2023-12-16T21:39:09.306613+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:48:19.817 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:48:19 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:48:19.391+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:48:20.567 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:48:20 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:48:20.346+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:09.306583+0000 front 2023-12-16T21:39:09.306633+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:48:20.567 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:48:20 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:48:20.346+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:09.306551+0000 front 2023-12-16T21:39:09.306652+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:48:20.567 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:48:20 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:48:20.346+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:09.306605+0000 front 2023-12-16T21:39:09.306613+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:48:20.567 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:48:20 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:48:20.346+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:48:20.568 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:48:20 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:48:20.363+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:05.622397+0000 front 2023-12-16T21:39:05.622556+0000 (oldest deadline 2023-12-16T21:39:30.322546+0000) 2023-12-16T21:48:20.568 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:48:20 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:48:20.363+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:05.622506+0000 front 2023-12-16T21:39:05.622528+0000 (oldest deadline 2023-12-16T21:39:30.322546+0000) 2023-12-16T21:48:20.568 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:48:20 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:48:20.363+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:05.622479+0000 front 2023-12-16T21:39:05.622580+0000 (oldest deadline 2023-12-16T21:39:30.322546+0000) 2023-12-16T21:48:20.568 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:48:20 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:48:20.364+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:48:20.568 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:48:20 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:48:20.183+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:09.828911+0000 front 2023-12-16T21:39:09.828958+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:48:20.569 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:48:20 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:48:20.183+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:09.828868+0000 front 2023-12-16T21:39:09.828878+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:48:20.569 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:48:20 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:48:20.183+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:09.828947+0000 front 2023-12-16T21:39:09.828993+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:48:20.569 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:48:20 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:48:20.184+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:48:21.402 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Dec 16 21:48:21 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-mon-smithi045[80759]: 2023-12-16T21:48:21.387+0000 7f45a1f8f700 -1 mon.smithi045@0(probing) e2 get_health_metrics reporting 6389 slow ops, oldest is monmgrreport(gid 14214, 0 checks, 1 progress events) 2023-12-16T21:48:21.403 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:48:21 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:48:21.301+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:09.306583+0000 front 2023-12-16T21:39:09.306633+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:48:21.403 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:48:21 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:48:21.301+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:09.306551+0000 front 2023-12-16T21:39:09.306652+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:48:21.403 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:48:21 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:48:21.301+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:09.306605+0000 front 2023-12-16T21:39:09.306613+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:48:21.403 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:48:21 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:48:21.302+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:48:21.404 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:48:21 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:48:21.147+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:09.828911+0000 front 2023-12-16T21:39:09.828958+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:48:21.404 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:48:21 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:48:21.147+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:09.828868+0000 front 2023-12-16T21:39:09.828878+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:48:21.404 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:48:21 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:48:21.147+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:09.828947+0000 front 2023-12-16T21:39:09.828993+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:48:21.404 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:48:21 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:48:21.148+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:48:21.658 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:48:21 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:48:21.411+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:05.622397+0000 front 2023-12-16T21:39:05.622556+0000 (oldest deadline 2023-12-16T21:39:30.322546+0000) 2023-12-16T21:48:21.659 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:48:21 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:48:21.411+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:05.622506+0000 front 2023-12-16T21:39:05.622528+0000 (oldest deadline 2023-12-16T21:39:30.322546+0000) 2023-12-16T21:48:21.659 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:48:21 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:48:21.411+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:05.622479+0000 front 2023-12-16T21:39:05.622580+0000 (oldest deadline 2023-12-16T21:39:30.322546+0000) 2023-12-16T21:48:21.659 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:48:21 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:48:21.412+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:48:22.419 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:48:22 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:48:22.324+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:09.306583+0000 front 2023-12-16T21:39:09.306633+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:48:22.420 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:48:22 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:48:22.324+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:09.306551+0000 front 2023-12-16T21:39:09.306652+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:48:22.420 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:48:22 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:48:22.325+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:09.306605+0000 front 2023-12-16T21:39:09.306613+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:48:22.420 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:48:22 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:48:22.325+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:48:22.420 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:48:22 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:48:22.117+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:09.828911+0000 front 2023-12-16T21:39:09.828958+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:48:22.420 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:48:22 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:48:22.117+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:09.828868+0000 front 2023-12-16T21:39:09.828878+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:48:22.420 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:48:22 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:48:22.117+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:09.828947+0000 front 2023-12-16T21:39:09.828993+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:48:22.420 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:48:22 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:48:22.118+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:48:22.816 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:48:22 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:48:22.416+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:05.622397+0000 front 2023-12-16T21:39:05.622556+0000 (oldest deadline 2023-12-16T21:39:30.322546+0000) 2023-12-16T21:48:22.817 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:48:22 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:48:22.416+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:05.622506+0000 front 2023-12-16T21:39:05.622528+0000 (oldest deadline 2023-12-16T21:39:30.322546+0000) 2023-12-16T21:48:22.817 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:48:22 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:48:22.416+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:05.622479+0000 front 2023-12-16T21:39:05.622580+0000 (oldest deadline 2023-12-16T21:39:30.322546+0000) 2023-12-16T21:48:22.817 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:48:22 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:48:22.417+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:48:23.443 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:48:23 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:48:23.301+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:09.306583+0000 front 2023-12-16T21:39:09.306633+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:48:23.443 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:48:23 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:48:23.301+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:09.306551+0000 front 2023-12-16T21:39:09.306652+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:48:23.443 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:48:23 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:48:23.301+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:09.306605+0000 front 2023-12-16T21:39:09.306613+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:48:23.443 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:48:23 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:48:23.302+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:48:23.444 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:48:23 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:48:23.135+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:09.828911+0000 front 2023-12-16T21:39:09.828958+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:48:23.444 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:48:23 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:48:23.135+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:09.828868+0000 front 2023-12-16T21:39:09.828878+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:48:23.444 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:48:23 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:48:23.135+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:09.828947+0000 front 2023-12-16T21:39:09.828993+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:48:23.444 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:48:23 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:48:23.136+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:48:23.816 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:48:23 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:48:23.440+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:05.622397+0000 front 2023-12-16T21:39:05.622556+0000 (oldest deadline 2023-12-16T21:39:30.322546+0000) 2023-12-16T21:48:23.817 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:48:23 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:48:23.440+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:05.622506+0000 front 2023-12-16T21:39:05.622528+0000 (oldest deadline 2023-12-16T21:39:30.322546+0000) 2023-12-16T21:48:23.817 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:48:23 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:48:23.440+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:05.622479+0000 front 2023-12-16T21:39:05.622580+0000 (oldest deadline 2023-12-16T21:39:30.322546+0000) 2023-12-16T21:48:23.817 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:48:23 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:48:23.440+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:48:24.410 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:48:24 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:48:24.092+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:09.828911+0000 front 2023-12-16T21:39:09.828958+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:48:24.410 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:48:24 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:48:24.092+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:09.828868+0000 front 2023-12-16T21:39:09.828878+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:48:24.410 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:48:24 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:48:24.092+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:09.828947+0000 front 2023-12-16T21:39:09.828993+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:48:24.410 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:48:24 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:48:24.092+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:48:24.410 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:48:24 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:48:24.310+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:09.306583+0000 front 2023-12-16T21:39:09.306633+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:48:24.410 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:48:24 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:48:24.310+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:09.306551+0000 front 2023-12-16T21:39:09.306652+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:48:24.410 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:48:24 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:48:24.310+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:09.306605+0000 front 2023-12-16T21:39:09.306613+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:48:24.410 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:48:24 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:48:24.311+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:48:24.816 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:48:24 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:48:24.406+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:05.622397+0000 front 2023-12-16T21:39:05.622556+0000 (oldest deadline 2023-12-16T21:39:30.322546+0000) 2023-12-16T21:48:24.817 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:48:24 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:48:24.406+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:05.622506+0000 front 2023-12-16T21:39:05.622528+0000 (oldest deadline 2023-12-16T21:39:30.322546+0000) 2023-12-16T21:48:24.817 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:48:24 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:48:24.406+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:05.622479+0000 front 2023-12-16T21:39:05.622580+0000 (oldest deadline 2023-12-16T21:39:30.322546+0000) 2023-12-16T21:48:24.817 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:48:24 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:48:24.407+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:48:25.566 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:48:25 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:48:25.319+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:09.306583+0000 front 2023-12-16T21:39:09.306633+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:48:25.567 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:48:25 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:48:25.319+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:09.306551+0000 front 2023-12-16T21:39:09.306652+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:48:25.567 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:48:25 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:48:25.319+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:09.306605+0000 front 2023-12-16T21:39:09.306613+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:48:25.567 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:48:25 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:48:25.320+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:48:25.567 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:48:25 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:48:25.136+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:09.828911+0000 front 2023-12-16T21:39:09.828958+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:48:25.567 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:48:25 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:48:25.136+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:09.828868+0000 front 2023-12-16T21:39:09.828878+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:48:25.567 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:48:25 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:48:25.136+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:09.828947+0000 front 2023-12-16T21:39:09.828993+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:48:25.567 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:48:25 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:48:25.137+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:48:25.568 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:48:25 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:48:25.375+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:05.622397+0000 front 2023-12-16T21:39:05.622556+0000 (oldest deadline 2023-12-16T21:39:30.322546+0000) 2023-12-16T21:48:25.568 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:48:25 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:48:25.375+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:05.622506+0000 front 2023-12-16T21:39:05.622528+0000 (oldest deadline 2023-12-16T21:39:30.322546+0000) 2023-12-16T21:48:25.568 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:48:25 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:48:25.375+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:05.622479+0000 front 2023-12-16T21:39:05.622580+0000 (oldest deadline 2023-12-16T21:39:30.322546+0000) 2023-12-16T21:48:25.568 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:48:25 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:48:25.376+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:48:26.391 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:48:26 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:48:26.089+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:09.828911+0000 front 2023-12-16T21:39:09.828958+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:48:26.392 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:48:26 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:48:26.089+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:09.828868+0000 front 2023-12-16T21:39:09.828878+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:48:26.392 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:48:26 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:48:26.089+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:09.828947+0000 front 2023-12-16T21:39:09.828993+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:48:26.392 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:48:26 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:48:26.090+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:48:26.392 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:48:26 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:48:26.297+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:09.306583+0000 front 2023-12-16T21:39:09.306633+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:48:26.392 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:48:26 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:48:26.297+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:09.306551+0000 front 2023-12-16T21:39:09.306652+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:48:26.392 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:48:26 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:48:26.297+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:09.306605+0000 front 2023-12-16T21:39:09.306613+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:48:26.392 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:48:26 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:48:26.298+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:48:26.392 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:48:26 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:48:26.332+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:05.622397+0000 front 2023-12-16T21:39:05.622556+0000 (oldest deadline 2023-12-16T21:39:30.322546+0000) 2023-12-16T21:48:26.393 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:48:26 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:48:26.332+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:05.622506+0000 front 2023-12-16T21:39:05.622528+0000 (oldest deadline 2023-12-16T21:39:30.322546+0000) 2023-12-16T21:48:26.393 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:48:26 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:48:26.332+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:05.622479+0000 front 2023-12-16T21:39:05.622580+0000 (oldest deadline 2023-12-16T21:39:30.322546+0000) 2023-12-16T21:48:26.393 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:48:26 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:48:26.334+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:48:26.816 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Dec 16 21:48:26 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-mon-smithi045[80759]: 2023-12-16T21:48:26.388+0000 7f45a1f8f700 -1 mon.smithi045@0(probing) e2 get_health_metrics reporting 6483 slow ops, oldest is monmgrreport(gid 14214, 0 checks, 1 progress events) 2023-12-16T21:48:27.316 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:48:27 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:48:27.042+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:09.828911+0000 front 2023-12-16T21:39:09.828958+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:48:27.316 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:48:27 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:48:27.042+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:09.828868+0000 front 2023-12-16T21:39:09.828878+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:48:27.317 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:48:27 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:48:27.042+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:09.828947+0000 front 2023-12-16T21:39:09.828993+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:48:27.317 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:48:27 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:48:27.043+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:48:27.317 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:48:27 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:48:27.291+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:09.306583+0000 front 2023-12-16T21:39:09.306633+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:48:27.317 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:48:27 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:48:27.291+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:09.306551+0000 front 2023-12-16T21:39:09.306652+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:48:27.317 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:48:27 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:48:27.291+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:09.306605+0000 front 2023-12-16T21:39:09.306613+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:48:27.317 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:48:27 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:48:27.292+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:48:27.816 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:48:27 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:48:27.365+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:05.622397+0000 front 2023-12-16T21:39:05.622556+0000 (oldest deadline 2023-12-16T21:39:30.322546+0000) 2023-12-16T21:48:27.817 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:48:27 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:48:27.365+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:05.622506+0000 front 2023-12-16T21:39:05.622528+0000 (oldest deadline 2023-12-16T21:39:30.322546+0000) 2023-12-16T21:48:27.817 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:48:27 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:48:27.365+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:05.622479+0000 front 2023-12-16T21:39:05.622580+0000 (oldest deadline 2023-12-16T21:39:30.322546+0000) 2023-12-16T21:48:27.817 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:48:27 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:48:27.366+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:48:28.302 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:48:28 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:48:28.038+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:09.828911+0000 front 2023-12-16T21:39:09.828958+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:48:28.302 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:48:28 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:48:28.038+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:09.828868+0000 front 2023-12-16T21:39:09.828878+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:48:28.302 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:48:28 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:48:28.038+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:09.828947+0000 front 2023-12-16T21:39:09.828993+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:48:28.302 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:48:28 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:48:28.039+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:48:28.566 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:48:28 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:48:28.409+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:05.622397+0000 front 2023-12-16T21:39:05.622556+0000 (oldest deadline 2023-12-16T21:39:30.322546+0000) 2023-12-16T21:48:28.567 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:48:28 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:48:28.409+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:05.622506+0000 front 2023-12-16T21:39:05.622528+0000 (oldest deadline 2023-12-16T21:39:30.322546+0000) 2023-12-16T21:48:28.567 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:48:28 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:48:28.409+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:05.622479+0000 front 2023-12-16T21:39:05.622580+0000 (oldest deadline 2023-12-16T21:39:30.322546+0000) 2023-12-16T21:48:28.567 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:48:28 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:48:28.410+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:48:28.567 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:48:28 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:48:28.299+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:09.306583+0000 front 2023-12-16T21:39:09.306633+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:48:28.567 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:48:28 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:48:28.299+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:09.306551+0000 front 2023-12-16T21:39:09.306652+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:48:28.567 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:48:28 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:48:28.299+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:09.306605+0000 front 2023-12-16T21:39:09.306613+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:48:28.567 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:48:28 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:48:28.300+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:48:29.345 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:48:29 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:48:29.076+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:09.828911+0000 front 2023-12-16T21:39:09.828958+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:48:29.345 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:48:29 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:48:29.076+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:09.828868+0000 front 2023-12-16T21:39:09.828878+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:48:29.345 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:48:29 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:48:29.076+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:09.828947+0000 front 2023-12-16T21:39:09.828993+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:48:29.345 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:48:29 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:48:29.077+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:48:29.816 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:48:29 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:48:29.341+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:09.306583+0000 front 2023-12-16T21:39:09.306633+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:48:29.817 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:48:29 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:48:29.342+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:09.306551+0000 front 2023-12-16T21:39:09.306652+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:48:29.817 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:48:29 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:48:29.342+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:09.306605+0000 front 2023-12-16T21:39:09.306613+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:48:29.817 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:48:29 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:48:29.343+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:48:29.817 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:48:29 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:48:29.430+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:05.622397+0000 front 2023-12-16T21:39:05.622556+0000 (oldest deadline 2023-12-16T21:39:30.322546+0000) 2023-12-16T21:48:29.817 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:48:29 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:48:29.430+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:05.622506+0000 front 2023-12-16T21:39:05.622528+0000 (oldest deadline 2023-12-16T21:39:30.322546+0000) 2023-12-16T21:48:29.817 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:48:29 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:48:29.430+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:05.622479+0000 front 2023-12-16T21:39:05.622580+0000 (oldest deadline 2023-12-16T21:39:30.322546+0000) 2023-12-16T21:48:29.817 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:48:29 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:48:29.431+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:48:30.387 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:48:30 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:48:30.127+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:09.828911+0000 front 2023-12-16T21:39:09.828958+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:48:30.388 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:48:30 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:48:30.127+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:09.828868+0000 front 2023-12-16T21:39:09.828878+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:48:30.388 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:48:30 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:48:30.127+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:09.828947+0000 front 2023-12-16T21:39:09.828993+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:48:30.388 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:48:30 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:48:30.128+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:48:30.816 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:48:30 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:48:30.384+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:09.306583+0000 front 2023-12-16T21:39:09.306633+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:48:30.817 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:48:30 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:48:30.384+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:09.306551+0000 front 2023-12-16T21:39:09.306652+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:48:30.817 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:48:30 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:48:30.384+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:09.306605+0000 front 2023-12-16T21:39:09.306613+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:48:30.817 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:48:30 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:48:30.385+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:48:30.817 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:48:30 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:48:30.409+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:05.622397+0000 front 2023-12-16T21:39:05.622556+0000 (oldest deadline 2023-12-16T21:39:30.322546+0000) 2023-12-16T21:48:30.817 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:48:30 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:48:30.409+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:05.622506+0000 front 2023-12-16T21:39:05.622528+0000 (oldest deadline 2023-12-16T21:39:30.322546+0000) 2023-12-16T21:48:30.817 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:48:30 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:48:30.409+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:05.622479+0000 front 2023-12-16T21:39:05.622580+0000 (oldest deadline 2023-12-16T21:39:30.322546+0000) 2023-12-16T21:48:30.817 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:48:30 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:48:30.411+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:48:31.432 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Dec 16 21:48:31 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-mon-smithi045[80759]: 2023-12-16T21:48:31.389+0000 7f45a1f8f700 -1 mon.smithi045@0(probing) e2 get_health_metrics reporting 6581 slow ops, oldest is monmgrreport(gid 14214, 0 checks, 1 progress events) 2023-12-16T21:48:31.433 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:48:31 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:48:31.175+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:09.828911+0000 front 2023-12-16T21:39:09.828958+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:48:31.433 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:48:31 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:48:31.176+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:09.828868+0000 front 2023-12-16T21:39:09.828878+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:48:31.433 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:48:31 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:48:31.176+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:09.828947+0000 front 2023-12-16T21:39:09.828993+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:48:31.433 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:48:31 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:48:31.177+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:48:31.433 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:48:31 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:48:31.396+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:05.622397+0000 front 2023-12-16T21:39:05.622556+0000 (oldest deadline 2023-12-16T21:39:30.322546+0000) 2023-12-16T21:48:31.433 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:48:31 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:48:31.396+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:05.622506+0000 front 2023-12-16T21:39:05.622528+0000 (oldest deadline 2023-12-16T21:39:30.322546+0000) 2023-12-16T21:48:31.434 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:48:31 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:48:31.396+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:05.622479+0000 front 2023-12-16T21:39:05.622580+0000 (oldest deadline 2023-12-16T21:39:30.322546+0000) 2023-12-16T21:48:31.434 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:48:31 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:48:31.397+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:48:31.816 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:48:31 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:48:31.429+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:09.306583+0000 front 2023-12-16T21:39:09.306633+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:48:31.817 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:48:31 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:48:31.429+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:09.306551+0000 front 2023-12-16T21:39:09.306652+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:48:31.817 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:48:31 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:48:31.429+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:09.306605+0000 front 2023-12-16T21:39:09.306613+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:48:31.817 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:48:31 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:48:31.431+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:48:32.473 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:48:32 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:48:32.378+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:05.622397+0000 front 2023-12-16T21:39:05.622556+0000 (oldest deadline 2023-12-16T21:39:30.322546+0000) 2023-12-16T21:48:32.473 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:48:32 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:48:32.378+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:05.622506+0000 front 2023-12-16T21:39:05.622528+0000 (oldest deadline 2023-12-16T21:39:30.322546+0000) 2023-12-16T21:48:32.473 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:48:32 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:48:32.378+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:05.622479+0000 front 2023-12-16T21:39:05.622580+0000 (oldest deadline 2023-12-16T21:39:30.322546+0000) 2023-12-16T21:48:32.474 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:48:32 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:48:32.379+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:48:32.474 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:48:32 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:48:32.154+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:09.828911+0000 front 2023-12-16T21:39:09.828958+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:48:32.474 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:48:32 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:48:32.154+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:09.828868+0000 front 2023-12-16T21:39:09.828878+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:48:32.474 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:48:32 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:48:32.154+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:09.828947+0000 front 2023-12-16T21:39:09.828993+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:48:32.474 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:48:32 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:48:32.155+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:48:32.816 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:48:32 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:48:32.470+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:09.306583+0000 front 2023-12-16T21:39:09.306633+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:48:32.817 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:48:32 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:48:32.470+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:09.306551+0000 front 2023-12-16T21:39:09.306652+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:48:32.817 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:48:32 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:48:32.470+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:09.306605+0000 front 2023-12-16T21:39:09.306613+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:48:32.817 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:48:32 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:48:32.471+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:48:33.420 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:48:33 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:48:33.110+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:09.828911+0000 front 2023-12-16T21:39:09.828958+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:48:33.420 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:48:33 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:48:33.111+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:09.828868+0000 front 2023-12-16T21:39:09.828878+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:48:33.421 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:48:33 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:48:33.111+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:09.828947+0000 front 2023-12-16T21:39:09.828993+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:48:33.421 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:48:33 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:48:33.111+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:48:33.816 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:48:33 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:48:33.423+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:09.306583+0000 front 2023-12-16T21:39:09.306633+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:48:33.817 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:48:33 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:48:33.423+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:09.306551+0000 front 2023-12-16T21:39:09.306652+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:48:33.817 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:48:33 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:48:33.423+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:09.306605+0000 front 2023-12-16T21:39:09.306613+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:48:33.817 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:48:33 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:48:33.423+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:48:33.817 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:48:33 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:48:33.417+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:05.622397+0000 front 2023-12-16T21:39:05.622556+0000 (oldest deadline 2023-12-16T21:39:30.322546+0000) 2023-12-16T21:48:33.817 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:48:33 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:48:33.417+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:05.622506+0000 front 2023-12-16T21:39:05.622528+0000 (oldest deadline 2023-12-16T21:39:30.322546+0000) 2023-12-16T21:48:33.817 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:48:33 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:48:33.417+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:05.622479+0000 front 2023-12-16T21:39:05.622580+0000 (oldest deadline 2023-12-16T21:39:30.322546+0000) 2023-12-16T21:48:33.817 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:48:33 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:48:33.418+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:48:34.435 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:48:34 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:48:34.390+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:05.622397+0000 front 2023-12-16T21:39:05.622556+0000 (oldest deadline 2023-12-16T21:39:30.322546+0000) 2023-12-16T21:48:34.435 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:48:34 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:48:34.390+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:05.622506+0000 front 2023-12-16T21:39:05.622528+0000 (oldest deadline 2023-12-16T21:39:30.322546+0000) 2023-12-16T21:48:34.435 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:48:34 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:48:34.390+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:05.622479+0000 front 2023-12-16T21:39:05.622580+0000 (oldest deadline 2023-12-16T21:39:30.322546+0000) 2023-12-16T21:48:34.435 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:48:34 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:48:34.391+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:48:34.435 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:48:34 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:48:34.161+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:09.828911+0000 front 2023-12-16T21:39:09.828958+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:48:34.435 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:48:34 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:48:34.161+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:09.828868+0000 front 2023-12-16T21:39:09.828878+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:48:34.435 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:48:34 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:48:34.161+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:09.828947+0000 front 2023-12-16T21:39:09.828993+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:48:34.435 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:48:34 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:48:34.161+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:48:34.816 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:48:34 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:48:34.431+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:09.306583+0000 front 2023-12-16T21:39:09.306633+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:48:34.817 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:48:34 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:48:34.431+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:09.306551+0000 front 2023-12-16T21:39:09.306652+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:48:34.817 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:48:34 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:48:34.431+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:09.306605+0000 front 2023-12-16T21:39:09.306613+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:48:34.817 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:48:34 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:48:34.433+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:48:35.443 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:48:35 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:48:35.143+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:09.828911+0000 front 2023-12-16T21:39:09.828958+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:48:35.443 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:48:35 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:48:35.143+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:09.828868+0000 front 2023-12-16T21:39:09.828878+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:48:35.444 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:48:35 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:48:35.143+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:09.828947+0000 front 2023-12-16T21:39:09.828993+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:48:35.444 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:48:35 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:48:35.145+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:48:35.816 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:48:35 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:48:35.440+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:05.622397+0000 front 2023-12-16T21:39:05.622556+0000 (oldest deadline 2023-12-16T21:39:30.322546+0000) 2023-12-16T21:48:35.817 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:48:35 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:48:35.440+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:05.622506+0000 front 2023-12-16T21:39:05.622528+0000 (oldest deadline 2023-12-16T21:39:30.322546+0000) 2023-12-16T21:48:35.817 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:48:35 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:48:35.440+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:05.622479+0000 front 2023-12-16T21:39:05.622580+0000 (oldest deadline 2023-12-16T21:39:30.322546+0000) 2023-12-16T21:48:35.817 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:48:35 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:48:35.441+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:48:35.817 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:48:35 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:48:35.449+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:09.306583+0000 front 2023-12-16T21:39:09.306633+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:48:35.817 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:48:35 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:48:35.449+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:09.306551+0000 front 2023-12-16T21:39:09.306652+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:48:35.817 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:48:35 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:48:35.449+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:09.306605+0000 front 2023-12-16T21:39:09.306613+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:48:35.817 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:48:35 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:48:35.450+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:48:36.492 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Dec 16 21:48:36 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-mon-smithi045[80759]: 2023-12-16T21:48:36.390+0000 7f45a1f8f700 -1 mon.smithi045@0(probing) e2 get_health_metrics reporting 6675 slow ops, oldest is monmgrreport(gid 14214, 0 checks, 1 progress events) 2023-12-16T21:48:36.492 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:48:36 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:48:36.165+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:09.828911+0000 front 2023-12-16T21:39:09.828958+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:48:36.492 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:48:36 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:48:36.165+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:09.828868+0000 front 2023-12-16T21:39:09.828878+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:48:36.492 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:48:36 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:48:36.165+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:09.828947+0000 front 2023-12-16T21:39:09.828993+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:48:36.492 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:48:36 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:48:36.166+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:48:36.816 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:48:36 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:48:36.492+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:09.306583+0000 front 2023-12-16T21:39:09.306633+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:48:36.817 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:48:36 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:48:36.492+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:09.306551+0000 front 2023-12-16T21:39:09.306652+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:48:36.817 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:48:36 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:48:36.492+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:09.306605+0000 front 2023-12-16T21:39:09.306613+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:48:36.817 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:48:36 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:48:36.492+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:48:36.817 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:48:36 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:48:36.489+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:05.622397+0000 front 2023-12-16T21:39:05.622556+0000 (oldest deadline 2023-12-16T21:39:30.322546+0000) 2023-12-16T21:48:36.817 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:48:36 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:48:36.489+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:05.622506+0000 front 2023-12-16T21:39:05.622528+0000 (oldest deadline 2023-12-16T21:39:30.322546+0000) 2023-12-16T21:48:36.817 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:48:36 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:48:36.489+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:05.622479+0000 front 2023-12-16T21:39:05.622580+0000 (oldest deadline 2023-12-16T21:39:30.322546+0000) 2023-12-16T21:48:36.817 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:48:36 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:48:36.490+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:48:37.455 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:48:37 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:48:37.202+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:09.828911+0000 front 2023-12-16T21:39:09.828958+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:48:37.455 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:48:37 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:48:37.202+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:09.828868+0000 front 2023-12-16T21:39:09.828878+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:48:37.456 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:48:37 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:48:37.202+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:09.828947+0000 front 2023-12-16T21:39:09.828993+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:48:37.456 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:48:37 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:48:37.202+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:48:37.456 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:48:37 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:48:37.447+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:09.306583+0000 front 2023-12-16T21:39:09.306633+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:48:37.456 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:48:37 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:48:37.447+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:09.306551+0000 front 2023-12-16T21:39:09.306652+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:48:37.456 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:48:37 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:48:37.447+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:09.306605+0000 front 2023-12-16T21:39:09.306613+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:48:37.456 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:48:37 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:48:37.448+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:48:37.816 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:48:37 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:48:37.452+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:05.622397+0000 front 2023-12-16T21:39:05.622556+0000 (oldest deadline 2023-12-16T21:39:30.322546+0000) 2023-12-16T21:48:37.817 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:48:37 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:48:37.452+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:05.622506+0000 front 2023-12-16T21:39:05.622528+0000 (oldest deadline 2023-12-16T21:39:30.322546+0000) 2023-12-16T21:48:37.817 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:48:37 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:48:37.452+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:05.622479+0000 front 2023-12-16T21:39:05.622580+0000 (oldest deadline 2023-12-16T21:39:30.322546+0000) 2023-12-16T21:48:37.817 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:48:37 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:48:37.453+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:48:38.567 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:48:38 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:48:38.469+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:05.622397+0000 front 2023-12-16T21:39:05.622556+0000 (oldest deadline 2023-12-16T21:39:30.322546+0000) 2023-12-16T21:48:38.567 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:48:38 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:48:38.469+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:05.622506+0000 front 2023-12-16T21:39:05.622528+0000 (oldest deadline 2023-12-16T21:39:30.322546+0000) 2023-12-16T21:48:38.567 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:48:38 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:48:38.469+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:05.622479+0000 front 2023-12-16T21:39:05.622580+0000 (oldest deadline 2023-12-16T21:39:30.322546+0000) 2023-12-16T21:48:38.567 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:48:38 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:48:38.470+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:48:38.568 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:48:38 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:48:38.229+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:09.828911+0000 front 2023-12-16T21:39:09.828958+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:48:38.568 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:48:38 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:48:38.229+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:09.828868+0000 front 2023-12-16T21:39:09.828878+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:48:38.568 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:48:38 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:48:38.229+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:09.828947+0000 front 2023-12-16T21:39:09.828993+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:48:38.568 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:48:38 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:48:38.230+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:48:38.568 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:48:38 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:48:38.467+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:09.306583+0000 front 2023-12-16T21:39:09.306633+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:48:38.568 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:48:38 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:48:38.467+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:09.306551+0000 front 2023-12-16T21:39:09.306652+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:48:38.568 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:48:38 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:48:38.467+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:09.306605+0000 front 2023-12-16T21:39:09.306613+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:48:38.568 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:48:38 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:48:38.468+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:48:39.567 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:48:39 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:48:39.476+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:05.622397+0000 front 2023-12-16T21:39:05.622556+0000 (oldest deadline 2023-12-16T21:39:30.322546+0000) 2023-12-16T21:48:39.567 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:48:39 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:48:39.476+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:05.622506+0000 front 2023-12-16T21:39:05.622528+0000 (oldest deadline 2023-12-16T21:39:30.322546+0000) 2023-12-16T21:48:39.567 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:48:39 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:48:39.476+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:05.622479+0000 front 2023-12-16T21:39:05.622580+0000 (oldest deadline 2023-12-16T21:39:30.322546+0000) 2023-12-16T21:48:39.567 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:48:39 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:48:39.477+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:48:39.567 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:48:39 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:48:39.453+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:09.306583+0000 front 2023-12-16T21:39:09.306633+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:48:39.567 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:48:39 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:48:39.453+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:09.306551+0000 front 2023-12-16T21:39:09.306652+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:48:39.567 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:48:39 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:48:39.453+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:09.306605+0000 front 2023-12-16T21:39:09.306613+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:48:39.567 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:48:39 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:48:39.454+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:48:39.568 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:48:39 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:48:39.274+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:09.828911+0000 front 2023-12-16T21:39:09.828958+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:48:39.568 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:48:39 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:48:39.274+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:09.828868+0000 front 2023-12-16T21:39:09.828878+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:48:39.568 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:48:39 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:48:39.274+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:09.828947+0000 front 2023-12-16T21:39:09.828993+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:48:39.568 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:48:39 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:48:39.275+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:48:40.566 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:48:40 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:48:40.489+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:09.306583+0000 front 2023-12-16T21:39:09.306633+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:48:40.567 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:48:40 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:48:40.489+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:09.306551+0000 front 2023-12-16T21:39:09.306652+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:48:40.567 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:48:40 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:48:40.489+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:09.306605+0000 front 2023-12-16T21:39:09.306613+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:48:40.567 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:48:40 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:48:40.491+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:48:40.567 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:48:40 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:48:40.310+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:09.828911+0000 front 2023-12-16T21:39:09.828958+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:48:40.567 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:48:40 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:48:40.310+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:09.828868+0000 front 2023-12-16T21:39:09.828878+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:48:40.567 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:48:40 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:48:40.310+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:09.828947+0000 front 2023-12-16T21:39:09.828993+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:48:40.567 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:48:40 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:48:40.311+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:48:40.567 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:48:40 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:48:40.503+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:05.622397+0000 front 2023-12-16T21:39:05.622556+0000 (oldest deadline 2023-12-16T21:39:30.322546+0000) 2023-12-16T21:48:40.567 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:48:40 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:48:40.503+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:05.622506+0000 front 2023-12-16T21:39:05.622528+0000 (oldest deadline 2023-12-16T21:39:30.322546+0000) 2023-12-16T21:48:40.568 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:48:40 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:48:40.503+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:05.622479+0000 front 2023-12-16T21:39:05.622580+0000 (oldest deadline 2023-12-16T21:39:30.322546+0000) 2023-12-16T21:48:40.568 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:48:40 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:48:40.503+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:48:41.659 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:48:41 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:48:41.339+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:09.828911+0000 front 2023-12-16T21:39:09.828958+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:48:41.660 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:48:41 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:48:41.339+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:09.828868+0000 front 2023-12-16T21:39:09.828878+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:48:41.660 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:48:41 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:48:41.339+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:09.828947+0000 front 2023-12-16T21:39:09.828993+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:48:41.660 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:48:41 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:48:41.340+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:48:41.660 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:48:41 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:48:41.494+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:09.306583+0000 front 2023-12-16T21:39:09.306633+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:48:41.660 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:48:41 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:48:41.494+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:09.306551+0000 front 2023-12-16T21:39:09.306652+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:48:41.660 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:48:41 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:48:41.494+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:09.306605+0000 front 2023-12-16T21:39:09.306613+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:48:41.660 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:48:41 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:48:41.495+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:48:41.661 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Dec 16 21:48:41 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-mon-smithi045[80759]: 2023-12-16T21:48:41.391+0000 7f45a1f8f700 -1 mon.smithi045@0(probing) e2 get_health_metrics reporting 6773 slow ops, oldest is monmgrreport(gid 14214, 0 checks, 1 progress events) 2023-12-16T21:48:41.661 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:48:41 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:48:41.528+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:05.622397+0000 front 2023-12-16T21:39:05.622556+0000 (oldest deadline 2023-12-16T21:39:30.322546+0000) 2023-12-16T21:48:41.661 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:48:41 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:48:41.528+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:05.622506+0000 front 2023-12-16T21:39:05.622528+0000 (oldest deadline 2023-12-16T21:39:30.322546+0000) 2023-12-16T21:48:41.661 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:48:41 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:48:41.528+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:05.622479+0000 front 2023-12-16T21:39:05.622580+0000 (oldest deadline 2023-12-16T21:39:30.322546+0000) 2023-12-16T21:48:41.661 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:48:41 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:48:41.529+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:48:42.817 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:48:42 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:48:42.327+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:09.828911+0000 front 2023-12-16T21:39:09.828958+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:48:42.817 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:48:42 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:48:42.327+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:09.828868+0000 front 2023-12-16T21:39:09.828878+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:48:42.817 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:48:42 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:48:42.327+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:09.828947+0000 front 2023-12-16T21:39:09.828993+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:48:42.817 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:48:42 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:48:42.328+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:48:42.818 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:48:42 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:48:42.490+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:05.622397+0000 front 2023-12-16T21:39:05.622556+0000 (oldest deadline 2023-12-16T21:39:30.322546+0000) 2023-12-16T21:48:42.818 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:48:42 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:48:42.490+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:05.622506+0000 front 2023-12-16T21:39:05.622528+0000 (oldest deadline 2023-12-16T21:39:30.322546+0000) 2023-12-16T21:48:42.818 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:48:42 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:48:42.490+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:05.622479+0000 front 2023-12-16T21:39:05.622580+0000 (oldest deadline 2023-12-16T21:39:30.322546+0000) 2023-12-16T21:48:42.818 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:48:42 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:48:42.491+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:48:42.818 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:48:42 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:48:42.520+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:09.306583+0000 front 2023-12-16T21:39:09.306633+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:48:42.818 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:48:42 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:48:42.520+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:09.306551+0000 front 2023-12-16T21:39:09.306652+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:48:42.818 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:48:42 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:48:42.520+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:09.306605+0000 front 2023-12-16T21:39:09.306613+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:48:42.818 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:48:42 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:48:42.521+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:48:43.665 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:48:43 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:48:43.524+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:05.622397+0000 front 2023-12-16T21:39:05.622556+0000 (oldest deadline 2023-12-16T21:39:30.322546+0000) 2023-12-16T21:48:43.666 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:48:43 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:48:43.524+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:05.622506+0000 front 2023-12-16T21:39:05.622528+0000 (oldest deadline 2023-12-16T21:39:30.322546+0000) 2023-12-16T21:48:43.666 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:48:43 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:48:43.524+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:05.622479+0000 front 2023-12-16T21:39:05.622580+0000 (oldest deadline 2023-12-16T21:39:30.322546+0000) 2023-12-16T21:48:43.666 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:48:43 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:48:43.524+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:48:43.666 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:48:43 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:48:43.317+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:09.828911+0000 front 2023-12-16T21:39:09.828958+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:48:43.666 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:48:43 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:48:43.317+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:09.828868+0000 front 2023-12-16T21:39:09.828878+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:48:43.666 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:48:43 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:48:43.317+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:09.828947+0000 front 2023-12-16T21:39:09.828993+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:48:43.666 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:48:43 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:48:43.318+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:48:43.666 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:48:43 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:48:43.484+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:09.306583+0000 front 2023-12-16T21:39:09.306633+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:48:43.667 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:48:43 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:48:43.484+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:09.306551+0000 front 2023-12-16T21:39:09.306652+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:48:43.667 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:48:43 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:48:43.484+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:09.306605+0000 front 2023-12-16T21:39:09.306613+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:48:43.667 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:48:43 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:48:43.485+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:48:44.816 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:48:44 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:48:44.561+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:05.622397+0000 front 2023-12-16T21:39:05.622556+0000 (oldest deadline 2023-12-16T21:39:30.322546+0000) 2023-12-16T21:48:44.816 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:48:44 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:48:44.561+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:05.622506+0000 front 2023-12-16T21:39:05.622528+0000 (oldest deadline 2023-12-16T21:39:30.322546+0000) 2023-12-16T21:48:44.817 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:48:44 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:48:44.561+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:05.622479+0000 front 2023-12-16T21:39:05.622580+0000 (oldest deadline 2023-12-16T21:39:30.322546+0000) 2023-12-16T21:48:44.817 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:48:44 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:48:44.562+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:48:44.817 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:48:44 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:48:44.332+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:09.828911+0000 front 2023-12-16T21:39:09.828958+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:48:44.817 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:48:44 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:48:44.332+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:09.828868+0000 front 2023-12-16T21:39:09.828878+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:48:44.817 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:48:44 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:48:44.332+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:09.828947+0000 front 2023-12-16T21:39:09.828993+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:48:44.817 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:48:44 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:48:44.332+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:48:44.817 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:48:44 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:48:44.444+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:09.306583+0000 front 2023-12-16T21:39:09.306633+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:48:44.818 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:48:44 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:48:44.444+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:09.306551+0000 front 2023-12-16T21:39:09.306652+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:48:44.818 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:48:44 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:48:44.444+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:09.306605+0000 front 2023-12-16T21:39:09.306613+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:48:44.818 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:48:44 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:48:44.445+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:48:45.817 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:48:45 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:48:45.537+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:05.622397+0000 front 2023-12-16T21:39:05.622556+0000 (oldest deadline 2023-12-16T21:39:30.322546+0000) 2023-12-16T21:48:45.817 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:48:45 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:48:45.537+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:05.622506+0000 front 2023-12-16T21:39:05.622528+0000 (oldest deadline 2023-12-16T21:39:30.322546+0000) 2023-12-16T21:48:45.817 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:48:45 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:48:45.537+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:05.622479+0000 front 2023-12-16T21:39:05.622580+0000 (oldest deadline 2023-12-16T21:39:30.322546+0000) 2023-12-16T21:48:45.817 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:48:45 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:48:45.537+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:48:45.817 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:48:45 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:48:45.472+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:09.306583+0000 front 2023-12-16T21:39:09.306633+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:48:45.817 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:48:45 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:48:45.472+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:09.306551+0000 front 2023-12-16T21:39:09.306652+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:48:45.818 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:48:45 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:48:45.472+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:09.306605+0000 front 2023-12-16T21:39:09.306613+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:48:45.818 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:48:45 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:48:45.472+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:48:45.818 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:48:45 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:48:45.361+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:09.828911+0000 front 2023-12-16T21:39:09.828958+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:48:45.818 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:48:45 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:48:45.361+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:09.828868+0000 front 2023-12-16T21:39:09.828878+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:48:45.818 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:48:45 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:48:45.361+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:09.828947+0000 front 2023-12-16T21:39:09.828993+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:48:45.818 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:48:45 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:48:45.361+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:48:46.817 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Dec 16 21:48:46 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-mon-smithi045[80759]: 2023-12-16T21:48:46.391+0000 7f45a1f8f700 -1 mon.smithi045@0(probing) e2 get_health_metrics reporting 6867 slow ops, oldest is monmgrreport(gid 14214, 0 checks, 1 progress events) 2023-12-16T21:48:46.817 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:48:46 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:48:46.581+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:05.622397+0000 front 2023-12-16T21:39:05.622556+0000 (oldest deadline 2023-12-16T21:39:30.322546+0000) 2023-12-16T21:48:46.817 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:48:46 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:48:46.581+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:05.622506+0000 front 2023-12-16T21:39:05.622528+0000 (oldest deadline 2023-12-16T21:39:30.322546+0000) 2023-12-16T21:48:46.817 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:48:46 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:48:46.581+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:05.622479+0000 front 2023-12-16T21:39:05.622580+0000 (oldest deadline 2023-12-16T21:39:30.322546+0000) 2023-12-16T21:48:46.817 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:48:46 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:48:46.582+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:48:46.818 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:48:46 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:48:46.398+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:09.828911+0000 front 2023-12-16T21:39:09.828958+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:48:46.818 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:48:46 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:48:46.398+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:09.828868+0000 front 2023-12-16T21:39:09.828878+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:48:46.818 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:48:46 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:48:46.398+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:09.828947+0000 front 2023-12-16T21:39:09.828993+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:48:46.818 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:48:46 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:48:46.399+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:48:46.818 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:48:46 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:48:46.454+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:09.306583+0000 front 2023-12-16T21:39:09.306633+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:48:46.818 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:48:46 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:48:46.454+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:09.306551+0000 front 2023-12-16T21:39:09.306652+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:48:46.818 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:48:46 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:48:46.454+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:09.306605+0000 front 2023-12-16T21:39:09.306613+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:48:46.819 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:48:46 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:48:46.454+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:48:47.817 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:48:47 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:48:47.599+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:05.622397+0000 front 2023-12-16T21:39:05.622556+0000 (oldest deadline 2023-12-16T21:39:30.322546+0000) 2023-12-16T21:48:47.817 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:48:47 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:48:47.599+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:05.622506+0000 front 2023-12-16T21:39:05.622528+0000 (oldest deadline 2023-12-16T21:39:30.322546+0000) 2023-12-16T21:48:47.817 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:48:47 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:48:47.599+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:05.622479+0000 front 2023-12-16T21:39:05.622580+0000 (oldest deadline 2023-12-16T21:39:30.322546+0000) 2023-12-16T21:48:47.817 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:48:47 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:48:47.601+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:48:47.817 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:48:47 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:48:47.406+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:09.828911+0000 front 2023-12-16T21:39:09.828958+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:48:47.817 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:48:47 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:48:47.406+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:09.828868+0000 front 2023-12-16T21:39:09.828878+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:48:47.817 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:48:47 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:48:47.406+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:09.828947+0000 front 2023-12-16T21:39:09.828993+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:48:47.818 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:48:47 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:48:47.407+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:48:47.818 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:48:47 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:48:47.448+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:09.306583+0000 front 2023-12-16T21:39:09.306633+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:48:47.818 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:48:47 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:48:47.448+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:09.306551+0000 front 2023-12-16T21:39:09.306652+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:48:47.818 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:48:47 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:48:47.448+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:09.306605+0000 front 2023-12-16T21:39:09.306613+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:48:47.818 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:48:47 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:48:47.449+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:48:48.722 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:48:48 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:48:48.403+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:09.306583+0000 front 2023-12-16T21:39:09.306633+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:48:48.723 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:48:48 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:48:48.403+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:09.306551+0000 front 2023-12-16T21:39:09.306652+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:48:48.723 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:48:48 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:48:48.403+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:09.306605+0000 front 2023-12-16T21:39:09.306613+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:48:48.723 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:48:48 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:48:48.403+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:48:48.723 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:48:48 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:48:48.419+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:09.828911+0000 front 2023-12-16T21:39:09.828958+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:48:48.723 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:48:48 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:48:48.419+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:09.828868+0000 front 2023-12-16T21:39:09.828878+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:48:48.723 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:48:48 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:48:48.419+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:09.828947+0000 front 2023-12-16T21:39:09.828993+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:48:48.723 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:48:48 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:48:48.421+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:48:48.724 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:48:48 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:48:48.613+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:05.622397+0000 front 2023-12-16T21:39:05.622556+0000 (oldest deadline 2023-12-16T21:39:30.322546+0000) 2023-12-16T21:48:48.724 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:48:48 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:48:48.613+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:05.622506+0000 front 2023-12-16T21:39:05.622528+0000 (oldest deadline 2023-12-16T21:39:30.322546+0000) 2023-12-16T21:48:48.724 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:48:48 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:48:48.613+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:05.622479+0000 front 2023-12-16T21:39:05.622580+0000 (oldest deadline 2023-12-16T21:39:30.322546+0000) 2023-12-16T21:48:48.724 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:48:48 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:48:48.613+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:48:49.817 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:48:49 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:48:49.365+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:09.306583+0000 front 2023-12-16T21:39:09.306633+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:48:49.817 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:48:49 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:48:49.365+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:09.306551+0000 front 2023-12-16T21:39:09.306652+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:48:49.817 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:48:49 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:48:49.365+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:09.306605+0000 front 2023-12-16T21:39:09.306613+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:48:49.817 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:48:49 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:48:49.366+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:48:49.817 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:48:49 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:48:49.574+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:05.622397+0000 front 2023-12-16T21:39:05.622556+0000 (oldest deadline 2023-12-16T21:39:30.322546+0000) 2023-12-16T21:48:49.817 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:48:49 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:48:49.574+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:05.622506+0000 front 2023-12-16T21:39:05.622528+0000 (oldest deadline 2023-12-16T21:39:30.322546+0000) 2023-12-16T21:48:49.817 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:48:49 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:48:49.574+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:05.622479+0000 front 2023-12-16T21:39:05.622580+0000 (oldest deadline 2023-12-16T21:39:30.322546+0000) 2023-12-16T21:48:49.818 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:48:49 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:48:49.575+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:48:49.818 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:48:49 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:48:49.398+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:09.828911+0000 front 2023-12-16T21:39:09.828958+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:48:49.818 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:48:49 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:48:49.398+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:09.828868+0000 front 2023-12-16T21:39:09.828878+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:48:49.818 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:48:49 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:48:49.398+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:09.828947+0000 front 2023-12-16T21:39:09.828993+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:48:49.818 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:48:49 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:48:49.398+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:48:50.817 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:48:50 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:48:50.334+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:09.306583+0000 front 2023-12-16T21:39:09.306633+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:48:50.817 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:48:50 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:48:50.334+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:09.306551+0000 front 2023-12-16T21:39:09.306652+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:48:50.817 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:48:50 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:48:50.334+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:09.306605+0000 front 2023-12-16T21:39:09.306613+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:48:50.817 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:48:50 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:48:50.335+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:48:50.817 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:48:50 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:48:50.561+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:05.622397+0000 front 2023-12-16T21:39:05.622556+0000 (oldest deadline 2023-12-16T21:39:30.322546+0000) 2023-12-16T21:48:50.817 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:48:50 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:48:50.561+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:05.622506+0000 front 2023-12-16T21:39:05.622528+0000 (oldest deadline 2023-12-16T21:39:30.322546+0000) 2023-12-16T21:48:50.817 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:48:50 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:48:50.561+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:05.622479+0000 front 2023-12-16T21:39:05.622580+0000 (oldest deadline 2023-12-16T21:39:30.322546+0000) 2023-12-16T21:48:50.818 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:48:50 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:48:50.562+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:48:50.818 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:48:50 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:48:50.395+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:09.828911+0000 front 2023-12-16T21:39:09.828958+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:48:50.818 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:48:50 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:48:50.395+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:09.828868+0000 front 2023-12-16T21:39:09.828878+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:48:50.818 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:48:50 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:48:50.395+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:09.828947+0000 front 2023-12-16T21:39:09.828993+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:48:50.818 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:48:50 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:48:50.396+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:48:51.660 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Dec 16 21:48:51 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-mon-smithi045[80759]: 2023-12-16T21:48:51.393+0000 7f45a1f8f700 -1 mon.smithi045@0(probing) e2 get_health_metrics reporting 6965 slow ops, oldest is monmgrreport(gid 14214, 0 checks, 1 progress events) 2023-12-16T21:48:51.661 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:48:51 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:48:51.372+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:09.306583+0000 front 2023-12-16T21:39:09.306633+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:48:51.661 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:48:51 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:48:51.372+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:09.306551+0000 front 2023-12-16T21:39:09.306652+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:48:51.661 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:48:51 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:48:51.372+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:09.306605+0000 front 2023-12-16T21:39:09.306613+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:48:51.661 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:48:51 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:48:51.373+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:48:51.661 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:48:51 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:48:51.561+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:05.622397+0000 front 2023-12-16T21:39:05.622556+0000 (oldest deadline 2023-12-16T21:39:30.322546+0000) 2023-12-16T21:48:51.661 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:48:51 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:48:51.561+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:05.622506+0000 front 2023-12-16T21:39:05.622528+0000 (oldest deadline 2023-12-16T21:39:30.322546+0000) 2023-12-16T21:48:51.662 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:48:51 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:48:51.561+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:05.622479+0000 front 2023-12-16T21:39:05.622580+0000 (oldest deadline 2023-12-16T21:39:30.322546+0000) 2023-12-16T21:48:51.662 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:48:51 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:48:51.562+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:48:51.662 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:48:51 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:48:51.432+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:09.828911+0000 front 2023-12-16T21:39:09.828958+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:48:51.662 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:48:51 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:48:51.432+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:09.828868+0000 front 2023-12-16T21:39:09.828878+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:48:51.662 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:48:51 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:48:51.432+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:09.828947+0000 front 2023-12-16T21:39:09.828993+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:48:51.662 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:48:51 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:48:51.433+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:48:52.816 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:48:52 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:48:52.422+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:09.306583+0000 front 2023-12-16T21:39:09.306633+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:48:52.817 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:48:52 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:48:52.422+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:09.306551+0000 front 2023-12-16T21:39:09.306652+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:48:52.817 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:48:52 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:48:52.422+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:09.306605+0000 front 2023-12-16T21:39:09.306613+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:48:52.817 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:48:52 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:48:52.423+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:48:52.817 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:48:52 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:48:52.567+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:05.622397+0000 front 2023-12-16T21:39:05.622556+0000 (oldest deadline 2023-12-16T21:39:30.322546+0000) 2023-12-16T21:48:52.817 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:48:52 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:48:52.567+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:05.622506+0000 front 2023-12-16T21:39:05.622528+0000 (oldest deadline 2023-12-16T21:39:30.322546+0000) 2023-12-16T21:48:52.817 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:48:52 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:48:52.567+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:05.622479+0000 front 2023-12-16T21:39:05.622580+0000 (oldest deadline 2023-12-16T21:39:30.322546+0000) 2023-12-16T21:48:52.817 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:48:52 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:48:52.568+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:48:52.817 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:48:52 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:48:52.395+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:09.828911+0000 front 2023-12-16T21:39:09.828958+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:48:52.818 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:48:52 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:48:52.395+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:09.828868+0000 front 2023-12-16T21:39:09.828878+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:48:52.818 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:48:52 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:48:52.395+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:09.828947+0000 front 2023-12-16T21:39:09.828993+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:48:52.818 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:48:52 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:48:52.396+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:48:53.817 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:48:53 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:48:53.589+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:05.622397+0000 front 2023-12-16T21:39:05.622556+0000 (oldest deadline 2023-12-16T21:39:30.322546+0000) 2023-12-16T21:48:53.817 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:48:53 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:48:53.589+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:05.622506+0000 front 2023-12-16T21:39:05.622528+0000 (oldest deadline 2023-12-16T21:39:30.322546+0000) 2023-12-16T21:48:53.817 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:48:53 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:48:53.589+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:05.622479+0000 front 2023-12-16T21:39:05.622580+0000 (oldest deadline 2023-12-16T21:39:30.322546+0000) 2023-12-16T21:48:53.817 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:48:53 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:48:53.590+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:48:53.817 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:48:53 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:48:53.389+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:09.828911+0000 front 2023-12-16T21:39:09.828958+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:48:53.817 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:48:53 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:48:53.389+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:09.828868+0000 front 2023-12-16T21:39:09.828878+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:48:53.817 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:48:53 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:48:53.389+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:09.828947+0000 front 2023-12-16T21:39:09.828993+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:48:53.817 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:48:53 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:48:53.389+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:48:53.818 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:48:53 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:48:53.448+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:09.306583+0000 front 2023-12-16T21:39:09.306633+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:48:53.818 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:48:53 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:48:53.448+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:09.306551+0000 front 2023-12-16T21:39:09.306652+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:48:53.818 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:48:53 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:48:53.448+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:09.306605+0000 front 2023-12-16T21:39:09.306613+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:48:53.818 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:48:53 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:48:53.449+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:48:54.817 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:48:54 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:48:54.566+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:05.622397+0000 front 2023-12-16T21:39:05.622556+0000 (oldest deadline 2023-12-16T21:39:30.322546+0000) 2023-12-16T21:48:54.817 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:48:54 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:48:54.566+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:05.622506+0000 front 2023-12-16T21:39:05.622528+0000 (oldest deadline 2023-12-16T21:39:30.322546+0000) 2023-12-16T21:48:54.817 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:48:54 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:48:54.566+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:05.622479+0000 front 2023-12-16T21:39:05.622580+0000 (oldest deadline 2023-12-16T21:39:30.322546+0000) 2023-12-16T21:48:54.817 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:48:54 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:48:54.567+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:48:54.817 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:48:54 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:48:54.384+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:09.828911+0000 front 2023-12-16T21:39:09.828958+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:48:54.817 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:48:54 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:48:54.384+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:09.828868+0000 front 2023-12-16T21:39:09.828878+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:48:54.817 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:48:54 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:48:54.384+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:09.828947+0000 front 2023-12-16T21:39:09.828993+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:48:54.817 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:48:54 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:48:54.385+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:48:54.818 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:48:54 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:48:54.424+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:09.306583+0000 front 2023-12-16T21:39:09.306633+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:48:54.818 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:48:54 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:48:54.424+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:09.306551+0000 front 2023-12-16T21:39:09.306652+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:48:54.818 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:48:54 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:48:54.424+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:09.306605+0000 front 2023-12-16T21:39:09.306613+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:48:54.818 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:48:54 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:48:54.424+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:48:55.817 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:48:55 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:48:55.521+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:05.622397+0000 front 2023-12-16T21:39:05.622556+0000 (oldest deadline 2023-12-16T21:39:30.322546+0000) 2023-12-16T21:48:55.817 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:48:55 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:48:55.521+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:05.622506+0000 front 2023-12-16T21:39:05.622528+0000 (oldest deadline 2023-12-16T21:39:30.322546+0000) 2023-12-16T21:48:55.817 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:48:55 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:48:55.521+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:05.622479+0000 front 2023-12-16T21:39:05.622580+0000 (oldest deadline 2023-12-16T21:39:30.322546+0000) 2023-12-16T21:48:55.817 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:48:55 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:48:55.523+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:48:55.817 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:48:55 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:48:55.440+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:09.306583+0000 front 2023-12-16T21:39:09.306633+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:48:55.817 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:48:55 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:48:55.440+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:09.306551+0000 front 2023-12-16T21:39:09.306652+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:48:55.818 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:48:55 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:48:55.440+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:09.306605+0000 front 2023-12-16T21:39:09.306613+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:48:55.818 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:48:55 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:48:55.441+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:48:55.818 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:48:55 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:48:55.384+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:09.828911+0000 front 2023-12-16T21:39:09.828958+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:48:55.818 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:48:55 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:48:55.384+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:09.828868+0000 front 2023-12-16T21:39:09.828878+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:48:55.818 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:48:55 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:48:55.384+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:09.828947+0000 front 2023-12-16T21:39:09.828993+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:48:55.818 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:48:55 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:48:55.385+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:48:56.816 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Dec 16 21:48:56 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-mon-smithi045[80759]: 2023-12-16T21:48:56.394+0000 7f45a1f8f700 -1 mon.smithi045@0(probing) e2 get_health_metrics reporting 7059 slow ops, oldest is monmgrreport(gid 14214, 0 checks, 1 progress events) 2023-12-16T21:48:56.817 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:48:56 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:48:56.484+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:09.306583+0000 front 2023-12-16T21:39:09.306633+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:48:56.817 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:48:56 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:48:56.484+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:09.306551+0000 front 2023-12-16T21:39:09.306652+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:48:56.817 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:48:56 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:48:56.484+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:09.306605+0000 front 2023-12-16T21:39:09.306613+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:48:56.817 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:48:56 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:48:56.485+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:48:56.817 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:48:56 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:48:56.389+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:09.828911+0000 front 2023-12-16T21:39:09.828958+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:48:56.817 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:48:56 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:48:56.389+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:09.828868+0000 front 2023-12-16T21:39:09.828878+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:48:56.817 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:48:56 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:48:56.389+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:09.828947+0000 front 2023-12-16T21:39:09.828993+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:48:56.818 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:48:56 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:48:56.390+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:48:56.818 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:48:56 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:48:56.474+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:05.622397+0000 front 2023-12-16T21:39:05.622556+0000 (oldest deadline 2023-12-16T21:39:30.322546+0000) 2023-12-16T21:48:56.818 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:48:56 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:48:56.474+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:05.622506+0000 front 2023-12-16T21:39:05.622528+0000 (oldest deadline 2023-12-16T21:39:30.322546+0000) 2023-12-16T21:48:56.818 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:48:56 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:48:56.474+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:05.622479+0000 front 2023-12-16T21:39:05.622580+0000 (oldest deadline 2023-12-16T21:39:30.322546+0000) 2023-12-16T21:48:56.818 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:48:56 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:48:56.474+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:48:57.816 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:48:57 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:48:57.450+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:05.622397+0000 front 2023-12-16T21:39:05.622556+0000 (oldest deadline 2023-12-16T21:39:30.322546+0000) 2023-12-16T21:48:57.817 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:48:57 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:48:57.450+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:05.622506+0000 front 2023-12-16T21:39:05.622528+0000 (oldest deadline 2023-12-16T21:39:30.322546+0000) 2023-12-16T21:48:57.817 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:48:57 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:48:57.450+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:05.622479+0000 front 2023-12-16T21:39:05.622580+0000 (oldest deadline 2023-12-16T21:39:30.322546+0000) 2023-12-16T21:48:57.817 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:48:57 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:48:57.451+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:48:57.817 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:48:57 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:48:57.519+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:09.306583+0000 front 2023-12-16T21:39:09.306633+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:48:57.817 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:48:57 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:48:57.519+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:09.306551+0000 front 2023-12-16T21:39:09.306652+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:48:57.817 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:48:57 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:48:57.519+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:09.306605+0000 front 2023-12-16T21:39:09.306613+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:48:57.817 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:48:57 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:48:57.519+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:48:57.818 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:48:57 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:48:57.381+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:09.828911+0000 front 2023-12-16T21:39:09.828958+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:48:57.818 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:48:57 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:48:57.381+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:09.828868+0000 front 2023-12-16T21:39:09.828878+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:48:57.818 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:48:57 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:48:57.381+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:09.828947+0000 front 2023-12-16T21:39:09.828993+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:48:57.818 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:48:57 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:48:57.381+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:48:58.723 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:48:58 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:48:58.501+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:09.306583+0000 front 2023-12-16T21:39:09.306633+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:48:58.724 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:48:58 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:48:58.501+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:09.306551+0000 front 2023-12-16T21:39:09.306652+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:48:58.724 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:48:58 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:48:58.501+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:09.306605+0000 front 2023-12-16T21:39:09.306613+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:48:58.724 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:48:58 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:48:58.502+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:48:58.724 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:48:58 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:48:58.491+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:05.622397+0000 front 2023-12-16T21:39:05.622556+0000 (oldest deadline 2023-12-16T21:39:30.322546+0000) 2023-12-16T21:48:58.724 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:48:58 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:48:58.491+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:05.622506+0000 front 2023-12-16T21:39:05.622528+0000 (oldest deadline 2023-12-16T21:39:30.322546+0000) 2023-12-16T21:48:58.724 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:48:58 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:48:58.491+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:05.622479+0000 front 2023-12-16T21:39:05.622580+0000 (oldest deadline 2023-12-16T21:39:30.322546+0000) 2023-12-16T21:48:58.724 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:48:58 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:48:58.491+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:48:58.724 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:48:58 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:48:58.402+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:09.828911+0000 front 2023-12-16T21:39:09.828958+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:48:58.725 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:48:58 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:48:58.402+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:09.828868+0000 front 2023-12-16T21:39:09.828878+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:48:58.725 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:48:58 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:48:58.402+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:09.828947+0000 front 2023-12-16T21:39:09.828993+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:48:58.725 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:48:58 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:48:58.402+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:48:59.816 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:48:59 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:48:59.498+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:09.306583+0000 front 2023-12-16T21:39:09.306633+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:48:59.817 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:48:59 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:48:59.498+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:09.306551+0000 front 2023-12-16T21:39:09.306652+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:48:59.817 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:48:59 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:48:59.498+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:09.306605+0000 front 2023-12-16T21:39:09.306613+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:48:59.817 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:48:59 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:48:59.499+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:48:59.817 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:48:59 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:48:59.458+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:05.622397+0000 front 2023-12-16T21:39:05.622556+0000 (oldest deadline 2023-12-16T21:39:30.322546+0000) 2023-12-16T21:48:59.817 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:48:59 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:48:59.458+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:05.622506+0000 front 2023-12-16T21:39:05.622528+0000 (oldest deadline 2023-12-16T21:39:30.322546+0000) 2023-12-16T21:48:59.817 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:48:59 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:48:59.458+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:05.622479+0000 front 2023-12-16T21:39:05.622580+0000 (oldest deadline 2023-12-16T21:39:30.322546+0000) 2023-12-16T21:48:59.817 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:48:59 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:48:59.458+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:48:59.818 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:48:59 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:48:59.425+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:09.828911+0000 front 2023-12-16T21:39:09.828958+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:48:59.818 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:48:59 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:48:59.425+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:09.828868+0000 front 2023-12-16T21:39:09.828878+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:48:59.818 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:48:59 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:48:59.425+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:09.828947+0000 front 2023-12-16T21:39:09.828993+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:48:59.818 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:48:59 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:48:59.426+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:49:00.816 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:49:00 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:49:00.466+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:09.306583+0000 front 2023-12-16T21:39:09.306633+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:49:00.817 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:49:00 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:49:00.466+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:09.306551+0000 front 2023-12-16T21:39:09.306652+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:49:00.817 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:49:00 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:49:00.467+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:09.306605+0000 front 2023-12-16T21:39:09.306613+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:49:00.817 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:49:00 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:49:00.467+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:49:00.817 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:49:00 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:49:00.492+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:05.622397+0000 front 2023-12-16T21:39:05.622556+0000 (oldest deadline 2023-12-16T21:39:30.322546+0000) 2023-12-16T21:49:00.817 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:49:00 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:49:00.492+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:05.622506+0000 front 2023-12-16T21:39:05.622528+0000 (oldest deadline 2023-12-16T21:39:30.322546+0000) 2023-12-16T21:49:00.817 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:49:00 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:49:00.492+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:05.622479+0000 front 2023-12-16T21:39:05.622580+0000 (oldest deadline 2023-12-16T21:39:30.322546+0000) 2023-12-16T21:49:00.818 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:49:00 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:49:00.493+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:49:00.818 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:49:00 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:49:00.462+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:09.828911+0000 front 2023-12-16T21:39:09.828958+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:49:00.818 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:49:00 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:49:00.462+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:09.828868+0000 front 2023-12-16T21:39:09.828878+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:49:00.818 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:49:00 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:49:00.462+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:09.828947+0000 front 2023-12-16T21:39:09.828993+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:49:00.818 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:49:00 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:49:00.463+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:49:01.661 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Dec 16 21:49:01 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-mon-smithi045[80759]: 2023-12-16T21:49:01.395+0000 7f45a1f8f700 -1 mon.smithi045@0(probing) e2 get_health_metrics reporting 7157 slow ops, oldest is monmgrreport(gid 14214, 0 checks, 1 progress events) 2023-12-16T21:49:01.661 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:49:01 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:49:01.445+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:05.622397+0000 front 2023-12-16T21:39:05.622556+0000 (oldest deadline 2023-12-16T21:39:30.322546+0000) 2023-12-16T21:49:01.661 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:49:01 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:49:01.445+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:05.622506+0000 front 2023-12-16T21:39:05.622528+0000 (oldest deadline 2023-12-16T21:39:30.322546+0000) 2023-12-16T21:49:01.661 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:49:01 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:49:01.445+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:05.622479+0000 front 2023-12-16T21:39:05.622580+0000 (oldest deadline 2023-12-16T21:39:30.322546+0000) 2023-12-16T21:49:01.661 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:49:01 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:49:01.445+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:49:01.662 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:49:01 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:49:01.474+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:09.828911+0000 front 2023-12-16T21:39:09.828958+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:49:01.662 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:49:01 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:49:01.474+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:09.828868+0000 front 2023-12-16T21:39:09.828878+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:49:01.662 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:49:01 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:49:01.474+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:09.828947+0000 front 2023-12-16T21:39:09.828993+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:49:01.662 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:49:01 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:49:01.475+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:49:01.662 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:49:01 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:49:01.429+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:09.306583+0000 front 2023-12-16T21:39:09.306633+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:49:01.662 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:49:01 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:49:01.429+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:09.306551+0000 front 2023-12-16T21:39:09.306652+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:49:01.662 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:49:01 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:49:01.429+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:09.306605+0000 front 2023-12-16T21:39:09.306613+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:49:01.662 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:49:01 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:49:01.430+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:49:02.816 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:49:02 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:49:02.456+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:09.306583+0000 front 2023-12-16T21:39:09.306633+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:49:02.817 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:49:02 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:49:02.456+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:09.306551+0000 front 2023-12-16T21:39:09.306652+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:49:02.817 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:49:02 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:49:02.456+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:09.306605+0000 front 2023-12-16T21:39:09.306613+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:49:02.817 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:49:02 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:49:02.456+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:49:02.818 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:49:02 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:49:02.491+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:05.622397+0000 front 2023-12-16T21:39:05.622556+0000 (oldest deadline 2023-12-16T21:39:30.322546+0000) 2023-12-16T21:49:02.818 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:49:02 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:49:02.491+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:05.622506+0000 front 2023-12-16T21:39:05.622528+0000 (oldest deadline 2023-12-16T21:39:30.322546+0000) 2023-12-16T21:49:02.818 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:49:02 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:49:02.491+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:05.622479+0000 front 2023-12-16T21:39:05.622580+0000 (oldest deadline 2023-12-16T21:39:30.322546+0000) 2023-12-16T21:49:02.819 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:49:02 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:49:02.492+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:49:02.819 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:49:02 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:49:02.523+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:09.828911+0000 front 2023-12-16T21:39:09.828958+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:49:02.819 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:49:02 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:49:02.523+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:09.828868+0000 front 2023-12-16T21:39:09.828878+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:49:02.819 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:49:02 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:49:02.523+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:09.828947+0000 front 2023-12-16T21:39:09.828993+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:49:02.819 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:49:02 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:49:02.524+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:49:03.817 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:49:03 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:49:03.456+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:09.306583+0000 front 2023-12-16T21:39:09.306633+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:49:03.817 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:49:03 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:49:03.456+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:09.306551+0000 front 2023-12-16T21:39:09.306652+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:49:03.817 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:49:03 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:49:03.456+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:09.306605+0000 front 2023-12-16T21:39:09.306613+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:49:03.817 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:49:03 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:49:03.457+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:49:03.818 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:49:03 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:49:03.569+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:09.828911+0000 front 2023-12-16T21:39:09.828958+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:49:03.818 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:49:03 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:49:03.569+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:09.828868+0000 front 2023-12-16T21:39:09.828878+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:49:03.818 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:49:03 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:49:03.569+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:09.828947+0000 front 2023-12-16T21:39:09.828993+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:49:03.818 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:49:03 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:49:03.570+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:49:03.819 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:49:03 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:49:03.501+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:05.622397+0000 front 2023-12-16T21:39:05.622556+0000 (oldest deadline 2023-12-16T21:39:30.322546+0000) 2023-12-16T21:49:03.819 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:49:03 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:49:03.501+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:05.622506+0000 front 2023-12-16T21:39:05.622528+0000 (oldest deadline 2023-12-16T21:39:30.322546+0000) 2023-12-16T21:49:03.819 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:49:03 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:49:03.501+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:05.622479+0000 front 2023-12-16T21:39:05.622580+0000 (oldest deadline 2023-12-16T21:39:30.322546+0000) 2023-12-16T21:49:03.819 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:49:03 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:49:03.501+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:49:04.816 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:49:04 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:49:04.504+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:09.306583+0000 front 2023-12-16T21:39:09.306633+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:49:04.817 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:49:04 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:49:04.504+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:09.306551+0000 front 2023-12-16T21:39:09.306652+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:49:04.817 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:49:04 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:49:04.504+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:09.306605+0000 front 2023-12-16T21:39:09.306613+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:49:04.817 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:49:04 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:49:04.505+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:49:04.817 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:49:04 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:49:04.452+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:05.622397+0000 front 2023-12-16T21:39:05.622556+0000 (oldest deadline 2023-12-16T21:39:30.322546+0000) 2023-12-16T21:49:04.817 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:49:04 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:49:04.452+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:05.622506+0000 front 2023-12-16T21:39:05.622528+0000 (oldest deadline 2023-12-16T21:39:30.322546+0000) 2023-12-16T21:49:04.817 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:49:04 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:49:04.452+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:05.622479+0000 front 2023-12-16T21:39:05.622580+0000 (oldest deadline 2023-12-16T21:39:30.322546+0000) 2023-12-16T21:49:04.817 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:49:04 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:49:04.453+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:49:04.818 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:49:04 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:49:04.618+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:09.828911+0000 front 2023-12-16T21:39:09.828958+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:49:04.818 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:49:04 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:49:04.618+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:09.828868+0000 front 2023-12-16T21:39:09.828878+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:49:04.818 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:49:04 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:49:04.618+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:09.828947+0000 front 2023-12-16T21:39:09.828993+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:49:04.818 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:49:04 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:49:04.620+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:49:05.817 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:49:05 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:49:05.506+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:09.306583+0000 front 2023-12-16T21:39:09.306633+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:49:05.817 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:49:05 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:49:05.506+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:09.306551+0000 front 2023-12-16T21:39:09.306652+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:49:05.817 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:49:05 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:49:05.506+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:09.306605+0000 front 2023-12-16T21:39:09.306613+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:49:05.817 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:49:05 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:49:05.507+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:49:05.817 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:49:05 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:49:05.501+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:05.622397+0000 front 2023-12-16T21:39:05.622556+0000 (oldest deadline 2023-12-16T21:39:30.322546+0000) 2023-12-16T21:49:05.817 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:49:05 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:49:05.501+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:05.622506+0000 front 2023-12-16T21:39:05.622528+0000 (oldest deadline 2023-12-16T21:39:30.322546+0000) 2023-12-16T21:49:05.817 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:49:05 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:49:05.501+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:05.622479+0000 front 2023-12-16T21:39:05.622580+0000 (oldest deadline 2023-12-16T21:39:30.322546+0000) 2023-12-16T21:49:05.818 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:49:05 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:49:05.502+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:49:05.818 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:49:05 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:49:05.591+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:09.828911+0000 front 2023-12-16T21:39:09.828958+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:49:05.818 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:49:05 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:49:05.591+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:09.828868+0000 front 2023-12-16T21:39:09.828878+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:49:05.818 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:49:05 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:49:05.591+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:09.828947+0000 front 2023-12-16T21:39:09.828993+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:49:05.818 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:49:05 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:49:05.592+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:49:06.816 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Dec 16 21:49:06 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-mon-smithi045[80759]: 2023-12-16T21:49:06.396+0000 7f45a1f8f700 -1 mon.smithi045@0(probing) e2 get_health_metrics reporting 7251 slow ops, oldest is monmgrreport(gid 14214, 0 checks, 1 progress events) 2023-12-16T21:49:06.817 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:49:06 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:49:06.555+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:09.306583+0000 front 2023-12-16T21:39:09.306633+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:49:06.817 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:49:06 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:49:06.555+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:09.306551+0000 front 2023-12-16T21:39:09.306652+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:49:06.817 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:49:06 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:49:06.555+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:09.306605+0000 front 2023-12-16T21:39:09.306613+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:49:06.817 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:49:06 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:49:06.555+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:49:06.817 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:49:06 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:49:06.531+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:05.622397+0000 front 2023-12-16T21:39:05.622556+0000 (oldest deadline 2023-12-16T21:39:30.322546+0000) 2023-12-16T21:49:06.818 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:49:06 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:49:06.531+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:05.622506+0000 front 2023-12-16T21:39:05.622528+0000 (oldest deadline 2023-12-16T21:39:30.322546+0000) 2023-12-16T21:49:06.818 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:49:06 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:49:06.531+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:05.622479+0000 front 2023-12-16T21:39:05.622580+0000 (oldest deadline 2023-12-16T21:39:30.322546+0000) 2023-12-16T21:49:06.818 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:49:06 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:49:06.532+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:49:06.818 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:49:06 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:49:06.590+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:09.828911+0000 front 2023-12-16T21:39:09.828958+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:49:06.818 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:49:06 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:49:06.590+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:09.828868+0000 front 2023-12-16T21:39:09.828878+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:49:06.818 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:49:06 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:49:06.590+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:09.828947+0000 front 2023-12-16T21:39:09.828993+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:49:06.818 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:49:06 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:49:06.591+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:49:07.816 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:49:07 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:49:07.527+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:09.306583+0000 front 2023-12-16T21:39:09.306633+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:49:07.817 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:49:07 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:49:07.527+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:09.306551+0000 front 2023-12-16T21:39:09.306652+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:49:07.817 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:49:07 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:49:07.527+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:09.306605+0000 front 2023-12-16T21:39:09.306613+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:49:07.817 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:49:07 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:49:07.527+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:49:07.817 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:49:07 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:49:07.623+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:09.828911+0000 front 2023-12-16T21:39:09.828958+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:49:07.817 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:49:07 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:49:07.623+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:09.828868+0000 front 2023-12-16T21:39:09.828878+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:49:07.817 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:49:07 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:49:07.623+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:09.828947+0000 front 2023-12-16T21:39:09.828993+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:49:07.818 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:49:07 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:49:07.624+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:49:07.818 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:49:07 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:49:07.508+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:05.622397+0000 front 2023-12-16T21:39:05.622556+0000 (oldest deadline 2023-12-16T21:39:30.322546+0000) 2023-12-16T21:49:07.818 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:49:07 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:49:07.508+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:05.622506+0000 front 2023-12-16T21:39:05.622528+0000 (oldest deadline 2023-12-16T21:39:30.322546+0000) 2023-12-16T21:49:07.818 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:49:07 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:49:07.508+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:05.622479+0000 front 2023-12-16T21:39:05.622580+0000 (oldest deadline 2023-12-16T21:39:30.322546+0000) 2023-12-16T21:49:07.818 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:49:07 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:49:07.508+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:49:08.494 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:49:08 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:49:08.468+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:05.622397+0000 front 2023-12-16T21:39:05.622556+0000 (oldest deadline 2023-12-16T21:39:30.322546+0000) 2023-12-16T21:49:08.494 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:49:08 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:49:08.468+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:05.622506+0000 front 2023-12-16T21:39:05.622528+0000 (oldest deadline 2023-12-16T21:39:30.322546+0000) 2023-12-16T21:49:08.494 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:49:08 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:49:08.468+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:05.622479+0000 front 2023-12-16T21:39:05.622580+0000 (oldest deadline 2023-12-16T21:39:30.322546+0000) 2023-12-16T21:49:08.494 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:49:08 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:49:08.469+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:49:08.816 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:49:08 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:49:08.601+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:09.828911+0000 front 2023-12-16T21:39:09.828958+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:49:08.817 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:49:08 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:49:08.601+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:09.828868+0000 front 2023-12-16T21:39:09.828878+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:49:08.817 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:49:08 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:49:08.601+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:09.828947+0000 front 2023-12-16T21:39:09.828993+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:49:08.817 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:49:08 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:49:08.601+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:49:08.817 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:49:08 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:49:08.491+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:09.306583+0000 front 2023-12-16T21:39:09.306633+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:49:08.817 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:49:08 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:49:08.491+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:09.306551+0000 front 2023-12-16T21:39:09.306652+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:49:08.817 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:49:08 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:49:08.491+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:09.306605+0000 front 2023-12-16T21:39:09.306613+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:49:08.817 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:49:08 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:49:08.492+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:49:09.816 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:49:09 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:49:09.613+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:09.828911+0000 front 2023-12-16T21:39:09.828958+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:49:09.817 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:49:09 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:49:09.613+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:09.828868+0000 front 2023-12-16T21:39:09.828878+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:49:09.817 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:49:09 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:49:09.613+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:09.828947+0000 front 2023-12-16T21:39:09.828993+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:49:09.817 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:49:09 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:49:09.615+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:49:09.818 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:49:09 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:49:09.481+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:09.306583+0000 front 2023-12-16T21:39:09.306633+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:49:09.818 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:49:09 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:49:09.481+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:09.306551+0000 front 2023-12-16T21:39:09.306652+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:49:09.818 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:49:09 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:49:09.481+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:09.306605+0000 front 2023-12-16T21:39:09.306613+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:49:09.818 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:49:09 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:49:09.483+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:49:09.818 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:49:09 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:49:09.493+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:05.622397+0000 front 2023-12-16T21:39:05.622556+0000 (oldest deadline 2023-12-16T21:39:30.322546+0000) 2023-12-16T21:49:09.818 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:49:09 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:49:09.493+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:05.622506+0000 front 2023-12-16T21:39:05.622528+0000 (oldest deadline 2023-12-16T21:39:30.322546+0000) 2023-12-16T21:49:09.818 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:49:09 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:49:09.493+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:05.622479+0000 front 2023-12-16T21:39:05.622580+0000 (oldest deadline 2023-12-16T21:39:30.322546+0000) 2023-12-16T21:49:09.818 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:49:09 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:49:09.494+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:49:10.817 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:49:10 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:49:10.456+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:05.622397+0000 front 2023-12-16T21:39:05.622556+0000 (oldest deadline 2023-12-16T21:39:30.322546+0000) 2023-12-16T21:49:10.817 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:49:10 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:49:10.456+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:05.622506+0000 front 2023-12-16T21:39:05.622528+0000 (oldest deadline 2023-12-16T21:39:30.322546+0000) 2023-12-16T21:49:10.817 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:49:10 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:49:10.456+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:05.622479+0000 front 2023-12-16T21:39:05.622580+0000 (oldest deadline 2023-12-16T21:39:30.322546+0000) 2023-12-16T21:49:10.817 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:49:10 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:49:10.457+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:49:10.817 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:49:10 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:49:10.598+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:09.828911+0000 front 2023-12-16T21:39:09.828958+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:49:10.817 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:49:10 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:49:10.598+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:09.828868+0000 front 2023-12-16T21:39:09.828878+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:49:10.818 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:49:10 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:49:10.598+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:09.828947+0000 front 2023-12-16T21:39:09.828993+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:49:10.818 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:49:10 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:49:10.599+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:49:10.818 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:49:10 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:49:10.495+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:09.306583+0000 front 2023-12-16T21:39:09.306633+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:49:10.818 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:49:10 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:49:10.495+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:09.306551+0000 front 2023-12-16T21:39:09.306652+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:49:10.818 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:49:10 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:49:10.495+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:09.306605+0000 front 2023-12-16T21:39:09.306613+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:49:10.818 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:49:10 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:49:10.497+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:49:11.662 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Dec 16 21:49:11 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-mon-smithi045[80759]: 2023-12-16T21:49:11.397+0000 7f45a1f8f700 -1 mon.smithi045@0(probing) e2 get_health_metrics reporting 7349 slow ops, oldest is monmgrreport(gid 14214, 0 checks, 1 progress events) 2023-12-16T21:49:11.662 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:49:11 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:49:11.574+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:09.828911+0000 front 2023-12-16T21:39:09.828958+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:49:11.662 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:49:11 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:49:11.574+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:09.828868+0000 front 2023-12-16T21:39:09.828878+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:49:11.662 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:49:11 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:49:11.574+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:09.828947+0000 front 2023-12-16T21:39:09.828993+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:49:11.662 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:49:11 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:49:11.575+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:49:11.662 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:49:11 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:49:11.538+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:09.306583+0000 front 2023-12-16T21:39:09.306633+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:49:11.662 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:49:11 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:49:11.538+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:09.306551+0000 front 2023-12-16T21:39:09.306652+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:49:11.663 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:49:11 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:49:11.538+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:09.306605+0000 front 2023-12-16T21:39:09.306613+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:49:11.663 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:49:11 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:49:11.540+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:49:11.663 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:49:11 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:49:11.416+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:05.622397+0000 front 2023-12-16T21:39:05.622556+0000 (oldest deadline 2023-12-16T21:39:30.322546+0000) 2023-12-16T21:49:11.663 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:49:11 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:49:11.416+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:05.622506+0000 front 2023-12-16T21:39:05.622528+0000 (oldest deadline 2023-12-16T21:39:30.322546+0000) 2023-12-16T21:49:11.663 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:49:11 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:49:11.416+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:05.622479+0000 front 2023-12-16T21:39:05.622580+0000 (oldest deadline 2023-12-16T21:39:30.322546+0000) 2023-12-16T21:49:11.663 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:49:11 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:49:11.417+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:49:12.816 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:49:12 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:49:12.585+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:09.828911+0000 front 2023-12-16T21:39:09.828958+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:49:12.817 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:49:12 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:49:12.585+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:09.828868+0000 front 2023-12-16T21:39:09.828878+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:49:12.817 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:49:12 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:49:12.585+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:09.828947+0000 front 2023-12-16T21:39:09.828993+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:49:12.817 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:49:12 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:49:12.586+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:49:12.817 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:49:12 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:49:12.525+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:09.306583+0000 front 2023-12-16T21:39:09.306633+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:49:12.817 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:49:12 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:49:12.525+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:09.306551+0000 front 2023-12-16T21:39:09.306652+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:49:12.817 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:49:12 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:49:12.525+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:09.306605+0000 front 2023-12-16T21:39:09.306613+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:49:12.817 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:49:12 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:49:12.526+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:49:12.818 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:49:12 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:49:12.386+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:05.622397+0000 front 2023-12-16T21:39:05.622556+0000 (oldest deadline 2023-12-16T21:39:30.322546+0000) 2023-12-16T21:49:12.818 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:49:12 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:49:12.386+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:05.622506+0000 front 2023-12-16T21:39:05.622528+0000 (oldest deadline 2023-12-16T21:39:30.322546+0000) 2023-12-16T21:49:12.818 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:49:12 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:49:12.386+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:05.622479+0000 front 2023-12-16T21:39:05.622580+0000 (oldest deadline 2023-12-16T21:39:30.322546+0000) 2023-12-16T21:49:12.818 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:49:12 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:49:12.388+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:49:13.817 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:49:13 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:49:13.518+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:09.306583+0000 front 2023-12-16T21:39:09.306633+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:49:13.817 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:49:13 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:49:13.518+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:09.306551+0000 front 2023-12-16T21:39:09.306652+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:49:13.817 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:49:13 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:49:13.518+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:09.306605+0000 front 2023-12-16T21:39:09.306613+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:49:13.817 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:49:13 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:49:13.520+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:49:13.817 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:49:13 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:49:13.422+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:05.622397+0000 front 2023-12-16T21:39:05.622556+0000 (oldest deadline 2023-12-16T21:39:30.322546+0000) 2023-12-16T21:49:13.817 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:49:13 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:49:13.422+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:05.622506+0000 front 2023-12-16T21:39:05.622528+0000 (oldest deadline 2023-12-16T21:39:30.322546+0000) 2023-12-16T21:49:13.817 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:49:13 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:49:13.422+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:05.622479+0000 front 2023-12-16T21:39:05.622580+0000 (oldest deadline 2023-12-16T21:39:30.322546+0000) 2023-12-16T21:49:13.818 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:49:13 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:49:13.423+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:49:13.818 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:49:13 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:49:13.557+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:09.828911+0000 front 2023-12-16T21:39:09.828958+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:49:13.818 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:49:13 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:49:13.557+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:09.828868+0000 front 2023-12-16T21:39:09.828878+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:49:13.818 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:49:13 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:49:13.557+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:09.828947+0000 front 2023-12-16T21:39:09.828993+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:49:13.818 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:49:13 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:49:13.558+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:49:14.816 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:49:14 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:49:14.595+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:09.828911+0000 front 2023-12-16T21:39:09.828958+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:49:14.817 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:49:14 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:49:14.595+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:09.828868+0000 front 2023-12-16T21:39:09.828878+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:49:14.817 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:49:14 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:49:14.595+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:09.828947+0000 front 2023-12-16T21:39:09.828993+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:49:14.817 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:49:14 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:49:14.596+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:49:14.817 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:49:14 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:49:14.430+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:05.622397+0000 front 2023-12-16T21:39:05.622556+0000 (oldest deadline 2023-12-16T21:39:30.322546+0000) 2023-12-16T21:49:14.817 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:49:14 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:49:14.430+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:05.622506+0000 front 2023-12-16T21:39:05.622528+0000 (oldest deadline 2023-12-16T21:39:30.322546+0000) 2023-12-16T21:49:14.817 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:49:14 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:49:14.430+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:05.622479+0000 front 2023-12-16T21:39:05.622580+0000 (oldest deadline 2023-12-16T21:39:30.322546+0000) 2023-12-16T21:49:14.817 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:49:14 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:49:14.431+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:49:14.818 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:49:14 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:49:14.488+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:09.306583+0000 front 2023-12-16T21:39:09.306633+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:49:14.818 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:49:14 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:49:14.488+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:09.306551+0000 front 2023-12-16T21:39:09.306652+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:49:14.818 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:49:14 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:49:14.488+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:09.306605+0000 front 2023-12-16T21:39:09.306613+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:49:14.818 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:49:14 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:49:14.489+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:49:15.817 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:49:15 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:49:15.489+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:09.306583+0000 front 2023-12-16T21:39:09.306633+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:49:15.817 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:49:15 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:49:15.489+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:09.306551+0000 front 2023-12-16T21:39:09.306652+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:49:15.817 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:49:15 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:49:15.489+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:09.306605+0000 front 2023-12-16T21:39:09.306613+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:49:15.817 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:49:15 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:49:15.490+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:49:15.817 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:49:15 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:49:15.432+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:05.622397+0000 front 2023-12-16T21:39:05.622556+0000 (oldest deadline 2023-12-16T21:39:30.322546+0000) 2023-12-16T21:49:15.817 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:49:15 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:49:15.432+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:05.622506+0000 front 2023-12-16T21:39:05.622528+0000 (oldest deadline 2023-12-16T21:39:30.322546+0000) 2023-12-16T21:49:15.817 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:49:15 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:49:15.432+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:05.622479+0000 front 2023-12-16T21:39:05.622580+0000 (oldest deadline 2023-12-16T21:39:30.322546+0000) 2023-12-16T21:49:15.818 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:49:15 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:49:15.432+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:49:15.818 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:49:15 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:49:15.621+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:09.828911+0000 front 2023-12-16T21:39:09.828958+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:49:15.818 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:49:15 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:49:15.621+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:09.828868+0000 front 2023-12-16T21:39:09.828878+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:49:15.818 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:49:15 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:49:15.621+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:09.828947+0000 front 2023-12-16T21:39:09.828993+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:49:15.818 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:49:15 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:49:15.622+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:49:16.659 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Dec 16 21:49:16 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-mon-smithi045[80759]: 2023-12-16T21:49:16.398+0000 7f45a1f8f700 -1 mon.smithi045@0(probing) e2 get_health_metrics reporting 7442 slow ops, oldest is monmgrreport(gid 14214, 0 checks, 1 progress events) 2023-12-16T21:49:16.659 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:49:16 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:49:16.456+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:05.622397+0000 front 2023-12-16T21:39:05.622556+0000 (oldest deadline 2023-12-16T21:39:30.322546+0000) 2023-12-16T21:49:16.659 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:49:16 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:49:16.456+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:05.622506+0000 front 2023-12-16T21:39:05.622528+0000 (oldest deadline 2023-12-16T21:39:30.322546+0000) 2023-12-16T21:49:16.659 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:49:16 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:49:16.456+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:05.622479+0000 front 2023-12-16T21:39:05.622580+0000 (oldest deadline 2023-12-16T21:39:30.322546+0000) 2023-12-16T21:49:16.659 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:49:16 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:49:16.457+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:49:16.659 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:49:16 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:49:16.509+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:09.306583+0000 front 2023-12-16T21:39:09.306633+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:49:16.660 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:49:16 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:49:16.509+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:09.306551+0000 front 2023-12-16T21:39:09.306652+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:49:16.660 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:49:16 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:49:16.509+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:09.306605+0000 front 2023-12-16T21:39:09.306613+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:49:16.660 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:49:16 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:49:16.510+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:49:17.066 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:49:16 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:49:16.655+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:09.828911+0000 front 2023-12-16T21:39:09.828958+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:49:17.066 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:49:16 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:49:16.655+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:09.828868+0000 front 2023-12-16T21:39:09.828878+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:49:17.067 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:49:16 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:49:16.655+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:09.828947+0000 front 2023-12-16T21:39:09.828993+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:49:17.067 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:49:16 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:49:16.656+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:49:17.817 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:49:17 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:49:17.686+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:09.828911+0000 front 2023-12-16T21:39:09.828958+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:49:17.817 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:49:17 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:49:17.686+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:09.828868+0000 front 2023-12-16T21:39:09.828878+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:49:17.817 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:49:17 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:49:17.686+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:09.828947+0000 front 2023-12-16T21:39:09.828993+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:49:17.817 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:49:17 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:49:17.686+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:49:17.817 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:49:17 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:49:17.476+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:09.306583+0000 front 2023-12-16T21:39:09.306633+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:49:17.817 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:49:17 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:49:17.476+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:09.306551+0000 front 2023-12-16T21:39:09.306652+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:49:17.817 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:49:17 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:49:17.476+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:09.306605+0000 front 2023-12-16T21:39:09.306613+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:49:17.818 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:49:17 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:49:17.477+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:49:17.818 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:49:17 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:49:17.460+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:05.622397+0000 front 2023-12-16T21:39:05.622556+0000 (oldest deadline 2023-12-16T21:39:30.322546+0000) 2023-12-16T21:49:17.818 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:49:17 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:49:17.460+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:05.622506+0000 front 2023-12-16T21:39:05.622528+0000 (oldest deadline 2023-12-16T21:39:30.322546+0000) 2023-12-16T21:49:17.818 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:49:17 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:49:17.460+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:05.622479+0000 front 2023-12-16T21:39:05.622580+0000 (oldest deadline 2023-12-16T21:39:30.322546+0000) 2023-12-16T21:49:17.818 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:49:17 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:49:17.462+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:49:18.722 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:49:18 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:49:18.458+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:05.622397+0000 front 2023-12-16T21:39:05.622556+0000 (oldest deadline 2023-12-16T21:39:30.322546+0000) 2023-12-16T21:49:18.722 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:49:18 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:49:18.458+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:05.622506+0000 front 2023-12-16T21:39:05.622528+0000 (oldest deadline 2023-12-16T21:39:30.322546+0000) 2023-12-16T21:49:18.722 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:49:18 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:49:18.458+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:05.622479+0000 front 2023-12-16T21:39:05.622580+0000 (oldest deadline 2023-12-16T21:39:30.322546+0000) 2023-12-16T21:49:18.723 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:49:18 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:49:18.458+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:49:18.723 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:49:18 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:49:18.657+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:09.828911+0000 front 2023-12-16T21:39:09.828958+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:49:18.723 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:49:18 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:49:18.657+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:09.828868+0000 front 2023-12-16T21:39:09.828878+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:49:18.723 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:49:18 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:49:18.657+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:09.828947+0000 front 2023-12-16T21:39:09.828993+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:49:18.723 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:49:18 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:49:18.658+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:49:18.723 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:49:18 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:49:18.525+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:09.306583+0000 front 2023-12-16T21:39:09.306633+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:49:18.723 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:49:18 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:49:18.525+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:09.306551+0000 front 2023-12-16T21:39:09.306652+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:49:18.724 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:49:18 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:49:18.525+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:09.306605+0000 front 2023-12-16T21:39:09.306613+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:49:18.724 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:49:18 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:49:18.527+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:49:19.816 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:49:19 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:49:19.535+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:09.306583+0000 front 2023-12-16T21:39:09.306633+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:49:19.817 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:49:19 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:49:19.535+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:09.306551+0000 front 2023-12-16T21:39:09.306652+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:49:19.817 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:49:19 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:49:19.535+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:09.306605+0000 front 2023-12-16T21:39:09.306613+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:49:19.817 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:49:19 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:49:19.536+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:49:19.817 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:49:19 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:49:19.465+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:05.622397+0000 front 2023-12-16T21:39:05.622556+0000 (oldest deadline 2023-12-16T21:39:30.322546+0000) 2023-12-16T21:49:19.817 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:49:19 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:49:19.465+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:05.622506+0000 front 2023-12-16T21:39:05.622528+0000 (oldest deadline 2023-12-16T21:39:30.322546+0000) 2023-12-16T21:49:19.817 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:49:19 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:49:19.465+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:05.622479+0000 front 2023-12-16T21:39:05.622580+0000 (oldest deadline 2023-12-16T21:39:30.322546+0000) 2023-12-16T21:49:19.817 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:49:19 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:49:19.466+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:49:19.817 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:49:19 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:49:19.662+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:09.828911+0000 front 2023-12-16T21:39:09.828958+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:49:19.818 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:49:19 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:49:19.662+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:09.828868+0000 front 2023-12-16T21:39:09.828878+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:49:19.818 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:49:19 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:49:19.662+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:09.828947+0000 front 2023-12-16T21:39:09.828993+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:49:19.818 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:49:19 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:49:19.663+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:49:20.816 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:49:20 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:49:20.526+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:09.306583+0000 front 2023-12-16T21:39:09.306633+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:49:20.817 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:49:20 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:49:20.526+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:09.306551+0000 front 2023-12-16T21:39:09.306652+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:49:20.817 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:49:20 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:49:20.526+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:09.306605+0000 front 2023-12-16T21:39:09.306613+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:49:20.817 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:49:20 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:49:20.527+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:49:20.817 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:49:20 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:49:20.443+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:05.622397+0000 front 2023-12-16T21:39:05.622556+0000 (oldest deadline 2023-12-16T21:39:30.322546+0000) 2023-12-16T21:49:20.817 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:49:20 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:49:20.443+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:05.622506+0000 front 2023-12-16T21:39:05.622528+0000 (oldest deadline 2023-12-16T21:39:30.322546+0000) 2023-12-16T21:49:20.817 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:49:20 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:49:20.443+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:05.622479+0000 front 2023-12-16T21:39:05.622580+0000 (oldest deadline 2023-12-16T21:39:30.322546+0000) 2023-12-16T21:49:20.817 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:49:20 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:49:20.444+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:49:20.818 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:49:20 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:49:20.620+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:09.828911+0000 front 2023-12-16T21:39:09.828958+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:49:20.818 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:49:20 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:49:20.620+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:09.828868+0000 front 2023-12-16T21:39:09.828878+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:49:20.818 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:49:20 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:49:20.620+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:09.828947+0000 front 2023-12-16T21:39:09.828993+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:49:20.818 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:49:20 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:49:20.621+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:49:21.662 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Dec 16 21:49:21 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-mon-smithi045[80759]: 2023-12-16T21:49:21.399+0000 7f45a1f8f700 -1 mon.smithi045@0(probing) e2 get_health_metrics reporting 7540 slow ops, oldest is monmgrreport(gid 14214, 0 checks, 1 progress events) 2023-12-16T21:49:21.662 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:49:21 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:49:21.563+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:09.306583+0000 front 2023-12-16T21:39:09.306633+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:49:21.662 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:49:21 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:49:21.563+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:09.306551+0000 front 2023-12-16T21:39:09.306652+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:49:21.662 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:49:21 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:49:21.563+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:09.306605+0000 front 2023-12-16T21:39:09.306613+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:49:21.662 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:49:21 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:49:21.564+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:49:21.663 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:49:21 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:49:21.409+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:05.622397+0000 front 2023-12-16T21:39:05.622556+0000 (oldest deadline 2023-12-16T21:39:30.322546+0000) 2023-12-16T21:49:21.663 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:49:21 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:49:21.409+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:05.622506+0000 front 2023-12-16T21:39:05.622528+0000 (oldest deadline 2023-12-16T21:39:30.322546+0000) 2023-12-16T21:49:21.663 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:49:21 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:49:21.409+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:05.622479+0000 front 2023-12-16T21:39:05.622580+0000 (oldest deadline 2023-12-16T21:39:30.322546+0000) 2023-12-16T21:49:21.663 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:49:21 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:49:21.410+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:49:21.663 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:49:21 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:49:21.573+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:09.828911+0000 front 2023-12-16T21:39:09.828958+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:49:21.663 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:49:21 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:49:21.573+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:09.828868+0000 front 2023-12-16T21:39:09.828878+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:49:21.663 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:49:21 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:49:21.573+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:09.828947+0000 front 2023-12-16T21:39:09.828993+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:49:21.663 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:49:21 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:49:21.574+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:49:22.817 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:49:22 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:49:22.604+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:09.306583+0000 front 2023-12-16T21:39:09.306633+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:49:22.817 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:49:22 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:49:22.604+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:09.306551+0000 front 2023-12-16T21:39:09.306652+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:49:22.817 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:49:22 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:49:22.604+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:09.306605+0000 front 2023-12-16T21:39:09.306613+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:49:22.817 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:49:22 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:49:22.605+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:49:22.818 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:49:22 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:49:22.407+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:05.622397+0000 front 2023-12-16T21:39:05.622556+0000 (oldest deadline 2023-12-16T21:39:30.322546+0000) 2023-12-16T21:49:22.818 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:49:22 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:49:22.407+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:05.622506+0000 front 2023-12-16T21:39:05.622528+0000 (oldest deadline 2023-12-16T21:39:30.322546+0000) 2023-12-16T21:49:22.818 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:49:22 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:49:22.407+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:05.622479+0000 front 2023-12-16T21:39:05.622580+0000 (oldest deadline 2023-12-16T21:39:30.322546+0000) 2023-12-16T21:49:22.818 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:49:22 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:49:22.408+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:49:22.819 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:49:22 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:49:22.554+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:09.828911+0000 front 2023-12-16T21:39:09.828958+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:49:22.819 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:49:22 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:49:22.554+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:09.828868+0000 front 2023-12-16T21:39:09.828878+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:49:22.819 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:49:22 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:49:22.554+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:09.828947+0000 front 2023-12-16T21:39:09.828993+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:49:22.819 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:49:22 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:49:22.555+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:49:23.817 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:49:23 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:49:23.556+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:09.306583+0000 front 2023-12-16T21:39:09.306633+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:49:23.817 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:49:23 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:49:23.556+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:09.306551+0000 front 2023-12-16T21:39:09.306652+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:49:23.817 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:49:23 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:49:23.556+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:09.306605+0000 front 2023-12-16T21:39:09.306613+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:49:23.817 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:49:23 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:49:23.558+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:49:23.817 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:49:23 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:49:23.385+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:05.622397+0000 front 2023-12-16T21:39:05.622556+0000 (oldest deadline 2023-12-16T21:39:30.322546+0000) 2023-12-16T21:49:23.817 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:49:23 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:49:23.385+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:05.622506+0000 front 2023-12-16T21:39:05.622528+0000 (oldest deadline 2023-12-16T21:39:30.322546+0000) 2023-12-16T21:49:23.817 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:49:23 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:49:23.385+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:05.622479+0000 front 2023-12-16T21:39:05.622580+0000 (oldest deadline 2023-12-16T21:39:30.322546+0000) 2023-12-16T21:49:23.818 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:49:23 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:49:23.386+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:49:23.818 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:49:23 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:49:23.585+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:09.828911+0000 front 2023-12-16T21:39:09.828958+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:49:23.818 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:49:23 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:49:23.585+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:09.828868+0000 front 2023-12-16T21:39:09.828878+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:49:23.818 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:49:23 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:49:23.585+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:09.828947+0000 front 2023-12-16T21:39:09.828993+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:49:23.818 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:49:23 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:49:23.586+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:49:24.606 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:49:24 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:49:24.578+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:09.306583+0000 front 2023-12-16T21:39:09.306633+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:49:24.606 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:49:24 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:49:24.578+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:09.306551+0000 front 2023-12-16T21:39:09.306652+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:49:24.606 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:49:24 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:49:24.578+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:09.306605+0000 front 2023-12-16T21:39:09.306613+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:49:24.606 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:49:24 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:49:24.579+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:49:24.606 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:49:24 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:49:24.340+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:05.622397+0000 front 2023-12-16T21:39:05.622556+0000 (oldest deadline 2023-12-16T21:39:30.322546+0000) 2023-12-16T21:49:24.606 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:49:24 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:49:24.340+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:05.622506+0000 front 2023-12-16T21:39:05.622528+0000 (oldest deadline 2023-12-16T21:39:30.322546+0000) 2023-12-16T21:49:24.606 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:49:24 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:49:24.340+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:05.622479+0000 front 2023-12-16T21:39:05.622580+0000 (oldest deadline 2023-12-16T21:39:30.322546+0000) 2023-12-16T21:49:24.606 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:49:24 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:49:24.341+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:49:24.861 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:49:24 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:49:24.602+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:09.828911+0000 front 2023-12-16T21:39:09.828958+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:49:24.861 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:49:24 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:49:24.602+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:09.828868+0000 front 2023-12-16T21:39:09.828878+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:49:24.861 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:49:24 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:49:24.602+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:09.828947+0000 front 2023-12-16T21:39:09.828993+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:49:24.861 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:49:24 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:49:24.603+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:49:24.862 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:49:24.857+0000 7f7465fa0700 0 monclient(hunting): authenticate timed out after 300 2023-12-16T21:49:24.862 INFO:teuthology.orchestra.run.smithi045.stderr:[errno 110] RADOS timed out (error connecting to the cluster) 2023-12-16T21:49:25.136 DEBUG:teuthology.orchestra.run.smithi045:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 88ff363c-9c5a-11ee-95a5-87774f69a715 -e sha1=c7586e95519b7be76f3b6e3a4b333888baf46bc9 -- bash -c 'ceph versions | jq -e '"'"'.mgr | keys'"'"' | grep $sha1' 2023-12-16T21:49:25.418 INFO:teuthology.orchestra.run.smithi045.stderr:Inferring config /var/lib/ceph/88ff363c-9c5a-11ee-95a5-87774f69a715/mon.smithi045/config 2023-12-16T21:49:25.509 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:49:25 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:49:25.369+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:05.622397+0000 front 2023-12-16T21:39:05.622556+0000 (oldest deadline 2023-12-16T21:39:30.322546+0000) 2023-12-16T21:49:25.509 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:49:25 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:49:25.369+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:05.622506+0000 front 2023-12-16T21:39:05.622528+0000 (oldest deadline 2023-12-16T21:39:30.322546+0000) 2023-12-16T21:49:25.509 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:49:25 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:49:25.369+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:05.622479+0000 front 2023-12-16T21:39:05.622580+0000 (oldest deadline 2023-12-16T21:39:30.322546+0000) 2023-12-16T21:49:25.509 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:49:25 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:49:25.369+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:49:25.817 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:49:25 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:49:25.562+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:09.306583+0000 front 2023-12-16T21:39:09.306633+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:49:25.817 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:49:25 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:49:25.562+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:09.306551+0000 front 2023-12-16T21:39:09.306652+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:49:25.817 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:49:25 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:49:25.562+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:09.306605+0000 front 2023-12-16T21:39:09.306613+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:49:25.817 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:49:25 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:49:25.563+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:49:25.818 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:49:25 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:49:25.588+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:09.828911+0000 front 2023-12-16T21:39:09.828958+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:49:25.818 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:49:25 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:49:25.588+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:09.828868+0000 front 2023-12-16T21:39:09.828878+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:49:25.818 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:49:25 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:49:25.588+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:09.828947+0000 front 2023-12-16T21:39:09.828993+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:49:25.818 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:49:25 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:49:25.589+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:49:26.531 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Dec 16 21:49:26 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-mon-smithi045[80759]: 2023-12-16T21:49:26.400+0000 7f45a1f8f700 -1 mon.smithi045@0(probing) e2 get_health_metrics reporting 7633 slow ops, oldest is monmgrreport(gid 14214, 0 checks, 1 progress events) 2023-12-16T21:49:26.531 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:49:26 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:49:26.349+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:05.622397+0000 front 2023-12-16T21:39:05.622556+0000 (oldest deadline 2023-12-16T21:39:30.322546+0000) 2023-12-16T21:49:26.531 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:49:26 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:49:26.349+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:05.622506+0000 front 2023-12-16T21:39:05.622528+0000 (oldest deadline 2023-12-16T21:39:30.322546+0000) 2023-12-16T21:49:26.531 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:49:26 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:49:26.349+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:05.622479+0000 front 2023-12-16T21:39:05.622580+0000 (oldest deadline 2023-12-16T21:39:30.322546+0000) 2023-12-16T21:49:26.531 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:49:26 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:49:26.350+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:49:26.817 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:49:26 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:49:26.540+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:09.306583+0000 front 2023-12-16T21:39:09.306633+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:49:26.817 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:49:26 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:49:26.540+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:09.306551+0000 front 2023-12-16T21:39:09.306652+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:49:26.817 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:49:26 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:49:26.540+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:09.306605+0000 front 2023-12-16T21:39:09.306613+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:49:26.817 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:49:26 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:49:26.541+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:49:26.817 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:49:26 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:49:26.613+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:09.828911+0000 front 2023-12-16T21:39:09.828958+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:49:26.817 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:49:26 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:49:26.613+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:09.828868+0000 front 2023-12-16T21:39:09.828878+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:49:26.818 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:49:26 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:49:26.613+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:09.828947+0000 front 2023-12-16T21:39:09.828993+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:49:26.818 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:49:26 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:49:26.614+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:49:27.657 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:49:27 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:49:27.567+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:09.306583+0000 front 2023-12-16T21:39:09.306633+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:49:27.658 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:49:27 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:49:27.567+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:09.306551+0000 front 2023-12-16T21:39:09.306652+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:49:27.658 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:49:27 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:49:27.567+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:09.306605+0000 front 2023-12-16T21:39:09.306613+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:49:27.658 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:49:27 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:49:27.567+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:49:27.658 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:49:27 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:49:27.361+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:05.622397+0000 front 2023-12-16T21:39:05.622556+0000 (oldest deadline 2023-12-16T21:39:30.322546+0000) 2023-12-16T21:49:27.658 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:49:27 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:49:27.361+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:05.622506+0000 front 2023-12-16T21:39:05.622528+0000 (oldest deadline 2023-12-16T21:39:30.322546+0000) 2023-12-16T21:49:27.658 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:49:27 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:49:27.361+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:05.622479+0000 front 2023-12-16T21:39:05.622580+0000 (oldest deadline 2023-12-16T21:39:30.322546+0000) 2023-12-16T21:49:27.658 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:49:27 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:49:27.361+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:49:27.948 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:49:27 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:49:27.654+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:09.828911+0000 front 2023-12-16T21:39:09.828958+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:49:27.948 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:49:27 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:49:27.654+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:09.828868+0000 front 2023-12-16T21:39:09.828878+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:49:27.948 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:49:27 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:49:27.654+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:09.828947+0000 front 2023-12-16T21:39:09.828993+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:49:27.948 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:49:27 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:49:27.655+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:49:28.620 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:49:28 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:49:28.315+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:05.622397+0000 front 2023-12-16T21:39:05.622556+0000 (oldest deadline 2023-12-16T21:39:30.322546+0000) 2023-12-16T21:49:28.620 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:49:28 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:49:28.315+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:05.622506+0000 front 2023-12-16T21:39:05.622528+0000 (oldest deadline 2023-12-16T21:39:30.322546+0000) 2023-12-16T21:49:28.620 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:49:28 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:49:28.315+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:05.622479+0000 front 2023-12-16T21:39:05.622580+0000 (oldest deadline 2023-12-16T21:39:30.322546+0000) 2023-12-16T21:49:28.620 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:49:28 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:49:28.316+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:49:29.066 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:49:28 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:49:28.616+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:09.306583+0000 front 2023-12-16T21:39:09.306633+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:49:29.067 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:49:28 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:49:28.616+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:09.306551+0000 front 2023-12-16T21:39:09.306652+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:49:29.067 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:49:28 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:49:28.616+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:09.306605+0000 front 2023-12-16T21:39:09.306613+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:49:29.067 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:49:28 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:49:28.617+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:49:29.067 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:49:28 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:49:28.653+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:09.828911+0000 front 2023-12-16T21:39:09.828958+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:49:29.067 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:49:28 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:49:28.653+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:09.828868+0000 front 2023-12-16T21:39:09.828878+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:49:29.067 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:49:28 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:49:28.653+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:09.828947+0000 front 2023-12-16T21:39:09.828993+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:49:29.067 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:49:28 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:49:28.654+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:49:29.566 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:49:29 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:49:29.271+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:05.622397+0000 front 2023-12-16T21:39:05.622556+0000 (oldest deadline 2023-12-16T21:39:30.322546+0000) 2023-12-16T21:49:29.567 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:49:29 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:49:29.271+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:05.622506+0000 front 2023-12-16T21:39:05.622528+0000 (oldest deadline 2023-12-16T21:39:30.322546+0000) 2023-12-16T21:49:29.567 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:49:29 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:49:29.271+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:05.622479+0000 front 2023-12-16T21:39:05.622580+0000 (oldest deadline 2023-12-16T21:39:30.322546+0000) 2023-12-16T21:49:29.567 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:49:29 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:49:29.272+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:49:30.067 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:49:29 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:49:29.662+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:09.828911+0000 front 2023-12-16T21:39:09.828958+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:49:30.067 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:49:29 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:49:29.662+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:09.828868+0000 front 2023-12-16T21:39:09.828878+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:49:30.067 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:49:29 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:49:29.662+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:09.828947+0000 front 2023-12-16T21:39:09.828993+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:49:30.067 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:49:29 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:49:29.663+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:49:30.067 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:49:29 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:49:29.632+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:09.306583+0000 front 2023-12-16T21:39:09.306633+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:49:30.067 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:49:29 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:49:29.632+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:09.306551+0000 front 2023-12-16T21:39:09.306652+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:49:30.067 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:49:29 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:49:29.632+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:09.306605+0000 front 2023-12-16T21:39:09.306613+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:49:30.068 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:49:29 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:49:29.633+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:49:30.567 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:49:30 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:49:30.229+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:05.622397+0000 front 2023-12-16T21:39:05.622556+0000 (oldest deadline 2023-12-16T21:39:30.322546+0000) 2023-12-16T21:49:30.567 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:49:30 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:49:30.229+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:05.622506+0000 front 2023-12-16T21:39:05.622528+0000 (oldest deadline 2023-12-16T21:39:30.322546+0000) 2023-12-16T21:49:30.567 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:49:30 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:49:30.229+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:05.622479+0000 front 2023-12-16T21:39:05.622580+0000 (oldest deadline 2023-12-16T21:39:30.322546+0000) 2023-12-16T21:49:30.567 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:49:30 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:49:30.231+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:49:31.067 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:49:30 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:49:30.648+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:09.828911+0000 front 2023-12-16T21:39:09.828958+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:49:31.067 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:49:30 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:49:30.648+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:09.828868+0000 front 2023-12-16T21:39:09.828878+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:49:31.067 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:49:30 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:49:30.648+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:09.828947+0000 front 2023-12-16T21:39:09.828993+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:49:31.067 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:49:30 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:49:30.649+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:49:31.067 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:49:30 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:49:30.589+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:09.306583+0000 front 2023-12-16T21:39:09.306633+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:49:31.067 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:49:30 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:49:30.589+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:09.306551+0000 front 2023-12-16T21:39:09.306652+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:49:31.067 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:49:30 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:49:30.589+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:09.306605+0000 front 2023-12-16T21:39:09.306613+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:49:31.067 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:49:30 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:49:30.589+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:49:31.529 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:49:31 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:49:31.272+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:49:31.529 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Dec 16 21:49:31 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-mon-smithi045[80759]: 2023-12-16T21:49:31.401+0000 7f45a1f8f700 -1 mon.smithi045@0(probing) e2 get_health_metrics reporting 7730 slow ops, oldest is monmgrreport(gid 14214, 0 checks, 1 progress events) 2023-12-16T21:49:31.817 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:49:31 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:49:31.660+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:09.828911+0000 front 2023-12-16T21:39:09.828958+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:49:31.817 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:49:31 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:49:31.660+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:09.828868+0000 front 2023-12-16T21:39:09.828878+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:49:31.817 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:49:31 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:49:31.660+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:09.828947+0000 front 2023-12-16T21:39:09.828993+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:49:31.817 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:49:31 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:49:31.661+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:49:31.817 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:49:31 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:49:31.567+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:09.306583+0000 front 2023-12-16T21:39:09.306633+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:49:31.817 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:49:31 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:49:31.567+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:09.306551+0000 front 2023-12-16T21:39:09.306652+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:49:31.817 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:49:31 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:49:31.567+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:09.306605+0000 front 2023-12-16T21:39:09.306613+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:49:31.818 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:49:31 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:49:31.569+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:49:32.569 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:49:32 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:49:32.225+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:49:33.066 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:49:32 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:49:32.619+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:09.828911+0000 front 2023-12-16T21:39:09.828958+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:49:33.067 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:49:32 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:49:32.619+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:09.828868+0000 front 2023-12-16T21:39:09.828878+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:49:33.067 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:49:32 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:49:32.619+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:09.828947+0000 front 2023-12-16T21:39:09.828993+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:49:33.067 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:49:32 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:49:32.620+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:49:33.067 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:49:32 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:49:32.588+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:09.306583+0000 front 2023-12-16T21:39:09.306633+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:49:33.067 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:49:32 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:49:32.588+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:09.306551+0000 front 2023-12-16T21:39:09.306652+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:49:33.067 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:49:32 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:49:32.588+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:09.306605+0000 front 2023-12-16T21:39:09.306613+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:49:33.067 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:49:32 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:49:32.589+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:49:33.553 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:49:33 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:49:33.195+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:49:33.817 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:49:33 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:49:33.567+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:09.306583+0000 front 2023-12-16T21:39:09.306633+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:49:33.817 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:49:33 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:49:33.567+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:09.306551+0000 front 2023-12-16T21:39:09.306652+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:49:33.817 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:49:33 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:49:33.567+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:09.306605+0000 front 2023-12-16T21:39:09.306613+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:49:33.817 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:49:33 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:49:33.568+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:49:33.817 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:49:33 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:49:33.573+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:09.828911+0000 front 2023-12-16T21:39:09.828958+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:49:33.817 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:49:33 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:49:33.573+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:09.828868+0000 front 2023-12-16T21:39:09.828878+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:49:33.817 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:49:33 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:49:33.573+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:09.828947+0000 front 2023-12-16T21:39:09.828993+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:49:33.817 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:49:33 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:49:33.573+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:49:34.566 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:49:34 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:49:34.189+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:49:35.067 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:49:34 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:49:34.586+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:09.306583+0000 front 2023-12-16T21:39:09.306633+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:49:35.067 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:49:34 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:49:34.586+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:09.306551+0000 front 2023-12-16T21:39:09.306652+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:49:35.067 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:49:34 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:49:34.586+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:09.306605+0000 front 2023-12-16T21:39:09.306613+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:49:35.067 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:49:34 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:49:34.587+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:49:35.067 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:49:34 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:49:34.575+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:09.828911+0000 front 2023-12-16T21:39:09.828958+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:49:35.067 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:49:34 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:49:34.575+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:09.828868+0000 front 2023-12-16T21:39:09.828878+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:49:35.068 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:49:34 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:49:34.575+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:09.828947+0000 front 2023-12-16T21:39:09.828993+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:49:35.068 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:49:34 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:49:34.576+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:49:35.531 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:49:35 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:49:35.201+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:49:35.816 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:49:35 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:49:35.528+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:09.828911+0000 front 2023-12-16T21:39:09.828958+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:49:35.817 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:49:35 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:49:35.528+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:09.828868+0000 front 2023-12-16T21:39:09.828878+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:49:35.817 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:49:35 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:49:35.528+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:09.828947+0000 front 2023-12-16T21:39:09.828993+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:49:35.817 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:49:35 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:49:35.529+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:49:35.817 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:49:35 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:49:35.542+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:09.306583+0000 front 2023-12-16T21:39:09.306633+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:49:35.817 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:49:35 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:49:35.542+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:09.306551+0000 front 2023-12-16T21:39:09.306652+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:49:35.817 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:49:35 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:49:35.542+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:09.306605+0000 front 2023-12-16T21:39:09.306613+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:49:35.817 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:49:35 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:49:35.543+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:49:36.495 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Dec 16 21:49:36 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-mon-smithi045[80759]: 2023-12-16T21:49:36.402+0000 7f45a1f8f700 -1 mon.smithi045@0(probing) e2 get_health_metrics reporting 7823 slow ops, oldest is monmgrreport(gid 14214, 0 checks, 1 progress events) 2023-12-16T21:49:36.495 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:49:36 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:49:36.230+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:49:36.817 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:49:36 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:49:36.580+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:09.306583+0000 front 2023-12-16T21:39:09.306633+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:49:36.817 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:49:36 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:49:36.580+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:09.306551+0000 front 2023-12-16T21:39:09.306652+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:49:36.817 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:49:36 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:49:36.580+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:09.306605+0000 front 2023-12-16T21:39:09.306613+0000 (oldest deadline 2023-12-16T21:39:33.406545+0000) 2023-12-16T21:49:36.817 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:49:36 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:49:36.580+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:49:36.817 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:49:36 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:49:36.491+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:09.828911+0000 front 2023-12-16T21:39:09.828958+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:49:36.817 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:49:36 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:49:36.491+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:09.828868+0000 front 2023-12-16T21:39:09.828878+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:49:36.817 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:49:36 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:49:36.491+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:09.828947+0000 front 2023-12-16T21:39:09.828993+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:49:36.817 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:49:36 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:49:36.493+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:49:37.536 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:49:37 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:49:37.195+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:49:37.816 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:49:37 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:49:37.565+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:49:37.817 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:49:37 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:49:37.533+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:09.828911+0000 front 2023-12-16T21:39:09.828958+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:49:37.817 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:49:37 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:49:37.533+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:09.828868+0000 front 2023-12-16T21:39:09.828878+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:49:37.817 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:49:37 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:49:37.533+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:09.828947+0000 front 2023-12-16T21:39:09.828993+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:49:37.817 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:49:37 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:49:37.533+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:49:38.468 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:49:38 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:49:38.210+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:49:38.723 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:49:38 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:49:38.559+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:49:38.723 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:49:38 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:49:38.498+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:09.828911+0000 front 2023-12-16T21:39:09.828958+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:49:38.723 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:49:38 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:49:38.498+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:09.828868+0000 front 2023-12-16T21:39:09.828878+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:49:38.723 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:49:38 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:49:38.498+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:09.828947+0000 front 2023-12-16T21:39:09.828993+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:49:38.723 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:49:38 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:49:38.499+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:49:39.517 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:49:39 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:49:39.246+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:49:39.816 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:49:39 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:49:39.592+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:49:39.817 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:49:39 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:49:39.514+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:09.828911+0000 front 2023-12-16T21:39:09.828958+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:49:39.817 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:49:39 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:49:39.514+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:09.828868+0000 front 2023-12-16T21:39:09.828878+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:49:39.817 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:49:39 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:49:39.514+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:09.828947+0000 front 2023-12-16T21:39:09.828993+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:49:39.817 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:49:39 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:49:39.515+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:49:40.563 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:49:40 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:49:40.295+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:49:40.563 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:49:40 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:49:40.526+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:09.828911+0000 front 2023-12-16T21:39:09.828958+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:49:40.563 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:49:40 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:49:40.526+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:09.828868+0000 front 2023-12-16T21:39:09.828878+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:49:40.563 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:49:40 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:49:40.526+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:09.828947+0000 front 2023-12-16T21:39:09.828993+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:49:40.563 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:49:40 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:49:40.527+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:49:40.816 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:49:40 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:49:40.560+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:49:41.584 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Dec 16 21:49:41 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-mon-smithi045[80759]: 2023-12-16T21:49:41.403+0000 7f45a1f8f700 -1 mon.smithi045@0(probing) e2 get_health_metrics reporting 7920 slow ops, oldest is monmgrreport(gid 14214, 0 checks, 1 progress events) 2023-12-16T21:49:41.585 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:49:41 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:49:41.331+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:49:41.585 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:49:41 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:49:41.483+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:09.828911+0000 front 2023-12-16T21:39:09.828958+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:49:41.585 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:49:41 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:49:41.483+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:09.828868+0000 front 2023-12-16T21:39:09.828878+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:49:41.585 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:49:41 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:49:41.483+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:09.828947+0000 front 2023-12-16T21:39:09.828993+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:49:41.585 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:49:41 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:49:41.485+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:49:42.067 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:49:41 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:49:41.581+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:49:42.566 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:49:42 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:49:42.295+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:49:42.567 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:49:42 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:49:42.508+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:09.828911+0000 front 2023-12-16T21:39:09.828958+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:49:42.567 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:49:42 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:49:42.508+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:09.828868+0000 front 2023-12-16T21:39:09.828878+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:49:42.567 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:49:42 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:49:42.508+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:09.828947+0000 front 2023-12-16T21:39:09.828993+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:49:42.567 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:49:42 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:49:42.509+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:49:43.066 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:49:42 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:49:42.599+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:49:43.553 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:49:43 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:49:43.479+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:09.828911+0000 front 2023-12-16T21:39:09.828958+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:49:43.553 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:49:43 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:49:43.479+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:09.828868+0000 front 2023-12-16T21:39:09.828878+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:49:43.553 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:49:43 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:49:43.479+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:09.828947+0000 front 2023-12-16T21:39:09.828993+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:49:43.553 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:49:43 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:49:43.479+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:49:43.554 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:49:43 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:49:43.259+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:49:43.816 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:49:43 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:49:43.635+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:49:44.566 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:49:44 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:49:44.268+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:49:44.567 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:49:44 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:49:44.456+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:09.828911+0000 front 2023-12-16T21:39:09.828958+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:49:44.567 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:49:44 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:49:44.456+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:09.828868+0000 front 2023-12-16T21:39:09.828878+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:49:44.567 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:49:44 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:49:44.456+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:09.828947+0000 front 2023-12-16T21:39:09.828993+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:49:44.567 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:49:44 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:49:44.457+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:49:45.066 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:49:44 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:49:44.601+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:49:45.480 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:49:45 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:49:45.281+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:49:45.816 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:49:45 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:49:45.569+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:49:45.817 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:49:45 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:49:45.477+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:09.828911+0000 front 2023-12-16T21:39:09.828958+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:49:45.817 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:49:45 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:49:45.477+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:09.828868+0000 front 2023-12-16T21:39:09.828878+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:49:45.817 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:49:45 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:49:45.477+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:09.828947+0000 front 2023-12-16T21:39:09.828993+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:49:45.817 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:49:45 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:49:45.478+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:49:46.553 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:49:46 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:49:46.277+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:49:46.554 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Dec 16 21:49:46 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-mon-smithi045[80759]: 2023-12-16T21:49:46.405+0000 7f45a1f8f700 -1 mon.smithi045@0(probing) e2 get_health_metrics reporting 8013 slow ops, oldest is monmgrreport(gid 14214, 0 checks, 1 progress events) 2023-12-16T21:49:46.554 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:49:46 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:49:46.518+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:09.828911+0000 front 2023-12-16T21:39:09.828958+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:49:46.554 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:49:46 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:49:46.518+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:09.828868+0000 front 2023-12-16T21:39:09.828878+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:49:46.554 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:49:46 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:49:46.518+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:09.828947+0000 front 2023-12-16T21:39:09.828993+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:49:46.554 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:49:46 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:49:46.519+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:49:46.816 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:49:46 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:49:46.550+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:49:47.528 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:49:47 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:49:47.241+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:49:47.816 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:49:47 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:49:47.538+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:49:47.816 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:49:47 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:49:47.523+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:09.828911+0000 front 2023-12-16T21:39:09.828958+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:49:47.817 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:49:47 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:49:47.524+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:09.828868+0000 front 2023-12-16T21:39:09.828878+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:49:47.817 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:49:47 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:49:47.524+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:09.828947+0000 front 2023-12-16T21:39:09.828993+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:49:47.817 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:49:47 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:49:47.524+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:49:48.558 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:49:48 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:49:48.289+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:49:48.816 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:49:48 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:49:48.562+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:49:48.817 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:49:48 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:49:48.554+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:09.828911+0000 front 2023-12-16T21:39:09.828958+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:49:48.817 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:49:48 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:49:48.554+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:09.828868+0000 front 2023-12-16T21:39:09.828878+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:49:48.817 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:49:48 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:49:48.554+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:09.828947+0000 front 2023-12-16T21:39:09.828993+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:49:48.817 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:49:48 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:49:48.555+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:49:49.598 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:49:49 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:49:49.321+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:49:49.858 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:49:49 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:49:49.594+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:49:49.859 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:49:49 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:49:49.605+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:09.828911+0000 front 2023-12-16T21:39:09.828958+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:49:49.859 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:49:49 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:49:49.605+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:09.828868+0000 front 2023-12-16T21:39:09.828878+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:49:49.859 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:49:49 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:49:49.605+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:09.828947+0000 front 2023-12-16T21:39:09.828993+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:49:49.859 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:49:49 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:49:49.605+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:49:50.561 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:49:50 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:49:50.287+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:49:50.816 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:49:50 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:49:50.611+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:49:50.817 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:49:50 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:49:50.557+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:09.828911+0000 front 2023-12-16T21:39:09.828958+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:49:50.817 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:49:50 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:49:50.557+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:09.828868+0000 front 2023-12-16T21:39:09.828878+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:49:50.817 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:49:50 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:49:50.557+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:09.828947+0000 front 2023-12-16T21:39:09.828993+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:49:50.817 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:49:50 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:49:50.558+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:49:51.566 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Dec 16 21:49:51 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-mon-smithi045[80759]: 2023-12-16T21:49:51.405+0000 7f45a1f8f700 -1 mon.smithi045@0(probing) e2 get_health_metrics reporting 8110 slow ops, oldest is monmgrreport(gid 14214, 0 checks, 1 progress events) 2023-12-16T21:49:51.567 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:49:51 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:49:51.286+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:49:52.066 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:49:51 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:49:51.591+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:49:52.067 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:49:51 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:49:51.606+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 since back 2023-12-16T21:39:09.828911+0000 front 2023-12-16T21:39:09.828958+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:49:52.067 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:49:51 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:49:51.606+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 since back 2023-12-16T21:39:09.828868+0000 front 2023-12-16T21:39:09.828878+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:49:52.067 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:49:51 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:49:51.606+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 since back 2023-12-16T21:39:09.828947+0000 front 2023-12-16T21:39:09.828993+0000 (oldest deadline 2023-12-16T21:39:32.129018+0000) 2023-12-16T21:49:52.067 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:49:51 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:49:51.606+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:49:52.599 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:49:52 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:49:52.337+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:49:52.599 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:49:52 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:49:52.337+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:49:52.599 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:49:52 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:49:52.337+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:49:52.599 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:49:52 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:49:52.337+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:49:53.066 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:49:52 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:49:52.594+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:49:53.067 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:49:52 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:49:52.634+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:49:53.554 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:49:53 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:49:53.292+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:49:53.554 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:49:53 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:49:53.292+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:49:53.554 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:49:53 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:49:53.292+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:49:53.554 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:49:53 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:49:53.293+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:49:53.816 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:49:53 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:49:53.556+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:49:53.816 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:49:53 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:49:53.654+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:49:54.561 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:49:54 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:49:54.269+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:49:54.561 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:49:54 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:49:54.269+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:49:54.561 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:49:54 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:49:54.269+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:49:54.561 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:49:54 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:49:54.270+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:49:54.816 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:49:54 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:49:54.558+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:49:54.816 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:49:54 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:49:54.671+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:49:55.685 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:49:55 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:49:55.546+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:49:55.685 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:49:55 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:49:55.314+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:49:55.685 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:49:55 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:49:55.314+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:49:55.685 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:49:55 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:49:55.314+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:49:55.685 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:49:55 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:49:55.314+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:49:56.066 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:49:55 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:49:55.682+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:49:56.717 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Dec 16 21:49:56 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-mon-smithi045[80759]: 2023-12-16T21:49:56.407+0000 7f45a1f8f700 -1 mon.smithi045@0(probing) e2 get_health_metrics reporting 8203 slow ops, oldest is monmgrreport(gid 14214, 0 checks, 1 progress events) 2023-12-16T21:49:56.717 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:49:56 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:49:56.592+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:49:56.718 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:49:56 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:49:56.353+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:49:56.718 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:49:56 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:49:56.353+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:49:56.718 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:49:56 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:49:56.353+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:49:56.718 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:49:56 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:49:56.353+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:49:57.066 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:49:56 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:49:56.714+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:49:57.608 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:49:57 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:49:57.317+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:49:57.609 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:49:57 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:49:57.317+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:49:57.609 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:49:57 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:49:57.317+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:49:57.609 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:49:57 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:49:57.318+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:49:58.066 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:49:57 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:49:57.745+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:49:58.066 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:49:57 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:49:57.605+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:49:58.566 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:49:58 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:49:58.271+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:49:58.566 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:49:58 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:49:58.271+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:49:58.566 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:49:58 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:49:58.271+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:49:58.567 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:49:58 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:49:58.272+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:49:59.066 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:49:58 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:49:58.711+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:49:59.066 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:49:58 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:49:58.629+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:49:59.067 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:49:58 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:49:58.629+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:49:59.067 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:49:58 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:49:58.629+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:49:59.067 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:49:58 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:49:58.630+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:49:59.566 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:49:59 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:49:59.305+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:49:59.566 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:49:59 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:49:59.305+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:49:59.566 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:49:59 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:49:59.305+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:49:59.567 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:49:59 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:49:59.306+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:50:00.066 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:49:59 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:49:59.749+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:50:00.067 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:49:59 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:49:59.661+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:50:00.067 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:49:59 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:49:59.661+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:50:00.067 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:49:59 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:49:59.661+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:50:00.067 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:49:59 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:49:59.662+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:50:00.566 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:50:00 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:50:00.339+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:50:00.566 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:50:00 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:50:00.339+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:50:00.567 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:50:00 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:50:00.339+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:50:00.567 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:50:00 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:50:00.340+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:50:01.066 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:50:00 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:50:00.741+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:50:01.067 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:50:00 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:50:00.677+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:50:01.067 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:50:00 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:50:00.677+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:50:01.067 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:50:00 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:50:00.677+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:50:01.067 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:50:00 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:50:00.678+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:50:01.664 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Dec 16 21:50:01 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-mon-smithi045[80759]: 2023-12-16T21:50:01.408+0000 7f45a1f8f700 -1 mon.smithi045@0(probing) e2 get_health_metrics reporting 8300 slow ops, oldest is monmgrreport(gid 14214, 0 checks, 1 progress events) 2023-12-16T21:50:01.665 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:50:01 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:50:01.344+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:50:01.665 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:50:01 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:50:01.344+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:50:01.665 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:50:01 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:50:01.344+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:50:01.665 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:50:01 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:50:01.345+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:50:02.066 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:50:01 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:50:01.727+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:50:02.066 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:50:01 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:50:01.688+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:50:02.067 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:50:01 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:50:01.688+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:50:02.067 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:50:01 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:50:01.688+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:50:02.067 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:50:01 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:50:01.689+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:50:02.647 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:50:02 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:50:02.336+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:50:02.647 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:50:02 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:50:02.336+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:50:02.648 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:50:02 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:50:02.336+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:50:02.648 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:50:02 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:50:02.337+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:50:03.066 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:50:02 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:50:02.745+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:50:03.067 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:50:02 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:50:02.644+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:50:03.067 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:50:02 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:50:02.644+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:50:03.067 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:50:02 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:50:02.644+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:50:03.067 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:50:02 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:50:02.645+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:50:03.616 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:50:03 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:50:03.342+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:50:03.616 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:50:03 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:50:03.342+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:50:03.616 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:50:03 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:50:03.342+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:50:03.616 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:50:03 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:50:03.343+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:50:04.066 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:50:03 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:50:03.753+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:50:04.066 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:50:03 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:50:03.613+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:50:04.067 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:50:03 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:50:03.613+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:50:04.067 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:50:03 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:50:03.613+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:50:04.067 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:50:03 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:50:03.614+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:50:04.629 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:50:04 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:50:04.335+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:50:04.629 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:50:04 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:50:04.335+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:50:04.629 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:50:04 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:50:04.335+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:50:04.629 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:50:04 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:50:04.336+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:50:05.066 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:50:04 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:50:04.775+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:50:05.067 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:50:04 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:50:04.625+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:50:05.067 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:50:04 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:50:04.626+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:50:05.067 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:50:04 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:50:04.626+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:50:05.067 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:50:04 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:50:04.627+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:50:05.566 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:50:05 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:50:05.311+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:50:05.566 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:50:05 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:50:05.311+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:50:05.567 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:50:05 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:50:05.311+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:50:05.567 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:50:05 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:50:05.313+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:50:06.066 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:50:05 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:50:05.785+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:50:06.066 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:50:05 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:50:05.660+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:50:06.067 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:50:05 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:50:05.660+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:50:06.067 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:50:05 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:50:05.660+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:50:06.067 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:50:05 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:50:05.661+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:50:06.609 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:50:06 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:50:06.334+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:50:06.609 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:50:06 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:50:06.334+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:50:06.609 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:50:06 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:50:06.334+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:50:06.609 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:50:06 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:50:06.335+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:50:06.610 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Dec 16 21:50:06 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-mon-smithi045[80759]: 2023-12-16T21:50:06.409+0000 7f45a1f8f700 -1 mon.smithi045@0(probing) e2 get_health_metrics reporting 8394 slow ops, oldest is monmgrreport(gid 14214, 0 checks, 1 progress events) 2023-12-16T21:50:07.067 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:50:06 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:50:06.770+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:50:07.067 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:50:06 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:50:06.621+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:50:07.067 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:50:06 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:50:06.621+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:50:07.067 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:50:06 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:50:06.621+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:50:07.067 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:50:06 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:50:06.621+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:50:07.592 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:50:07 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:50:07.334+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:50:07.592 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:50:07 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:50:07.334+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:50:07.592 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:50:07 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:50:07.334+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:50:07.592 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:50:07 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:50:07.334+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:50:08.066 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:50:07 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:50:07.742+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:50:08.066 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:50:07 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:50:07.589+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:50:08.067 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:50:07 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:50:07.589+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:50:08.067 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:50:07 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:50:07.589+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:50:08.067 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:50:07 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:50:07.590+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:50:08.605 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:50:08 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:50:08.325+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:50:08.605 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:50:08 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:50:08.325+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:50:08.605 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:50:08 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:50:08.325+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:50:08.605 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:50:08 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:50:08.325+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:50:09.066 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:50:08 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:50:08.746+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:50:09.066 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:50:08 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:50:08.602+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:50:09.067 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:50:08 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:50:08.602+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:50:09.067 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:50:08 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:50:08.602+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:50:09.067 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:50:08 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:50:08.602+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:50:09.681 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:50:09 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:50:09.566+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:50:09.682 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:50:09 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:50:09.566+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:50:09.682 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:50:09 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:50:09.566+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:50:09.682 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:50:09 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:50:09.567+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:50:09.682 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:50:09 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:50:09.360+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:50:09.682 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:50:09 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:50:09.360+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:50:09.682 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:50:09 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:50:09.360+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:50:09.682 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:50:09 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:50:09.362+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:50:10.066 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:50:09 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:50:09.779+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:50:10.770 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:50:10 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:50:10.386+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:50:10.770 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:50:10 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:50:10.386+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:50:10.770 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:50:10 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:50:10.386+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:50:10.770 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:50:10 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:50:10.386+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:50:10.770 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:50:10 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:50:10.575+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:50:10.771 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:50:10 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:50:10.575+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:50:10.771 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:50:10 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:50:10.575+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:50:10.771 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:50:10 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:50:10.576+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:50:11.066 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:50:10 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:50:10.767+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:50:11.665 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Dec 16 21:50:11 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-mon-smithi045[80759]: 2023-12-16T21:50:11.410+0000 7f45a1f8f700 -1 mon.smithi045@0(probing) e2 get_health_metrics reporting 8491 slow ops, oldest is monmgrreport(gid 14214, 0 checks, 1 progress events) 2023-12-16T21:50:11.665 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:50:11 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:50:11.352+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:50:11.665 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:50:11 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:50:11.352+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:50:11.665 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:50:11 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:50:11.352+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:50:11.666 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:50:11 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:50:11.353+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:50:11.666 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:50:11 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:50:11.572+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:50:11.666 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:50:11 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:50:11.572+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:50:11.666 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:50:11 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:50:11.572+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:50:11.666 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:50:11 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:50:11.573+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:50:12.066 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:50:11 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:50:11.720+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:50:12.706 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:50:12 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:50:12.618+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:50:12.706 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:50:12 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:50:12.618+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:50:12.706 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:50:12 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:50:12.618+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:50:12.706 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:50:12 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:50:12.620+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:50:12.707 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:50:12 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:50:12.377+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:50:12.707 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:50:12 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:50:12.377+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:50:12.707 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:50:12 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:50:12.377+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:50:12.707 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:50:12 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:50:12.378+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:50:13.066 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:50:12 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:50:12.703+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:50:13.697 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:50:13 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:50:13.629+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:50:13.697 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:50:13 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:50:13.629+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:50:13.697 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:50:13 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:50:13.629+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:50:13.697 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:50:13 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:50:13.630+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:50:13.697 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:50:13 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:50:13.416+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:50:13.697 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:50:13 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:50:13.416+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:50:13.698 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:50:13 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:50:13.416+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:50:13.698 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:50:13 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:50:13.417+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:50:14.066 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:50:13 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:50:13.694+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:50:14.743 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:50:14 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:50:14.643+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:50:14.743 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:50:14 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:50:14.643+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:50:14.743 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:50:14 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:50:14.643+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:50:14.743 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:50:14 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:50:14.643+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:50:14.744 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:50:14 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:50:14.442+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:50:14.744 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:50:14 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:50:14.442+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:50:14.744 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:50:14 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:50:14.442+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:50:14.744 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:50:14 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:50:14.442+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:50:15.066 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:50:14 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:50:14.740+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:50:15.712 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:50:15 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:50:15.406+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:50:15.712 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:50:15 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:50:15.406+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:50:15.712 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:50:15 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:50:15.406+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:50:15.712 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:50:15 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:50:15.407+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:50:15.713 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:50:15 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:50:15.622+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:50:15.713 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:50:15 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:50:15.622+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:50:15.713 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:50:15 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:50:15.622+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:50:15.713 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:50:15 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:50:15.623+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:50:16.066 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:50:15 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:50:15.709+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T21:50:16.066 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:50:15 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:50:15.709+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T21:50:16.066 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:50:15 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:50:15.709+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T21:50:16.067 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:50:15 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:50:15.710+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:50:16.730 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:50:16 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:50:16.616+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:50:16.730 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:50:16 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:50:16.616+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:50:16.730 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:50:16 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:50:16.616+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:50:16.730 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:50:16 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:50:16.617+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:50:16.731 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Dec 16 21:50:16 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-mon-smithi045[80759]: 2023-12-16T21:50:16.411+0000 7f45a1f8f700 -1 mon.smithi045@0(probing) e2 get_health_metrics reporting 8585 slow ops, oldest is monmgrreport(gid 14214, 0 checks, 1 progress events) 2023-12-16T21:50:16.731 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:50:16 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:50:16.445+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:50:16.731 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:50:16 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:50:16.445+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:50:16.731 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:50:16 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:50:16.445+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:50:16.731 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:50:16 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:50:16.446+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:50:17.069 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:50:16 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:50:16.727+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T21:50:17.069 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:50:16 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:50:16.727+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T21:50:17.069 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:50:16 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:50:16.727+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T21:50:17.069 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:50:16 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:50:16.728+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:50:17.816 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:50:17 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:50:17.607+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:50:17.817 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:50:17 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:50:17.607+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:50:17.817 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:50:17 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:50:17.607+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:50:17.817 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:50:17 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:50:17.608+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:50:17.817 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:50:17 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:50:17.479+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:50:17.817 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:50:17 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:50:17.479+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:50:17.817 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:50:17 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:50:17.479+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:50:17.817 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:50:17 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:50:17.480+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:50:17.817 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:50:17 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:50:17.722+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T21:50:17.818 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:50:17 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:50:17.722+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T21:50:17.818 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:50:17 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:50:17.722+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T21:50:17.818 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:50:17 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:50:17.723+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:50:18.816 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:50:18 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:50:18.574+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:50:18.816 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:50:18 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:50:18.574+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:50:18.817 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:50:18 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:50:18.574+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:50:18.817 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:50:18 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:50:18.575+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:50:18.817 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:50:18 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:50:18.736+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T21:50:18.817 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:50:18 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:50:18.736+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T21:50:18.817 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:50:18 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:50:18.736+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T21:50:18.817 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:50:18 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:50:18.737+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:50:18.817 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:50:18 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:50:18.507+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:50:18.817 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:50:18 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:50:18.507+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:50:18.818 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:50:18 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:50:18.507+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:50:18.818 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:50:18 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:50:18.508+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:50:19.816 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:50:19 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:50:19.526+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:50:19.817 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:50:19 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:50:19.526+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:50:19.817 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:50:19 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:50:19.526+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:50:19.817 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:50:19 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:50:19.527+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:50:19.817 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:50:19 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:50:19.531+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:50:19.817 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:50:19 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:50:19.531+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:50:19.817 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:50:19 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:50:19.531+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:50:19.817 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:50:19 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:50:19.532+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:50:19.818 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:50:19 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:50:19.742+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T21:50:19.818 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:50:19 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:50:19.742+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T21:50:19.818 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:50:19 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:50:19.742+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T21:50:19.818 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:50:19 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:50:19.743+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:50:20.816 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:50:20 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:50:20.525+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:50:20.816 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:50:20 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:50:20.525+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:50:20.816 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:50:20 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:50:20.525+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:50:20.817 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:50:20 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:50:20.526+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:50:20.817 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:50:20 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:50:20.571+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:50:20.817 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:50:20 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:50:20.571+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:50:20.817 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:50:20 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:50:20.571+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:50:20.817 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:50:20 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:50:20.571+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:50:20.817 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:50:20 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:50:20.702+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T21:50:20.817 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:50:20 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:50:20.702+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T21:50:20.817 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:50:20 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:50:20.702+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T21:50:20.818 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:50:20 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:50:20.703+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:50:21.666 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:50:21 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:50:21.525+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:50:21.666 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:50:21 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:50:21.525+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:50:21.666 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:50:21 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:50:21.525+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:50:21.666 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:50:21 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:50:21.526+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:50:21.667 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:50:21 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:50:21.527+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:50:21.667 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:50:21 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:50:21.527+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:50:21.667 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:50:21 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:50:21.527+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:50:21.667 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:50:21 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:50:21.528+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:50:21.667 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Dec 16 21:50:21 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-mon-smithi045[80759]: 2023-12-16T21:50:21.412+0000 7f45a1f8f700 -1 mon.smithi045@0(probing) e2 get_health_metrics reporting 8682 slow ops, oldest is monmgrreport(gid 14214, 0 checks, 1 progress events) 2023-12-16T21:50:21.969 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:50:21 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:50:21.739+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T21:50:21.969 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:50:21 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:50:21.739+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T21:50:21.969 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:50:21 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:50:21.739+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T21:50:21.969 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:50:21 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:50:21.741+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:50:22.816 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:50:22 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:50:22.501+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:50:22.816 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:50:22 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:50:22.501+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:50:22.816 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:50:22 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:50:22.501+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:50:22.817 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:50:22 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:50:22.501+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:50:22.817 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:50:22 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:50:22.559+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:50:22.817 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:50:22 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:50:22.559+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:50:22.817 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:50:22 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:50:22.559+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:50:22.817 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:50:22 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:50:22.559+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:50:22.817 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:50:22 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:50:22.745+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T21:50:22.817 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:50:22 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:50:22.745+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T21:50:22.817 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:50:22 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:50:22.745+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T21:50:22.818 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:50:22 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:50:22.746+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:50:23.816 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:50:23 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:50:23.765+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T21:50:23.816 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:50:23 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:50:23.765+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T21:50:23.816 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:50:23 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:50:23.765+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T21:50:23.817 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:50:23 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:50:23.766+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:50:23.817 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:50:23 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:50:23.544+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:50:23.817 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:50:23 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:50:23.544+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:50:23.817 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:50:23 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:50:23.544+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:50:23.817 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:50:23 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:50:23.544+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:50:23.817 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:50:23 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:50:23.523+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:50:23.818 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:50:23 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:50:23.523+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:50:23.818 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:50:23 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:50:23.523+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:50:23.818 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:50:23 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:50:23.524+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:50:24.816 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:50:24 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:50:24.513+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:50:24.816 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:50:24 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:50:24.513+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:50:24.816 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:50:24 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:50:24.513+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:50:24.817 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:50:24 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:50:24.514+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:50:24.817 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:50:24 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:50:24.734+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T21:50:24.817 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:50:24 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:50:24.734+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T21:50:24.817 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:50:24 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:50:24.734+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T21:50:24.817 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:50:24 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:50:24.734+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:50:24.817 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:50:24 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:50:24.523+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:50:24.817 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:50:24 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:50:24.523+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:50:24.817 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:50:24 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:50:24.523+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:50:24.818 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:50:24 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:50:24.524+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:50:25.816 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:50:25 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:50:25.509+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:50:25.816 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:50:25 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:50:25.509+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:50:25.817 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:50:25 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:50:25.509+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:50:25.817 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:50:25 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:50:25.510+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:50:25.817 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:50:25 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:50:25.721+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T21:50:25.817 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:50:25 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:50:25.721+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T21:50:25.817 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:50:25 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:50:25.721+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T21:50:25.817 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:50:25 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:50:25.722+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:50:25.817 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:50:25 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:50:25.532+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:50:25.818 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:50:25 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:50:25.532+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:50:25.818 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:50:25 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:50:25.532+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:50:25.818 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:50:25 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:50:25.534+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:50:26.719 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Dec 16 21:50:26 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-mon-smithi045[80759]: 2023-12-16T21:50:26.413+0000 7f45a1f8f700 -1 mon.smithi045@0(probing) e2 get_health_metrics reporting 8776 slow ops, oldest is monmgrreport(gid 14214, 0 checks, 1 progress events) 2023-12-16T21:50:26.719 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:50:26 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:50:26.492+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:50:26.720 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:50:26 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:50:26.492+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:50:26.720 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:50:26 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:50:26.492+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:50:26.720 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:50:26 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:50:26.493+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:50:26.720 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:50:26 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:50:26.522+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:50:26.720 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:50:26 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:50:26.522+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:50:26.720 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:50:26 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:50:26.522+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:50:26.720 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:50:26 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:50:26.524+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:50:27.066 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:50:26 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:50:26.716+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T21:50:27.066 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:50:26 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:50:26.716+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T21:50:27.066 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:50:26 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:50:26.716+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T21:50:27.066 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:50:26 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:50:26.717+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:50:27.766 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:50:27 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:50:27.482+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:50:27.766 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:50:27 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:50:27.482+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:50:27.767 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:50:27 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:50:27.482+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:50:27.767 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:50:27 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:50:27.482+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:50:27.767 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:50:27 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:50:27.533+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:50:27.767 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:50:27 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:50:27.533+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:50:27.767 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:50:27 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:50:27.533+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:50:27.767 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:50:27 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:50:27.534+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:50:28.066 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:50:27 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:50:27.763+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T21:50:28.066 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:50:27 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:50:27.763+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T21:50:28.066 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:50:27 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:50:27.763+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T21:50:28.066 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:50:27 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:50:27.764+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:50:28.722 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:50:28 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:50:28.485+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:50:28.722 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:50:28 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:50:28.485+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:50:28.723 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:50:28 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:50:28.485+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:50:28.723 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:50:28 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:50:28.486+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:50:28.723 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:50:28 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:50:28.456+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:50:28.723 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:50:28 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:50:28.456+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:50:28.723 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:50:28 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:50:28.456+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:50:28.723 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:50:28 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:50:28.456+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:50:29.066 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:50:28 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:50:28.754+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T21:50:29.066 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:50:28 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:50:28.754+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T21:50:29.066 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:50:28 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:50:28.754+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T21:50:29.066 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:50:28 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:50:28.755+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:50:29.720 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:50:29 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:50:29.441+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:50:29.720 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:50:29 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:50:29.441+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:50:29.721 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:50:29 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:50:29.441+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:50:29.721 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:50:29 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:50:29.442+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:50:29.721 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:50:29 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:50:29.407+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:50:29.721 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:50:29 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:50:29.407+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:50:29.721 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:50:29 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:50:29.407+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:50:29.721 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:50:29 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:50:29.408+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:50:30.066 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:50:29 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:50:29.717+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T21:50:30.066 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:50:29 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:50:29.717+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T21:50:30.066 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:50:29 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:50:29.717+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T21:50:30.067 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:50:29 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:50:29.718+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:50:30.702 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:50:30 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:50:30.441+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:50:30.702 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:50:30 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:50:30.441+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:50:30.702 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:50:30 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:50:30.441+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:50:30.702 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:50:30 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:50:30.442+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:50:30.703 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:50:30 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:50:30.376+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:50:30.703 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:50:30 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:50:30.376+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:50:30.703 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:50:30 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:50:30.376+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:50:30.703 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:50:30 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:50:30.376+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:50:31.066 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:50:30 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:50:30.699+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T21:50:31.066 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:50:30 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:50:30.699+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T21:50:31.066 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:50:30 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:50:30.699+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T21:50:31.066 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:50:30 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:50:30.700+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:50:31.666 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Dec 16 21:50:31 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-mon-smithi045[80759]: 2023-12-16T21:50:31.414+0000 7f45a1f8f700 -1 mon.smithi045@0(probing) e2 get_health_metrics reporting 8873 slow ops, oldest is monmgrreport(gid 14214, 0 checks, 1 progress events) 2023-12-16T21:50:31.667 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:50:31 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:50:31.443+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:50:31.667 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:50:31 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:50:31.443+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:50:31.667 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:50:31 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:50:31.443+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:50:31.667 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:50:31 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:50:31.444+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:50:31.667 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:50:31 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:50:31.349+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:50:31.667 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:50:31 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:50:31.349+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:50:31.667 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:50:31 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:50:31.349+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:50:31.667 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:50:31 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:50:31.351+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:50:32.066 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:50:31 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:50:31.721+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T21:50:32.066 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:50:31 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:50:31.721+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T21:50:32.066 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:50:31 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:50:31.721+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T21:50:32.067 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:50:31 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:50:31.721+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:50:32.722 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:50:32 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:50:32.451+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:50:32.722 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:50:32 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:50:32.451+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:50:32.722 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:50:32 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:50:32.451+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:50:32.722 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:50:32 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:50:32.452+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:50:32.722 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:50:32 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:50:32.347+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:50:32.722 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:50:32 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:50:32.347+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:50:32.722 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:50:32 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:50:32.347+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:50:32.722 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:50:32 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:50:32.348+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:50:33.066 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:50:32 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:50:32.719+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T21:50:33.066 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:50:32 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:50:32.719+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T21:50:33.066 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:50:32 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:50:32.719+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T21:50:33.066 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:50:32 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:50:32.720+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:50:33.756 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:50:33 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:50:33.410+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:50:33.756 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:50:33 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:50:33.410+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:50:33.756 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:50:33 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:50:33.410+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:50:33.756 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:50:33 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:50:33.411+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:50:33.756 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:50:33 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:50:33.379+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:50:33.756 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:50:33 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:50:33.379+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:50:33.756 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:50:33 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:50:33.379+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:50:33.756 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:50:33 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:50:33.380+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:50:34.066 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:50:33 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:50:33.753+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T21:50:34.066 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:50:33 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:50:33.753+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T21:50:34.066 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:50:33 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:50:33.753+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T21:50:34.066 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:50:33 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:50:33.754+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:50:34.566 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:50:34 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:50:34.376+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:50:34.566 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:50:34 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:50:34.376+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:50:34.566 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:50:34 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:50:34.376+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:50:34.567 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:50:34 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:50:34.378+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:50:34.567 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:50:34 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:50:34.360+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:50:34.567 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:50:34 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:50:34.360+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:50:34.567 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:50:34 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:50:34.360+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:50:34.567 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:50:34 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:50:34.361+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:50:35.066 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:50:34 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:50:34.756+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T21:50:35.066 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:50:34 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:50:34.756+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T21:50:35.066 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:50:34 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:50:34.756+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T21:50:35.066 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:50:34 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:50:34.757+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:50:35.716 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:50:35 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:50:35.345+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:50:35.717 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:50:35 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:50:35.345+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:50:35.717 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:50:35 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:50:35.345+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:50:35.717 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:50:35 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:50:35.346+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:50:35.717 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:50:35 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:50:35.383+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:50:35.717 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:50:35 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:50:35.383+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:50:35.717 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:50:35 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:50:35.383+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:50:35.717 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:50:35 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:50:35.384+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:50:36.066 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:50:35 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:50:35.713+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T21:50:36.067 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:50:35 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:50:35.713+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T21:50:36.067 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:50:35 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:50:35.713+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T21:50:36.067 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:50:35 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:50:35.714+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:50:36.743 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:50:36 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:50:36.343+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:50:36.744 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:50:36 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:50:36.343+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:50:36.744 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:50:36 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:50:36.343+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:50:36.744 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:50:36 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:50:36.343+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:50:36.744 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:50:36 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:50:36.403+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:50:36.744 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:50:36 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:50:36.403+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:50:36.744 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:50:36 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:50:36.403+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:50:36.744 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:50:36 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:50:36.403+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:50:36.744 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Dec 16 21:50:36 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-mon-smithi045[80759]: 2023-12-16T21:50:36.415+0000 7f45a1f8f700 -1 mon.smithi045@0(probing) e2 get_health_metrics reporting 8968 slow ops, oldest is monmgrreport(gid 14214, 0 checks, 1 progress events) 2023-12-16T21:50:37.066 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:50:36 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:50:36.740+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T21:50:37.066 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:50:36 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:50:36.740+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T21:50:37.066 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:50:36 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:50:36.740+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T21:50:37.067 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:50:36 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:50:36.741+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:50:37.720 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:50:37 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:50:37.357+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:50:37.720 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:50:37 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:50:37.357+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:50:37.720 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:50:37 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:50:37.357+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:50:37.720 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:50:37 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:50:37.357+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:50:37.720 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:50:37 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:50:37.412+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:50:37.720 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:50:37 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:50:37.412+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:50:37.720 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:50:37 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:50:37.412+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:50:37.720 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:50:37 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:50:37.412+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:50:38.066 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:50:37 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:50:37.717+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T21:50:38.066 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:50:37 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:50:37.717+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T21:50:38.066 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:50:37 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:50:37.717+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T21:50:38.066 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:50:37 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:50:37.717+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:50:38.701 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:50:38 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:50:38.354+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:50:38.701 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:50:38 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:50:38.354+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:50:38.701 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:50:38 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:50:38.354+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:50:38.701 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:50:38 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:50:38.355+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:50:38.701 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:50:38 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:50:38.432+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:50:38.701 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:50:38 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:50:38.432+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:50:38.701 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:50:38 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:50:38.432+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:50:38.702 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:50:38 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:50:38.433+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:50:39.066 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:50:38 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:50:38.698+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T21:50:39.066 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:50:38 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:50:38.698+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T21:50:39.066 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:50:38 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:50:38.698+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T21:50:39.066 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:50:38 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:50:38.699+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:50:39.694 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:50:39 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:50:39.398+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:50:39.694 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:50:39 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:50:39.398+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:50:39.694 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:50:39 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:50:39.398+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:50:39.695 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:50:39 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:50:39.399+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:50:39.695 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:50:39 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:50:39.397+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:50:39.695 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:50:39 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:50:39.397+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:50:39.695 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:50:39 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:50:39.397+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:50:39.695 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:50:39 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:50:39.398+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:50:40.066 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:50:39 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:50:39.691+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T21:50:40.066 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:50:39 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:50:39.691+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T21:50:40.067 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:50:39 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:50:39.691+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T21:50:40.067 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:50:39 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:50:39.692+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:50:40.691 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:50:40 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:50:40.431+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:50:40.692 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:50:40 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:50:40.431+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:50:40.692 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:50:40 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:50:40.431+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:50:40.692 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:50:40 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:50:40.432+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:50:40.692 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:50:40 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:50:40.392+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:50:40.692 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:50:40 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:50:40.392+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:50:40.692 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:50:40 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:50:40.392+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:50:40.692 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:50:40 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:50:40.393+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:50:41.066 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:50:40 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:50:40.688+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T21:50:41.066 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:50:40 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:50:40.688+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T21:50:41.066 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:50:40 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:50:40.688+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T21:50:41.066 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:50:40 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:50:40.689+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:50:41.667 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:50:41 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:50:41.361+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:50:41.667 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:50:41 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:50:41.361+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:50:41.667 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:50:41 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:50:41.361+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:50:41.667 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:50:41 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:50:41.362+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:50:41.668 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:50:41 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:50:41.426+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:50:41.668 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:50:41 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:50:41.426+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:50:41.668 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:50:41 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:50:41.426+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:50:41.668 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:50:41 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:50:41.428+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:50:41.668 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Dec 16 21:50:41 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-mon-smithi045[80759]: 2023-12-16T21:50:41.416+0000 7f45a1f8f700 -1 mon.smithi045@0(probing) e2 get_health_metrics reporting 9066 slow ops, oldest is monmgrreport(gid 14214, 0 checks, 1 progress events) 2023-12-16T21:50:42.066 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:50:41 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:50:41.690+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T21:50:42.067 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:50:41 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:50:41.690+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T21:50:42.067 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:50:41 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:50:41.690+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T21:50:42.067 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:50:41 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:50:41.691+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:50:42.649 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:50:42 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:50:42.390+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:50:42.649 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:50:42 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:50:42.390+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:50:42.649 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:50:42 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:50:42.390+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:50:42.649 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:50:42 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:50:42.391+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:50:42.649 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:50:42 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:50:42.399+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:50:42.650 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:50:42 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:50:42.399+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:50:42.650 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:50:42 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:50:42.399+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:50:42.650 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:50:42 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:50:42.400+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:50:43.066 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:50:42 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:50:42.646+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T21:50:43.066 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:50:42 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:50:42.646+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T21:50:43.067 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:50:42 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:50:42.646+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T21:50:43.067 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:50:42 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:50:42.647+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:50:43.685 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:50:43 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:50:43.353+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:50:43.685 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:50:43 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:50:43.353+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:50:43.685 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:50:43 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:50:43.353+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:50:43.685 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:50:43 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:50:43.353+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:50:43.685 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:50:43 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:50:43.381+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:50:43.685 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:50:43 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:50:43.381+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:50:43.686 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:50:43 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:50:43.381+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:50:43.686 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:50:43 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:50:43.382+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:50:44.066 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:50:43 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:50:43.682+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T21:50:44.066 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:50:43 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:50:43.682+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T21:50:44.067 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:50:43 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:50:43.682+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T21:50:44.067 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:50:43 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:50:43.683+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:50:44.566 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:50:44 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:50:44.345+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:50:44.566 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:50:44 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:50:44.345+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:50:44.566 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:50:44 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:50:44.345+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:50:44.567 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:50:44 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:50:44.345+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:50:44.567 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:50:44 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:50:44.304+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:50:44.567 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:50:44 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:50:44.304+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:50:44.567 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:50:44 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:50:44.304+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:50:44.567 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:50:44 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:50:44.304+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:50:45.066 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:50:44 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:50:44.717+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T21:50:45.066 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:50:44 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:50:44.717+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T21:50:45.066 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:50:44 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:50:44.717+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T21:50:45.066 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:50:44 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:50:44.718+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:50:45.566 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:50:45 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:50:45.341+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:50:45.566 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:50:45 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:50:45.341+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:50:45.566 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:50:45 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:50:45.341+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:50:45.566 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:50:45 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:50:45.341+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:50:45.567 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:50:45 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:50:45.272+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:50:45.567 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:50:45 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:50:45.272+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:50:45.567 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:50:45 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:50:45.272+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:50:45.567 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:50:45 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:50:45.273+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:50:46.066 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:50:45 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:50:45.736+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T21:50:46.067 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:50:45 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:50:45.736+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T21:50:46.067 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:50:45 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:50:45.736+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T21:50:46.067 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:50:45 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:50:45.737+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:50:46.543 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Dec 16 21:50:46 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-mon-smithi045[80759]: 2023-12-16T21:50:46.417+0000 7f45a1f8f700 -1 mon.smithi045@0(probing) e2 get_health_metrics reporting 9160 slow ops, oldest is monmgrreport(gid 14214, 0 checks, 1 progress events) 2023-12-16T21:50:46.544 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:50:46 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:50:46.355+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:50:46.544 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:50:46 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:50:46.355+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:50:46.544 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:50:46 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:50:46.355+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:50:46.544 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:50:46 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:50:46.355+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:50:46.544 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:50:46 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:50:46.257+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:50:46.544 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:50:46 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:50:46.257+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:50:46.544 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:50:46 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:50:46.257+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:50:46.545 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:50:46 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:50:46.257+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:50:46.816 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:50:46 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:50:46.726+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T21:50:46.816 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:50:46 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:50:46.726+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T21:50:46.816 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:50:46 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:50:46.726+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T21:50:46.816 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:50:46 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:50:46.727+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:50:47.566 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:50:47 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:50:47.392+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:50:47.566 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:50:47 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:50:47.392+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:50:47.566 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:50:47 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:50:47.392+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:50:47.567 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:50:47 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:50:47.392+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:50:47.567 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:50:47 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:50:47.267+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:50:47.567 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:50:47 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:50:47.267+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:50:47.567 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:50:47 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:50:47.267+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:50:47.567 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:50:47 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:50:47.268+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:50:48.066 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:50:47 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:50:47.738+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T21:50:48.067 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:50:47 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:50:47.738+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T21:50:48.067 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:50:47 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:50:47.738+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T21:50:48.067 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:50:47 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:50:47.739+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:50:48.566 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:50:48 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:50:48.273+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:50:48.566 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:50:48 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:50:48.273+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:50:48.566 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:50:48 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:50:48.273+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:50:48.566 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:50:48 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:50:48.273+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:50:48.567 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:50:48 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:50:48.384+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:50:48.567 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:50:48 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:50:48.384+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:50:48.567 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:50:48 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:50:48.384+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:50:48.567 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:50:48 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:50:48.385+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:50:49.066 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:50:48 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:50:48.778+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T21:50:49.066 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:50:48 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:50:48.778+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T21:50:49.066 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:50:48 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:50:48.778+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T21:50:49.067 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:50:48 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:50:48.779+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:50:49.346 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:50:49 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:50:49.282+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:50:49.346 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:50:49 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:50:49.282+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:50:49.346 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:50:49 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:50:49.282+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:50:49.347 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:50:49 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:50:49.283+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:50:49.795 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:50:49 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:50:49.343+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:50:49.795 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:50:49 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:50:49.343+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:50:49.796 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:50:49 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:50:49.343+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:50:49.796 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:50:49 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:50:49.345+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:50:50.066 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:50:49 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:50:49.792+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T21:50:50.066 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:50:49 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:50:49.792+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T21:50:50.066 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:50:49 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:50:49.792+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T21:50:50.066 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:50:49 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:50:49.793+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:50:50.566 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:50:50 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:50:50.258+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:50:50.566 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:50:50 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:50:50.258+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:50:50.566 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:50:50 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:50:50.258+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:50:50.567 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:50:50 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:50:50.259+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:50:50.567 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:50:50 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:50:50.327+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:50:50.567 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:50:50 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:50:50.327+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:50:50.567 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:50:50 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:50:50.327+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:50:50.567 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:50:50 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:50:50.329+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:50:51.066 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:50:50 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:50:50.747+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T21:50:51.067 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:50:50 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:50:50.747+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T21:50:51.067 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:50:50 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:50:50.747+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T21:50:51.067 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:50:50 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:50:50.747+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:50:51.566 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Dec 16 21:50:51 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-mon-smithi045[80759]: 2023-12-16T21:50:51.419+0000 7f45a1f8f700 -1 mon.smithi045@0(probing) e2 get_health_metrics reporting 9258 slow ops, oldest is monmgrreport(gid 14214, 0 checks, 1 progress events) 2023-12-16T21:50:51.567 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:50:51 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:50:51.298+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:50:51.567 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:50:51 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:50:51.298+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:50:51.567 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:50:51 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:50:51.298+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:50:51.567 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:50:51 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:50:51.299+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:50:51.567 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:50:51 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:50:51.323+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:50:51.567 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:50:51 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:50:51.323+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:50:51.567 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:50:51 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:50:51.323+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:50:51.567 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:50:51 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:50:51.324+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:50:52.066 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:50:51 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:50:51.781+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T21:50:52.066 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:50:51 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:50:51.781+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T21:50:52.066 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:50:51 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:50:51.781+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T21:50:52.067 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:50:51 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:50:51.781+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:50:52.689 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:50:52 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:50:52.348+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:50:52.689 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:50:52 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:50:52.348+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:50:52.689 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:50:52 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:50:52.348+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:50:52.689 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:50:52 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:50:52.349+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:50:52.689 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:50:52 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:50:52.334+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:50:52.689 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:50:52 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:50:52.334+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:50:52.690 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:50:52 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:50:52.334+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:50:52.690 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:50:52 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:50:52.335+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:50:53.066 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:50:52 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:50:52.771+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T21:50:53.066 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:50:52 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:50:52.771+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T21:50:53.066 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:50:52 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:50:52.771+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T21:50:53.066 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:50:52 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:50:52.772+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:50:53.558 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:50:53 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:50:53.301+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:50:53.558 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:50:53 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:50:53.301+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:50:53.558 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:50:53 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:50:53.301+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:50:53.558 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:50:53 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:50:53.302+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:50:53.558 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:50:53 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:50:53.300+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:50:53.558 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:50:53 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:50:53.300+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:50:53.559 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:50:53 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:50:53.300+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:50:53.559 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:50:53 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:50:53.301+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:50:53.816 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:50:53 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:50:53.796+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T21:50:53.817 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:50:53 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:50:53.796+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T21:50:53.817 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:50:53 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:50:53.796+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T21:50:53.817 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:50:53 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:50:53.796+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:50:54.566 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:50:54 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:50:54.296+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:50:54.566 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:50:54 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:50:54.296+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:50:54.566 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:50:54 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:50:54.296+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:50:54.566 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:50:54 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:50:54.296+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:50:54.567 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:50:54 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:50:54.302+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:50:54.567 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:50:54 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:50:54.302+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:50:54.567 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:50:54 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:50:54.302+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:50:54.567 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:50:54 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:50:54.304+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:50:55.302 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:50:54 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:50:54.832+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T21:50:55.302 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:50:54 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:50:54.833+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T21:50:55.302 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:50:54 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:50:54.833+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T21:50:55.302 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:50:54 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:50:54.834+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:50:55.566 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:50:55 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:50:55.338+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:50:55.566 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:50:55 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:50:55.338+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:50:55.566 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:50:55 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:50:55.338+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:50:55.566 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:50:55 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:50:55.338+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:50:55.567 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:50:55 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:50:55.298+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:50:55.567 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:50:55 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:50:55.298+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:50:55.567 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:50:55 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:50:55.298+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:50:55.567 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:50:55 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:50:55.299+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:50:56.066 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:50:55 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:50:55.861+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T21:50:56.066 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:50:55 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:50:55.861+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T21:50:56.066 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:50:55 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:50:55.861+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T21:50:56.066 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:50:55 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:50:55.862+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:50:56.816 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Dec 16 21:50:56 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-mon-smithi045[80759]: 2023-12-16T21:50:56.419+0000 7f45a1f8f700 -1 mon.smithi045@0(probing) e2 get_health_metrics reporting 9352 slow ops, oldest is monmgrreport(gid 14214, 0 checks, 1 progress events) 2023-12-16T21:50:56.816 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:50:56 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:50:56.317+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:50:56.816 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:50:56 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:50:56.317+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:50:56.816 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:50:56 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:50:56.317+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:50:56.817 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:50:56 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:50:56.317+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:50:56.817 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:50:56 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:50:56.342+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:50:56.817 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:50:56 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:50:56.342+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:50:56.817 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:50:56 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:50:56.342+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:50:56.817 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:50:56 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:50:56.343+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:50:57.285 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:50:56 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:50:56.836+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T21:50:57.285 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:50:56 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:50:56.836+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T21:50:57.285 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:50:56 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:50:56.836+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T21:50:57.285 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:50:56 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:50:56.837+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:50:57.566 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:50:57 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:50:57.282+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:50:57.566 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:50:57 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:50:57.282+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:50:57.566 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:50:57 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:50:57.282+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:50:57.566 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:50:57 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:50:57.282+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:50:57.567 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:50:57 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:50:57.310+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:50:57.567 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:50:57 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:50:57.310+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:50:57.567 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:50:57 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:50:57.310+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:50:57.567 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:50:57 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:50:57.310+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:50:58.276 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:50:57 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:50:57.841+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T21:50:58.277 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:50:57 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:50:57.841+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T21:50:58.277 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:50:57 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:50:57.841+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T21:50:58.277 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:50:57 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:50:57.842+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:50:58.566 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:50:58 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:50:58.274+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:50:58.566 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:50:58 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:50:58.274+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:50:58.566 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:50:58 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:50:58.274+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:50:58.566 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:50:58 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:50:58.274+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:50:58.567 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:50:58 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:50:58.314+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:50:58.567 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:50:58 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:50:58.314+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:50:58.567 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:50:58 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:50:58.314+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:50:58.567 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:50:58 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:50:58.315+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:50:59.066 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:50:58 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:50:58.839+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T21:50:59.066 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:50:58 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:50:58.839+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T21:50:59.066 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:50:58 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:50:58.839+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T21:50:59.067 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:50:58 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:50:58.840+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:50:59.566 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:50:59 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:50:59.268+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:50:59.566 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:50:59 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:50:59.268+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:50:59.566 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:50:59 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:50:59.268+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:50:59.566 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:50:59 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:50:59.268+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:50:59.567 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:50:59 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:50:59.228+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:50:59.567 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:50:59 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:50:59.228+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:50:59.567 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:50:59 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:50:59.228+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:50:59.567 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:50:59 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:50:59.229+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:51:00.222 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:50:59 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:50:59.834+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T21:51:00.222 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:50:59 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:50:59.834+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T21:51:00.223 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:50:59 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:50:59.834+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T21:51:00.223 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:50:59 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:50:59.835+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:51:00.566 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:51:00 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:51:00.219+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:51:00.566 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:51:00 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:51:00.219+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:51:00.566 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:51:00 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:51:00.219+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:51:00.566 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:51:00 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:51:00.221+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:51:00.567 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:51:00 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:51:00.238+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:51:00.567 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:51:00 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:51:00.238+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:51:00.567 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:51:00 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:51:00.238+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:51:00.567 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:51:00 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:51:00.239+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:51:01.235 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:51:00 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:51:00.852+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T21:51:01.235 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:51:00 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:51:00.852+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T21:51:01.235 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:51:00 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:51:00.852+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T21:51:01.235 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:51:00 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:51:00.853+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:51:01.548 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:51:01 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:51:01.257+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:51:01.548 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:51:01 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:51:01.257+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:51:01.548 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:51:01 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:51:01.257+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:51:01.548 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:51:01 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:51:01.257+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:51:01.549 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:51:01 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:51:01.232+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:51:01.549 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:51:01 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:51:01.232+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:51:01.549 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:51:01 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:51:01.232+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:51:01.549 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:51:01 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:51:01.233+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:51:01.550 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Dec 16 21:51:01 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-mon-smithi045[80759]: 2023-12-16T21:51:01.421+0000 7f45a1f8f700 -1 mon.smithi045@0(probing) e2 get_health_metrics reporting 9450 slow ops, oldest is monmgrreport(gid 14214, 0 checks, 1 progress events) 2023-12-16T21:51:02.217 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:51:01 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:51:01.819+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T21:51:02.217 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:51:01 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:51:01.819+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T21:51:02.217 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:51:01 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:51:01.819+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T21:51:02.217 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:51:01 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:51:01.820+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:51:02.566 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:51:02 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:51:02.235+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:51:02.566 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:51:02 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:51:02.235+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:51:02.566 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:51:02 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:51:02.235+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:51:02.566 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:51:02 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:51:02.236+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:51:02.567 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:51:02 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:51:02.214+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:51:02.567 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:51:02 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:51:02.214+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:51:02.567 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:51:02 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:51:02.214+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:51:02.567 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:51:02 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:51:02.215+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:51:03.195 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:51:02 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:51:02.836+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T21:51:03.195 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:51:02 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:51:02.836+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T21:51:03.195 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:51:02 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:51:02.836+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T21:51:03.195 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:51:02 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:51:02.837+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:51:03.559 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:51:03 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:51:03.213+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:51:03.559 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:51:03 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:51:03.213+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:51:03.559 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:51:03 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:51:03.213+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:51:03.559 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:51:03 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:51:03.213+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:51:03.559 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:51:03 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:51:03.192+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:51:03.560 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:51:03 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:51:03.192+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:51:03.560 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:51:03 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:51:03.192+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:51:03.560 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:51:03 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:51:03.193+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:51:04.150 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:51:03 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:51:03.881+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T21:51:04.150 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:51:03 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:51:03.881+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T21:51:04.150 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:51:03 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:51:03.881+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T21:51:04.150 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:51:03 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:51:03.882+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:51:04.567 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:51:04 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:51:04.246+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:51:04.567 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:51:04 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:51:04.246+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:51:04.567 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:51:04 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:51:04.246+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:51:04.567 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:51:04 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:51:04.247+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:51:04.567 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:51:04 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:51:04.147+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:51:04.567 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:51:04 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:51:04.147+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:51:04.567 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:51:04 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:51:04.147+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:51:04.567 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:51:04 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:51:04.148+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:51:05.175 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:51:04 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:51:04.905+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T21:51:05.175 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:51:04 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:51:04.905+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T21:51:05.175 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:51:04 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:51:04.905+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T21:51:05.175 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:51:04 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:51:04.906+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:51:05.566 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:51:05 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:51:05.172+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:51:05.566 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:51:05 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:51:05.172+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:51:05.566 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:51:05 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:51:05.172+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:51:05.566 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:51:05 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:51:05.173+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:51:05.566 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:51:05 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:51:05.251+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:51:05.567 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:51:05 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:51:05.251+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:51:05.567 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:51:05 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:51:05.251+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:51:05.567 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:51:05 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:51:05.252+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:51:06.294 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:51:06 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:51:06.139+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:51:06.294 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:51:06 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:51:06.139+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:51:06.294 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:51:06 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:51:06.139+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:51:06.295 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:51:06 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:51:06.140+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:51:06.295 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:51:05 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:51:05.938+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T21:51:06.295 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:51:05 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:51:05.938+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T21:51:06.295 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:51:05 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:51:05.938+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T21:51:06.295 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:51:05 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:51:05.940+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:51:06.548 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Dec 16 21:51:06 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-mon-smithi045[80759]: 2023-12-16T21:51:06.422+0000 7f45a1f8f700 -1 mon.smithi045@0(probing) e2 get_health_metrics reporting 9545 slow ops, oldest is monmgrreport(gid 14214, 0 checks, 1 progress events) 2023-12-16T21:51:06.549 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:51:06 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:51:06.291+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:51:06.549 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:51:06 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:51:06.291+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:51:06.549 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:51:06 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:51:06.291+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:51:06.549 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:51:06 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:51:06.292+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:51:07.316 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:51:07 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:51:07.155+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:51:07.316 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:51:07 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:51:07.155+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:51:07.316 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:51:07 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:51:07.155+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:51:07.316 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:51:07 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:51:07.156+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:51:07.317 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:51:06 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:51:06.919+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T21:51:07.317 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:51:06 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:51:06.919+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T21:51:07.317 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:51:06 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:51:06.919+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T21:51:07.317 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:51:06 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:51:06.920+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:51:07.816 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:51:07 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:51:07.340+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:51:07.816 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:51:07 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:51:07.340+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:51:07.816 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:51:07 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:51:07.340+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:51:07.816 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:51:07 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:51:07.341+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:51:08.308 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:51:07 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:51:07.952+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T21:51:08.308 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:51:07 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:51:07.952+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T21:51:08.308 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:51:07 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:51:07.952+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T21:51:08.308 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:51:07 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:51:07.954+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:51:08.308 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:51:08 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:51:08.166+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:51:08.308 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:51:08 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:51:08.166+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:51:08.308 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:51:08 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:51:08.166+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:51:08.308 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:51:08 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:51:08.167+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:51:08.566 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:51:08 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:51:08.305+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:51:08.566 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:51:08 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:51:08.305+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:51:08.566 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:51:08 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:51:08.305+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:51:08.566 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:51:08 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:51:08.306+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:51:09.316 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:51:09 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:51:09.203+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:51:09.316 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:51:09 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:51:09.203+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:51:09.316 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:51:09 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:51:09.203+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:51:09.317 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:51:09 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:51:09.203+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:51:09.317 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:51:08 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:51:08.988+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T21:51:09.317 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:51:08 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:51:08.988+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T21:51:09.317 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:51:08 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:51:08.988+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T21:51:09.317 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:51:08 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:51:08.989+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:51:09.816 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:51:09 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:51:09.317+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:51:09.816 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:51:09 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:51:09.317+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:51:09.816 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:51:09 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:51:09.317+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:51:09.816 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:51:09 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:51:09.318+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:51:10.316 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:51:10 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:51:10.242+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:51:10.316 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:51:10 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:51:10.242+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:51:10.316 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:51:10 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:51:10.242+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:51:10.316 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:51:10 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:51:10.242+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:51:10.317 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:51:10 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:51:10.031+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T21:51:10.317 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:51:10 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:51:10.031+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T21:51:10.317 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:51:10 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:51:10.031+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T21:51:10.317 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:51:10 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:51:10.032+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:51:10.816 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:51:10 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:51:10.327+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:51:10.816 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:51:10 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:51:10.327+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:51:10.816 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:51:10 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:51:10.327+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:51:10.816 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:51:10 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:51:10.328+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:51:11.316 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:51:11 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:51:11.204+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:51:11.316 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:51:11 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:51:11.204+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:51:11.316 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:51:11 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:51:11.204+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:51:11.316 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:51:11 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:51:11.204+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:51:11.317 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:51:11 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:51:11.000+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T21:51:11.317 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:51:11 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:51:11.000+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T21:51:11.317 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:51:11 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:51:11.000+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T21:51:11.317 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:51:11 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:51:11.001+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:51:11.669 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Dec 16 21:51:11 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-mon-smithi045[80759]: 2023-12-16T21:51:11.423+0000 7f45a1f8f700 -1 mon.smithi045@0(probing) e2 get_health_metrics reporting 9644 slow ops, oldest is monmgrreport(gid 14214, 0 checks, 1 progress events) 2023-12-16T21:51:11.670 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:51:11 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:51:11.329+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:51:11.670 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:51:11 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:51:11.329+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:51:11.670 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:51:11 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:51:11.329+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:51:11.670 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:51:11 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:51:11.330+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:51:12.316 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:51:12 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:51:12.236+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:51:12.316 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:51:12 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:51:12.236+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:51:12.316 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:51:12 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:51:12.236+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:51:12.316 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:51:12 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:51:12.237+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:51:12.317 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:51:12 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:51:12.043+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T21:51:12.317 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:51:12 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:51:12.043+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T21:51:12.317 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:51:12 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:51:12.043+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T21:51:12.317 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:51:12 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:51:12.043+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:51:12.816 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:51:12 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:51:12.355+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:51:12.816 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:51:12 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:51:12.355+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:51:12.816 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:51:12 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:51:12.355+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:51:12.816 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:51:12 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:51:12.356+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:51:13.316 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:51:13 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:51:13.012+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T21:51:13.317 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:51:13 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:51:13.012+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T21:51:13.317 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:51:13 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:51:13.012+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T21:51:13.317 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:51:13 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:51:13.013+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:51:13.317 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:51:13 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:51:13.234+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:51:13.317 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:51:13 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:51:13.234+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:51:13.317 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:51:13 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:51:13.234+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:51:13.317 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:51:13 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:51:13.235+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:51:13.816 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:51:13 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:51:13.398+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:51:13.817 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:51:13 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:51:13.398+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:51:13.817 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:51:13 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:51:13.398+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:51:13.817 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:51:13 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:51:13.399+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:51:14.316 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:51:14 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:51:14.253+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:51:14.316 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:51:14 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:51:14.253+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:51:14.316 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:51:14 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:51:14.253+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:51:14.316 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:51:14 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:51:14.253+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:51:14.317 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:51:14 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:51:14.036+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T21:51:14.317 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:51:14 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:51:14.036+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T21:51:14.317 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:51:14 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:51:14.036+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T21:51:14.317 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:51:14 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:51:14.037+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:51:14.816 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:51:14 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:51:14.401+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:51:14.816 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:51:14 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:51:14.401+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:51:14.816 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:51:14 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:51:14.401+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:51:14.816 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:51:14 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:51:14.401+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:51:15.265 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:51:15 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:51:15.008+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T21:51:15.265 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:51:15 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:51:15.008+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T21:51:15.265 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:51:15 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:51:15.008+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T21:51:15.265 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:51:15 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:51:15.008+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:51:15.566 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:51:15 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:51:15.262+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:51:15.566 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:51:15 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:51:15.262+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:51:15.566 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:51:15 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:51:15.262+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:51:15.566 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:51:15 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:51:15.263+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:51:15.567 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:51:15 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:51:15.415+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:51:15.567 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:51:15 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:51:15.415+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:51:15.567 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:51:15 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:51:15.415+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:51:15.567 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:51:15 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:51:15.416+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:51:16.296 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:51:15 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:51:15.992+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T21:51:16.296 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:51:15 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:51:15.992+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T21:51:16.296 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:51:15 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:51:15.992+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T21:51:16.296 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:51:15 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:51:15.993+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:51:16.553 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Dec 16 21:51:16 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-mon-smithi045[80759]: 2023-12-16T21:51:16.425+0000 7f45a1f8f700 -1 mon.smithi045@0(probing) e2 get_health_metrics reporting 9739 slow ops, oldest is monmgrreport(gid 14214, 0 checks, 1 progress events) 2023-12-16T21:51:16.554 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:51:16 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:51:16.405+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:51:16.554 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:51:16 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:51:16.405+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:51:16.554 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:51:16 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:51:16.405+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:51:16.554 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:51:16 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:51:16.405+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:51:16.554 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:51:16 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:51:16.293+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:51:16.554 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:51:16 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:51:16.293+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:51:16.554 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:51:16 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:51:16.293+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:51:16.554 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:51:16 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:51:16.294+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:51:17.265 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:51:16 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:51:16.993+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T21:51:17.265 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:51:16 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:51:16.993+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T21:51:17.266 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:51:16 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:51:16.993+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T21:51:17.266 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:51:16 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:51:16.993+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:51:17.816 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:51:17 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:51:17.447+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:51:17.816 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:51:17 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:51:17.447+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:51:17.816 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:51:17 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:51:17.447+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:51:17.816 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:51:17 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:51:17.447+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:51:17.816 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:51:17 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:51:17.324+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:51:17.817 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:51:17 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:51:17.324+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:51:17.817 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:51:17 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:51:17.324+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:51:17.817 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:51:17 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:51:17.325+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:51:18.316 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:51:18 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:51:18.002+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T21:51:18.317 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:51:18 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:51:18.002+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T21:51:18.317 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:51:18 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:51:18.002+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T21:51:18.317 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:51:18 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:51:18.003+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:51:18.722 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:51:18 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:51:18.470+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:51:18.722 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:51:18 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:51:18.470+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:51:18.722 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:51:18 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:51:18.470+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:51:18.722 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:51:18 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:51:18.471+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:51:18.722 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:51:18 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:51:18.352+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:51:18.722 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:51:18 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:51:18.352+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:51:18.722 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:51:18 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:51:18.352+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:51:18.723 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:51:18 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:51:18.353+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:51:19.066 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:51:18 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:51:18.955+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T21:51:19.066 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:51:18 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:51:18.955+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T21:51:19.066 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:51:18 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:51:18.955+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T21:51:19.067 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:51:18 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:51:18.955+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:51:19.816 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:51:19 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:51:19.400+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:51:19.816 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:51:19 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:51:19.400+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:51:19.816 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:51:19 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:51:19.400+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:51:19.816 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:51:19 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:51:19.401+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:51:19.817 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:51:19 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:51:19.450+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:51:19.817 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:51:19 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:51:19.450+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:51:19.817 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:51:19 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:51:19.450+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:51:19.817 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:51:19 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:51:19.451+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:51:20.316 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:51:19 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:51:19.975+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T21:51:20.316 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:51:19 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:51:19.975+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T21:51:20.316 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:51:19 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:51:19.975+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T21:51:20.316 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:51:19 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:51:19.976+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:51:20.816 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:51:20 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:51:20.354+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:51:20.816 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:51:20 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:51:20.354+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:51:20.816 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:51:20 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:51:20.354+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:51:20.816 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:51:20 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:51:20.354+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:51:20.817 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:51:20 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:51:20.417+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:51:20.817 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:51:20 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:51:20.417+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:51:20.817 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:51:20 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:51:20.417+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:51:20.817 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:51:20 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:51:20.418+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:51:21.316 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:51:21 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:51:21.002+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T21:51:21.316 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:51:21 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:51:21.002+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T21:51:21.316 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:51:21 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:51:21.002+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T21:51:21.316 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:51:21 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:51:21.003+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:51:21.669 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:51:21 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:51:21.435+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:51:21.669 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:51:21 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:51:21.435+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:51:21.670 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:51:21 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:51:21.435+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:51:21.670 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:51:21 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:51:21.437+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:51:21.670 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:51:21 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:51:21.354+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:51:21.670 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:51:21 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:51:21.354+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:51:21.670 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:51:21 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:51:21.354+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:51:21.670 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:51:21 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:51:21.355+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:51:21.670 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Dec 16 21:51:21 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-mon-smithi045[80759]: 2023-12-16T21:51:21.426+0000 7f45a1f8f700 -1 mon.smithi045@0(probing) e2 get_health_metrics reporting 9838 slow ops, oldest is monmgrreport(gid 14214, 0 checks, 1 progress events) 2023-12-16T21:51:22.316 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:51:21 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:51:21.975+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T21:51:22.316 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:51:21 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:51:21.975+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T21:51:22.316 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:51:21 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:51:21.975+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T21:51:22.316 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:51:21 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:51:21.976+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:51:22.816 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:51:22 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:51:22.401+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:51:22.816 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:51:22 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:51:22.401+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:51:22.816 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:51:22 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:51:22.401+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:51:22.816 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:51:22 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:51:22.402+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:51:22.817 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:51:22 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:51:22.477+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:51:22.817 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:51:22 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:51:22.477+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:51:22.817 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:51:22 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:51:22.477+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:51:22.817 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:51:22 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:51:22.479+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:51:23.316 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:51:22 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:51:22.980+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T21:51:23.316 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:51:22 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:51:22.980+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T21:51:23.316 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:51:22 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:51:22.980+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T21:51:23.316 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:51:22 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:51:22.981+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:51:23.816 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:51:23 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:51:23.372+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:51:23.817 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:51:23 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:51:23.372+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:51:23.817 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:51:23 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:51:23.372+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:51:23.817 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:51:23 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:51:23.373+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:51:23.817 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:51:23 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:51:23.487+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:51:23.818 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:51:23 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:51:23.487+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:51:23.818 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:51:23 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:51:23.487+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:51:23.818 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:51:23 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:51:23.488+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:51:24.316 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:51:24 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:51:24.007+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T21:51:24.316 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:51:24 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:51:24.007+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T21:51:24.316 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:51:24 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:51:24.007+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T21:51:24.316 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:51:24 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:51:24.008+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:51:24.816 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:51:24 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:51:24.497+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:51:24.816 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:51:24 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:51:24.497+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:51:24.817 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:51:24 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:51:24.497+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:51:24.817 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:51:24 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:51:24.498+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:51:24.817 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:51:24 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:51:24.342+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:51:24.817 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:51:24 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:51:24.342+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:51:24.817 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:51:24 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:51:24.342+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:51:24.817 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:51:24 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:51:24.344+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:51:25.316 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:51:24 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:51:24.973+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T21:51:25.316 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:51:24 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:51:24.973+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T21:51:25.316 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:51:24 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:51:24.973+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T21:51:25.316 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:51:24 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:51:24.975+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:51:25.816 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:51:25 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:51:25.472+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:51:25.817 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:51:25 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:51:25.472+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:51:25.817 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:51:25 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:51:25.472+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:51:25.817 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:51:25 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:51:25.473+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:51:25.817 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:51:25 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:51:25.369+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:51:25.817 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:51:25 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:51:25.369+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:51:25.817 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:51:25 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:51:25.369+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:51:25.817 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:51:25 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:51:25.370+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:51:26.316 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:51:25 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:51:25.933+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T21:51:26.316 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:51:25 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:51:25.933+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T21:51:26.316 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:51:25 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:51:25.933+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T21:51:26.316 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:51:25 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:51:25.933+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:51:26.816 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Dec 16 21:51:26 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-mon-smithi045[80759]: 2023-12-16T21:51:26.427+0000 7f45a1f8f700 -1 mon.smithi045@0(probing) e2 get_health_metrics reporting 9933 slow ops, oldest is monmgrreport(gid 14214, 0 checks, 1 progress events) 2023-12-16T21:51:26.816 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:51:26 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:51:26.509+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:51:26.816 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:51:26 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:51:26.509+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:51:26.816 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:51:26 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:51:26.509+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:51:26.816 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:51:26 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:51:26.509+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:51:26.817 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:51:26 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:51:26.362+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:51:26.817 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:51:26 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:51:26.362+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:51:26.817 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:51:26 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:51:26.362+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:51:26.817 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:51:26 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:51:26.363+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:51:27.316 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:51:26 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:51:26.913+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T21:51:27.316 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:51:26 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:51:26.913+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T21:51:27.316 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:51:26 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:51:26.913+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T21:51:27.316 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:51:26 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:51:26.913+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:51:27.816 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:51:27 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:51:27.335+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:51:27.816 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:51:27 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:51:27.335+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:51:27.816 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:51:27 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:51:27.335+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:51:27.817 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:51:27 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:51:27.336+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:51:27.817 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:51:27 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:51:27.498+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:51:27.817 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:51:27 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:51:27.498+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:51:27.817 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:51:27 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:51:27.498+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:51:27.817 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:51:27 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:51:27.499+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:51:28.316 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:51:27 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:51:27.921+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T21:51:28.316 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:51:27 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:51:27.921+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T21:51:28.316 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:51:27 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:51:27.921+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T21:51:28.316 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:51:27 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:51:27.922+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:51:28.723 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:51:28 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:51:28.506+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:51:28.723 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:51:28 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:51:28.506+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:51:28.723 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:51:28 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:51:28.506+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:51:28.723 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:51:28 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:51:28.507+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:51:28.723 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:51:28 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:51:28.346+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:51:28.723 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:51:28 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:51:28.346+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:51:28.723 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:51:28 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:51:28.346+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:51:28.723 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:51:28 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:51:28.347+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:51:29.066 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:51:28 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:51:28.931+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T21:51:29.066 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:51:28 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:51:28.931+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T21:51:29.067 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:51:28 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:51:28.931+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T21:51:29.067 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:51:28 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:51:28.932+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:51:29.566 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:51:29 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:51:29.530+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:51:29.566 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:51:29 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:51:29.530+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:51:29.566 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:51:29 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:51:29.530+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:51:29.566 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:51:29 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:51:29.531+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:51:29.566 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:51:29 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:51:29.308+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:51:29.566 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:51:29 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:51:29.308+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:51:29.566 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:51:29 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:51:29.308+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:51:29.567 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:51:29 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:51:29.309+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:51:30.267 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:51:29 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:51:29.927+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T21:51:30.267 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:51:29 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:51:29.927+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T21:51:30.267 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:51:29 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:51:29.927+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T21:51:30.267 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:51:29 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:51:29.929+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:51:30.566 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:51:30 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:51:30.264+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:51:30.566 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:51:30 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:51:30.264+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:51:30.566 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:51:30 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:51:30.264+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:51:30.566 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:51:30 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:51:30.265+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:51:30.900 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:51:30 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:51:30.571+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:51:30.900 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:51:30 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:51:30.571+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:51:30.900 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:51:30 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:51:30.571+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:51:30.900 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:51:30 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:51:30.573+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:51:31.221 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:51:30 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:51:30.897+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T21:51:31.221 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:51:30 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:51:30.897+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T21:51:31.221 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:51:30 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:51:30.897+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T21:51:31.221 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:51:30 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:51:30.898+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:51:31.547 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Dec 16 21:51:31 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-mon-smithi045[80759]: 2023-12-16T21:51:31.428+0000 7f45a1f8f700 -1 mon.smithi045@0(probing) e2 get_health_metrics reporting 10028 slow ops, oldest is monmgrreport(gid 14214, 0 checks, 1 progress events) 2023-12-16T21:51:31.547 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:51:31 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:51:31.218+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:51:31.547 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:51:31 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:51:31.218+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:51:31.547 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:51:31 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:51:31.218+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:51:31.547 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:51:31 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:51:31.219+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:51:31.816 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:51:31 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:51:31.544+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:51:31.816 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:51:31 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:51:31.544+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:51:31.816 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:51:31 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:51:31.544+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:51:31.816 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:51:31 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:51:31.545+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:51:32.231 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:51:31 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:51:31.891+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T21:51:32.231 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:51:31 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:51:31.891+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T21:51:32.231 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:51:31 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:51:31.891+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T21:51:32.231 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:51:31 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:51:31.892+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:51:32.514 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:51:32 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:51:32.228+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:51:32.514 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:51:32 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:51:32.228+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:51:32.514 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:51:32 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:51:32.228+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:51:32.514 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:51:32 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:51:32.229+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:51:32.816 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:51:32 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:51:32.511+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:51:32.816 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:51:32 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:51:32.511+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:51:32.816 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:51:32 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:51:32.511+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:51:32.816 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:51:32 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:51:32.512+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:51:33.211 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:51:32 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:51:32.902+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T21:51:33.212 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:51:32 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:51:32.902+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T21:51:33.212 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:51:32 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:51:32.902+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T21:51:33.212 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:51:32 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:51:32.903+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:51:33.492 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:51:33 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:51:33.208+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:51:33.493 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:51:33 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:51:33.208+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:51:33.493 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:51:33 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:51:33.208+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:51:33.493 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:51:33 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:51:33.209+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:51:33.816 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:51:33 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:51:33.490+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:51:33.816 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:51:33 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:51:33.490+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:51:33.816 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:51:33 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:51:33.490+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:51:33.817 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:51:33 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:51:33.491+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:51:34.169 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:51:33 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:51:33.879+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T21:51:34.169 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:51:33 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:51:33.879+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T21:51:34.169 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:51:33 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:51:33.879+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T21:51:34.169 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:51:33 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:51:33.880+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:51:34.454 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:51:34 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:51:34.166+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:51:34.454 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:51:34 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:51:34.166+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:51:34.454 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:51:34 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:51:34.166+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:51:34.455 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:51:34 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:51:34.167+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:51:34.817 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:51:34 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:51:34.451+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:51:34.817 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:51:34 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:51:34.451+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:51:34.817 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:51:34 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:51:34.451+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:51:34.817 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:51:34 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:51:34.453+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:51:35.194 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:51:34 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:51:34.878+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T21:51:35.195 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:51:34 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:51:34.878+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T21:51:35.195 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:51:34 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:51:34.878+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T21:51:35.195 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:51:34 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:51:34.879+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:51:35.566 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:51:35 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:51:35.436+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:51:35.566 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:51:35 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:51:35.436+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:51:35.566 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:51:35 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:51:35.436+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:51:35.566 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:51:35 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:51:35.437+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:51:35.566 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:51:35 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:51:35.192+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:51:35.566 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:51:35 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:51:35.192+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:51:35.567 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:51:35 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:51:35.192+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:51:35.567 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:51:35 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:51:35.193+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:51:36.066 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:51:35 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:51:35.873+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T21:51:36.066 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:51:35 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:51:35.873+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T21:51:36.066 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:51:35 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:51:35.873+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T21:51:36.067 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:51:35 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:51:35.873+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:51:36.432 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:51:36 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:51:36.169+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:51:36.432 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:51:36 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:51:36.169+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:51:36.432 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:51:36 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:51:36.169+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:51:36.433 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:51:36 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:51:36.171+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:51:36.816 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Dec 16 21:51:36 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-mon-smithi045[80759]: 2023-12-16T21:51:36.429+0000 7f45a1f8f700 -1 mon.smithi045@0(probing) e2 get_health_metrics reporting 10119 slow ops, oldest is monmgrreport(gid 14214, 0 checks, 1 progress events) 2023-12-16T21:51:36.816 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:51:36 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:51:36.468+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:51:36.816 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:51:36 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:51:36.468+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:51:36.816 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:51:36 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:51:36.468+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:51:36.817 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:51:36 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:51:36.469+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:51:37.190 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:51:36 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:51:36.852+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T21:51:37.190 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:51:36 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:51:36.852+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T21:51:37.190 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:51:36 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:51:36.852+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T21:51:37.190 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:51:36 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:51:36.852+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:51:37.501 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:51:37 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:51:37.187+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:51:37.501 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:51:37 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:51:37.187+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:51:37.501 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:51:37 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:51:37.187+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:51:37.501 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:51:37 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:51:37.187+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:51:37.816 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:51:37 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:51:37.498+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:51:37.816 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:51:37 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:51:37.498+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:51:37.816 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:51:37 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:51:37.498+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:51:37.816 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:51:37 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:51:37.499+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:51:38.184 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:51:37 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:51:37.870+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T21:51:38.184 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:51:37 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:51:37.870+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T21:51:38.184 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:51:37 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:51:37.870+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T21:51:38.184 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:51:37 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:51:37.871+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:51:38.513 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:51:38 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:51:38.181+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:51:38.513 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:51:38 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:51:38.181+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:51:38.514 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:51:38 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:51:38.182+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:51:38.514 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:51:38 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:51:38.183+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:51:38.816 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:51:38 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:51:38.511+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:51:38.816 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:51:38 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:51:38.511+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:51:38.816 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:51:38 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:51:38.511+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:51:38.816 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:51:38 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:51:38.511+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:51:39.173 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:51:38 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:51:38.915+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T21:51:39.173 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:51:38 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:51:38.915+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T21:51:39.173 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:51:38 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:51:38.915+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T21:51:39.173 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:51:38 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:51:38.916+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:51:39.494 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:51:39 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:51:39.170+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:51:39.494 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:51:39 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:51:39.170+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:51:39.494 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:51:39 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:51:39.170+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:51:39.494 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:51:39 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:51:39.171+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:51:39.816 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:51:39 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:51:39.491+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:51:39.816 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:51:39 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:51:39.491+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:51:39.816 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:51:39 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:51:39.491+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:51:39.816 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:51:39 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:51:39.492+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:51:40.316 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:51:40 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:51:40.171+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:51:40.316 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:51:40 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:51:40.171+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:51:40.316 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:51:40 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:51:40.171+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:51:40.316 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:51:40 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:51:40.172+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:51:40.317 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:51:39 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:51:39.930+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T21:51:40.317 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:51:39 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:51:39.930+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T21:51:40.317 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:51:39 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:51:39.930+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T21:51:40.317 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:51:39 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:51:39.932+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:51:40.816 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:51:40 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:51:40.487+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:51:40.816 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:51:40 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:51:40.487+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:51:40.816 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:51:40 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:51:40.487+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:51:40.816 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:51:40 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:51:40.488+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:51:41.198 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:51:40 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:51:40.910+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T21:51:41.198 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:51:40 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:51:40.910+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T21:51:41.198 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:51:40 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:51:40.910+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T21:51:41.198 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:51:40 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:51:40.910+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:51:41.455 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Dec 16 21:51:41 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-mon-smithi045[80759]: 2023-12-16T21:51:41.430+0000 7f45a1f8f700 -1 mon.smithi045@0(probing) e2 get_health_metrics reporting 10211 slow ops, oldest is monmgrreport(gid 14214, 0 checks, 1 progress events) 2023-12-16T21:51:41.456 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:51:41 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:51:41.195+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:51:41.456 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:51:41 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:51:41.195+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:51:41.456 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:51:41 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:51:41.195+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:51:41.456 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:51:41 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:51:41.196+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:51:41.816 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:51:41 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:51:41.520+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:51:41.816 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:51:41 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:51:41.520+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:51:41.816 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:51:41 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:51:41.520+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:51:41.816 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:51:41 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:51:41.522+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:51:42.240 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:51:41 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:51:41.947+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T21:51:42.240 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:51:41 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:51:41.947+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T21:51:42.240 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:51:41 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:51:41.947+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T21:51:42.240 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:51:41 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:51:41.948+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:51:42.522 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:51:42 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:51:42.237+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:51:42.522 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:51:42 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:51:42.237+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:51:42.522 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:51:42 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:51:42.237+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:51:42.522 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:51:42 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:51:42.238+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:51:42.816 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:51:42 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:51:42.519+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:51:42.816 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:51:42 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:51:42.519+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:51:42.816 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:51:42 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:51:42.519+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:51:42.816 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:51:42 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:51:42.520+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:51:43.252 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:51:42 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:51:42.929+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T21:51:43.253 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:51:42 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:51:42.929+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T21:51:43.253 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:51:42 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:51:42.929+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T21:51:43.253 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:51:42 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:51:42.930+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:51:43.531 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:51:43 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:51:43.249+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:51:43.531 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:51:43 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:51:43.250+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:51:43.531 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:51:43 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:51:43.250+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:51:43.531 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:51:43 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:51:43.250+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:51:43.816 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:51:43 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:51:43.528+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:51:43.816 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:51:43 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:51:43.528+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:51:43.816 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:51:43 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:51:43.528+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:51:43.816 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:51:43 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:51:43.529+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:51:44.316 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:51:44 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:51:44.201+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:51:44.316 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:51:44 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:51:44.201+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:51:44.317 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:51:44 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:51:44.201+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:51:44.317 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:51:44 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:51:44.202+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:51:44.317 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:51:43 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:51:43.976+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T21:51:44.317 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:51:43 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:51:43.976+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T21:51:44.317 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:51:43 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:51:43.976+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T21:51:44.317 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:51:43 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:51:43.977+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:51:44.816 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:51:44 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:51:44.526+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:51:44.816 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:51:44 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:51:44.526+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:51:44.816 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:51:44 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:51:44.526+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:51:44.816 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:51:44 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:51:44.526+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:51:45.242 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:51:44 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:51:44.995+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T21:51:45.242 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:51:44 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:51:44.995+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T21:51:45.242 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:51:44 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:51:44.995+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T21:51:45.242 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:51:44 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:51:44.997+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:51:45.536 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:51:45 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:51:45.239+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:51:45.536 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:51:45 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:51:45.239+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:51:45.536 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:51:45 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:51:45.239+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:51:45.536 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:51:45 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:51:45.240+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:51:45.816 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:51:45 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:51:45.533+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:51:45.816 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:51:45 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:51:45.533+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:51:45.816 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:51:45 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:51:45.533+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:51:45.817 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:51:45 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:51:45.534+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:51:46.316 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:51:46 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:51:46.191+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:51:46.316 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:51:46 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:51:46.191+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:51:46.316 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:51:46 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:51:46.191+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:51:46.317 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:51:46 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:51:46.192+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:51:46.317 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:51:45 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:51:45.968+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T21:51:46.317 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:51:45 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:51:45.968+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T21:51:46.317 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:51:45 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:51:45.968+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T21:51:46.318 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:51:45 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:51:45.969+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:51:46.816 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Dec 16 21:51:46 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-mon-smithi045[80759]: 2023-12-16T21:51:46.432+0000 7f45a1f8f700 -1 mon.smithi045@0(probing) e2 get_health_metrics reporting 10299 slow ops, oldest is monmgrreport(gid 14214, 0 checks, 1 progress events) 2023-12-16T21:51:46.816 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:51:46 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:51:46.584+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:51:46.816 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:51:46 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:51:46.584+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:51:46.816 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:51:46 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:51:46.584+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:51:46.816 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:51:46 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:51:46.585+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:51:47.316 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:51:46 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:51:46.982+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T21:51:47.316 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:51:46 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:51:46.982+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T21:51:47.316 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:51:46 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:51:46.982+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T21:51:47.316 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:51:46 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:51:46.983+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:51:47.317 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:51:47 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:51:47.166+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:51:47.317 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:51:47 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:51:47.166+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:51:47.317 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:51:47 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:51:47.166+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:51:47.317 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:51:47 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:51:47.167+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:51:47.936 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:51:47 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:51:47.568+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:51:47.936 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:51:47 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:51:47.568+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:51:47.936 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:51:47 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:51:47.568+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:51:47.936 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:51:47 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:51:47.569+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:51:48.219 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:51:47 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:51:47.933+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T21:51:48.219 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:51:47 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:51:47.933+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T21:51:48.219 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:51:47 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:51:47.933+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T21:51:48.219 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:51:47 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:51:47.934+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:51:48.566 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:51:48 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:51:48.216+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:51:48.566 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:51:48 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:51:48.216+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:51:48.566 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:51:48 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:51:48.216+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:51:48.566 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:51:48 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:51:48.217+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:51:48.923 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:51:48 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:51:48.616+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:51:48.923 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:51:48 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:51:48.616+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:51:48.924 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:51:48 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:51:48.616+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:51:48.924 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:51:48 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:51:48.617+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:51:49.248 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:51:48 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:51:48.920+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T21:51:49.248 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:51:48 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:51:48.921+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T21:51:49.248 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:51:48 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:51:48.921+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T21:51:49.249 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:51:48 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:51:48.922+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:51:49.566 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:51:49 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:51:49.246+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:51:49.566 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:51:49 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:51:49.246+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:51:49.566 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:51:49 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:51:49.246+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:51:49.566 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:51:49 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:51:49.246+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:51:49.938 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:51:49 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:51:49.619+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:51:49.938 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:51:49 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:51:49.619+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:51:49.938 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:51:49 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:51:49.619+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:51:49.938 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:51:49 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:51:49.620+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:51:50.267 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:51:49 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:51:49.935+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T21:51:50.267 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:51:49 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:51:49.935+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T21:51:50.267 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:51:49 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:51:49.935+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T21:51:50.267 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:51:49 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:51:49.936+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:51:50.566 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:51:50 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:51:50.264+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:51:50.566 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:51:50 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:51:50.264+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:51:50.566 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:51:50 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:51:50.264+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:51:50.566 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:51:50 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:51:50.265+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:51:50.986 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:51:50 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:51:50.622+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:51:50.986 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:51:50 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:51:50.622+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:51:50.986 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:51:50 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:51:50.622+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:51:50.986 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:51:50 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:51:50.623+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:51:51.243 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:51:50 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:51:50.983+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T21:51:51.244 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:51:50 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:51:50.983+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T21:51:51.244 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:51:50 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:51:50.983+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T21:51:51.244 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:51:50 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:51:50.984+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:51:51.560 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Dec 16 21:51:51 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-mon-smithi045[80759]: 2023-12-16T21:51:51.433+0000 7f45a1f8f700 -1 mon.smithi045@0(probing) e2 get_health_metrics reporting 10390 slow ops, oldest is monmgrreport(gid 14214, 0 checks, 1 progress events) 2023-12-16T21:51:51.560 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:51:51 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:51:51.241+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:51:51.560 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:51:51 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:51:51.241+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:51:51.560 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:51:51 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:51:51.241+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:51:51.560 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:51:51 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:51:51.242+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:51:51.816 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:51:51 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:51:51.601+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:51:51.816 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:51:51 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:51:51.601+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:51:51.816 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:51:51 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:51:51.601+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:51:51.816 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:51:51 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:51:51.602+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:51:52.316 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:51:52 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:51:52.225+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:51:52.316 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:51:52 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:51:52.225+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:51:52.316 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:51:52 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:51:52.225+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:51:52.316 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:51:52 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:51:52.226+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:51:52.317 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:51:52 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:51:52.012+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T21:51:52.317 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:51:52 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:51:52.012+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T21:51:52.317 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:51:52 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:51:52.012+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T21:51:52.317 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:51:52 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:51:52.013+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:51:53.044 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:51:52 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:51:52.584+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:51:53.044 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:51:52 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:51:52.584+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:51:53.044 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:51:52 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:51:52.584+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:51:53.045 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:51:52 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:51:52.585+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:51:53.316 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:51:53 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:51:53.244+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:51:53.316 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:51:53 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:51:53.244+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:51:53.316 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:51:53 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:51:53.244+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:51:53.316 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:51:53 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:51:53.244+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:51:53.316 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:51:53 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:51:53.042+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T21:51:53.317 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:51:53 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:51:53.042+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T21:51:53.317 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:51:53 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:51:53.042+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T21:51:53.317 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:51:53 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:51:53.043+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:51:53.816 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:51:53 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:51:53.562+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:51:53.816 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:51:53 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:51:53.562+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:51:53.816 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:51:53 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:51:53.562+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:51:53.816 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:51:53 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:51:53.563+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:51:54.557 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:51:54 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:51:54.251+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:51:54.557 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:51:54 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:51:54.251+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:51:54.557 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:51:54 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:51:54.251+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:51:54.557 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:51:54 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:51:54.252+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:51:54.558 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:51:54 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:51:54.077+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T21:51:54.558 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:51:54 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:51:54.077+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T21:51:54.558 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:51:54 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:51:54.077+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T21:51:54.558 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:51:54 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:51:54.078+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:51:54.816 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:51:54 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:51:54.554+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:51:54.816 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:51:54 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:51:54.554+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:51:54.816 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:51:54 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:51:54.554+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:51:54.816 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:51:54 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:51:54.555+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:51:55.316 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:51:55 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:51:55.295+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:51:55.316 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:51:55 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:51:55.295+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:51:55.316 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:51:55 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:51:55.295+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:51:55.316 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:51:55 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:51:55.296+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:51:55.316 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:51:55 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:51:55.061+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T21:51:55.317 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:51:55 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:51:55.061+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T21:51:55.317 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:51:55 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:51:55.061+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T21:51:55.317 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:51:55 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:51:55.062+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:51:56.066 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:51:55 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:51:55.593+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:51:56.066 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:51:55 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:51:55.593+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:51:56.066 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:51:55 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:51:55.593+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:51:56.066 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:51:55 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:51:55.594+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:51:56.437 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:51:56 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:51:56.065+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T21:51:56.437 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:51:56 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:51:56.065+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T21:51:56.437 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:51:56 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:51:56.065+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T21:51:56.437 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:51:56 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:51:56.066+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:51:56.437 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:51:56 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:51:56.302+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:51:56.438 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:51:56 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:51:56.302+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:51:56.438 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:51:56 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:51:56.302+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:51:56.438 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:51:56 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:51:56.304+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:51:56.816 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Dec 16 21:51:56 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-mon-smithi045[80759]: 2023-12-16T21:51:56.434+0000 7f45a1f8f700 -1 mon.smithi045@0(probing) e2 get_health_metrics reporting 10479 slow ops, oldest is monmgrreport(gid 14214, 0 checks, 1 progress events) 2023-12-16T21:51:56.816 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:51:56 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:51:56.594+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:51:56.816 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:51:56 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:51:56.594+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:51:56.816 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:51:56 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:51:56.594+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:51:56.816 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:51:56 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:51:56.594+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:51:57.316 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:51:57 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:51:57.028+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T21:51:57.316 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:51:57 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:51:57.028+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T21:51:57.316 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:51:57 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:51:57.028+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T21:51:57.316 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:51:57 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:51:57.028+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:51:57.634 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:51:57 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:51:57.333+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:51:57.635 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:51:57 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:51:57.333+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:51:57.635 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:51:57 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:51:57.333+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:51:57.635 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:51:57 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:51:57.334+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:51:58.029 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:51:57 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:51:57.631+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:51:58.029 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:51:57 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:51:57.631+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:51:58.029 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:51:57 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:51:57.631+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:51:58.029 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:51:57 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:51:57.632+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:51:58.316 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:51:58 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:51:58.025+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T21:51:58.316 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:51:58 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:51:58.025+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T21:51:58.316 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:51:58 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:51:58.025+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T21:51:58.316 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:51:58 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:51:58.027+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:51:58.616 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:51:58 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:51:58.357+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:51:58.616 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:51:58 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:51:58.357+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:51:58.616 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:51:58 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:51:58.357+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:51:58.616 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:51:58 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:51:58.358+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:51:59.006 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:51:58 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:51:58.612+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:51:59.006 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:51:58 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:51:58.612+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:51:59.006 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:51:58 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:51:58.612+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:51:59.006 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:51:58 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:51:58.613+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:51:59.316 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:51:59 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:51:59.002+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T21:51:59.316 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:51:59 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:51:59.002+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T21:51:59.316 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:51:59 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:51:59.002+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T21:51:59.316 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:51:59 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:51:59.003+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:51:59.590 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:51:59 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:51:59.329+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:51:59.590 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:51:59 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:51:59.329+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:51:59.590 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:51:59 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:51:59.329+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:51:59.590 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:51:59 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:51:59.329+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:51:59.994 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:51:59 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:51:59.587+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:51:59.994 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:51:59 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:51:59.587+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:51:59.994 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:51:59 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:51:59.587+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:51:59.995 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:51:59 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:51:59.587+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:52:00.273 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:51:59 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:51:59.991+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T21:52:00.273 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:51:59 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:51:59.991+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T21:52:00.273 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:51:59 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:51:59.991+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T21:52:00.273 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:51:59 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:51:59.991+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:52:00.566 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:52:00 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:52:00.286+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:52:00.566 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:52:00 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:52:00.286+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:52:00.566 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:52:00 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:52:00.286+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:52:00.566 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:52:00 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:52:00.287+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:52:00.970 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:52:00 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:52:00.621+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:52:00.970 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:52:00 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:52:00.621+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:52:00.970 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:52:00 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:52:00.621+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:52:00.971 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:52:00 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:52:00.622+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:52:01.260 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:52:00 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:52:00.967+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T21:52:01.260 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:52:00 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:52:00.967+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T21:52:01.260 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:52:00 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:52:00.967+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T21:52:01.260 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:52:00 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:52:00.968+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:52:01.564 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Dec 16 21:52:01 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-mon-smithi045[80759]: 2023-12-16T21:52:01.434+0000 7f45a1f8f700 -1 mon.smithi045@0(probing) e2 get_health_metrics reporting 10573 slow ops, oldest is monmgrreport(gid 14214, 0 checks, 1 progress events) 2023-12-16T21:52:01.564 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:52:01 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:52:01.256+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:52:01.564 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:52:01 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:52:01.256+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:52:01.564 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:52:01 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:52:01.256+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:52:01.565 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:52:01 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:52:01.257+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:52:01.816 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:52:01 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:52:01.616+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:52:01.816 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:52:01 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:52:01.616+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:52:01.816 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:52:01 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:52:01.616+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:52:01.816 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:52:01 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:52:01.617+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:52:02.254 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:52:01 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:52:01.983+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T21:52:02.254 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:52:01 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:52:01.983+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T21:52:02.254 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:52:01 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:52:01.983+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T21:52:02.254 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:52:01 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:52:01.983+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:52:02.566 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:52:02 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:52:02.250+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:52:02.566 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:52:02 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:52:02.250+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:52:02.566 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:52:02 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:52:02.250+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:52:02.566 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:52:02 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:52:02.251+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:52:03.009 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:52:02 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:52:02.579+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:52:03.009 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:52:02 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:52:02.579+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:52:03.009 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:52:02 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:52:02.579+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:52:03.009 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:52:02 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:52:02.581+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:52:03.316 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:52:03 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:52:03.247+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:52:03.316 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:52:03 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:52:03.247+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:52:03.316 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:52:03 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:52:03.247+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:52:03.316 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:52:03 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:52:03.248+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:52:03.316 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:52:03 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:52:03.006+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T21:52:03.316 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:52:03 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:52:03.006+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T21:52:03.316 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:52:03 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:52:03.006+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T21:52:03.316 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:52:03 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:52:03.007+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:52:03.816 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:52:03 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:52:03.577+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:52:03.816 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:52:03 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:52:03.577+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:52:03.816 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:52:03 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:52:03.577+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:52:03.816 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:52:03 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:52:03.577+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:52:04.251 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:52:03 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:52:03.971+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T21:52:04.251 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:52:03 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:52:03.971+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T21:52:04.251 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:52:03 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:52:03.971+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T21:52:04.251 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:52:03 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:52:03.972+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:52:04.566 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:52:04 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:52:04.248+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:52:04.566 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:52:04 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:52:04.248+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:52:04.566 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:52:04 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:52:04.248+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:52:04.566 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:52:04 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:52:04.249+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:52:04.987 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:52:04 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:52:04.566+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:52:04.987 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:52:04 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:52:04.566+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:52:04.987 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:52:04 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:52:04.566+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:52:04.987 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:52:04 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:52:04.566+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:52:05.273 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:52:04 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:52:04.984+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T21:52:05.273 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:52:04 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:52:04.984+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T21:52:05.273 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:52:04 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:52:04.984+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T21:52:05.273 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:52:04 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:52:04.985+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:52:05.537 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:52:05 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:52:05.270+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:52:05.537 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:52:05 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:52:05.270+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:52:05.538 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:52:05 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:52:05.270+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:52:05.538 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:52:05 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:52:05.271+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:52:05.816 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:52:05 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:52:05.535+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:52:05.816 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:52:05 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:52:05.535+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:52:05.816 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:52:05 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:52:05.535+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:52:05.816 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:52:05 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:52:05.536+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:52:06.316 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:52:06 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:52:06.021+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T21:52:06.316 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:52:06 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:52:06.021+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T21:52:06.316 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:52:06 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:52:06.021+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T21:52:06.317 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:52:06 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:52:06.022+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:52:06.317 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:52:06 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:52:06.232+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:52:06.317 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:52:06 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:52:06.232+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:52:06.317 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:52:06 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:52:06.232+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:52:06.317 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:52:06 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:52:06.233+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:52:06.816 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Dec 16 21:52:06 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-mon-smithi045[80759]: 2023-12-16T21:52:06.437+0000 7f45a1f8f700 -1 mon.smithi045@0(probing) e2 get_health_metrics reporting 10665 slow ops, oldest is monmgrreport(gid 14214, 0 checks, 1 progress events) 2023-12-16T21:52:06.816 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:52:06 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:52:06.499+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:52:06.816 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:52:06 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:52:06.499+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:52:06.816 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:52:06 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:52:06.499+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:52:06.816 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:52:06 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:52:06.499+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:52:07.316 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:52:07 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:52:07.186+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:52:07.316 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:52:07 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:52:07.186+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:52:07.316 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:52:07 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:52:07.186+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:52:07.316 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:52:07 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:52:07.187+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:52:07.316 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:52:06 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:52:06.985+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T21:52:07.317 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:52:06 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:52:06.985+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T21:52:07.317 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:52:06 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:52:06.985+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T21:52:07.317 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:52:06 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:52:06.986+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:52:07.816 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:52:07 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:52:07.470+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:52:07.816 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:52:07 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:52:07.470+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:52:07.816 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:52:07 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:52:07.470+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:52:07.817 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:52:07 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:52:07.471+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:52:08.227 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:52:07 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:52:07.963+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T21:52:08.227 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:52:07 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:52:07.963+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T21:52:08.227 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:52:07 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:52:07.963+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T21:52:08.227 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:52:07 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:52:07.963+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:52:08.508 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:52:08 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:52:08.224+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:52:08.508 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:52:08 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:52:08.224+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:52:08.508 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:52:08 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:52:08.224+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:52:08.508 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:52:08 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:52:08.225+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:52:08.816 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:52:08 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:52:08.505+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:52:08.816 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:52:08 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:52:08.505+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:52:08.816 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:52:08 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:52:08.505+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:52:08.816 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:52:08 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:52:08.506+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:52:09.277 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:52:08 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:52:08.952+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T21:52:09.278 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:52:08 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:52:08.952+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T21:52:09.278 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:52:08 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:52:08.952+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T21:52:09.278 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:52:08 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:52:08.953+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:52:09.566 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:52:09 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:52:09.496+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:52:09.566 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:52:09 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:52:09.496+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:52:09.566 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:52:09 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:52:09.496+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:52:09.566 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:52:09 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:52:09.497+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:52:09.566 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:52:09 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:52:09.275+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:52:09.567 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:52:09 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:52:09.275+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:52:09.567 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:52:09 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:52:09.275+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:52:09.567 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:52:09 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:52:09.276+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:52:10.298 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:52:09 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:52:09.964+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T21:52:10.298 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:52:09 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:52:09.964+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T21:52:10.298 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:52:09 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:52:09.964+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T21:52:10.298 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:52:09 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:52:09.965+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:52:10.566 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:52:10 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:52:10.453+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:52:10.566 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:52:10 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:52:10.453+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:52:10.566 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:52:10 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:52:10.453+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:52:10.566 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:52:10 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:52:10.453+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:52:10.566 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:52:10 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:52:10.295+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:52:10.566 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:52:10 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:52:10.295+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:52:10.566 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:52:10 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:52:10.295+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:52:10.566 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:52:10 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:52:10.296+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:52:11.286 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:52:10 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:52:10.961+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T21:52:11.286 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:52:10 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:52:10.961+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T21:52:11.286 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:52:10 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:52:10.961+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T21:52:11.287 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:52:10 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:52:10.962+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:52:11.566 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Dec 16 21:52:11 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-mon-smithi045[80759]: 2023-12-16T21:52:11.438+0000 7f45a1f8f700 -1 mon.smithi045@0(probing) e2 get_health_metrics reporting 10759 slow ops, oldest is monmgrreport(gid 14214, 0 checks, 1 progress events) 2023-12-16T21:52:11.566 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:52:11 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:52:11.412+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:52:11.566 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:52:11 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:52:11.412+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:52:11.566 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:52:11 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:52:11.412+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:52:11.566 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:52:11 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:52:11.412+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:52:11.567 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:52:11 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:52:11.284+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:52:11.567 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:52:11 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:52:11.284+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:52:11.567 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:52:11 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:52:11.284+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:52:11.567 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:52:11 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:52:11.285+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:52:12.262 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:52:11 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:52:11.968+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T21:52:12.262 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:52:11 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:52:11.968+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T21:52:12.262 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:52:11 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:52:11.968+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T21:52:12.262 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:52:11 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:52:11.969+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:52:12.561 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:52:12 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:52:12.379+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:52:12.561 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:52:12 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:52:12.379+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:52:12.561 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:52:12 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:52:12.379+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:52:12.561 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:52:12 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:52:12.379+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:52:12.561 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:52:12 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:52:12.259+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:52:12.561 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:52:12 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:52:12.259+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:52:12.562 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:52:12 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:52:12.259+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:52:12.562 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:52:12 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:52:12.260+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:52:13.294 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:52:12 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:52:12.997+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T21:52:13.294 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:52:12 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:52:12.997+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T21:52:13.294 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:52:12 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:52:12.997+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T21:52:13.294 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:52:12 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:52:12.997+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:52:13.563 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:52:13 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:52:13.375+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:52:13.564 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:52:13 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:52:13.375+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:52:13.564 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:52:13 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:52:13.375+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:52:13.564 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:52:13 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:52:13.375+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:52:13.564 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:52:13 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:52:13.291+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:52:13.564 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:52:13 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:52:13.291+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:52:13.564 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:52:13 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:52:13.291+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:52:13.564 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:52:13 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:52:13.293+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:52:14.316 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:52:14 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:52:14.005+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T21:52:14.316 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:52:14 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:52:14.005+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T21:52:14.316 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:52:14 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:52:14.005+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T21:52:14.316 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:52:14 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:52:14.005+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:52:14.816 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:52:14 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:52:14.388+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:52:14.816 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:52:14 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:52:14.388+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:52:14.816 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:52:14 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:52:14.388+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:52:14.816 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:52:14 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:52:14.389+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:52:14.817 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:52:14 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:52:14.318+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:52:14.817 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:52:14 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:52:14.318+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:52:14.817 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:52:14 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:52:14.318+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:52:14.817 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:52:14 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:52:14.319+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:52:15.316 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:52:15 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:52:15.008+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T21:52:15.316 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:52:15 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:52:15.008+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T21:52:15.316 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:52:15 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:52:15.008+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T21:52:15.316 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:52:15 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:52:15.009+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:52:15.632 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:52:15 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:52:15.423+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:52:15.633 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:52:15 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:52:15.423+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:52:15.633 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:52:15 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:52:15.423+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:52:15.633 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:52:15 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:52:15.425+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:52:15.633 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:52:15 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:52:15.362+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:52:15.633 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:52:15 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:52:15.362+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:52:15.633 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:52:15 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:52:15.362+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:52:15.633 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:52:15 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:52:15.363+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:52:16.316 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:52:16 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:52:16.022+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T21:52:16.316 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:52:16 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:52:16.022+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T21:52:16.316 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:52:16 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:52:16.022+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T21:52:16.316 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:52:16 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:52:16.023+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:52:16.816 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Dec 16 21:52:16 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-mon-smithi045[80759]: 2023-12-16T21:52:16.439+0000 7f45a1f8f700 -1 mon.smithi045@0(probing) e2 get_health_metrics reporting 10854 slow ops, oldest is monmgrreport(gid 14214, 0 checks, 1 progress events) 2023-12-16T21:52:16.816 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:52:16 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:52:16.388+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:52:16.816 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:52:16 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:52:16.388+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:52:16.816 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:52:16 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:52:16.388+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:52:16.816 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:52:16 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:52:16.389+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:52:16.817 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:52:16 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:52:16.346+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:52:16.817 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:52:16 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:52:16.346+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:52:16.817 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:52:16 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:52:16.346+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:52:16.817 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:52:16 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:52:16.347+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:52:17.316 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:52:17 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:52:16.999+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T21:52:17.316 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:52:17 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:52:16.999+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T21:52:17.316 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:52:17 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:52:16.999+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T21:52:17.316 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:52:17 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:52:17.000+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:52:17.816 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:52:17 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:52:17.343+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:52:17.816 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:52:17 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:52:17.343+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:52:17.816 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:52:17 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:52:17.343+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:52:17.817 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:52:17 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:52:17.344+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:52:17.817 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:52:17 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:52:17.412+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:52:17.817 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:52:17 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:52:17.412+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:52:17.817 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:52:17 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:52:17.412+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:52:17.817 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:52:17 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:52:17.413+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:52:18.316 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:52:18 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:52:18.032+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T21:52:18.316 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:52:18 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:52:18.032+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T21:52:18.316 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:52:18 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:52:18.032+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T21:52:18.316 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:52:18 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:52:18.033+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:52:18.722 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:52:18 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:52:18.408+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:52:18.723 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:52:18 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:52:18.408+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:52:18.723 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:52:18 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:52:18.408+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:52:18.723 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:52:18 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:52:18.409+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:52:18.723 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:52:18 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:52:18.340+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:52:18.723 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:52:18 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:52:18.340+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:52:18.724 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:52:18 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:52:18.340+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:52:18.724 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:52:18 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:52:18.341+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:52:19.316 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:52:19 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:52:19.013+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T21:52:19.316 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:52:19 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:52:19.013+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T21:52:19.316 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:52:19 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:52:19.013+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T21:52:19.316 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:52:19 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:52:19.014+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:52:19.816 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:52:19 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:52:19.410+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:52:19.816 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:52:19 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:52:19.410+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:52:19.816 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:52:19 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:52:19.411+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:52:19.816 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:52:19 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:52:19.411+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:52:19.816 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:52:19 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:52:19.319+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:52:19.816 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:52:19 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:52:19.319+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:52:19.817 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:52:19 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:52:19.319+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:52:19.817 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:52:19 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:52:19.319+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:52:20.316 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:52:20 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:52:20.030+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T21:52:20.316 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:52:20 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:52:20.030+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T21:52:20.316 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:52:20 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:52:20.030+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T21:52:20.316 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:52:20 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:52:20.031+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:52:20.816 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:52:20 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:52:20.444+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:52:20.816 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:52:20 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:52:20.444+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:52:20.816 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:52:20 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:52:20.444+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:52:20.816 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:52:20 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:52:20.445+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:52:20.816 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:52:20 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:52:20.354+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:52:20.817 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:52:20 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:52:20.354+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:52:20.817 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:52:20 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:52:20.354+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:52:20.817 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:52:20 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:52:20.354+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:52:21.316 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:52:21 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:52:21.032+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T21:52:21.316 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:52:21 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:52:21.032+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T21:52:21.316 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:52:21 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:52:21.032+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T21:52:21.316 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:52:21 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:52:21.033+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:52:21.673 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Dec 16 21:52:21 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-mon-smithi045[80759]: 2023-12-16T21:52:21.441+0000 7f45a1f8f700 -1 mon.smithi045@0(probing) e2 get_health_metrics reporting 10952 slow ops, oldest is monmgrreport(gid 14214, 0 checks, 1 progress events) 2023-12-16T21:52:21.673 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:52:21 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:52:21.405+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:52:21.674 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:52:21 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:52:21.405+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:52:21.674 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:52:21 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:52:21.405+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:52:21.674 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:52:21 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:52:21.407+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:52:21.674 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:52:21 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:52:21.375+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:52:21.674 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:52:21 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:52:21.375+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:52:21.674 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:52:21 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:52:21.375+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:52:21.674 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:52:21 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:52:21.376+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:52:22.316 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:52:22 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:52:22.029+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T21:52:22.316 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:52:22 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:52:22.029+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T21:52:22.316 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:52:22 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:52:22.030+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T21:52:22.316 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:52:22 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:52:22.030+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:52:22.816 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:52:22 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:52:22.414+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:52:22.816 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:52:22 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:52:22.414+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:52:22.816 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:52:22 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:52:22.414+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:52:22.816 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:52:22 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:52:22.415+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:52:22.816 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:52:22 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:52:22.400+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:52:22.816 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:52:22 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:52:22.400+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:52:22.817 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:52:22 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:52:22.400+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:52:22.817 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:52:22 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:52:22.401+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:52:23.316 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:52:23 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:52:23.041+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T21:52:23.316 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:52:23 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:52:23.041+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T21:52:23.316 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:52:23 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:52:23.041+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T21:52:23.316 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:52:23 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:52:23.042+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:52:23.816 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:52:23 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:52:23.372+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:52:23.816 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:52:23 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:52:23.372+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:52:23.816 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:52:23 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:52:23.372+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:52:23.816 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:52:23 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:52:23.373+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:52:23.816 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:52:23 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:52:23.419+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:52:23.817 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:52:23 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:52:23.419+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:52:23.817 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:52:23 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:52:23.419+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:52:23.817 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:52:23 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:52:23.420+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:52:24.316 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:52:24 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:52:24.023+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T21:52:24.316 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:52:24 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:52:24.023+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T21:52:24.316 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:52:24 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:52:24.023+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T21:52:24.316 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:52:24 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:52:24.024+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:52:24.817 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:52:24 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:52:24.454+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:52:24.817 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:52:24 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:52:24.454+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:52:24.817 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:52:24 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:52:24.454+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:52:24.817 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:52:24 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:52:24.455+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:52:24.818 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:52:24 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:52:24.371+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:52:24.818 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:52:24 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:52:24.371+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:52:24.818 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:52:24 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:52:24.371+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:52:24.818 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:52:24 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:52:24.372+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:52:25.316 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:52:25 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:52:25.043+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T21:52:25.316 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:52:25 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:52:25.043+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T21:52:25.316 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:52:25 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:52:25.043+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T21:52:25.316 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:52:25 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:52:25.043+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:52:25.816 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:52:25 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:52:25.402+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:52:25.816 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:52:25 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:52:25.402+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:52:25.816 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:52:25 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:52:25.402+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:52:25.816 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:52:25 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:52:25.403+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:52:25.817 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:52:25 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:52:25.416+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:52:25.817 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:52:25 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:52:25.417+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:52:25.817 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:52:25 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:52:25.417+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:52:25.817 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:52:25 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:52:25.417+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:52:26.361 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:52:26 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:52:26.081+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T21:52:26.362 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:52:26 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:52:26.081+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T21:52:26.362 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:52:26 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:52:26.081+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T21:52:26.362 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:52:26 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:52:26.081+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:52:26.816 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Dec 16 21:52:26 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-mon-smithi045[80759]: 2023-12-16T21:52:26.442+0000 7f45a1f8f700 -1 mon.smithi045@0(probing) e2 get_health_metrics reporting 11048 slow ops, oldest is monmgrreport(gid 14214, 0 checks, 1 progress events) 2023-12-16T21:52:26.816 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:52:26 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:52:26.421+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:52:26.816 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:52:26 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:52:26.421+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:52:26.816 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:52:26 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:52:26.421+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:52:26.816 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:52:26 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:52:26.422+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:52:26.817 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:52:26 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:52:26.357+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:52:26.817 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:52:26 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:52:26.357+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:52:26.817 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:52:26 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:52:26.357+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:52:26.817 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:52:26 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:52:26.358+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:52:27.377 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:52:27 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:52:27.101+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T21:52:27.377 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:52:27 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:52:27.102+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T21:52:27.377 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:52:27 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:52:27.102+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T21:52:27.377 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:52:27 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:52:27.103+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:52:27.377 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:52:27 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:52:27.323+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:52:27.377 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:52:27 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:52:27.323+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:52:27.377 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:52:27 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:52:27.323+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:52:27.377 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:52:27 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:52:27.323+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:52:27.816 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:52:27 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:52:27.374+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:52:27.816 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:52:27 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:52:27.374+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:52:27.816 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:52:27 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:52:27.374+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:52:27.816 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:52:27 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:52:27.375+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:52:28.566 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:52:28 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:52:28.115+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T21:52:28.566 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:52:28 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:52:28.115+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T21:52:28.566 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:52:28 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:52:28.115+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T21:52:28.566 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:52:28 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:52:28.117+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:52:28.566 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:52:28 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:52:28.281+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:52:28.567 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:52:28 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:52:28.281+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:52:28.567 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:52:28 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:52:28.281+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:52:28.567 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:52:28 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:52:28.281+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:52:28.567 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:52:28 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:52:28.349+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:52:28.567 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:52:28 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:52:28.349+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:52:28.567 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:52:28 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:52:28.349+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:52:28.567 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:52:28 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:52:28.349+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:52:29.566 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:52:29 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:52:29.362+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:52:29.566 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:52:29 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:52:29.362+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:52:29.566 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:52:29 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:52:29.362+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:52:29.566 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:52:29 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:52:29.362+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:52:29.566 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:52:29 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:52:29.279+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:52:29.567 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:52:29 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:52:29.279+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:52:29.567 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:52:29 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:52:29.279+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:52:29.567 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:52:29 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:52:29.280+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:52:29.567 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:52:29 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:52:29.162+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T21:52:29.567 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:52:29 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:52:29.162+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T21:52:29.567 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:52:29 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:52:29.162+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T21:52:29.567 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:52:29 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:52:29.163+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:52:30.566 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:52:30 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:52:30.154+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T21:52:30.566 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:52:30 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:52:30.154+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T21:52:30.566 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:52:30 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:52:30.154+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T21:52:30.566 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:52:30 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:52:30.155+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:52:30.567 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:52:30 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:52:30.333+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:52:30.567 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:52:30 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:52:30.333+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:52:30.567 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:52:30 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:52:30.333+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:52:30.567 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:52:30 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:52:30.333+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:52:30.567 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:52:30 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:52:30.239+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:52:30.567 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:52:30 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:52:30.239+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:52:30.567 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:52:30 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:52:30.239+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:52:30.567 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:52:30 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:52:30.239+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:52:31.446 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:52:31 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:52:31.343+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:52:31.446 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:52:31 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:52:31.343+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:52:31.446 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:52:31 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:52:31.343+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:52:31.446 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:52:31 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:52:31.344+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:52:31.446 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:52:31 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:52:31.144+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T21:52:31.446 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:52:31 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:52:31.144+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T21:52:31.447 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:52:31 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:52:31.144+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T21:52:31.447 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:52:31 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:52:31.145+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:52:31.447 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:52:31 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:52:31.261+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:52:31.447 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:52:31 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:52:31.261+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:52:31.447 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:52:31 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:52:31.261+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:52:31.447 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:52:31 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:52:31.262+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:52:31.816 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Dec 16 21:52:31 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-mon-smithi045[80759]: 2023-12-16T21:52:31.443+0000 7f45a1f8f700 -1 mon.smithi045@0(probing) e2 get_health_metrics reporting 11146 slow ops, oldest is monmgrreport(gid 14214, 0 checks, 1 progress events) 2023-12-16T21:52:32.566 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:52:32 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:52:32.321+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:52:32.566 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:52:32 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:52:32.321+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:52:32.566 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:52:32 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:52:32.321+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:52:32.566 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:52:32 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:52:32.322+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:52:32.567 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:52:32 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:52:32.242+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:52:32.567 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:52:32 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:52:32.242+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:52:32.567 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:52:32 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:52:32.242+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:52:32.567 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:52:32 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:52:32.243+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:52:32.567 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:52:32 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:52:32.135+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T21:52:32.567 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:52:32 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:52:32.135+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T21:52:32.567 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:52:32 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:52:32.135+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T21:52:32.567 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:52:32 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:52:32.135+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:52:33.346 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:52:33 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:52:33.093+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T21:52:33.346 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:52:33 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:52:33.093+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T21:52:33.346 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:52:33 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:52:33.093+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T21:52:33.346 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:52:33 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:52:33.094+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:52:33.347 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:52:33 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:52:33.215+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:52:33.347 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:52:33 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:52:33.215+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:52:33.347 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:52:33 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:52:33.215+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:52:33.347 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:52:33 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:52:33.216+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:52:33.816 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:52:33 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:52:33.343+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:52:33.816 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:52:33 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:52:33.343+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:52:33.816 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:52:33 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:52:33.344+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:52:33.816 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:52:33 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:52:33.345+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:52:34.316 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:52:34 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:52:34.124+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T21:52:34.316 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:52:34 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:52:34.124+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T21:52:34.316 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:52:34 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:52:34.124+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T21:52:34.316 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:52:34 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:52:34.125+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:52:34.316 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:52:34 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:52:34.237+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:52:34.316 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:52:34 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:52:34.237+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:52:34.316 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:52:34 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:52:34.237+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:52:34.316 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:52:34 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:52:34.237+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:52:34.816 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:52:34 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:52:34.331+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:52:34.816 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:52:34 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:52:34.331+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:52:34.816 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:52:34 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:52:34.331+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:52:34.816 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:52:34 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:52:34.332+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:52:35.566 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:52:35 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:52:35.291+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:52:35.566 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:52:35 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:52:35.291+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:52:35.566 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:52:35 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:52:35.291+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:52:35.566 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:52:35 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:52:35.291+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:52:35.566 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:52:35 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:52:35.245+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:52:35.566 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:52:35 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:52:35.245+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:52:35.566 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:52:35 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:52:35.245+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:52:35.566 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:52:35 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:52:35.245+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:52:35.567 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:52:35 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:52:35.147+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T21:52:35.567 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:52:35 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:52:35.147+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T21:52:35.567 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:52:35 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:52:35.147+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T21:52:35.567 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:52:35 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:52:35.149+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:52:36.447 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:52:36 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:52:36.261+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:52:36.447 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:52:36 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:52:36.261+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:52:36.448 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:52:36 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:52:36.261+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:52:36.448 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:52:36 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:52:36.263+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:52:36.448 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:52:36 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:52:36.182+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T21:52:36.448 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:52:36 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:52:36.182+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T21:52:36.448 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:52:36 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:52:36.182+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T21:52:36.448 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:52:36 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:52:36.183+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:52:36.448 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:52:36 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:52:36.223+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:52:36.448 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:52:36 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:52:36.223+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:52:36.449 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:52:36 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:52:36.223+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:52:36.449 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:52:36 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:52:36.224+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:52:36.816 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Dec 16 21:52:36 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-mon-smithi045[80759]: 2023-12-16T21:52:36.444+0000 7f45a1f8f700 -1 mon.smithi045@0(probing) e2 get_health_metrics reporting 11241 slow ops, oldest is monmgrreport(gid 14214, 0 checks, 1 progress events) 2023-12-16T21:52:37.566 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:52:37 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:52:37.197+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T21:52:37.566 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:52:37 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:52:37.197+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T21:52:37.566 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:52:37 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:52:37.197+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T21:52:37.566 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:52:37 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:52:37.198+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:52:37.566 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:52:37 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:52:37.241+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:52:37.566 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:52:37 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:52:37.241+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:52:37.566 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:52:37 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:52:37.241+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:52:37.567 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:52:37 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:52:37.242+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:52:37.567 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:52:37 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:52:37.210+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:52:37.567 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:52:37 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:52:37.210+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:52:37.567 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:52:37 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:52:37.210+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:52:37.567 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:52:37 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:52:37.211+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:52:38.566 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:52:38 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:52:38.189+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:52:38.566 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:52:38 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:52:38.189+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:52:38.566 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:52:38 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:52:38.189+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:52:38.566 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:52:38 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:52:38.189+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:52:38.567 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:52:38 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:52:38.200+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T21:52:38.567 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:52:38 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:52:38.200+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T21:52:38.567 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:52:38 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:52:38.200+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T21:52:38.567 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:52:38 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:52:38.201+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:52:38.567 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:52:38 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:52:38.255+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:52:38.567 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:52:38 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:52:38.255+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:52:38.567 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:52:38 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:52:38.255+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:52:38.567 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:52:38 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:52:38.256+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:52:39.566 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:52:39 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:52:39.157+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:52:39.566 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:52:39 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:52:39.157+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:52:39.566 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:52:39 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:52:39.157+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:52:39.566 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:52:39 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:52:39.157+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:52:39.567 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:52:39 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:52:39.266+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:52:39.567 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:52:39 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:52:39.266+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:52:39.567 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:52:39 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:52:39.266+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:52:39.567 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:52:39 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:52:39.266+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:52:39.567 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:52:39 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:52:39.247+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T21:52:39.567 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:52:39 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:52:39.247+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T21:52:39.567 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:52:39 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:52:39.247+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T21:52:39.567 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:52:39 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:52:39.248+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:52:40.566 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:52:40 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:52:40.259+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T21:52:40.566 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:52:40 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:52:40.259+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T21:52:40.566 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:52:40 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:52:40.259+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T21:52:40.566 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:52:40 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:52:40.259+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:52:40.567 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:52:40 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:52:40.273+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:52:40.567 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:52:40 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:52:40.273+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:52:40.567 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:52:40 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:52:40.273+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:52:40.567 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:52:40 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:52:40.274+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:52:40.567 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:52:40 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:52:40.108+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:52:40.567 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:52:40 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:52:40.108+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:52:40.567 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:52:40 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:52:40.108+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:52:40.567 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:52:40 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:52:40.109+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:52:41.448 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:52:41 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:52:41.253+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T21:52:41.448 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:52:41 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:52:41.253+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T21:52:41.449 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:52:41 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:52:41.253+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T21:52:41.449 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:52:41 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:52:41.254+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:52:41.449 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:52:41 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:52:41.316+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:52:41.449 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:52:41 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:52:41.316+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:52:41.449 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:52:41 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:52:41.316+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:52:41.449 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:52:41 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:52:41.317+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:52:41.450 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:52:41 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:52:41.069+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:52:41.450 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:52:41 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:52:41.069+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:52:41.450 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:52:41 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:52:41.069+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:52:41.450 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:52:41 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:52:41.070+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:52:41.816 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Dec 16 21:52:41 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-mon-smithi045[80759]: 2023-12-16T21:52:41.446+0000 7f45a1f8f700 -1 mon.smithi045@0(probing) e2 get_health_metrics reporting 11337 slow ops, oldest is monmgrreport(gid 14214, 0 checks, 1 progress events) 2023-12-16T21:52:42.337 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:52:42 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:52:42.066+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:52:42.337 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:52:42 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:52:42.066+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:52:42.337 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:52:42 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:52:42.066+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:52:42.337 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:52:42 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:52:42.067+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:52:42.338 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:52:42 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:52:42.214+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T21:52:42.338 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:52:42 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:52:42.214+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T21:52:42.338 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:52:42 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:52:42.214+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T21:52:42.338 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:52:42 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:52:42.215+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:52:42.816 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:52:42 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:52:42.331+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:52:42.816 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:52:42 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:52:42.331+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:52:42.816 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:52:42 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:52:42.331+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:52:42.816 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:52:42 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:52:42.332+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:52:43.316 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:52:43 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:52:43.037+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:52:43.316 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:52:43 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:52:43.037+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:52:43.316 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:52:43 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:52:43.037+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:52:43.316 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:52:43 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:52:43.038+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:52:43.316 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:52:43 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:52:43.285+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:52:43.316 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:52:43 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:52:43.285+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:52:43.317 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:52:43 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:52:43.285+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:52:43.317 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:52:43 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:52:43.286+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:52:43.317 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:52:43 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:52:43.201+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T21:52:43.317 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:52:43 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:52:43.201+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T21:52:43.317 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:52:43 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:52:43.201+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T21:52:43.317 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:52:43 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:52:43.202+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:52:44.566 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:52:44 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:52:44.071+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:52:44.566 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:52:44 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:52:44.071+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:52:44.566 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:52:44 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:52:44.071+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:52:44.566 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:52:44 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:52:44.071+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:52:44.567 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:52:44 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:52:44.167+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T21:52:44.567 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:52:44 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:52:44.167+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T21:52:44.567 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:52:44 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:52:44.167+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T21:52:44.567 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:52:44 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:52:44.169+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:52:44.567 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:52:44 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:52:44.246+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:52:44.567 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:52:44 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:52:44.246+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:52:44.567 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:52:44 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:52:44.246+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:52:44.567 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:52:44 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:52:44.247+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:52:45.316 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:52:45 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:52:45.028+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:52:45.316 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:52:45 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:52:45.028+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:52:45.316 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:52:45 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:52:45.028+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:52:45.316 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:52:45 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:52:45.029+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:52:45.317 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:52:45 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:52:45.124+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T21:52:45.317 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:52:45 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:52:45.124+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T21:52:45.317 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:52:45 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:52:45.124+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T21:52:45.317 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:52:45 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:52:45.125+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:52:45.317 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:52:45 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:52:45.238+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:52:45.317 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:52:45 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:52:45.238+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:52:45.317 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:52:45 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:52:45.238+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:52:45.317 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:52:45 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:52:45.239+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:52:46.352 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:52:46 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:52:46.136+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T21:52:46.353 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:52:46 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:52:46.136+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T21:52:46.353 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:52:46 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:52:46.136+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T21:52:46.353 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:52:46 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:52:46.137+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:52:46.353 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:52:46 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:52:46.255+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:52:46.353 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:52:46 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:52:46.255+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:52:46.353 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:52:46 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:52:46.255+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:52:46.353 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:52:46 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:52:46.256+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:52:46.354 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:52:46 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:52:46.073+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:52:46.354 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:52:46 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:52:46.073+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:52:46.354 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:52:46 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:52:46.073+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:52:46.354 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:52:46 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:52:46.075+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:52:46.816 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Dec 16 21:52:46 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-mon-smithi045[80759]: 2023-12-16T21:52:46.447+0000 7f45a1f8f700 -1 mon.smithi045@0(probing) e2 get_health_metrics reporting 11432 slow ops, oldest is monmgrreport(gid 14214, 0 checks, 1 progress events) 2023-12-16T21:52:47.566 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:52:47 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:52:47.246+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:52:47.566 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:52:47 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:52:47.246+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:52:47.566 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:52:47 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:52:47.246+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:52:47.566 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:52:47 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:52:47.247+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:52:47.567 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:52:47 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:52:47.074+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:52:47.567 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:52:47 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:52:47.074+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:52:47.567 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:52:47 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:52:47.074+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:52:47.567 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:52:47 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:52:47.075+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:52:47.567 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:52:47 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:52:47.153+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T21:52:47.567 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:52:47 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:52:47.153+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T21:52:47.567 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:52:47 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:52:47.153+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T21:52:47.567 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:52:47 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:52:47.154+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:52:48.316 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:52:48 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:52:48.250+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:52:48.316 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:52:48 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:52:48.250+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:52:48.316 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:52:48 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:52:48.250+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:52:48.316 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:52:48 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:52:48.251+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:52:48.316 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:52:48 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:52:48.038+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:52:48.317 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:52:48 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:52:48.038+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:52:48.317 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:52:48 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:52:48.038+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:52:48.317 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:52:48 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:52:48.039+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:52:48.317 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:52:48 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:52:48.172+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T21:52:48.317 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:52:48 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:52:48.172+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T21:52:48.317 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:52:48 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:52:48.172+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T21:52:48.317 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:52:48 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:52:48.173+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:52:49.287 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:52:49 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:52:49.159+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T21:52:49.287 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:52:49 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:52:49.159+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T21:52:49.287 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:52:49 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:52:49.159+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T21:52:49.287 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:52:49 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:52:49.160+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:52:49.287 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:52:49 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:52:49.012+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:52:49.287 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:52:49 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:52:49.013+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:52:49.287 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:52:49 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:52:49.013+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:52:49.288 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:52:49 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:52:49.014+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:52:49.566 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:52:49 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:52:49.284+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:52:49.566 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:52:49 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:52:49.284+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:52:49.566 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:52:49 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:52:49.284+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:52:49.566 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:52:49 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:52:49.285+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:52:50.316 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:52:50 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:52:50.289+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:52:50.316 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:52:50 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:52:50.289+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:52:50.316 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:52:50 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:52:50.289+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:52:50.316 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:52:50 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:52:50.290+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:52:50.317 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:52:50 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:52:50.060+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:52:50.317 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:52:50 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:52:50.060+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:52:50.317 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:52:50 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:52:50.060+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:52:50.317 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:52:50 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:52:50.061+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:52:50.318 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:52:50 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:52:50.154+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T21:52:50.318 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:52:50 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:52:50.154+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T21:52:50.318 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:52:50 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:52:50.154+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T21:52:50.318 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:52:50 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:52:50.155+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:52:51.451 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:52:51 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:52:51.281+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:52:51.451 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:52:51 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:52:51.281+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:52:51.451 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:52:51 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:52:51.281+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:52:51.451 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:52:51 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:52:51.282+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:52:51.452 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:52:51 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:52:51.133+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T21:52:51.452 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:52:51 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:52:51.133+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T21:52:51.452 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:52:51 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:52:51.133+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T21:52:51.452 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:52:51 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:52:51.133+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:52:51.453 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:52:51 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:52:51.090+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:52:51.453 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:52:51 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:52:51.090+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:52:51.453 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:52:51 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:52:51.090+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:52:51.453 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:52:51 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:52:51.091+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:52:51.816 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Dec 16 21:52:51 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-mon-smithi045[80759]: 2023-12-16T21:52:51.448+0000 7f45a1f8f700 -1 mon.smithi045@0(probing) e2 get_health_metrics reporting 11528 slow ops, oldest is monmgrreport(gid 14214, 0 checks, 1 progress events) 2023-12-16T21:52:52.316 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:52:52 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:52:52.046+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:52:52.316 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:52:52 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:52:52.046+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:52:52.316 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:52:52 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:52:52.046+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:52:52.316 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:52:52 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:52:52.047+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:52:52.316 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:52:52 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:52:52.099+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T21:52:52.316 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:52:52 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:52:52.099+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T21:52:52.316 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:52:52 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:52:52.099+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T21:52:52.316 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:52:52 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:52:52.100+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:52:52.816 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:52:52 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:52:52.321+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:52:52.816 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:52:52 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:52:52.321+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:52:52.816 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:52:52 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:52:52.321+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:52:52.816 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:52:52 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:52:52.321+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:52:53.316 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:52:53 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:52:53.136+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T21:52:53.316 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:52:53 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:52:53.136+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T21:52:53.316 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:52:53 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:52:53.136+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T21:52:53.316 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:52:53 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:52:53.137+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:52:53.316 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:52:53 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:52:53.034+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:52:53.316 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:52:53 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:52:53.034+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:52:53.317 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:52:53 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:52:53.034+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:52:53.317 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:52:53 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:52:53.034+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:52:53.816 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:52:53 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:52:53.335+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:52:53.816 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:52:53 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:52:53.335+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:52:53.816 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:52:53 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:52:53.335+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:52:53.816 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:52:53 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:52:53.336+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:52:54.316 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:52:54 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:52:54.163+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T21:52:54.316 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:52:54 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:52:54.163+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T21:52:54.316 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:52:54 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:52:54.163+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T21:52:54.316 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:52:54 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:52:54.163+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:52:54.316 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:52:54 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:52:54.020+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:52:54.316 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:52:54 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:52:54.020+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:52:54.316 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:52:54 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:52:54.020+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:52:54.316 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:52:54 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:52:54.021+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:52:54.816 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:52:54 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:52:54.382+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:52:54.816 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:52:54 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:52:54.382+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:52:54.816 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:52:54 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:52:54.382+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:52:54.816 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:52:54 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:52:54.383+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:52:55.316 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:52:55 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:52:55.004+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:52:55.316 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:52:55 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:52:55.004+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:52:55.316 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:52:55 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:52:55.004+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:52:55.316 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:52:55 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:52:55.005+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:52:55.316 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:52:55 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:52:55.128+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T21:52:55.316 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:52:55 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:52:55.128+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T21:52:55.316 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:52:55 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:52:55.128+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T21:52:55.317 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:52:55 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:52:55.129+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:52:55.816 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:52:55 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:52:55.422+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:52:55.816 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:52:55 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:52:55.422+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:52:55.816 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:52:55 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:52:55.422+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:52:55.816 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:52:55 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:52:55.423+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:52:56.316 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:52:56 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:52:56.026+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:52:56.316 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:52:56 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:52:56.026+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:52:56.316 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:52:56 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:52:56.026+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:52:56.316 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:52:56 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:52:56.027+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:52:56.316 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:52:56 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:52:56.133+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T21:52:56.316 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:52:56 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:52:56.133+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T21:52:56.316 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:52:56 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:52:56.133+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T21:52:56.317 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:52:56 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:52:56.135+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:52:56.815 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Dec 16 21:52:56 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-mon-smithi045[80759]: 2023-12-16T21:52:56.450+0000 7f45a1f8f700 -1 mon.smithi045@0(probing) e2 get_health_metrics reporting 11622 slow ops, oldest is monmgrreport(gid 14214, 0 checks, 1 progress events) 2023-12-16T21:52:56.816 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:52:56 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:52:56.456+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:52:56.816 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:52:56 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:52:56.456+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:52:56.816 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:52:56 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:52:56.456+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:52:56.816 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:52:56 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:52:56.457+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:52:57.317 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:52:57 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:52:57.000+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:52:57.317 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:52:57 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:52:57.000+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:52:57.317 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:52:57 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:52:57.000+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:52:57.317 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:52:57 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:52:57.001+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:52:57.317 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:52:57 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:52:57.184+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T21:52:57.318 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:52:57 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:52:57.184+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T21:52:57.318 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:52:57 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:52:57.184+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T21:52:57.318 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:52:57 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:52:57.185+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:52:57.816 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:52:57 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:52:57.412+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:52:57.816 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:52:57 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:52:57.412+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:52:57.816 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:52:57 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:52:57.412+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:52:57.816 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:52:57 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:52:57.413+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:52:58.316 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:52:58 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:52:58.018+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:52:58.316 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:52:58 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:52:58.019+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:52:58.316 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:52:58 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:52:58.019+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:52:58.316 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:52:58 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:52:58.019+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:52:58.316 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:52:58 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:52:58.164+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T21:52:58.316 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:52:58 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:52:58.164+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T21:52:58.316 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:52:58 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:52:58.164+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T21:52:58.316 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:52:58 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:52:58.165+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:52:58.722 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:52:58 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:52:58.422+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:52:58.722 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:52:58 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:52:58.422+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:52:58.722 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:52:58 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:52:58.422+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:52:58.722 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:52:58 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:52:58.423+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:52:59.315 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:52:59 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:52:59.152+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T21:52:59.316 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:52:59 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:52:59.152+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T21:52:59.316 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:52:59 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:52:59.152+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T21:52:59.316 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:52:59 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:52:59.153+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:52:59.316 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:52:59 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:52:59.010+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:52:59.316 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:52:59 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:52:59.010+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:52:59.316 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:52:59 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:52:59.010+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:52:59.316 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:52:59 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:52:59.011+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:52:59.815 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:52:59 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:52:59.392+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:52:59.816 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:52:59 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:52:59.392+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:52:59.816 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:52:59 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:52:59.392+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:52:59.816 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:52:59 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:52:59.393+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:53:00.316 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:52:59 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:52:59.991+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:53:00.316 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:52:59 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:52:59.991+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:53:00.316 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:52:59 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:52:59.991+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:53:00.316 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:52:59 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:52:59.992+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:53:00.316 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:53:00 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:53:00.170+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T21:53:00.316 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:53:00 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:53:00.170+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T21:53:00.316 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:53:00 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:53:00.170+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T21:53:00.316 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:53:00 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:53:00.171+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:53:00.816 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:53:00 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:53:00.381+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:53:00.816 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:53:00 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:53:00.381+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:53:00.816 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:53:00 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:53:00.381+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:53:00.816 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:53:00 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:53:00.383+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:53:01.316 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:53:01 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:53:01.144+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T21:53:01.316 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:53:01 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:53:01.144+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T21:53:01.316 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:53:01 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:53:01.144+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T21:53:01.316 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:53:01 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:53:01.145+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:53:01.316 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:53:00 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:53:00.980+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:53:01.316 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:53:00 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:53:00.980+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:53:01.316 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:53:00 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:53:00.980+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:53:01.316 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:53:00 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:53:00.981+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:53:01.676 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Dec 16 21:53:01 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-mon-smithi045[80759]: 2023-12-16T21:53:01.451+0000 7f45a1f8f700 -1 mon.smithi045@0(probing) e2 get_health_metrics reporting 11718 slow ops, oldest is monmgrreport(gid 14214, 0 checks, 1 progress events) 2023-12-16T21:53:01.676 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:53:01 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:53:01.345+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:53:01.677 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:53:01 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:53:01.345+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:53:01.677 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:53:01 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:53:01.345+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:53:01.677 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:53:01 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:53:01.345+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:53:02.307 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:53:02 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:53:02.104+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T21:53:02.307 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:53:02 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:53:02.104+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T21:53:02.307 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:53:02 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:53:02.104+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T21:53:02.307 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:53:02 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:53:02.105+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:53:02.307 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:53:01 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:53:01.939+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:53:02.307 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:53:01 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:53:01.939+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:53:02.308 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:53:01 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:53:01.939+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:53:02.308 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:53:01 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:53:01.939+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:53:02.566 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:53:02 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:53:02.304+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:53:02.566 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:53:02 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:53:02.304+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:53:02.566 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:53:02 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:53:02.304+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:53:02.566 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:53:02 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:53:02.305+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:53:03.316 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:53:02 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:53:02.924+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:53:03.316 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:53:02 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:53:02.924+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:53:03.316 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:53:02 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:53:02.924+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:53:03.316 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:53:02 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:53:02.925+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:53:03.317 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:53:03 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:53:03.109+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T21:53:03.317 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:53:03 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:53:03.109+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T21:53:03.317 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:53:03 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:53:03.109+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T21:53:03.317 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:53:03 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:53:03.110+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:53:03.816 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:53:03 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:53:03.332+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:53:03.816 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:53:03 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:53:03.332+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:53:03.816 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:53:03 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:53:03.332+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:53:03.816 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:53:03 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:53:03.333+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:53:04.316 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:53:04 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:53:04.083+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T21:53:04.316 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:53:04 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:53:04.083+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T21:53:04.316 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:53:04 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:53:04.083+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T21:53:04.316 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:53:04 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:53:04.083+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:53:04.316 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:53:03 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:53:03.964+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:53:04.316 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:53:03 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:53:03.964+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:53:04.316 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:53:03 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:53:03.964+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:53:04.316 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:53:03 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:53:03.965+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:53:04.784 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:53:04 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:53:04.321+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:53:04.784 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:53:04 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:53:04.321+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:53:04.784 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:53:04 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:53:04.321+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:53:04.784 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:53:04 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:53:04.322+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:53:05.065 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:53:04 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:53:04.984+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:53:05.066 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:53:04 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:53:04.984+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:53:05.066 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:53:04 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:53:04.984+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:53:05.066 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:53:04 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:53:04.985+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:53:05.566 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:53:05 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:53:05.274+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:53:05.566 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:53:05 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:53:05.274+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:53:05.566 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:53:05 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:53:05.274+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:53:05.566 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:53:05 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:53:05.275+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:53:05.566 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:53:05 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:53:05.105+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T21:53:05.566 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:53:05 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:53:05.105+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T21:53:05.566 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:53:05 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:53:05.105+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T21:53:05.566 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:53:05 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:53:05.106+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:53:06.286 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:53:06 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:53:06.126+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T21:53:06.286 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:53:06 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:53:06.126+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T21:53:06.286 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:53:06 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:53:06.126+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T21:53:06.286 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:53:06 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:53:06.126+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:53:06.287 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:53:06 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:53:06.020+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:53:06.287 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:53:06 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:53:06.020+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:53:06.287 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:53:06 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:53:06.020+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:53:06.287 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:53:06 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:53:06.021+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:53:06.566 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Dec 16 21:53:06 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-mon-smithi045[80759]: 2023-12-16T21:53:06.452+0000 7f45a1f8f700 -1 mon.smithi045@0(probing) e2 get_health_metrics reporting 11812 slow ops, oldest is monmgrreport(gid 14214, 0 checks, 1 progress events) 2023-12-16T21:53:06.566 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:53:06 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:53:06.283+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:53:06.566 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:53:06 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:53:06.283+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:53:06.566 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:53:06 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:53:06.283+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:53:06.566 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:53:06 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:53:06.283+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:53:07.295 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:53:07 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:53:07.101+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T21:53:07.296 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:53:07 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:53:07.101+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T21:53:07.296 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:53:07 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:53:07.101+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T21:53:07.296 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:53:07 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:53:07.101+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:53:07.296 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:53:07 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:53:07.019+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:53:07.296 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:53:07 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:53:07.019+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:53:07.296 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:53:07 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:53:07.019+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:53:07.296 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:53:07 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:53:07.019+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:53:07.566 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:53:07 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:53:07.293+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:53:07.566 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:53:07 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:53:07.293+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:53:07.566 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:53:07 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:53:07.293+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:53:07.566 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:53:07 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:53:07.294+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:53:08.259 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:53:08 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:53:08.099+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T21:53:08.259 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:53:08 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:53:08.099+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T21:53:08.259 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:53:08 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:53:08.099+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T21:53:08.259 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:53:08 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:53:08.100+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:53:08.259 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:53:07 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:53:07.978+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:53:08.259 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:53:07 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:53:07.978+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:53:08.259 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:53:07 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:53:07.978+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:53:08.259 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:53:07 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:53:07.978+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:53:08.566 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:53:08 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:53:08.256+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:53:08.566 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:53:08 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:53:08.256+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:53:08.566 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:53:08 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:53:08.256+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:53:08.566 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:53:08 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:53:08.258+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:53:09.316 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:53:08 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:53:08.995+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:53:09.316 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:53:08 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:53:08.995+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:53:09.316 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:53:08 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:53:08.995+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:53:09.316 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:53:08 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:53:08.997+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:53:09.317 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:53:09 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:53:09.073+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T21:53:09.317 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:53:09 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:53:09.073+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T21:53:09.317 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:53:09 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:53:09.073+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T21:53:09.317 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:53:09 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:53:09.074+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:53:09.317 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:53:09 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:53:09.223+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:53:09.317 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:53:09 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:53:09.223+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:53:09.317 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:53:09 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:53:09.223+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:53:09.317 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:53:09 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:53:09.224+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:53:10.316 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:53:10 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:53:10.103+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T21:53:10.316 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:53:10 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:53:10.103+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T21:53:10.316 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:53:10 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:53:10.103+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T21:53:10.316 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:53:10 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:53:10.105+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:53:10.316 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:53:10 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:53:10.226+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:53:10.317 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:53:10 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:53:10.226+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:53:10.317 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:53:10 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:53:10.226+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:53:10.317 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:53:10 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:53:10.227+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:53:10.317 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:53:10 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:53:10.001+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:53:10.317 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:53:10 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:53:10.001+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:53:10.317 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:53:10 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:53:10.001+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:53:10.317 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:53:10 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:53:10.002+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:53:11.225 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:53:11 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:53:11.146+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T21:53:11.225 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:53:11 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:53:11.146+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T21:53:11.225 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:53:11 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:53:11.146+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T21:53:11.226 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:53:11 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:53:11.147+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:53:11.226 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:53:10 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:53:10.957+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:53:11.226 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:53:10 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:53:10.957+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:53:11.226 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:53:10 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:53:10.957+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:53:11.226 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:53:10 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:53:10.958+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:53:11.480 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Dec 16 21:53:11 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-mon-smithi045[80759]: 2023-12-16T21:53:11.454+0000 7f45a1f8f700 -1 mon.smithi045@0(probing) e2 get_health_metrics reporting 11908 slow ops, oldest is monmgrreport(gid 14214, 0 checks, 1 progress events) 2023-12-16T21:53:11.480 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:53:11 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:53:11.223+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:53:11.480 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:53:11 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:53:11.223+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:53:11.481 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:53:11 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:53:11.223+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:53:11.481 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:53:11 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:53:11.223+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:53:12.228 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:53:11 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:53:11.936+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:53:12.228 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:53:11 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:53:11.936+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:53:12.228 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:53:11 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:53:11.937+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:53:12.228 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:53:11 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:53:11.937+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:53:12.228 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:53:12 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:53:12.149+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T21:53:12.229 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:53:12 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:53:12.149+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T21:53:12.229 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:53:12 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:53:12.149+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T21:53:12.229 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:53:12 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:53:12.150+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:53:12.566 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:53:12 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:53:12.225+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:53:12.566 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:53:12 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:53:12.225+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:53:12.566 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:53:12 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:53:12.225+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:53:12.566 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:53:12 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:53:12.226+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:53:13.260 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:53:13 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:53:13.163+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T21:53:13.260 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:53:13 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:53:13.163+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T21:53:13.260 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:53:13 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:53:13.163+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T21:53:13.260 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:53:13 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:53:13.164+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:53:13.260 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:53:12 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:53:12.974+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:53:13.260 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:53:12 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:53:12.974+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:53:13.261 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:53:12 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:53:12.974+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:53:13.261 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:53:12 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:53:12.975+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:53:13.566 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:53:13 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:53:13.257+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:53:13.566 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:53:13 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:53:13.257+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:53:13.566 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:53:13 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:53:13.257+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:53:13.566 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:53:13 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:53:13.258+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:53:14.280 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:53:14 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:53:14.125+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T21:53:14.281 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:53:14 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:53:14.125+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T21:53:14.281 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:53:14 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:53:14.125+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T21:53:14.281 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:53:14 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:53:14.126+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:53:14.281 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:53:13 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:53:13.957+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:53:14.281 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:53:13 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:53:13.957+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:53:14.281 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:53:13 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:53:13.957+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:53:14.281 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:53:13 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:53:13.958+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:53:14.566 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:53:14 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:53:14.278+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:53:14.566 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:53:14 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:53:14.278+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:53:14.566 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:53:14 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:53:14.278+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:53:14.566 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:53:14 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:53:14.279+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:53:15.253 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:53:15 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:53:15.096+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T21:53:15.253 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:53:15 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:53:15.096+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T21:53:15.254 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:53:15 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:53:15.096+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T21:53:15.254 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:53:15 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:53:15.096+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:53:15.254 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:53:14 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:53:14.943+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:53:15.254 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:53:14 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:53:14.943+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:53:15.254 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:53:14 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:53:14.943+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:53:15.254 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:53:14 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:53:14.944+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:53:15.565 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:53:15 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:53:15.251+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:53:15.566 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:53:15 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:53:15.251+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:53:15.566 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:53:15 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:53:15.251+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:53:15.566 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:53:15 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:53:15.252+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:53:16.210 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:53:16 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:53:16.059+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T21:53:16.210 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:53:16 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:53:16.059+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T21:53:16.210 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:53:16 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:53:16.059+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T21:53:16.210 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:53:16 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:53:16.060+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:53:16.211 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:53:15 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:53:15.931+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:53:16.211 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:53:15 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:53:15.931+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:53:16.211 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:53:15 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:53:15.931+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:53:16.211 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:53:15 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:53:15.932+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:53:16.471 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Dec 16 21:53:16 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-mon-smithi045[80759]: 2023-12-16T21:53:16.455+0000 7f45a1f8f700 -1 mon.smithi045@0(probing) e2 get_health_metrics reporting 12002 slow ops, oldest is monmgrreport(gid 14214, 0 checks, 1 progress events) 2023-12-16T21:53:16.471 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:53:16 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:53:16.207+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:53:16.471 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:53:16 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:53:16.208+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:53:16.472 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:53:16 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:53:16.208+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:53:16.472 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:53:16 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:53:16.208+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:53:17.235 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:53:16 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:53:16.927+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:53:17.235 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:53:16 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:53:16.927+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:53:17.235 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:53:16 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:53:16.927+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:53:17.235 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:53:16 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:53:16.928+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:53:17.235 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:53:17 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:53:17.046+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T21:53:17.235 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:53:17 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:53:17.046+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T21:53:17.236 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:53:17 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:53:17.046+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T21:53:17.236 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:53:17 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:53:17.047+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:53:17.566 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:53:17 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:53:17.232+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:53:17.566 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:53:17 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:53:17.232+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:53:17.566 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:53:17 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:53:17.232+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:53:17.566 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:53:17 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:53:17.233+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:53:18.199 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:53:18 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:53:18.011+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T21:53:18.199 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:53:18 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:53:18.011+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T21:53:18.199 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:53:18 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:53:18.011+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T21:53:18.199 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:53:18 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:53:18.013+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:53:18.200 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:53:17 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:53:17.888+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:53:18.200 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:53:17 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:53:17.888+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:53:18.200 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:53:17 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:53:17.888+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:53:18.200 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:53:17 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:53:17.889+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:53:18.565 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:53:18 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:53:18.196+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:53:18.566 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:53:18 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:53:18.196+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:53:18.566 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:53:18 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:53:18.196+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:53:18.566 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:53:18 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:53:18.197+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:53:18.987 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:53:18 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:53:18.932+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:53:18.987 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:53:18 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:53:18.932+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:53:18.987 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:53:18 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:53:18.932+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:53:18.987 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:53:18 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:53:18.933+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:53:19.316 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:53:19 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:53:19.154+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:53:19.316 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:53:19 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:53:19.154+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:53:19.316 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:53:19 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:53:19.154+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:53:19.316 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:53:19 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:53:19.155+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:53:19.316 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:53:18 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:53:18.984+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T21:53:19.316 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:53:18 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:53:18.984+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T21:53:19.316 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:53:18 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:53:18.984+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T21:53:19.316 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:53:18 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:53:18.985+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:53:20.316 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:53:20 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:53:20.017+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T21:53:20.316 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:53:20 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:53:20.017+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T21:53:20.316 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:53:20 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:53:20.017+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T21:53:20.316 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:53:20 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:53:20.018+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:53:20.316 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:53:20 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:53:20.163+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:53:20.316 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:53:20 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:53:20.163+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:53:20.316 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:53:20 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:53:20.163+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:53:20.316 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:53:20 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:53:20.164+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:53:20.317 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:53:19 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:53:19.935+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:53:20.317 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:53:19 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:53:19.935+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:53:20.317 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:53:19 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:53:19.935+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:53:20.317 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:53:19 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:53:19.935+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:53:21.316 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:53:20 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:53:20.962+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:53:21.316 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:53:20 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:53:20.962+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:53:21.316 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:53:20 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:53:20.962+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:53:21.316 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:53:20 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:53:20.962+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:53:21.316 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:53:21 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:53:21.191+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:53:21.316 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:53:21 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:53:21.191+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:53:21.316 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:53:21 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:53:21.191+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:53:21.316 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:53:21 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:53:21.191+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:53:21.317 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:53:21 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:53:21.045+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T21:53:21.317 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:53:21 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:53:21.045+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T21:53:21.317 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:53:21 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:53:21.045+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T21:53:21.317 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:53:21 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:53:21.045+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:53:21.818 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Dec 16 21:53:21 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-mon-smithi045[80759]: 2023-12-16T21:53:21.456+0000 7f45a1f8f700 -1 mon.smithi045@0(probing) e2 get_health_metrics reporting 12098 slow ops, oldest is monmgrreport(gid 14214, 0 checks, 1 progress events) 2023-12-16T21:53:22.316 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:53:22 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:53:22.179+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:53:22.316 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:53:22 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:53:22.179+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:53:22.316 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:53:22 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:53:22.179+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:53:22.316 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:53:22 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:53:22.180+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:53:22.316 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:53:21 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:53:21.993+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:53:22.316 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:53:21 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:53:21.993+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:53:22.316 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:53:21 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:53:21.993+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:53:22.316 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:53:21 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:53:21.994+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:53:22.317 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:53:22 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:53:22.072+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T21:53:22.317 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:53:22 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:53:22.072+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T21:53:22.317 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:53:22 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:53:22.072+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T21:53:22.317 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:53:22 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:53:22.072+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:53:23.222 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:53:23 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:53:23.075+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T21:53:23.222 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:53:23 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:53:23.075+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T21:53:23.222 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:53:23 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:53:23.075+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T21:53:23.222 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:53:23 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:53:23.075+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:53:23.222 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:53:22 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:53:22.967+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:53:23.222 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:53:22 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:53:22.967+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:53:23.222 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:53:22 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:53:22.967+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:53:23.222 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:53:22 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:53:22.967+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:53:23.566 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:53:23 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:53:23.219+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:53:23.566 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:53:23 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:53:23.219+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:53:23.566 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:53:23 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:53:23.219+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:53:23.566 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:53:23 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:53:23.220+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:53:24.239 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:53:23 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:53:23.955+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:53:24.239 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:53:23 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:53:23.955+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:53:24.239 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:53:23 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:53:23.955+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:53:24.240 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:53:23 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:53:23.956+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:53:24.240 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:53:24 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:53:24.091+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T21:53:24.240 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:53:24 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:53:24.091+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T21:53:24.240 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:53:24 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:53:24.091+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T21:53:24.240 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:53:24 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:53:24.092+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:53:24.565 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:53:24 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:53:24.237+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:53:24.566 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:53:24 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:53:24.237+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:53:24.566 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:53:24 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:53:24.237+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:53:24.566 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:53:24 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:53:24.238+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:53:25.191 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:53:24 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:53:24.923+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:53:25.191 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:53:24 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:53:24.923+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:53:25.192 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:53:24 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:53:24.923+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:53:25.192 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:53:24 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:53:24.924+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:53:25.192 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:53:25 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:53:25.062+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T21:53:25.192 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:53:25 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:53:25.062+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T21:53:25.192 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:53:25 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:53:25.062+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T21:53:25.192 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:53:25 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:53:25.063+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:53:25.565 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:53:25 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:53:25.189+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:53:25.566 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:53:25 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:53:25.189+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:53:25.566 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:53:25 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:53:25.189+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:53:25.566 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:53:25 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:53:25.190+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:53:26.169 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:53:25 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:53:25.905+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:53:26.169 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:53:25 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:53:25.905+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:53:26.169 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:53:25 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:53:25.905+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:53:26.169 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:53:25 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:53:25.906+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:53:26.169 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:53:26 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:53:26.046+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T21:53:26.170 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:53:26 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:53:26.046+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T21:53:26.170 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:53:26 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:53:26.046+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T21:53:26.170 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:53:26 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:53:26.048+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:53:26.460 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:53:26 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:53:26.166+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:53:26.460 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:53:26 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:53:26.166+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:53:26.460 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:53:26 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:53:26.166+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:53:26.460 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:53:26 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:53:26.168+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:53:26.815 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Dec 16 21:53:26 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-mon-smithi045[80759]: 2023-12-16T21:53:26.458+0000 7f45a1f8f700 -1 mon.smithi045@0(probing) e2 get_health_metrics reporting 12193 slow ops, oldest is monmgrreport(gid 14214, 0 checks, 1 progress events) 2023-12-16T21:53:27.315 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:53:27 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:53:27.038+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T21:53:27.316 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:53:27 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:53:27.038+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T21:53:27.316 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:53:27 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:53:27.038+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T21:53:27.316 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:53:27 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:53:27.038+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:53:27.316 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:53:26 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:53:26.900+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:53:27.316 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:53:26 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:53:26.900+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:53:27.316 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:53:26 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:53:26.900+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:53:27.316 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:53:26 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:53:26.901+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:53:27.316 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:53:27 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:53:27.148+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:53:27.317 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:53:27 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:53:27.148+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:53:27.317 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:53:27 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:53:27.148+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:53:27.317 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:53:27 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:53:27.150+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:53:28.193 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:53:27 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:53:27.919+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:53:28.193 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:53:27 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:53:27.919+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:53:28.193 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:53:27 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:53:27.919+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:53:28.193 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:53:27 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:53:27.919+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:53:28.194 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:53:28 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:53:28.045+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T21:53:28.194 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:53:28 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:53:28.045+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T21:53:28.194 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:53:28 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:53:28.045+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T21:53:28.194 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:53:28 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:53:28.046+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:53:28.565 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:53:28 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:53:28.191+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:53:28.566 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:53:28 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:53:28.191+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:53:28.566 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:53:28 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:53:28.191+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:53:28.566 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:53:28 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:53:28.192+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:53:29.032 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:53:28 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:53:28.923+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:53:29.032 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:53:28 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:53:28.923+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:53:29.033 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:53:28 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:53:28.923+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:53:29.033 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:53:28 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:53:28.924+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:53:29.316 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:53:29 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:53:29.030+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T21:53:29.316 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:53:29 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:53:29.030+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T21:53:29.316 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:53:29 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:53:29.030+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T21:53:29.316 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:53:29 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:53:29.031+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:53:29.316 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:53:29 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:53:29.216+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:53:29.316 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:53:29 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:53:29.216+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:53:29.316 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:53:29 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:53:29.216+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:53:29.316 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:53:29 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:53:29.217+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:53:30.227 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:53:29 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:53:29.899+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:53:30.227 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:53:29 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:53:29.899+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:53:30.227 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:53:29 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:53:29.899+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:53:30.227 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:53:29 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:53:29.900+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:53:30.227 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:53:30 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:53:30.037+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T21:53:30.227 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:53:30 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:53:30.037+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T21:53:30.228 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:53:30 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:53:30.037+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T21:53:30.228 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:53:30 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:53:30.037+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:53:30.566 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:53:30 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:53:30.224+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:53:30.566 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:53:30 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:53:30.224+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:53:30.566 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:53:30 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:53:30.224+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:53:30.566 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:53:30 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:53:30.225+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:53:31.212 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:53:31 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:53:31.068+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T21:53:31.212 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:53:31 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:53:31.068+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T21:53:31.212 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:53:31 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:53:31.068+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T21:53:31.212 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:53:31 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:53:31.069+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:53:31.212 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:53:30 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:53:30.882+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:53:31.212 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:53:30 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:53:30.882+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:53:31.212 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:53:30 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:53:30.882+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:53:31.213 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:53:30 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:53:30.883+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:53:31.476 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Dec 16 21:53:31 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-mon-smithi045[80759]: 2023-12-16T21:53:31.459+0000 7f45a1f8f700 -1 mon.smithi045@0(probing) e2 get_health_metrics reporting 12290 slow ops, oldest is monmgrreport(gid 14214, 0 checks, 1 progress events) 2023-12-16T21:53:31.476 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:53:31 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:53:31.209+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:53:31.476 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:53:31 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:53:31.209+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:53:31.476 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:53:31 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:53:31.209+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:53:31.476 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:53:31 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:53:31.210+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:53:32.227 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:53:31 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:53:31.887+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:53:32.227 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:53:31 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:53:31.887+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:53:32.227 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:53:31 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:53:31.887+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:53:32.227 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:53:31 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:53:31.887+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:53:32.228 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:53:32 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:53:32.022+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T21:53:32.228 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:53:32 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:53:32.022+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T21:53:32.228 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:53:32 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:53:32.022+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T21:53:32.228 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:53:32 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:53:32.023+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:53:32.565 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:53:32 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:53:32.225+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:53:32.566 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:53:32 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:53:32.225+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:53:32.566 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:53:32 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:53:32.225+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:53:32.566 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:53:32 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:53:32.226+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:53:33.249 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:53:33 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:53:33.066+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T21:53:33.249 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:53:33 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:53:33.066+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T21:53:33.249 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:53:33 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:53:33.066+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T21:53:33.249 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:53:33 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:53:33.067+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:53:33.249 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:53:32 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:53:32.851+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:53:33.249 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:53:32 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:53:32.851+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:53:33.250 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:53:32 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:53:32.851+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:53:33.250 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:53:32 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:53:32.853+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:53:33.565 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:53:33 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:53:33.246+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:53:33.566 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:53:33 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:53:33.247+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:53:33.566 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:53:33 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:53:33.247+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:53:33.566 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:53:33 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:53:33.248+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:53:34.224 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:53:34 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:53:34.046+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T21:53:34.224 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:53:34 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:53:34.046+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T21:53:34.224 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:53:34 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:53:34.046+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T21:53:34.225 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:53:34 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:53:34.047+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:53:34.225 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:53:33 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:53:33.840+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:53:34.225 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:53:33 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:53:33.840+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:53:34.225 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:53:33 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:53:33.840+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:53:34.225 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:53:33 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:53:33.840+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:53:34.566 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:53:34 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:53:34.222+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:53:34.566 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:53:34 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:53:34.222+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:53:34.566 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:53:34 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:53:34.222+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:53:34.566 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:53:34 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:53:34.223+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:53:35.233 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:53:35 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:53:35.003+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T21:53:35.233 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:53:35 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:53:35.003+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T21:53:35.233 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:53:35 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:53:35.003+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T21:53:35.233 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:53:35 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:53:35.004+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:53:35.234 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:53:34 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:53:34.879+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:53:35.234 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:53:34 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:53:34.879+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:53:35.234 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:53:34 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:53:34.879+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:53:35.234 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:53:34 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:53:34.880+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:53:35.505 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:53:35 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:53:35.231+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:53:35.505 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:53:35 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:53:35.231+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:53:35.505 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:53:35 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:53:35.231+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:53:35.505 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:53:35 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:53:35.232+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:53:36.272 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:53:36 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:53:36.037+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T21:53:36.272 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:53:36 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:53:36.037+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T21:53:36.272 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:53:36 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:53:36.037+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T21:53:36.272 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:53:36 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:53:36.038+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:53:36.272 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:53:35 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:53:35.869+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:53:36.272 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:53:35 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:53:35.869+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:53:36.272 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:53:35 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:53:35.869+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:53:36.273 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:53:35 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:53:35.870+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:53:36.565 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Dec 16 21:53:36 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-mon-smithi045[80759]: 2023-12-16T21:53:36.460+0000 7f45a1f8f700 -1 mon.smithi045@0(probing) e2 get_health_metrics reporting 12385 slow ops, oldest is monmgrreport(gid 14214, 0 checks, 1 progress events) 2023-12-16T21:53:36.566 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:53:36 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:53:36.269+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:53:36.566 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:53:36 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:53:36.269+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:53:36.566 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:53:36 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:53:36.269+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:53:36.566 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:53:36 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:53:36.270+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:53:37.239 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:53:37 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:53:37.031+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T21:53:37.239 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:53:37 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:53:37.031+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T21:53:37.239 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:53:37 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:53:37.031+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T21:53:37.239 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:53:37 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:53:37.032+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:53:37.239 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:53:36 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:53:36.900+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:53:37.239 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:53:36 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:53:36.900+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:53:37.240 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:53:36 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:53:36.900+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:53:37.240 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:53:36 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:53:36.901+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:53:37.566 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:53:37 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:53:37.236+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:53:37.566 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:53:37 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:53:37.236+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:53:37.566 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:53:37 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:53:37.236+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:53:37.566 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:53:37 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:53:37.238+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:53:38.275 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:53:38 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:53:38.063+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T21:53:38.275 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:53:38 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:53:38.063+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T21:53:38.275 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:53:38 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:53:38.063+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T21:53:38.276 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:53:38 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:53:38.064+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:53:38.276 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:53:37 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:53:37.905+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:53:38.276 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:53:37 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:53:37.905+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:53:38.276 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:53:37 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:53:37.905+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:53:38.276 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:53:37 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:53:37.905+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:53:38.566 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:53:38 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:53:38.273+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:53:38.566 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:53:38 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:53:38.273+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:53:38.566 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:53:38 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:53:38.273+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:53:38.566 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:53:38 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:53:38.274+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:53:39.066 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:53:38 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:53:38.875+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:53:39.066 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:53:38 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:53:38.875+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:53:39.066 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:53:38 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:53:38.875+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:53:39.066 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:53:38 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:53:38.876+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:53:39.566 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:53:39 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:53:39.084+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T21:53:39.566 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:53:39 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:53:39.084+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T21:53:39.566 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:53:39 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:53:39.084+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T21:53:39.566 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:53:39 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:53:39.084+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:53:39.566 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:53:39 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:53:39.307+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:53:39.566 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:53:39 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:53:39.307+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:53:39.566 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:53:39 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:53:39.307+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:53:39.566 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:53:39 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:53:39.308+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:53:40.287 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:53:40 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:53:40.119+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T21:53:40.287 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:53:40 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:53:40.119+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T21:53:40.288 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:53:40 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:53:40.119+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T21:53:40.288 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:53:40 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:53:40.120+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:53:40.288 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:53:39 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:53:39.910+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:53:40.288 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:53:39 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:53:39.910+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:53:40.288 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:53:39 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:53:39.910+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:53:40.288 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:53:39 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:53:39.911+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:53:40.565 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:53:40 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:53:40.285+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:53:40.566 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:53:40 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:53:40.285+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:53:40.566 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:53:40 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:53:40.285+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:53:40.566 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:53:40 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:53:40.286+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:53:41.162 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:53:40 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:53:40.877+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:53:41.162 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:53:40 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:53:40.877+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:53:41.163 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:53:40 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:53:40.877+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:53:41.163 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:53:40 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:53:40.878+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:53:41.464 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:53:41 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:53:41.321+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:53:41.464 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:53:41 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:53:41.321+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:53:41.464 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:53:41 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:53:41.321+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:53:41.464 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:53:41 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:53:41.322+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:53:41.465 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:53:41 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:53:41.160+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T21:53:41.465 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:53:41 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:53:41.160+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T21:53:41.465 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:53:41 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:53:41.160+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T21:53:41.465 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:53:41 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:53:41.161+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:53:41.815 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Dec 16 21:53:41 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-mon-smithi045[80759]: 2023-12-16T21:53:41.462+0000 7f45a1f8f700 -1 mon.smithi045@0(probing) e2 get_health_metrics reporting 12482 slow ops, oldest is monmgrreport(gid 14214, 0 checks, 1 progress events) 2023-12-16T21:53:42.150 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:53:41 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:53:41.845+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:53:42.150 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:53:41 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:53:41.845+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:53:42.150 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:53:41 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:53:41.845+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:53:42.150 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:53:41 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:53:41.846+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:53:42.566 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:53:42 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:53:42.147+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T21:53:42.566 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:53:42 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:53:42.147+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T21:53:42.566 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:53:42 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:53:42.147+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T21:53:42.566 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:53:42 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:53:42.148+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:53:42.566 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:53:42 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:53:42.360+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:53:42.566 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:53:42 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:53:42.360+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:53:42.566 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:53:42 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:53:42.360+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:53:42.566 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:53:42 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:53:42.362+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:53:43.159 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:53:42 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:53:42.857+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:53:43.160 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:53:42 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:53:42.857+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:53:43.160 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:53:42 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:53:42.857+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:53:43.160 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:53:42 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:53:42.858+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:53:43.566 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:53:43 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:53:43.336+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:53:43.566 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:53:43 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:53:43.336+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:53:43.566 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:53:43 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:53:43.336+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:53:43.566 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:53:43 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:53:43.338+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:53:43.566 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:53:43 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:53:43.157+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T21:53:43.566 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:53:43 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:53:43.157+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T21:53:43.566 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:53:43 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:53:43.157+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T21:53:43.566 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:53:43 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:53:43.158+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:53:44.149 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:53:43 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:53:43.844+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:53:44.149 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:53:43 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:53:43.844+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:53:44.149 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:53:43 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:53:43.844+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:53:44.150 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:53:43 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:53:43.845+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:53:44.566 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:53:44 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:53:44.147+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T21:53:44.566 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:53:44 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:53:44.147+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T21:53:44.566 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:53:44 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:53:44.147+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T21:53:44.566 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:53:44 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:53:44.148+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:53:44.566 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:53:44 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:53:44.333+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:53:44.567 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:53:44 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:53:44.333+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:53:44.567 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:53:44 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:53:44.333+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:53:44.567 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:53:44 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:53:44.334+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:53:45.065 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:53:44 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:53:44.871+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:53:45.066 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:53:44 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:53:44.871+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:53:45.066 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:53:44 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:53:44.871+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:53:45.066 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:53:44 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:53:44.872+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:53:45.565 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:53:45 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:53:45.190+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T21:53:45.566 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:53:45 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:53:45.190+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T21:53:45.566 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:53:45 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:53:45.190+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T21:53:45.566 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:53:45 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:53:45.192+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:53:45.566 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:53:45 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:53:45.302+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:53:45.566 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:53:45 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:53:45.302+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:53:45.566 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:53:45 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:53:45.302+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:53:45.566 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:53:45 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:53:45.302+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:53:46.210 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:53:45 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:53:45.861+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:53:46.210 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:53:45 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:53:45.861+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:53:46.210 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:53:45 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:53:45.861+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:53:46.211 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:53:45 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:53:45.862+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:53:46.465 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:53:46 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:53:46.208+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T21:53:46.466 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:53:46 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:53:46.208+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T21:53:46.466 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:53:46 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:53:46.208+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T21:53:46.466 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:53:46 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:53:46.209+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:53:46.466 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:53:46 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:53:46.287+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:53:46.466 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:53:46 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:53:46.287+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:53:46.466 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:53:46 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:53:46.287+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:53:46.466 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:53:46 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:53:46.288+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:53:46.815 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Dec 16 21:53:46 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-mon-smithi045[80759]: 2023-12-16T21:53:46.463+0000 7f45a1f8f700 -1 mon.smithi045@0(probing) e2 get_health_metrics reporting 12577 slow ops, oldest is monmgrreport(gid 14214, 0 checks, 1 progress events) 2023-12-16T21:53:47.222 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:53:46 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:53:46.907+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:53:47.222 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:53:46 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:53:46.907+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:53:47.222 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:53:46 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:53:46.907+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:53:47.223 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:53:46 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:53:46.907+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:53:47.566 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:53:47 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:53:47.220+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T21:53:47.566 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:53:47 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:53:47.220+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T21:53:47.566 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:53:47 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:53:47.220+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T21:53:47.566 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:53:47 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:53:47.221+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:53:47.566 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:53:47 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:53:47.258+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:53:47.566 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:53:47 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:53:47.258+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:53:47.566 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:53:47 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:53:47.258+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:53:47.566 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:53:47 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:53:47.260+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:53:48.184 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:53:47 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:53:47.871+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:53:48.184 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:53:47 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:53:47.871+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:53:48.184 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:53:47 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:53:47.871+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:53:48.184 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:53:47 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:53:47.872+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:53:48.565 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:53:48 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:53:48.232+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:53:48.566 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:53:48 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:53:48.232+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:53:48.566 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:53:48 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:53:48.232+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:53:48.566 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:53:48 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:53:48.233+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:53:48.566 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:53:48 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:53:48.182+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T21:53:48.566 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:53:48 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:53:48.182+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T21:53:48.566 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:53:48 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:53:48.182+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T21:53:48.566 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:53:48 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:53:48.182+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:53:49.066 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:53:48 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:53:48.832+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:53:49.066 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:53:48 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:53:48.832+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:53:49.066 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:53:48 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:53:48.832+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:53:49.066 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:53:48 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:53:48.833+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:53:49.566 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:53:49 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:53:49.211+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:53:49.566 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:53:49 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:53:49.211+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:53:49.566 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:53:49 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:53:49.211+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:53:49.566 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:53:49 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:53:49.212+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:53:49.566 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:53:49 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:53:49.177+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T21:53:49.567 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:53:49 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:53:49.177+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T21:53:49.567 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:53:49 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:53:49.177+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T21:53:49.567 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:53:49 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:53:49.178+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:53:50.066 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:53:49 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:53:49.786+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:53:50.066 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:53:49 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:53:49.786+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:53:50.066 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:53:49 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:53:49.786+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:53:50.066 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:53:49 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:53:49.788+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:53:50.566 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:53:50 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:53:50.147+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T21:53:50.566 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:53:50 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:53:50.147+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T21:53:50.566 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:53:50 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:53:50.147+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T21:53:50.566 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:53:50 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:53:50.147+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:53:50.566 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:53:50 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:53:50.203+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:53:50.566 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:53:50 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:53:50.203+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:53:50.566 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:53:50 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:53:50.203+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:53:50.566 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:53:50 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:53:50.204+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:53:51.065 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:53:50 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:53:50.754+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:53:51.066 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:53:50 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:53:50.754+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:53:51.066 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:53:50 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:53:50.754+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:53:51.066 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:53:50 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:53:50.754+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:53:51.467 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:53:51 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:53:51.185+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:53:51.467 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:53:51 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:53:51.185+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:53:51.467 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:53:51 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:53:51.185+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:53:51.467 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:53:51 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:53:51.186+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:53:51.467 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:53:51 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:53:51.194+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T21:53:51.467 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:53:51 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:53:51.194+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T21:53:51.468 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:53:51 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:53:51.194+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T21:53:51.468 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:53:51 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:53:51.195+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:53:51.741 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Dec 16 21:53:51 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-mon-smithi045[80759]: 2023-12-16T21:53:51.464+0000 7f45a1f8f700 -1 mon.smithi045@0(probing) e2 get_health_metrics reporting 12674 slow ops, oldest is monmgrreport(gid 14214, 0 checks, 1 progress events) 2023-12-16T21:53:52.065 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:53:51 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:53:51.739+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:53:52.066 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:53:51 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:53:51.739+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:53:52.066 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:53:51 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:53:51.739+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:53:52.066 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:53:51 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:53:51.740+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:53:52.565 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:53:52 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:53:52.186+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:53:52.566 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:53:52 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:53:52.186+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:53:52.566 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:53:52 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:53:52.186+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:53:52.566 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:53:52 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:53:52.187+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:53:52.566 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:53:52 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:53:52.226+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T21:53:52.566 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:53:52 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:53:52.226+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T21:53:52.566 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:53:52 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:53:52.226+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T21:53:52.566 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:53:52 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:53:52.227+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:53:53.065 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:53:52 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:53:52.704+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:53:53.066 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:53:52 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:53:52.704+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:53:53.066 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:53:52 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:53:52.704+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:53:53.066 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:53:52 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:53:52.705+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:53:53.565 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:53:53 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:53:53.229+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T21:53:53.566 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:53:53 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:53:53.229+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T21:53:53.566 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:53:53 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:53:53.229+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T21:53:53.566 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:53:53 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:53:53.230+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:53:53.566 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:53:53 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:53:53.195+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:53:53.566 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:53:53 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:53:53.195+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:53:53.566 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:53:53 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:53:53.195+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:53:53.566 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:53:53 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:53:53.196+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:53:53.936 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:53:53 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:53:53.739+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:53:53.936 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:53:53 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:53:53.739+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:53:53.936 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:53:53 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:53:53.739+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:53:53.936 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:53:53 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:53:53.740+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:53:54.250 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:53:54 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:53:54.159+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:53:54.250 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:53:54 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:53:54.159+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:53:54.250 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:53:54 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:53:54.159+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:53:54.250 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:53:54 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:53:54.160+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:53:54.565 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:53:54 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:53:54.247+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T21:53:54.566 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:53:54 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:53:54.247+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T21:53:54.566 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:53:54 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:53:54.247+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T21:53:54.566 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:53:54 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:53:54.248+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:53:55.066 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:53:54 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:53:54.729+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:53:55.066 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:53:54 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:53:54.729+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:53:55.066 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:53:54 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:53:54.729+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:53:55.066 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:53:54 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:53:54.730+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:53:55.566 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:53:55 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:53:55.119+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:53:55.566 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:53:55 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:53:55.119+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:53:55.566 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:53:55 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:53:55.119+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:53:55.566 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:53:55 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:53:55.120+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:53:55.567 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:53:55 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:53:55.260+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T21:53:55.567 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:53:55 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:53:55.260+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T21:53:55.567 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:53:55 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:53:55.260+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T21:53:55.567 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:53:55 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:53:55.261+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:53:56.065 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:53:55 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:53:55.683+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:53:56.066 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:53:55 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:53:55.683+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:53:56.066 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:53:55 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:53:55.683+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:53:56.066 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:53:55 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:53:55.684+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:53:56.468 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:53:56 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:53:56.109+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:53:56.468 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:53:56 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:53:56.109+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:53:56.468 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:53:56 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:53:56.109+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:53:56.469 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:53:56 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:53:56.110+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:53:56.469 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:53:56 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:53:56.247+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T21:53:56.469 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:53:56 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:53:56.247+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T21:53:56.469 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:53:56 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:53:56.247+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T21:53:56.469 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:53:56 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:53:56.248+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:53:56.733 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Dec 16 21:53:56 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-mon-smithi045[80759]: 2023-12-16T21:53:56.466+0000 7f45a1f8f700 -1 mon.smithi045@0(probing) e2 get_health_metrics reporting 12769 slow ops, oldest is monmgrreport(gid 14214, 0 checks, 1 progress events) 2023-12-16T21:53:57.065 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:53:56 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:53:56.731+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:53:57.066 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:53:56 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:53:56.731+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:53:57.066 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:53:56 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:53:56.731+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:53:57.066 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:53:56 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:53:56.732+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:53:57.566 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:53:57 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:53:57.094+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:53:57.566 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:53:57 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:53:57.094+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:53:57.566 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:53:57 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:53:57.094+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:53:57.566 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:53:57 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:53:57.094+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:53:57.566 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:53:57 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:53:57.206+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T21:53:57.566 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:53:57 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:53:57.206+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T21:53:57.566 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:53:57 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:53:57.206+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T21:53:57.566 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:53:57 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:53:57.206+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:53:58.065 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:53:57 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:53:57.757+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:53:58.066 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:53:57 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:53:57.757+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:53:58.066 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:53:57 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:53:57.757+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:53:58.066 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:53:57 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:53:57.758+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:53:58.565 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:53:58 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:53:58.136+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:53:58.566 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:53:58 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:53:58.136+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:53:58.566 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:53:58 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:53:58.136+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:53:58.566 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:53:58 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:53:58.137+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:53:58.566 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:53:58 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:53:58.170+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T21:53:58.566 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:53:58 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:53:58.170+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T21:53:58.566 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:53:58 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:53:58.170+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T21:53:58.566 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:53:58 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:53:58.171+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:53:59.065 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:53:58 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:53:58.807+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:53:59.066 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:53:58 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:53:58.807+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:53:59.066 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:53:58 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:53:58.807+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:53:59.066 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:53:58 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:53:58.808+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:53:59.566 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:53:59 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:53:59.090+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:53:59.566 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:53:59 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:53:59.090+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:53:59.566 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:53:59 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:53:59.090+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:53:59.566 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:53:59 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:53:59.090+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:53:59.566 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:53:59 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:53:59.195+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T21:53:59.566 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:53:59 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:53:59.195+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T21:53:59.566 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:53:59 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:53:59.195+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T21:53:59.566 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:53:59 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:53:59.196+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:54:00.065 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:53:59 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:53:59.764+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:54:00.066 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:53:59 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:53:59.764+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:54:00.066 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:53:59 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:53:59.764+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:54:00.066 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:53:59 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:53:59.766+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:54:00.566 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:54:00 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:54:00.231+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T21:54:00.566 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:54:00 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:54:00.231+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T21:54:00.566 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:54:00 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:54:00.231+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T21:54:00.566 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:54:00 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:54:00.232+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:54:00.566 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:54:00 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:54:00.133+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:54:00.566 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:54:00 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:54:00.133+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:54:00.566 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:54:00 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:54:00.133+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:54:00.566 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:54:00 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:54:00.134+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:54:01.065 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:54:00 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:54:00.762+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:54:01.066 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:54:00 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:54:00.762+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:54:01.066 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:54:00 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:54:00.762+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:54:01.066 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:54:00 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:54:00.764+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:54:01.470 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:54:01 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:54:01.157+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:54:01.470 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:54:01 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:54:01.157+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:54:01.470 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:54:01 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:54:01.157+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:54:01.470 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:54:01 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:54:01.158+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:54:01.471 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:54:01 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:54:01.262+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T21:54:01.471 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:54:01 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:54:01.262+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T21:54:01.471 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:54:01 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:54:01.262+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T21:54:01.471 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:54:01 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:54:01.264+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:54:01.772 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Dec 16 21:54:01 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-mon-smithi045[80759]: 2023-12-16T21:54:01.467+0000 7f45a1f8f700 -1 mon.smithi045@0(probing) e2 get_health_metrics reporting 12866 slow ops, oldest is monmgrreport(gid 14214, 0 checks, 1 progress events) 2023-12-16T21:54:02.065 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:54:01 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:54:01.770+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:54:02.065 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:54:01 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:54:01.770+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:54:02.066 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:54:01 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:54:01.770+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:54:02.066 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:54:01 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:54:01.771+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:54:02.565 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:54:02 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:54:02.221+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T21:54:02.566 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:54:02 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:54:02.221+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T21:54:02.566 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:54:02 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:54:02.221+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T21:54:02.566 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:54:02 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:54:02.222+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:54:02.566 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:54:02 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:54:02.151+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:54:02.566 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:54:02 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:54:02.151+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:54:02.566 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:54:02 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:54:02.151+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:54:02.566 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:54:02 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:54:02.152+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:54:03.065 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:54:02 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:54:02.813+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:54:03.066 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:54:02 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:54:02.813+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:54:03.066 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:54:02 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:54:02.813+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:54:03.066 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:54:02 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:54:02.814+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:54:03.565 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:54:03 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:54:03.191+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:54:03.566 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:54:03 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:54:03.191+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:54:03.566 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:54:03 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:54:03.191+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:54:03.566 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:54:03 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:54:03.192+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:54:03.566 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:54:03 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:54:03.244+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T21:54:03.566 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:54:03 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:54:03.244+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T21:54:03.566 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:54:03 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:54:03.244+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T21:54:03.566 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:54:03 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:54:03.246+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:54:04.065 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:54:03 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:54:03.776+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:54:04.066 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:54:03 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:54:03.776+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:54:04.066 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:54:03 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:54:03.776+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:54:04.066 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:54:03 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:54:03.777+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:54:04.565 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:54:04 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:54:04.188+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:54:04.566 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:54:04 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:54:04.188+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:54:04.566 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:54:04 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:54:04.188+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:54:04.566 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:54:04 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:54:04.188+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:54:04.566 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:54:04 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:54:04.231+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T21:54:04.566 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:54:04 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:54:04.231+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T21:54:04.566 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:54:04 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:54:04.231+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T21:54:04.566 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:54:04 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:54:04.232+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:54:05.065 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:54:04 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:54:04.796+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:54:05.066 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:54:04 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:54:04.796+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:54:05.066 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:54:04 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:54:04.796+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:54:05.066 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:54:04 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:54:04.797+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:54:05.565 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:54:05 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:54:05.249+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T21:54:05.566 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:54:05 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:54:05.249+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T21:54:05.566 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:54:05 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:54:05.249+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T21:54:05.566 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:54:05 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:54:05.250+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:54:05.566 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:54:05 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:54:05.192+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:54:05.566 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:54:05 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:54:05.192+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:54:05.566 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:54:05 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:54:05.192+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:54:05.566 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:54:05 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:54:05.192+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:54:06.065 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:54:05 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:54:05.777+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:54:06.066 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:54:05 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:54:05.777+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:54:06.066 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:54:05 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:54:05.777+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:54:06.066 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:54:05 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:54:05.778+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:54:06.472 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:54:06 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:54:06.197+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:54:06.473 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:54:06 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:54:06.197+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:54:06.473 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:54:06 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:54:06.197+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:54:06.473 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:54:06 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:54:06.198+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:54:06.474 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:54:06 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:54:06.259+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T21:54:06.474 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:54:06 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:54:06.259+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T21:54:06.474 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:54:06 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:54:06.259+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T21:54:06.474 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:54:06 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:54:06.260+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:54:06.759 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Dec 16 21:54:06 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-mon-smithi045[80759]: 2023-12-16T21:54:06.469+0000 7f45a1f8f700 -1 mon.smithi045@0(probing) e2 get_health_metrics reporting 12961 slow ops, oldest is monmgrreport(gid 14214, 0 checks, 1 progress events) 2023-12-16T21:54:07.065 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:54:06 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:54:06.756+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:54:07.066 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:54:06 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:54:06.756+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:54:07.066 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:54:06 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:54:06.756+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:54:07.066 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:54:06 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:54:06.758+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:54:07.565 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:54:07 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:54:07.221+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T21:54:07.566 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:54:07 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:54:07.221+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T21:54:07.566 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:54:07 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:54:07.221+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T21:54:07.566 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:54:07 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:54:07.222+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:54:07.566 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:54:07 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:54:07.158+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:54:07.566 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:54:07 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:54:07.158+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:54:07.566 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:54:07 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:54:07.158+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:54:07.566 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:54:07 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:54:07.159+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:54:08.065 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:54:07 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:54:07.712+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:54:08.066 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:54:07 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:54:07.712+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:54:08.066 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:54:07 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:54:07.712+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:54:08.066 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:54:07 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:54:07.713+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:54:08.565 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:54:08 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:54:08.250+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T21:54:08.566 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:54:08 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:54:08.250+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T21:54:08.566 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:54:08 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:54:08.250+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T21:54:08.566 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:54:08 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:54:08.251+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:54:08.566 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:54:08 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:54:08.176+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:54:08.566 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:54:08 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:54:08.176+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:54:08.566 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:54:08 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:54:08.176+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:54:08.566 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:54:08 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:54:08.177+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:54:09.066 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:54:08 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:54:08.677+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:54:09.066 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:54:08 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:54:08.678+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:54:09.066 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:54:08 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:54:08.678+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:54:09.066 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:54:08 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:54:08.678+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:54:09.566 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:54:09 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:54:09.190+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:54:09.566 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:54:09 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:54:09.190+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:54:09.566 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:54:09 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:54:09.190+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:54:09.566 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:54:09 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:54:09.191+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:54:09.566 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:54:09 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:54:09.214+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T21:54:09.566 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:54:09 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:54:09.214+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T21:54:09.566 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:54:09 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:54:09.214+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T21:54:09.566 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:54:09 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:54:09.215+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:54:10.066 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:54:09 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:54:09.683+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:54:10.066 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:54:09 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:54:09.683+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:54:10.067 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:54:09 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:54:09.683+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:54:10.067 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:54:09 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:54:09.684+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:54:10.565 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:54:10 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:54:10.202+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T21:54:10.566 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:54:10 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:54:10.202+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T21:54:10.566 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:54:10 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:54:10.202+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T21:54:10.566 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:54:10 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:54:10.202+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:54:10.566 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:54:10 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:54:10.201+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:54:10.566 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:54:10 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:54:10.201+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:54:10.566 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:54:10 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:54:10.201+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:54:10.566 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:54:10 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:54:10.201+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:54:11.015 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:54:10 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:54:10.695+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:54:11.015 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:54:10 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:54:10.695+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:54:11.015 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:54:10 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:54:10.695+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:54:11.015 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:54:10 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:54:10.695+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:54:11.315 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:54:11 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:54:11.230+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T21:54:11.316 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:54:11 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:54:11.230+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T21:54:11.316 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:54:11 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:54:11.230+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T21:54:11.316 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:54:11 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:54:11.231+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:54:11.316 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:54:11 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:54:11.220+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:54:11.316 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:54:11 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:54:11.220+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:54:11.316 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:54:11 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:54:11.220+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:54:11.316 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:54:11 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:54:11.221+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:54:11.729 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Dec 16 21:54:11 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-mon-smithi045[80759]: 2023-12-16T21:54:11.469+0000 7f45a1f8f700 -1 mon.smithi045@0(probing) e2 get_health_metrics reporting 13058 slow ops, oldest is monmgrreport(gid 14214, 0 checks, 1 progress events) 2023-12-16T21:54:12.065 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:54:11 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:54:11.726+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:54:12.066 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:54:11 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:54:11.726+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:54:12.066 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:54:11 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:54:11.726+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:54:12.066 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:54:11 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:54:11.727+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:54:12.565 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:54:12 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:54:12.220+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:54:12.566 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:54:12 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:54:12.220+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:54:12.566 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:54:12 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:54:12.220+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:54:12.566 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:54:12 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:54:12.221+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:54:12.566 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:54:12 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:54:12.191+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T21:54:12.566 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:54:12 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:54:12.191+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T21:54:12.566 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:54:12 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:54:12.191+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T21:54:12.566 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:54:12 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:54:12.191+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:54:13.066 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:54:12 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:54:12.727+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:54:13.066 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:54:12 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:54:12.727+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:54:13.066 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:54:12 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:54:12.727+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:54:13.066 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:54:12 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:54:12.727+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:54:13.566 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:54:13 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:54:13.243+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:54:13.566 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:54:13 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:54:13.243+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:54:13.566 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:54:13 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:54:13.243+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:54:13.566 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:54:13 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:54:13.243+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:54:13.566 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:54:13 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:54:13.221+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T21:54:13.566 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:54:13 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:54:13.221+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T21:54:13.566 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:54:13 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:54:13.221+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T21:54:13.567 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:54:13 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:54:13.222+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:54:14.066 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:54:13 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:54:13.702+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:54:14.066 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:54:13 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:54:13.702+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:54:14.066 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:54:13 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:54:13.702+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:54:14.066 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:54:13 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:54:13.704+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:54:14.565 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:54:14 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:54:14.213+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T21:54:14.566 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:54:14 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:54:14.213+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T21:54:14.566 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:54:14 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:54:14.213+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T21:54:14.566 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:54:14 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:54:14.214+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:54:14.566 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:54:14 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:54:14.237+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:54:14.566 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:54:14 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:54:14.237+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:54:14.566 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:54:14 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:54:14.237+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:54:14.566 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:54:14 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:54:14.238+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:54:15.065 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:54:14 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:54:14.672+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:54:15.066 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:54:14 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:54:14.672+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:54:15.066 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:54:14 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:54:14.672+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:54:15.066 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:54:14 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:54:14.673+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:54:15.565 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:54:15 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:54:15.253+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:54:15.566 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:54:15 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:54:15.253+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:54:15.566 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:54:15 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:54:15.253+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:54:15.566 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:54:15 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:54:15.255+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:54:15.566 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:54:15 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:54:15.215+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T21:54:15.566 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:54:15 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:54:15.215+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T21:54:15.566 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:54:15 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:54:15.215+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T21:54:15.566 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:54:15 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:54:15.217+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:54:16.066 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:54:15 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:54:15.629+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:54:16.066 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:54:15 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:54:15.629+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:54:16.066 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:54:15 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:54:15.629+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:54:16.066 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:54:15 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:54:15.630+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:54:16.496 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Dec 16 21:54:16 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-mon-smithi045[80759]: 2023-12-16T21:54:16.472+0000 7f45a1f8f700 -1 mon.smithi045@0(probing) e2 get_health_metrics reporting 13152 slow ops, oldest is monmgrreport(gid 14214, 0 checks, 1 progress events) 2023-12-16T21:54:16.496 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:54:16 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:54:16.297+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:54:16.497 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:54:16 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:54:16.297+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:54:16.497 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:54:16 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:54:16.297+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:54:16.497 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:54:16 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:54:16.297+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:54:16.497 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:54:16 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:54:16.241+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T21:54:16.497 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:54:16 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:54:16.241+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T21:54:16.497 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:54:16 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:54:16.241+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T21:54:16.497 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:54:16 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:54:16.241+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:54:16.816 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:54:16 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:54:16.666+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:54:16.816 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:54:16 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:54:16.666+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:54:16.816 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:54:16 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:54:16.666+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:54:16.816 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:54:16 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:54:16.667+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:54:17.565 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:54:17 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:54:17.271+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T21:54:17.566 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:54:17 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:54:17.271+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T21:54:17.566 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:54:17 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:54:17.271+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T21:54:17.566 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:54:17 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:54:17.272+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:54:17.566 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:54:17 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:54:17.311+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:54:17.566 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:54:17 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:54:17.311+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:54:17.566 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:54:17 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:54:17.311+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:54:17.566 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:54:17 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:54:17.311+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:54:18.065 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:54:17 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:54:17.662+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:54:18.066 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:54:17 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:54:17.662+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:54:18.066 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:54:17 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:54:17.662+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:54:18.066 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:54:17 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:54:17.662+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:54:18.513 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:54:18 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:54:18.226+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T21:54:18.513 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:54:18 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:54:18.226+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T21:54:18.513 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:54:18 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:54:18.226+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T21:54:18.513 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:54:18 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:54:18.227+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:54:18.513 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:54:18 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:54:18.347+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:54:18.513 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:54:18 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:54:18.347+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:54:18.513 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:54:18 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:54:18.347+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:54:18.513 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:54:18 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:54:18.348+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:54:19.065 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:54:18 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:54:18.712+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:54:19.066 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:54:18 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:54:18.712+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:54:19.066 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:54:18 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:54:18.712+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:54:19.066 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:54:18 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:54:18.713+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:54:19.368 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:54:19 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:54:19.245+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T21:54:19.369 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:54:19 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:54:19.245+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T21:54:19.369 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:54:19 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:54:19.245+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T21:54:19.369 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:54:19 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:54:19.245+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:54:19.752 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:54:19 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:54:19.366+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:54:19.753 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:54:19 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:54:19.366+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:54:19.753 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:54:19 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:54:19.366+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:54:19.753 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:54:19 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:54:19.367+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:54:20.065 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:54:19 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:54:19.750+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:54:20.065 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:54:19 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:54:19.750+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:54:20.065 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:54:19 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:54:19.750+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:54:20.066 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:54:19 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:54:19.751+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:54:20.565 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:54:20 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:54:20.288+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T21:54:20.566 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:54:20 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:54:20.288+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T21:54:20.566 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:54:20 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:54:20.288+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T21:54:20.566 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:54:20 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:54:20.289+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:54:20.566 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:54:20 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:54:20.320+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:54:20.566 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:54:20 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:54:20.320+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:54:20.566 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:54:20 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:54:20.320+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:54:20.566 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:54:20 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:54:20.321+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:54:21.065 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:54:20 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:54:20.711+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:54:21.066 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:54:20 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:54:20.711+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:54:21.066 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:54:20 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:54:20.711+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:54:21.066 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:54:20 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:54:20.711+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:54:21.565 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Dec 16 21:54:21 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-mon-smithi045[80759]: 2023-12-16T21:54:21.473+0000 7f45a1f8f700 -1 mon.smithi045@0(probing) e2 get_health_metrics reporting 13248 slow ops, oldest is monmgrreport(gid 14214, 0 checks, 1 progress events) 2023-12-16T21:54:21.566 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:54:21 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:54:21.348+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:54:21.566 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:54:21 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:54:21.348+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:54:21.566 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:54:21 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:54:21.348+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:54:21.566 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:54:21 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:54:21.349+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:54:21.566 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:54:21 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:54:21.307+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T21:54:21.566 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:54:21 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:54:21.307+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T21:54:21.566 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:54:21 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:54:21.307+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T21:54:21.567 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:54:21 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:54:21.309+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:54:21.830 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:54:21 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:54:21.703+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:54:21.830 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:54:21 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:54:21.703+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:54:21.830 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:54:21 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:54:21.703+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:54:21.830 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:54:21 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:54:21.704+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:54:22.725 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:54:22 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:54:22.316+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T21:54:22.725 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:54:22 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:54:22.316+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T21:54:22.725 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:54:22 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:54:22.316+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T21:54:22.726 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:54:22 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:54:22.316+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:54:22.726 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:54:22 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:54:22.371+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:54:22.726 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:54:22 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:54:22.371+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:54:22.726 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:54:22 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:54:22.371+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:54:22.726 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:54:22 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:54:22.372+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:54:23.065 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:54:22 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:54:22.723+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:54:23.066 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:54:22 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:54:22.723+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:54:23.066 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:54:22 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:54:22.723+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:54:23.066 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:54:22 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:54:22.724+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:54:23.565 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:54:23 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:54:23.309+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T21:54:23.566 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:54:23 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:54:23.309+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T21:54:23.566 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:54:23 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:54:23.309+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T21:54:23.566 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:54:23 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:54:23.310+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:54:23.566 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:54:23 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:54:23.372+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:54:23.566 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:54:23 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:54:23.372+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:54:23.566 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:54:23 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:54:23.372+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:54:23.566 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:54:23 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:54:23.373+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:54:24.065 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:54:23 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:54:23.725+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:54:24.066 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:54:23 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:54:23.725+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:54:24.066 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:54:23 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:54:23.725+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:54:24.066 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:54:23 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:54:23.726+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:54:24.656 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:54:24 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:54:24.344+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T21:54:24.656 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:54:24 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:54:24.345+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T21:54:24.656 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:54:24 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:54:24.345+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T21:54:24.656 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:54:24 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:54:24.345+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:54:24.657 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:54:24 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:54:24.415+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:54:24.657 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:54:24 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:54:24.415+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:54:24.657 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:54:24 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:54:24.415+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:54:24.657 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:54:24 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:54:24.416+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:54:25.065 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:54:24 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:54:24.686+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:54:25.066 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:54:24 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:54:24.686+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:54:25.066 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:54:24 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:54:24.686+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:54:25.066 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:54:24 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:54:24.687+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:54:25.709 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:54:25 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:54:25.455+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:54:25.709 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:54:25 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:54:25.455+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:54:25.710 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:54:25 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:54:25.455+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:54:25.710 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:54:25 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:54:25.456+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:54:25.710 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:54:25 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:54:25.395+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T21:54:25.710 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:54:25 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:54:25.395+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T21:54:25.710 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:54:25 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:54:25.395+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T21:54:25.710 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:54:25 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:54:25.396+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:54:26.066 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:54:25 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:54:25.707+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:54:26.066 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:54:25 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:54:25.707+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:54:26.066 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:54:25 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:54:25.707+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:54:26.066 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:54:25 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:54:25.708+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:54:26.345 INFO:teuthology.orchestra.run.smithi045.stderr:2023-12-16T21:54:26.341+0000 7fb1b96fc700 0 monclient(hunting): authenticate timed out after 300 2023-12-16T21:54:26.345 INFO:teuthology.orchestra.run.smithi045.stderr:[errno 110] RADOS timed out (error connecting to the cluster) 2023-12-16T21:54:26.599 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:54:26 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:54:26.460+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:54:26.599 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:54:26 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:54:26.460+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:54:26.599 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:54:26 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:54:26.460+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:54:26.599 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:54:26 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:54:26.461+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:54:26.599 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:54:26 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:54:26.426+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T21:54:26.600 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:54:26 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:54:26.426+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T21:54:26.600 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:54:26 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:54:26.426+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T21:54:26.600 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:54:26 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:54:26.426+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:54:26.600 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Dec 16 21:54:26 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-mon-smithi045[80759]: 2023-12-16T21:54:26.475+0000 7f45a1f8f700 -1 mon.smithi045@0(probing) e2 get_health_metrics reporting 13338 slow ops, oldest is monmgrreport(gid 14214, 0 checks, 1 progress events) 2023-12-16T21:54:26.798 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-12-16T21:54:26.799 ERROR:teuthology.run_tasks:Saw exception from tasks. Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_5f3731ee8689ef3b0ef7d571e4cc2cb968c78d73/teuthology/run_tasks.py", line 105, in run_tasks manager = run_one_task(taskname, ctx=ctx, config=config) File "/home/teuthworker/src/git.ceph.com_teuthology_5f3731ee8689ef3b0ef7d571e4cc2cb968c78d73/teuthology/run_tasks.py", line 83, in run_one_task return task(**kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_5f3731ee8689ef3b0ef7d571e4cc2cb968c78d73/teuthology/task/parallel.py", line 56, in task p.spawn(_run_spawned, ctx, confg, taskname) File "/home/teuthworker/src/git.ceph.com_teuthology_5f3731ee8689ef3b0ef7d571e4cc2cb968c78d73/teuthology/parallel.py", line 84, in __exit__ for result in self: File "/home/teuthworker/src/git.ceph.com_teuthology_5f3731ee8689ef3b0ef7d571e4cc2cb968c78d73/teuthology/parallel.py", line 98, in __next__ resurrect_traceback(result) File "/home/teuthworker/src/git.ceph.com_teuthology_5f3731ee8689ef3b0ef7d571e4cc2cb968c78d73/teuthology/parallel.py", line 30, in resurrect_traceback raise exc.exc_info[1] File "/home/teuthworker/src/git.ceph.com_teuthology_5f3731ee8689ef3b0ef7d571e4cc2cb968c78d73/teuthology/parallel.py", line 23, in capture_traceback return func(*args, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_5f3731ee8689ef3b0ef7d571e4cc2cb968c78d73/teuthology/task/parallel.py", line 64, in _run_spawned mgr = run_tasks.run_one_task(taskname, ctx=ctx, config=config) File "/home/teuthworker/src/git.ceph.com_teuthology_5f3731ee8689ef3b0ef7d571e4cc2cb968c78d73/teuthology/run_tasks.py", line 83, in run_one_task return task(**kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_5f3731ee8689ef3b0ef7d571e4cc2cb968c78d73/teuthology/task/sequential.py", line 47, in task mgr = run_tasks.run_one_task(taskname, ctx=ctx, config=confg) File "/home/teuthworker/src/git.ceph.com_teuthology_5f3731ee8689ef3b0ef7d571e4cc2cb968c78d73/teuthology/run_tasks.py", line 83, in run_one_task return task(**kwargs) File "/home/teuthworker/src/github.com_ceph_ceph-c_c7586e95519b7be76f3b6e3a4b333888baf46bc9/qa/tasks/cephadm.py", line 1351, in shell _shell(ctx, cluster_name, remote, File "/home/teuthworker/src/github.com_ceph_ceph-c_c7586e95519b7be76f3b6e3a4b333888baf46bc9/qa/tasks/cephadm.py", line 40, in _shell return remote.run( File "/home/teuthworker/src/git.ceph.com_teuthology_5f3731ee8689ef3b0ef7d571e4cc2cb968c78d73/teuthology/orchestra/remote.py", line 523, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_5f3731ee8689ef3b0ef7d571e4cc2cb968c78d73/teuthology/orchestra/run.py", line 455, in run r.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_5f3731ee8689ef3b0ef7d571e4cc2cb968c78d73/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_5f3731ee8689ef3b0ef7d571e4cc2cb968c78d73/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi045 with status 1: 'sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 88ff363c-9c5a-11ee-95a5-87774f69a715 -e sha1=c7586e95519b7be76f3b6e3a4b333888baf46bc9 -- bash -c \'ceph versions | jq -e \'"\'"\'.mgr | keys\'"\'"\' | grep $sha1\'' 2023-12-16T21:54:26.877 ERROR:teuthology.util.sentry: Sentry event: https://sentry.ceph.com/organizations/ceph/?query=c83a120f77f048c9bbb43efdd1f904ee Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_5f3731ee8689ef3b0ef7d571e4cc2cb968c78d73/teuthology/run_tasks.py", line 105, in run_tasks manager = run_one_task(taskname, ctx=ctx, config=config) File "/home/teuthworker/src/git.ceph.com_teuthology_5f3731ee8689ef3b0ef7d571e4cc2cb968c78d73/teuthology/run_tasks.py", line 83, in run_one_task return task(**kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_5f3731ee8689ef3b0ef7d571e4cc2cb968c78d73/teuthology/task/parallel.py", line 56, in task p.spawn(_run_spawned, ctx, confg, taskname) File "/home/teuthworker/src/git.ceph.com_teuthology_5f3731ee8689ef3b0ef7d571e4cc2cb968c78d73/teuthology/parallel.py", line 84, in __exit__ for result in self: File "/home/teuthworker/src/git.ceph.com_teuthology_5f3731ee8689ef3b0ef7d571e4cc2cb968c78d73/teuthology/parallel.py", line 98, in __next__ resurrect_traceback(result) File "/home/teuthworker/src/git.ceph.com_teuthology_5f3731ee8689ef3b0ef7d571e4cc2cb968c78d73/teuthology/parallel.py", line 30, in resurrect_traceback raise exc.exc_info[1] File "/home/teuthworker/src/git.ceph.com_teuthology_5f3731ee8689ef3b0ef7d571e4cc2cb968c78d73/teuthology/parallel.py", line 23, in capture_traceback return func(*args, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_5f3731ee8689ef3b0ef7d571e4cc2cb968c78d73/teuthology/task/parallel.py", line 64, in _run_spawned mgr = run_tasks.run_one_task(taskname, ctx=ctx, config=config) File "/home/teuthworker/src/git.ceph.com_teuthology_5f3731ee8689ef3b0ef7d571e4cc2cb968c78d73/teuthology/run_tasks.py", line 83, in run_one_task return task(**kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_5f3731ee8689ef3b0ef7d571e4cc2cb968c78d73/teuthology/task/sequential.py", line 47, in task mgr = run_tasks.run_one_task(taskname, ctx=ctx, config=confg) File "/home/teuthworker/src/git.ceph.com_teuthology_5f3731ee8689ef3b0ef7d571e4cc2cb968c78d73/teuthology/run_tasks.py", line 83, in run_one_task return task(**kwargs) File "/home/teuthworker/src/github.com_ceph_ceph-c_c7586e95519b7be76f3b6e3a4b333888baf46bc9/qa/tasks/cephadm.py", line 1351, in shell _shell(ctx, cluster_name, remote, File "/home/teuthworker/src/github.com_ceph_ceph-c_c7586e95519b7be76f3b6e3a4b333888baf46bc9/qa/tasks/cephadm.py", line 40, in _shell return remote.run( File "/home/teuthworker/src/git.ceph.com_teuthology_5f3731ee8689ef3b0ef7d571e4cc2cb968c78d73/teuthology/orchestra/remote.py", line 523, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_5f3731ee8689ef3b0ef7d571e4cc2cb968c78d73/teuthology/orchestra/run.py", line 455, in run r.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_5f3731ee8689ef3b0ef7d571e4cc2cb968c78d73/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_5f3731ee8689ef3b0ef7d571e4cc2cb968c78d73/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi045 with status 1: 'sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 88ff363c-9c5a-11ee-95a5-87774f69a715 -e sha1=c7586e95519b7be76f3b6e3a4b333888baf46bc9 -- bash -c \'ceph versions | jq -e \'"\'"\'.mgr | keys\'"\'"\' | grep $sha1\'' 2023-12-16T21:54:26.881 DEBUG:teuthology.run_tasks:Unwinding manager kclient 2023-12-16T21:54:26.892 INFO:tasks.kclient:Unmounting kernel clients... 2023-12-16T21:54:26.893 DEBUG:teuthology.orchestra.run.smithi045:> set -ex 2023-12-16T21:54:26.893 DEBUG:teuthology.orchestra.run.smithi045:> dd if=/proc/self/mounts of=/dev/stdout 2023-12-16T21:54:26.926 DEBUG:teuthology.orchestra.run.smithi045:> set -ex 2023-12-16T21:54:26.926 DEBUG:teuthology.orchestra.run.smithi045:> dd if=/proc/self/mounts of=/dev/stdout 2023-12-16T21:54:26.985 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-12-16T21:54:27.056 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:54:26 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:54:26.719+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:54:27.056 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:54:26 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:54:26.719+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:54:27.056 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:54:26 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:54:26.719+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:54:27.056 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:54:26 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:54:26.719+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:54:27.729 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:54:27 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:54:27.405+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T21:54:27.729 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:54:27 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:54:27.405+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T21:54:27.729 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:54:27 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:54:27.405+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T21:54:27.729 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:54:27 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:54:27.405+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:54:27.730 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:54:27 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:54:27.499+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:54:27.730 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:54:27 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:54:27.499+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:54:27.730 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:54:27 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:54:27.499+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:54:27.730 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:54:27 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:54:27.499+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:54:28.065 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:54:27 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:54:27.754+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:54:28.065 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:54:27 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:54:27.754+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:54:28.066 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:54:27 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:54:27.754+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:54:28.066 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:54:27 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:54:27.755+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:54:28.721 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:54:28 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:54:28.358+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T21:54:28.721 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:54:28 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:54:28.358+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T21:54:28.722 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:54:28 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:54:28.358+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T21:54:28.722 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:54:28 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:54:28.358+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:54:28.722 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:54:28 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:54:28.497+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:54:28.722 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:54:28 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:54:28.497+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:54:28.722 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:54:28 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:54:28.497+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:54:28.722 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:54:28 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:54:28.498+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:54:29.065 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:54:28 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:54:28.749+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:54:29.066 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:54:28 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:54:28.749+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:54:29.066 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:54:28 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:54:28.749+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:54:29.066 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:54:28 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:54:28.750+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:54:29.736 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:54:29 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:54:29.533+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:54:29.736 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:54:29 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:54:29.533+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:54:29.736 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:54:29 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:54:29.533+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:54:29.736 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:54:29 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:54:29.534+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:54:29.736 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:54:29 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:54:29.317+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T21:54:29.737 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:54:29 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:54:29.317+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T21:54:29.737 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:54:29 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:54:29.317+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T21:54:29.737 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:54:29 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:54:29.318+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:54:30.066 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:54:29 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:54:29.732+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:54:30.066 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:54:29 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:54:29.732+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:54:30.066 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:54:29 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:54:29.732+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:54:30.066 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:54:29 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:54:29.733+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:54:30.538 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:54:30 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:54:30.350+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T21:54:30.538 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:54:30 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:54:30.350+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T21:54:30.539 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:54:30 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:54:30.350+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T21:54:30.539 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:54:30 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:54:30.351+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:54:30.800 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:54:30 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:54:30.536+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:54:30.800 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:54:30 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:54:30.536+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:54:30.801 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:54:30 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:54:30.536+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:54:30.801 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:54:30 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:54:30.537+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:54:30.801 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:54:30 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:54:30.699+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:54:30.801 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:54:30 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:54:30.699+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:54:30.801 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:54:30 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:54:30.699+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:54:30.801 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:54:30 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:54:30.699+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:54:31.566 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Dec 16 21:54:31 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-mon-smithi045[80759]: 2023-12-16T21:54:31.476+0000 7f45a1f8f700 -1 mon.smithi045@0(probing) e2 get_health_metrics reporting 13434 slow ops, oldest is monmgrreport(gid 14214, 0 checks, 1 progress events) 2023-12-16T21:54:31.566 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:54:31 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:54:31.494+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:54:31.566 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:54:31 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:54:31.494+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:54:31.566 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:54:31 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:54:31.494+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:54:31.567 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:54:31 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:54:31.495+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:54:31.567 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:54:31 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:54:31.311+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T21:54:31.567 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:54:31 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:54:31.311+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T21:54:31.568 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:54:31 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:54:31.311+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T21:54:31.568 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:54:31 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:54:31.312+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:54:32.066 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:54:31 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:54:31.726+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:54:32.066 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:54:31 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:54:31.726+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:54:32.066 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:54:31 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:54:31.726+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:54:32.067 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:54:31 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:54:31.727+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:54:32.565 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:54:32 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:54:32.306+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T21:54:32.566 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:54:32 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:54:32.306+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T21:54:32.566 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:54:32 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:54:32.306+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T21:54:32.566 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:54:32 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:54:32.307+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:54:32.566 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:54:32 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:54:32.496+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:54:32.566 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:54:32 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:54:32.496+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:54:32.566 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:54:32 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:54:32.496+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:54:32.567 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:54:32 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:54:32.497+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:54:33.066 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:54:32 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:54:32.690+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:54:33.066 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:54:32 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:54:32.690+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:54:33.066 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:54:32 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:54:32.690+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:54:33.066 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:54:32 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:54:32.691+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:54:33.565 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:54:33 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:54:33.511+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:54:33.566 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:54:33 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:54:33.511+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:54:33.566 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:54:33 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:54:33.511+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:54:33.566 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:54:33 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:54:33.512+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:54:33.566 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:54:33 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:54:33.299+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T21:54:33.566 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:54:33 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:54:33.299+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T21:54:33.567 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:54:33 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:54:33.299+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T21:54:33.567 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:54:33 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:54:33.299+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:54:33.872 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:54:33 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:54:33.694+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:54:33.872 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:54:33 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:54:33.694+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:54:33.872 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:54:33 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:54:33.694+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:54:33.873 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:54:33 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:54:33.695+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:54:34.689 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:54:34 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:54:34.495+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:54:34.689 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:54:34 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:54:34.495+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:54:34.689 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:54:34 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:54:34.495+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:54:34.690 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:54:34 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:54:34.495+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:54:34.690 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:54:34 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:54:34.343+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T21:54:34.690 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:54:34 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:54:34.343+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T21:54:34.690 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:54:34 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:54:34.343+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T21:54:34.691 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:54:34 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:54:34.345+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:54:35.065 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:54:34 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:54:34.687+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:54:35.066 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:54:34 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:54:34.687+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:54:35.066 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:54:34 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:54:34.687+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:54:35.066 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:54:34 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:54:34.688+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:54:35.566 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:54:35 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:54:35.297+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T21:54:35.566 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:54:35 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:54:35.297+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T21:54:35.566 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:54:35 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:54:35.297+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T21:54:35.566 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:54:35 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:54:35.298+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:54:35.567 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:54:35 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:54:35.533+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:54:35.567 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:54:35 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:54:35.533+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:54:35.567 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:54:35 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:54:35.533+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:54:35.567 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:54:35 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:54:35.535+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:54:36.065 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:54:35 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:54:35.694+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:54:36.066 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:54:35 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:54:35.694+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:54:36.066 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:54:35 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:54:35.694+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:54:36.066 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:54:35 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:54:35.695+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:54:36.604 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Dec 16 21:54:36 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-mon-smithi045[80759]: 2023-12-16T21:54:36.478+0000 7f45a1f8f700 -1 mon.smithi045@0(probing) e2 get_health_metrics reporting 13528 slow ops, oldest is monmgrreport(gid 14214, 0 checks, 1 progress events) 2023-12-16T21:54:36.604 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:54:36 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:54:36.327+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T21:54:36.605 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:54:36 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:54:36.327+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T21:54:36.605 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:54:36 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:54:36.327+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T21:54:36.605 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:54:36 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:54:36.328+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:54:36.605 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:54:36 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:54:36.525+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:54:36.605 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:54:36 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:54:36.525+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:54:36.605 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:54:36 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:54:36.525+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:54:36.605 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:54:36 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:54:36.526+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:54:37.065 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:54:36 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:54:36.718+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:54:37.066 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:54:36 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:54:36.718+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:54:37.066 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:54:36 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:54:36.718+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:54:37.066 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:54:36 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:54:36.719+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:54:37.739 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:54:37 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:54:37.548+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:54:37.740 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:54:37 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:54:37.548+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:54:37.740 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:54:37 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:54:37.548+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:54:37.740 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:54:37 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:54:37.549+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:54:37.740 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:54:37 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:54:37.373+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T21:54:37.740 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:54:37 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:54:37.373+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T21:54:37.740 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:54:37 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:54:37.373+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T21:54:37.741 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:54:37 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:54:37.374+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:54:38.065 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:54:37 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:54:37.737+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:54:38.066 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:54:37 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:54:37.737+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:54:38.066 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:54:37 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:54:37.737+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:54:38.066 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:54:37 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:54:37.738+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:54:38.721 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:54:38 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:54:38.519+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:54:38.722 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:54:38 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:54:38.519+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:54:38.722 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:54:38 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:54:38.519+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:54:38.722 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:54:38 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:54:38.520+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:54:38.722 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:54:38 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:54:38.370+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T21:54:38.722 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:54:38 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:54:38.370+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T21:54:38.722 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:54:38 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:54:38.370+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T21:54:38.723 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:54:38 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:54:38.371+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:54:39.065 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:54:38 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:54:38.731+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:54:39.066 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:54:38 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:54:38.731+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:54:39.066 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:54:38 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:54:38.731+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:54:39.066 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:54:38 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:54:38.732+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:54:39.716 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:54:39 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:54:39.569+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:54:39.716 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:54:39 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:54:39.569+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:54:39.717 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:54:39 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:54:39.569+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:54:39.717 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:54:39 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:54:39.570+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:54:39.717 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:54:39 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:54:39.372+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T21:54:39.717 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:54:39 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:54:39.372+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T21:54:39.717 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:54:39 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:54:39.372+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T21:54:39.717 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:54:39 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:54:39.372+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:54:40.065 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:54:39 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:54:39.714+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:54:40.066 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:54:39 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:54:39.714+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:54:40.066 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:54:39 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:54:39.714+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:54:40.066 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:54:39 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:54:39.716+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:54:40.531 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:54:40 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:54:40.326+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T21:54:40.531 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:54:40 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:54:40.326+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T21:54:40.532 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:54:40 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:54:40.326+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T21:54:40.532 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:54:40 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:54:40.327+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:54:40.816 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:54:40 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:54:40.745+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:54:40.816 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:54:40 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:54:40.745+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:54:40.816 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:54:40 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:54:40.745+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:54:40.816 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:54:40 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:54:40.746+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:54:40.817 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:54:40 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:54:40.529+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:54:40.817 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:54:40 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:54:40.529+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:54:40.817 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:54:40 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:54:40.529+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:54:40.817 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:54:40 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:54:40.530+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:54:41.565 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Dec 16 21:54:41 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-mon-smithi045[80759]: 2023-12-16T21:54:41.479+0000 7f45a1f8f700 -1 mon.smithi045@0(probing) e2 get_health_metrics reporting 13624 slow ops, oldest is monmgrreport(gid 14214, 0 checks, 1 progress events) 2023-12-16T21:54:41.566 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:54:41 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:54:41.499+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:54:41.566 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:54:41 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:54:41.499+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:54:41.566 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:54:41 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:54:41.499+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:54:41.566 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:54:41 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:54:41.500+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:54:41.566 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:54:41 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:54:41.283+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T21:54:41.567 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:54:41 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:54:41.283+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T21:54:41.567 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:54:41 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:54:41.283+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T21:54:41.567 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:54:41 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:54:41.284+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:54:42.067 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:54:41 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:54:41.707+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:54:42.067 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:54:41 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:54:41.707+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:54:42.067 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:54:41 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:54:41.707+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:54:42.067 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:54:41 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:54:41.708+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:54:42.565 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:54:42 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:54:42.304+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T21:54:42.566 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:54:42 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:54:42.304+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T21:54:42.566 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:54:42 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:54:42.304+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T21:54:42.566 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:54:42 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:54:42.305+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:54:42.566 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:54:42 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:54:42.502+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:54:42.566 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:54:42 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:54:42.502+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:54:42.567 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:54:42 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:54:42.502+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:54:42.567 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:54:42 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:54:42.503+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:54:43.065 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:54:42 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:54:42.709+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:54:43.066 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:54:42 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:54:42.709+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:54:43.066 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:54:42 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:54:42.709+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:54:43.066 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:54:42 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:54:42.710+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:54:43.509 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:54:43 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:54:43.262+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T21:54:43.509 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:54:43 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:54:43.262+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T21:54:43.509 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:54:43 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:54:43.262+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T21:54:43.509 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:54:43 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:54:43.263+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:54:43.815 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:54:43 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:54:43.506+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:54:43.816 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:54:43 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:54:43.506+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:54:43.816 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:54:43 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:54:43.506+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:54:43.816 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:54:43 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:54:43.508+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:54:43.816 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:54:43 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:54:43.678+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:54:43.816 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:54:43 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:54:43.678+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:54:43.816 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:54:43 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:54:43.678+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:54:43.817 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:54:43 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:54:43.679+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:54:44.546 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:54:44 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:54:44.262+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T21:54:44.546 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:54:44 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:54:44.262+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T21:54:44.546 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:54:44 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:54:44.262+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T21:54:44.546 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:54:44 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:54:44.262+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:54:44.815 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:54:44 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:54:44.665+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:54:44.815 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:54:44 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:54:44.665+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:54:44.816 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:54:44 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:54:44.665+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:54:44.816 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:54:44 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:54:44.666+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:54:44.816 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:54:44 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:54:44.543+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:54:44.816 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:54:44 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:54:44.543+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:54:44.816 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:54:44 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:54:44.543+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:54:44.816 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:54:44 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:54:44.544+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:54:45.565 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:54:45 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:54:45.279+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T21:54:45.566 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:54:45 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:54:45.279+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T21:54:45.566 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:54:45 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:54:45.279+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T21:54:45.566 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:54:45 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:54:45.280+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:54:46.065 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:54:45 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:54:45.699+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:54:46.066 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:54:45 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:54:45.699+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:54:46.066 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:54:45 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:54:45.699+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:54:46.066 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:54:45 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:54:45.700+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:54:46.066 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:54:45 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:54:45.572+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:54:46.066 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:54:45 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:54:45.572+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:54:46.066 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:54:45 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:54:45.572+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:54:46.067 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:54:45 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:54:45.573+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:54:46.483 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:54:46 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:54:46.269+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T21:54:46.484 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:54:46 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:54:46.269+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T21:54:46.484 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:54:46 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:54:46.269+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T21:54:46.484 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:54:46 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:54:46.270+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:54:46.740 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Dec 16 21:54:46 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-mon-smithi045[80759]: 2023-12-16T21:54:46.481+0000 7f45a1f8f700 -1 mon.smithi045@0(probing) e2 get_health_metrics reporting 13717 slow ops, oldest is monmgrreport(gid 14214, 0 checks, 1 progress events) 2023-12-16T21:54:46.741 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:54:46 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:54:46.532+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:54:46.741 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:54:46 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:54:46.532+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:54:46.741 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:54:46 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:54:46.532+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:54:46.741 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:54:46 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:54:46.532+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:54:47.065 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:54:46 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:54:46.738+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:54:47.065 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:54:46 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:54:46.738+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:54:47.066 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:54:46 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:54:46.738+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:54:47.066 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:54:46 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:54:46.740+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:54:47.565 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:54:47 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:54:47.312+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T21:54:47.566 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:54:47 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:54:47.312+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T21:54:47.566 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:54:47 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:54:47.312+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T21:54:47.566 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:54:47 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:54:47.313+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:54:48.066 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:54:47 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:54:47.579+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:54:48.066 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:54:47 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:54:47.579+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:54:48.066 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:54:47 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:54:47.579+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:54:48.066 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:54:47 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:54:47.580+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:54:48.066 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:54:47 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:54:47.748+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:54:48.067 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:54:47 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:54:47.748+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:54:48.067 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:54:47 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:54:47.748+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:54:48.067 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:54:47 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:54:47.748+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:54:48.722 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:54:48 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:54:48.595+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:54:48.722 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:54:48 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:54:48.595+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:54:48.722 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:54:48 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:54:48.595+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:54:48.722 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:54:48 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:54:48.596+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:54:48.723 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:54:48 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:54:48.351+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T21:54:48.723 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:54:48 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:54:48.351+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T21:54:48.723 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:54:48 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:54:48.351+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T21:54:48.723 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:54:48 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:54:48.352+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:54:49.065 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:54:48 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:54:48.771+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:54:49.066 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:54:48 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:54:48.771+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:54:49.066 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:54:48 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:54:48.771+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:54:49.066 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:54:48 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:54:48.772+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:54:49.798 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:54:49 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:54:49.613+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:54:49.798 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:54:49 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:54:49.613+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:54:49.798 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:54:49 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:54:49.613+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:54:49.799 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:54:49 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:54:49.614+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:54:49.799 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:54:49 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:54:49.372+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T21:54:49.799 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:54:49 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:54:49.372+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T21:54:49.799 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:54:49 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:54:49.372+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T21:54:49.799 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:54:49 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:54:49.373+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:54:50.065 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:54:49 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:54:49.796+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:54:50.066 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:54:49 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:54:49.796+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:54:50.066 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:54:49 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:54:49.796+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:54:50.066 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:54:49 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:54:49.797+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:54:50.666 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:54:50 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:54:50.397+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T21:54:50.666 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:54:50 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:54:50.397+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T21:54:50.666 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:54:50 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:54:50.397+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T21:54:50.666 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:54:50 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:54:50.398+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:54:51.065 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:54:50 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:54:50.663+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:54:51.066 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:54:50 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:54:50.663+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:54:51.066 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:54:50 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:54:50.663+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:54:51.066 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:54:50 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:54:50.664+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:54:51.066 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:54:50 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:54:50.790+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:54:51.066 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:54:50 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:54:50.790+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:54:51.067 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:54:50 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:54:50.790+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:54:51.067 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:54:50 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:54:50.791+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:54:51.621 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Dec 16 21:54:51 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-mon-smithi045[80759]: 2023-12-16T21:54:51.482+0000 7f45a1f8f700 -1 mon.smithi045@0(probing) e2 get_health_metrics reporting 13814 slow ops, oldest is monmgrreport(gid 14214, 0 checks, 1 progress events) 2023-12-16T21:54:51.621 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:54:51 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:54:51.358+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T21:54:51.621 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:54:51 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:54:51.358+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T21:54:51.621 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:54:51 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:54:51.358+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T21:54:51.622 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:54:51 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:54:51.359+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:54:52.065 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:54:51 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:54:51.619+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:54:52.065 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:54:51 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:54:51.619+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:54:52.066 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:54:51 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:54:51.619+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:54:52.066 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:54:51 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:54:51.620+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:54:52.066 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:54:51 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:54:51.813+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:54:52.066 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:54:51 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:54:51.813+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:54:52.066 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:54:51 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:54:51.813+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:54:52.066 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:54:51 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:54:51.814+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:54:52.565 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:54:52 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:54:52.347+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T21:54:52.566 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:54:52 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:54:52.347+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T21:54:52.566 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:54:52 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:54:52.347+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T21:54:52.566 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:54:52 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:54:52.348+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:54:53.065 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:54:52 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:54:52.583+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:54:53.066 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:54:52 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:54:52.584+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:54:53.066 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:54:52 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:54:52.584+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:54:53.066 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:54:52 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:54:52.584+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:54:53.066 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:54:52 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:54:52.782+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:54:53.066 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:54:52 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:54:52.782+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:54:53.066 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:54:52 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:54:52.782+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:54:53.067 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:54:52 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:54:52.783+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:54:53.565 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:54:53 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:54:53.302+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T21:54:53.566 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:54:53 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:54:53.302+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T21:54:53.566 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:54:53 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:54:53.302+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T21:54:53.566 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:54:53 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:54:53.302+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:54:54.065 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:54:53 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:54:53.627+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:54:54.066 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:54:53 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:54:53.627+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:54:54.066 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:54:53 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:54:53.627+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:54:54.066 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:54:53 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:54:53.628+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:54:54.066 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:54:53 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:54:53.764+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:54:54.066 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:54:53 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:54:53.765+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:54:54.067 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:54:53 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:54:53.765+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:54:54.067 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:54:53 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:54:53.766+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:54:54.666 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:54:54 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:54:54.347+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T21:54:54.666 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:54:54 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:54:54.347+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T21:54:54.666 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:54:54 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:54:54.347+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T21:54:54.666 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:54:54 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:54:54.348+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:54:55.065 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:54:54 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:54:54.663+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:54:55.066 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:54:54 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:54:54.663+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:54:55.066 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:54:54 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:54:54.663+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:54:55.066 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:54:54 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:54:54.664+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:54:55.066 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:54:54 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:54:54.718+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:54:55.066 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:54:54 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:54:54.718+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:54:55.066 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:54:54 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:54:54.718+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:54:55.067 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:54:54 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:54:54.718+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:54:55.641 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:54:55 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:54:55.366+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T21:54:55.642 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:54:55 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:54:55.366+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T21:54:55.642 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:54:55 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:54:55.366+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T21:54:55.642 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:54:55 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:54:55.368+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:54:56.065 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:54:55 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:54:55.761+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:54:56.066 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:54:55 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:54:55.761+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:54:56.066 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:54:55 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:54:55.761+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:54:56.066 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:54:55 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:54:55.762+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:54:56.066 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:54:55 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:54:55.639+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:54:56.066 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:54:55 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:54:55.639+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:54:56.066 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:54:55 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:54:55.639+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:54:56.067 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:54:55 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:54:55.640+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:54:56.811 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:54:56 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:54:56.594+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:54:56.811 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:54:56 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:54:56.594+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:54:56.811 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:54:56 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:54:56.594+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:54:56.811 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:54:56 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:54:56.595+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:54:56.812 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:54:56 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:54:56.364+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T21:54:56.812 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:54:56 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:54:56.364+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T21:54:56.812 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:54:56 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:54:56.364+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T21:54:56.812 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:54:56 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:54:56.365+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:54:56.812 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Dec 16 21:54:56 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-mon-smithi045[80759]: 2023-12-16T21:54:56.484+0000 7f45a1f8f700 -1 mon.smithi045@0(probing) e2 get_health_metrics reporting 13907 slow ops, oldest is monmgrreport(gid 14214, 0 checks, 1 progress events) 2023-12-16T21:54:57.065 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:54:56 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:54:56.808+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:54:57.066 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:54:56 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:54:56.808+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:54:57.066 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:54:56 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:54:56.808+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:54:57.066 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:54:56 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:54:56.810+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:54:57.815 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:54:57 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:54:57.546+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:54:57.816 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:54:57 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:54:57.546+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:54:57.816 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:54:57 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:54:57.546+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:54:57.816 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:54:57 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:54:57.546+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:54:57.816 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:54:57 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:54:57.368+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T21:54:57.816 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:54:57 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:54:57.368+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T21:54:57.817 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:54:57 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:54:57.369+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T21:54:57.817 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:54:57 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:54:57.370+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:54:58.315 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:54:57 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:54:57.817+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:54:58.316 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:54:57 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:54:57.817+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:54:58.316 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:54:57 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:54:57.817+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:54:58.316 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:54:57 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:54:57.818+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:54:58.721 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:54:58 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:54:58.378+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T21:54:58.722 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:54:58 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:54:58.378+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T21:54:58.722 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:54:58 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:54:58.378+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T21:54:58.722 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:54:58 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:54:58.379+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:54:58.723 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:54:58 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:54:58.544+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:54:58.723 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:54:58 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:54:58.544+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:54:58.723 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:54:58 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:54:58.544+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:54:58.723 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:54:58 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:54:58.544+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:54:59.065 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:54:58 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:54:58.864+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:54:59.066 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:54:58 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:54:58.864+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:54:59.066 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:54:58 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:54:58.864+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:54:59.066 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:54:58 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:54:58.864+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:54:59.815 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:54:59 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:54:59.569+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:54:59.816 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:54:59 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:54:59.569+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:54:59.816 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:54:59 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:54:59.569+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:54:59.816 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:54:59 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:54:59.570+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:54:59.816 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:54:59 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:54:59.429+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T21:54:59.816 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:54:59 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:54:59.429+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T21:54:59.817 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:54:59 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:54:59.429+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T21:54:59.817 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:54:59 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:54:59.430+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:55:00.152 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:54:59 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:54:59.885+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:55:00.152 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:54:59 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:54:59.885+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:55:00.152 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:54:59 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:54:59.885+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:55:00.152 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:54:59 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:54:59.887+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:55:00.566 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:55:00 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:55:00.393+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T21:55:00.566 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:55:00 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:55:00.393+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T21:55:00.566 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:55:00 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:55:00.393+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T21:55:00.566 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:55:00 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:55:00.394+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:55:00.878 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:55:00 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:55:00.577+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:55:00.878 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:55:00 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:55:00.577+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:55:00.878 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:55:00 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:55:00.577+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:55:00.879 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:55:00 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:55:00.578+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:55:01.315 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:55:00 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:55:00.876+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:55:01.316 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:55:00 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:55:00.876+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:55:01.316 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:55:00 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:55:00.876+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:55:01.316 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:55:00 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:55:00.877+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:55:01.681 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Dec 16 21:55:01 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-mon-smithi045[80759]: 2023-12-16T21:55:01.485+0000 7f45a1f8f700 -1 mon.smithi045@0(probing) e2 get_health_metrics reporting 14005 slow ops, oldest is monmgrreport(gid 14214, 0 checks, 1 progress events) 2023-12-16T21:55:01.681 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:55:01 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:55:01.399+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T21:55:01.682 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:55:01 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:55:01.399+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T21:55:01.682 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:55:01 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:55:01.399+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T21:55:01.682 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:55:01 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:55:01.400+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:55:01.682 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:55:01 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:55:01.545+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:55:01.682 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:55:01 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:55:01.545+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:55:01.682 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:55:01 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:55:01.545+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:55:01.683 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:55:01 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:55:01.546+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:55:02.065 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:55:01 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:55:01.886+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:55:02.066 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:55:01 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:55:01.886+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:55:02.066 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:55:01 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:55:01.886+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:55:02.066 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:55:01 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:55:01.888+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:55:02.816 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:55:02 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:55:02.557+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:55:02.816 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:55:02 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:55:02.557+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:55:02.816 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:55:02 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:55:02.557+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:55:02.817 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:55:02 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:55:02.557+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:55:02.817 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:55:02 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:55:02.363+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T21:55:02.817 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:55:02 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:55:02.363+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T21:55:02.817 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:55:02 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:55:02.363+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T21:55:02.817 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:55:02 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:55:02.364+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:55:03.315 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:55:02 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:55:02.898+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:55:03.316 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:55:02 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:55:02.898+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:55:03.316 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:55:02 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:55:02.898+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:55:03.316 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:55:02 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:55:02.900+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:55:03.815 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:55:03 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:55:03.566+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:55:03.816 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:55:03 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:55:03.566+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:55:03.816 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:55:03 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:55:03.566+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:55:03.816 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:55:03 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:55:03.566+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:55:03.816 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:55:03 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:55:03.340+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T21:55:03.816 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:55:03 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:55:03.340+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T21:55:03.816 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:55:03 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:55:03.340+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T21:55:03.817 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:55:03 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:55:03.341+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:55:04.315 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:55:03 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:55:03.932+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:55:04.315 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:55:03 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:55:03.932+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:55:04.316 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:55:03 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:55:03.932+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:55:04.316 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:55:03 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:55:03.933+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:55:04.592 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:55:04 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:55:04.327+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T21:55:04.592 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:55:04 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:55:04.327+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T21:55:04.592 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:55:04 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:55:04.327+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T21:55:04.593 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:55:04 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:55:04.328+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:55:04.593 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:55:04 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:55:04.544+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:55:04.593 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:55:04 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:55:04.544+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:55:04.593 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:55:04 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:55:04.544+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:55:04.593 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:55:04 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:55:04.544+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:55:05.315 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:55:04 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:55:04.886+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:55:05.316 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:55:04 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:55:04.886+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:55:05.316 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:55:04 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:55:04.886+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:55:05.316 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:55:04 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:55:04.887+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:55:05.785 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:55:05 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:55:05.566+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:55:05.785 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:55:05 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:55:05.566+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:55:05.785 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:55:05 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:55:05.566+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:55:05.785 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:55:05 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:55:05.566+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:55:05.786 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:55:05 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:55:05.329+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T21:55:05.786 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:55:05 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:55:05.329+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T21:55:05.786 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:55:05 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:55:05.329+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T21:55:05.786 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:55:05 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:55:05.329+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:55:06.065 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:55:05 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:55:05.872+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:55:06.065 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:55:05 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:55:05.872+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:55:06.066 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:55:05 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:55:05.872+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:55:06.066 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:55:05 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:55:05.873+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:55:06.572 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Dec 16 21:55:06 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-mon-smithi045[80759]: 2023-12-16T21:55:06.487+0000 7f45a1f8f700 -1 mon.smithi045@0(probing) e2 get_health_metrics reporting 14098 slow ops, oldest is monmgrreport(gid 14214, 0 checks, 1 progress events) 2023-12-16T21:55:06.572 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:55:06 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:55:06.318+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T21:55:06.572 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:55:06 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:55:06.318+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T21:55:06.572 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:55:06 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:55:06.318+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T21:55:06.573 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:55:06 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:55:06.319+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:55:06.865 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:55:06 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:55:06.570+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:55:06.865 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:55:06 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:55:06.570+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:55:06.866 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:55:06 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:55:06.570+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:55:06.866 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:55:06 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:55:06.571+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:55:07.303 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:55:06 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:55:06.863+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:55:07.304 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:55:06 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:55:06.863+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:55:07.304 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:55:06 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:55:06.863+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:55:07.304 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:55:06 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:55:06.864+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:55:07.565 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:55:07 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:55:07.542+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:55:07.566 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:55:07 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:55:07.542+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:55:07.566 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:55:07 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:55:07.542+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:55:07.566 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:55:07 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:55:07.543+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:55:07.566 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:55:07 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:55:07.301+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T21:55:07.566 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:55:07 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:55:07.301+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T21:55:07.566 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:55:07 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:55:07.301+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T21:55:07.567 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:55:07 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:55:07.302+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:55:08.065 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:55:07 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:55:07.894+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:55:08.066 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:55:07 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:55:07.894+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:55:08.066 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:55:07 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:55:07.894+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:55:08.066 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:55:07 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:55:07.895+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:55:08.565 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:55:08 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:55:08.540+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:55:08.566 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:55:08 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:55:08.540+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:55:08.566 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:55:08 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:55:08.540+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:55:08.566 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:55:08 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:55:08.541+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:55:08.566 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:55:08 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:55:08.302+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T21:55:08.566 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:55:08 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:55:08.302+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T21:55:08.566 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:55:08 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:55:08.302+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T21:55:08.567 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:55:08 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:55:08.303+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:55:09.065 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:55:08 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:55:08.944+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:55:09.066 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:55:08 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:55:08.944+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:55:09.066 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:55:08 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:55:08.944+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:55:09.066 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:55:08 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:55:08.945+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:55:09.815 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:55:09 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:55:09.564+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:55:09.816 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:55:09 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:55:09.564+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:55:09.816 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:55:09 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:55:09.564+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:55:09.816 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:55:09 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:55:09.564+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:55:09.816 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:55:09 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:55:09.321+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T21:55:09.816 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:55:09 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:55:09.321+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T21:55:09.816 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:55:09 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:55:09.321+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T21:55:09.817 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:55:09 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:55:09.322+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:55:10.315 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:55:09 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:55:09.973+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:55:10.316 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:55:09 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:55:09.973+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:55:10.316 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:55:09 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:55:09.973+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:55:10.316 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:55:09 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:55:09.974+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:55:10.736 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:55:10 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:55:10.613+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:55:10.736 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:55:10 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:55:10.613+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:55:10.736 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:55:10 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:55:10.613+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:55:10.736 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:55:10 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:55:10.614+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:55:10.737 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:55:10 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:55:10.365+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T21:55:10.737 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:55:10 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:55:10.365+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T21:55:10.737 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:55:10 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:55:10.365+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T21:55:10.737 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:55:10 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:55:10.366+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:55:11.316 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:55:10 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:55:10.985+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:55:11.316 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:55:10 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:55:10.985+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:55:11.316 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:55:10 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:55:10.985+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:55:11.316 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:55:10 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:55:10.986+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:55:11.592 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Dec 16 21:55:11 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-mon-smithi045[80759]: 2023-12-16T21:55:11.489+0000 7f45a1f8f700 -1 mon.smithi045@0(probing) e2 get_health_metrics reporting 14195 slow ops, oldest is monmgrreport(gid 14214, 0 checks, 1 progress events) 2023-12-16T21:55:11.593 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:55:11 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:55:11.338+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T21:55:11.593 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:55:11 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:55:11.338+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T21:55:11.593 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:55:11 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:55:11.338+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T21:55:11.593 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:55:11 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:55:11.339+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:55:12.030 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:55:11 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:55:11.590+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:55:12.030 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:55:11 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:55:11.590+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:55:12.030 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:55:11 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:55:11.590+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:55:12.031 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:55:11 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:55:11.592+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:55:12.315 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:55:12 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:55:12.028+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:55:12.316 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:55:12 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:55:12.028+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:55:12.316 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:55:12 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:55:12.028+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:55:12.316 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:55:12 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:55:12.029+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:55:12.815 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:55:12 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:55:12.570+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:55:12.816 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:55:12 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:55:12.570+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:55:12.816 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:55:12 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:55:12.570+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:55:12.816 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:55:12 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:55:12.571+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:55:12.816 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:55:12 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:55:12.325+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T21:55:12.816 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:55:12 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:55:12.325+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T21:55:12.816 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:55:12 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:55:12.325+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T21:55:12.816 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:55:12 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:55:12.326+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:55:13.284 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:55:13 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:55:12.999+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:55:13.284 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:55:13 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:55:12.999+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:55:13.284 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:55:13 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:55:12.999+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:55:13.284 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:55:13 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:55:13.000+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:55:13.566 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:55:13 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:55:13.281+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T21:55:13.566 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:55:13 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:55:13.281+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T21:55:13.566 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:55:13 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:55:13.281+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T21:55:13.566 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:55:13 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:55:13.282+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:55:13.964 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:55:13 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:55:13.582+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:55:13.965 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:55:13 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:55:13.582+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:55:13.965 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:55:13 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:55:13.582+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:55:13.965 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:55:13 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:55:13.582+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:55:14.315 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:55:13 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:55:13.961+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:55:14.316 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:55:13 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:55:13.961+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:55:14.316 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:55:13 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:55:13.961+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:55:14.316 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:55:13 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:55:13.962+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:55:14.612 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:55:14 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:55:14.323+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T21:55:14.613 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:55:14 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:55:14.323+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T21:55:14.613 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:55:14 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:55:14.323+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T21:55:14.613 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:55:14 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:55:14.324+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:55:14.960 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:55:14 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:55:14.610+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:55:14.960 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:55:14 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:55:14.610+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:55:14.960 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:55:14 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:55:14.610+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:55:14.960 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:55:14 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:55:14.611+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:55:15.315 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:55:14 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:55:14.956+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:55:15.316 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:55:14 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:55:14.956+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:55:15.316 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:55:14 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:55:14.956+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:55:15.316 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:55:14 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:55:14.957+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:55:15.815 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:55:15 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:55:15.578+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:55:15.816 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:55:15 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:55:15.578+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:55:15.816 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:55:15 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:55:15.578+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:55:15.816 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:55:15 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:55:15.578+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:55:15.816 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:55:15 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:55:15.358+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T21:55:15.816 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:55:15 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:55:15.358+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T21:55:15.817 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:55:15 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:55:15.358+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T21:55:15.817 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:55:15 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:55:15.359+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:55:16.315 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:55:15 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:55:15.922+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:55:16.316 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:55:15 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:55:15.922+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:55:16.316 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:55:15 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:55:15.922+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:55:16.316 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:55:15 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:55:15.922+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:55:16.815 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Dec 16 21:55:16 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-mon-smithi045[80759]: 2023-12-16T21:55:16.489+0000 7f45a1f8f700 -1 mon.smithi045@0(probing) e2 get_health_metrics reporting 14288 slow ops, oldest is monmgrreport(gid 14214, 0 checks, 1 progress events) 2023-12-16T21:55:16.816 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:55:16 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:55:16.583+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:55:16.816 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:55:16 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:55:16.583+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:55:16.816 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:55:16 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:55:16.583+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:55:16.816 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:55:16 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:55:16.584+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:55:16.816 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:55:16 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:55:16.403+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T21:55:16.817 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:55:16 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:55:16.403+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T21:55:16.817 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:55:16 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:55:16.403+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T21:55:16.817 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:55:16 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:55:16.404+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:55:17.315 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:55:16 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:55:16.891+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:55:17.316 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:55:16 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:55:16.891+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:55:17.316 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:55:16 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:55:16.891+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:55:17.316 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:55:16 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:55:16.891+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:55:17.815 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:55:17 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:55:17.402+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T21:55:17.816 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:55:17 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:55:17.402+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T21:55:17.816 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:55:17 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:55:17.402+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T21:55:17.816 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:55:17 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:55:17.402+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:55:17.816 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:55:17 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:55:17.557+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:55:17.816 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:55:17 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:55:17.557+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:55:17.816 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:55:17 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:55:17.557+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:55:17.817 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:55:17 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:55:17.558+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:55:18.315 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:55:17 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:55:17.936+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:55:18.316 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:55:17 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:55:17.936+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:55:18.316 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:55:17 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:55:17.936+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:55:18.316 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:55:17 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:55:17.936+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:55:18.722 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:55:18 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:55:18.561+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:55:18.722 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:55:18 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:55:18.561+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:55:18.722 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:55:18 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:55:18.561+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:55:18.722 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:55:18 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:55:18.562+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:55:18.723 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:55:18 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:55:18.450+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T21:55:18.723 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:55:18 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:55:18.450+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T21:55:18.723 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:55:18 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:55:18.450+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T21:55:18.723 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:55:18 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:55:18.450+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:55:19.065 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:55:18 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:55:18.924+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:55:19.065 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:55:18 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:55:18.924+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:55:19.066 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:55:18 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:55:18.924+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:55:19.066 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:55:18 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:55:18.925+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:55:19.815 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:55:19 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:55:19.594+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:55:19.816 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:55:19 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:55:19.594+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:55:19.816 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:55:19 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:55:19.594+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:55:19.816 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:55:19 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:55:19.596+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:55:19.816 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:55:19 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:55:19.493+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T21:55:19.816 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:55:19 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:55:19.493+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T21:55:19.817 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:55:19 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:55:19.493+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T21:55:19.817 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:55:19 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:55:19.494+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:55:20.315 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:55:19 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:55:19.932+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:55:20.316 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:55:19 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:55:19.932+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:55:20.316 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:55:19 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:55:19.932+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:55:20.316 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:55:19 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:55:19.933+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:55:20.815 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:55:20 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:55:20.504+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T21:55:20.816 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:55:20 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:55:20.504+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T21:55:20.816 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:55:20 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:55:20.504+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T21:55:20.816 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:55:20 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:55:20.505+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:55:20.816 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:55:20 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:55:20.570+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:55:20.816 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:55:20 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:55:20.570+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:55:20.816 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:55:20 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:55:20.570+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:55:20.817 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:55:20 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:55:20.572+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:55:21.315 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:55:20 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:55:20.964+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:55:21.316 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:55:20 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:55:20.964+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:55:21.316 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:55:20 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:55:20.964+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:55:21.316 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:55:20 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:55:20.965+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:55:21.817 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Dec 16 21:55:21 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-mon-smithi045[80759]: 2023-12-16T21:55:21.492+0000 7f45a1f8f700 -1 mon.smithi045@0(probing) e2 get_health_metrics reporting 14385 slow ops, oldest is monmgrreport(gid 14214, 0 checks, 1 progress events) 2023-12-16T21:55:21.817 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:55:21 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:55:21.473+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T21:55:21.818 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:55:21 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:55:21.473+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T21:55:21.818 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:55:21 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:55:21.473+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T21:55:21.818 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:55:21 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:55:21.473+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:55:21.818 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:55:21 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:55:21.551+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:55:21.818 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:55:21 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:55:21.551+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:55:21.818 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:55:21 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:55:21.551+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:55:21.819 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:55:21 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:55:21.552+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:55:22.315 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:55:21 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:55:21.966+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:55:22.315 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:55:21 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:55:21.966+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:55:22.316 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:55:21 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:55:21.966+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:55:22.316 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:55:21 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:55:21.967+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:55:22.815 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:55:22 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:55:22.550+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:55:22.816 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:55:22 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:55:22.550+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:55:22.816 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:55:22 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:55:22.550+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:55:22.816 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:55:22 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:55:22.551+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:55:22.816 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:55:22 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:55:22.475+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T21:55:22.816 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:55:22 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:55:22.475+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T21:55:22.816 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:55:22 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:55:22.475+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T21:55:22.816 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:55:22 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:55:22.476+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:55:23.315 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:55:23 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:55:23.000+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:55:23.315 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:55:23 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:55:23.000+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:55:23.316 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:55:23 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:55:23.000+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:55:23.316 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:55:23 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:55:23.001+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:55:23.575 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:55:23 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:55:23.529+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:55:23.575 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:55:23 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:55:23.529+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:55:23.575 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:55:23 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:55:23.529+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:55:23.575 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:55:23 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:55:23.531+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:55:23.575 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:55:23 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:55:23.437+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T21:55:23.576 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:55:23 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:55:23.437+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T21:55:23.576 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:55:23 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:55:23.437+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T21:55:23.576 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:55:23 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:55:23.438+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:55:24.315 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:55:23 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:55:23.986+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:55:24.316 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:55:23 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:55:23.986+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:55:24.316 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:55:23 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:55:23.986+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:55:24.316 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:55:23 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:55:23.987+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:55:24.816 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:55:24 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:55:24.431+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T21:55:24.816 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:55:24 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:55:24.431+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T21:55:24.816 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:55:24 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:55:24.431+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T21:55:24.816 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:55:24 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:55:24.431+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:55:24.816 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:55:24 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:55:24.563+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:55:24.817 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:55:24 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:55:24.563+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:55:24.817 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:55:24 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:55:24.563+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:55:24.817 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:55:24 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:55:24.564+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:55:25.315 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:55:24 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:55:24.964+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:55:25.316 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:55:24 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:55:24.964+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:55:25.316 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:55:24 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:55:24.964+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:55:25.316 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:55:24 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:55:24.965+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:55:25.815 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:55:25 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:55:25.609+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:55:25.816 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:55:25 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:55:25.609+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:55:25.816 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:55:25 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:55:25.609+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:55:25.816 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:55:25 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:55:25.611+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:55:25.816 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:55:25 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:55:25.440+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T21:55:25.816 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:55:25 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:55:25.440+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T21:55:25.816 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:55:25 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:55:25.440+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T21:55:25.816 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:55:25 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:55:25.441+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:55:26.315 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:55:25 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:55:25.929+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:55:26.316 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:55:25 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:55:25.929+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:55:26.316 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:55:25 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:55:25.929+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:55:26.316 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:55:25 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:55:25.930+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:55:26.815 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Dec 16 21:55:26 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-mon-smithi045[80759]: 2023-12-16T21:55:26.493+0000 7f45a1f8f700 -1 mon.smithi045@0(probing) e2 get_health_metrics reporting 14478 slow ops, oldest is monmgrreport(gid 14214, 0 checks, 1 progress events) 2023-12-16T21:55:26.816 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:55:26 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:55:26.598+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:55:26.816 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:55:26 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:55:26.598+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:55:26.816 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:55:26 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:55:26.598+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:55:26.816 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:55:26 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:55:26.599+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:55:26.816 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:55:26 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:55:26.449+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T21:55:26.817 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:55:26 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:55:26.449+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T21:55:26.817 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:55:26 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:55:26.449+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T21:55:26.817 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:55:26 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:55:26.451+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:55:27.315 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:55:26 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:55:26.954+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:55:27.316 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:55:26 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:55:26.954+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:55:27.316 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:55:26 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:55:26.954+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:55:27.316 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:55:26 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:55:26.954+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:55:27.815 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:55:27 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:55:27.649+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:55:27.815 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:55:27 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:55:27.649+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:55:27.816 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:55:27 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:55:27.649+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:55:27.816 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:55:27 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:55:27.650+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:55:27.816 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:55:27 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:55:27.489+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T21:55:27.816 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:55:27 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:55:27.489+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T21:55:27.816 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:55:27 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:55:27.489+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T21:55:27.816 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:55:27 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:55:27.490+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:55:28.315 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:55:27 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:55:27.929+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:55:28.315 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:55:27 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:55:27.929+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:55:28.316 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:55:27 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:55:27.929+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:55:28.316 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:55:27 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:55:27.930+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:55:28.815 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:55:28 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:55:28.685+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:55:28.816 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:55:28 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:55:28.685+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:55:28.816 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:55:28 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:55:28.685+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:55:28.816 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:55:28 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:55:28.686+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:55:28.816 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:55:28 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:55:28.476+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T21:55:28.816 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:55:28 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:55:28.476+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T21:55:28.817 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:55:28 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:55:28.476+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T21:55:28.817 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:55:28 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:55:28.476+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:55:29.315 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:55:28 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:55:28.934+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:55:29.316 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:55:28 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:55:28.934+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:55:29.316 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:55:28 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:55:28.934+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:55:29.316 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:55:28 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:55:28.935+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:55:29.815 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:55:29 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:55:29.736+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:55:29.816 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:55:29 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:55:29.736+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:55:29.816 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:55:29 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:55:29.736+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:55:29.816 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:55:29 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:55:29.737+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:55:29.817 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:55:29 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:55:29.517+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T21:55:29.817 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:55:29 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:55:29.517+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T21:55:29.817 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:55:29 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:55:29.517+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T21:55:29.817 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:55:29 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:55:29.518+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:55:30.315 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:55:29 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:55:29.981+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:55:30.315 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:55:29 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:55:29.981+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:55:30.316 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:55:29 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:55:29.981+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:55:30.316 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:55:29 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:55:29.981+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:55:30.815 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:55:30 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:55:30.694+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:55:30.816 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:55:30 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:55:30.694+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:55:30.816 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:55:30 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:55:30.694+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:55:30.816 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:55:30 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:55:30.695+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:55:30.816 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:55:30 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:55:30.528+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T21:55:30.816 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:55:30 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:55:30.528+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T21:55:30.816 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:55:30 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:55:30.528+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T21:55:30.816 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:55:30 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:55:30.529+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:55:31.315 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:55:31 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:55:31.015+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:55:31.315 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:55:31 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:55:31.015+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:55:31.316 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:55:31 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:55:31.015+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:55:31.316 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:55:31 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:55:31.016+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:55:31.815 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Dec 16 21:55:31 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-mon-smithi045[80759]: 2023-12-16T21:55:31.495+0000 7f45a1f8f700 -1 mon.smithi045@0(probing) e2 get_health_metrics reporting 14575 slow ops, oldest is monmgrreport(gid 14214, 0 checks, 1 progress events) 2023-12-16T21:55:31.816 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:55:31 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:55:31.680+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:55:31.816 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:55:31 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:55:31.680+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:55:31.816 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:55:31 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:55:31.680+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:55:31.816 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:55:31 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:55:31.681+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:55:31.817 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:55:31 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:55:31.482+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T21:55:31.817 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:55:31 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:55:31.482+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T21:55:31.817 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:55:31 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:55:31.482+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T21:55:31.817 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:55:31 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:55:31.483+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:55:32.316 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:55:32 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:55:32.016+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:55:32.316 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:55:32 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:55:32.016+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:55:32.316 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:55:32 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:55:32.016+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:55:32.316 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:55:32 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:55:32.017+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:55:32.815 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:55:32 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:55:32.634+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:55:32.815 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:55:32 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:55:32.634+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:55:32.816 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:55:32 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:55:32.634+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:55:32.816 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:55:32 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:55:32.635+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:55:32.816 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:55:32 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:55:32.509+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T21:55:32.816 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:55:32 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:55:32.509+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T21:55:32.816 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:55:32 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:55:32.509+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T21:55:32.816 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:55:32 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:55:32.511+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:55:33.315 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:55:33 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:55:33.052+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:55:33.316 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:55:33 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:55:33.052+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:55:33.316 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:55:33 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:55:33.052+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:55:33.316 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:55:33 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:55:33.053+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:55:33.815 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:55:33 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:55:33.623+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:55:33.816 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:55:33 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:55:33.623+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:55:33.816 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:55:33 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:55:33.623+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:55:33.816 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:55:33 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:55:33.624+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:55:33.816 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:55:33 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:55:33.510+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T21:55:33.816 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:55:33 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:55:33.510+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T21:55:33.816 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:55:33 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:55:33.510+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T21:55:33.816 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:55:33 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:55:33.511+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:55:34.315 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:55:34 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:55:34.037+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:55:34.316 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:55:34 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:55:34.038+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:55:34.316 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:55:34 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:55:34.038+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:55:34.316 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:55:34 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:55:34.038+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:55:34.815 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:55:34 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:55:34.507+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T21:55:34.815 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:55:34 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:55:34.507+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T21:55:34.816 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:55:34 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:55:34.507+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T21:55:34.816 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:55:34 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:55:34.509+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:55:34.816 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:55:34 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:55:34.645+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:55:34.816 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:55:34 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:55:34.645+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:55:34.816 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:55:34 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:55:34.645+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:55:34.816 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:55:34 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:55:34.646+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:55:35.312 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:55:35 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:55:35.003+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:55:35.312 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:55:35 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:55:35.003+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:55:35.312 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:55:35 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:55:35.003+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:55:35.312 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:55:35 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:55:35.004+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:55:35.565 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:55:35 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:55:35.472+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T21:55:35.566 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:55:35 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:55:35.472+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T21:55:35.566 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:55:35 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:55:35.472+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T21:55:35.566 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:55:35 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:55:35.472+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:55:35.987 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:55:35 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:55:35.602+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:55:35.988 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:55:35 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:55:35.602+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:55:35.988 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:55:35 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:55:35.602+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:55:35.989 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:55:35 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:55:35.602+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:55:36.315 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:55:35 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:55:35.985+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:55:36.316 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:55:35 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:55:35.985+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:55:36.316 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:55:35 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:55:35.985+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:55:36.316 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:55:35 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:55:35.986+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:55:36.815 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Dec 16 21:55:36 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-mon-smithi045[80759]: 2023-12-16T21:55:36.496+0000 7f45a1f8f700 -1 mon.smithi045@0(probing) e2 get_health_metrics reporting 14668 slow ops, oldest is monmgrreport(gid 14214, 0 checks, 1 progress events) 2023-12-16T21:55:36.816 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:55:36 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:55:36.586+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:55:36.816 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:55:36 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:55:36.586+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:55:36.816 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:55:36 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:55:36.586+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:55:36.816 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:55:36 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:55:36.587+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:55:36.816 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:55:36 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:55:36.522+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T21:55:36.816 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:55:36 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:55:36.522+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T21:55:36.816 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:55:36 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:55:36.522+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T21:55:36.817 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:55:36 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:55:36.523+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:55:37.315 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:55:37 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:55:37.035+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:55:37.315 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:55:37 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:55:37.035+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:55:37.316 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:55:37 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:55:37.035+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:55:37.316 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:55:37 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:55:37.036+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:55:38.065 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:55:37 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:55:37.591+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:55:38.066 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:55:37 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:55:37.591+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:55:38.066 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:55:37 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:55:37.591+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:55:38.066 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:55:37 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:55:37.592+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:55:38.066 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:55:37 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:55:37.567+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T21:55:38.066 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:55:37 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:55:37.567+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T21:55:38.066 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:55:37 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:55:37.567+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T21:55:38.066 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:55:37 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:55:37.568+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:55:38.565 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:55:38 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:55:38.077+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:55:38.566 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:55:38 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:55:38.077+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:55:38.566 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:55:38 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:55:38.077+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:55:38.566 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:55:38 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:55:38.078+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:55:39.062 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:55:38 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:55:38.629+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:55:39.062 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:55:38 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:55:38.629+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:55:39.062 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:55:38 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:55:38.629+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:55:39.062 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:55:38 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:55:38.630+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:55:39.062 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:55:38 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:55:38.606+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T21:55:39.062 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:55:38 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:55:38.606+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T21:55:39.063 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:55:38 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:55:38.606+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T21:55:39.063 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:55:38 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:55:38.607+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:55:39.315 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:55:39 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:55:39.059+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:55:39.315 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:55:39 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:55:39.059+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:55:39.315 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:55:39 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:55:39.059+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:55:39.316 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:55:39 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:55:39.060+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:55:40.041 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:55:39 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:55:39.591+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T21:55:40.041 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:55:39 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:55:39.591+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T21:55:40.042 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:55:39 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:55:39.591+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T21:55:40.042 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:55:39 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:55:39.592+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:55:40.042 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:55:39 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:55:39.583+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:55:40.042 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:55:39 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:55:39.583+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:55:40.042 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:55:39 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:55:39.583+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:55:40.042 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:55:39 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:55:39.584+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:55:40.315 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:55:40 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:55:40.039+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:55:40.315 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:55:40 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:55:40.039+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:55:40.315 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:55:40 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:55:40.039+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:55:40.316 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:55:40 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:55:40.040+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:55:41.039 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:55:40 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:55:40.604+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:55:41.039 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:55:40 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:55:40.604+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:55:41.039 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:55:40 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:55:40.604+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:55:41.039 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:55:40 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:55:40.605+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:55:41.040 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:55:40 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:55:40.569+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T21:55:41.040 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:55:40 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:55:40.569+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T21:55:41.040 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:55:40 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:55:40.569+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T21:55:41.040 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:55:40 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:55:40.570+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:55:41.315 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:55:41 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:55:41.037+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:55:41.315 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:55:41 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:55:41.037+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:55:41.315 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:55:41 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:55:41.037+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:55:41.316 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:55:41 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:55:41.038+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:55:41.815 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Dec 16 21:55:41 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-mon-smithi045[80759]: 2023-12-16T21:55:41.498+0000 7f45a1f8f700 -1 mon.smithi045@0(probing) e2 get_health_metrics reporting 14765 slow ops, oldest is monmgrreport(gid 14214, 0 checks, 1 progress events) 2023-12-16T21:55:41.816 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:55:41 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:55:41.632+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:55:41.816 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:55:41 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:55:41.632+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:55:41.816 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:55:41 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:55:41.632+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:55:41.816 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:55:41 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:55:41.633+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:55:41.816 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:55:41 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:55:41.561+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T21:55:41.816 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:55:41 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:55:41.561+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T21:55:41.817 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:55:41 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:55:41.561+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T21:55:41.817 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:55:41 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:55:41.562+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:55:42.315 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:55:42 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:55:42.057+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:55:42.316 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:55:42 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:55:42.057+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:55:42.316 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:55:42 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:55:42.057+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:55:42.316 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:55:42 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:55:42.058+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:55:42.815 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:55:42 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:55:42.548+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T21:55:42.816 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:55:42 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:55:42.548+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T21:55:42.816 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:55:42 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:55:42.548+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T21:55:42.816 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:55:42 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:55:42.548+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:55:42.816 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:55:42 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:55:42.634+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:55:42.816 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:55:42 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:55:42.634+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:55:42.816 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:55:42 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:55:42.634+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:55:42.816 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:55:42 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:55:42.634+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:55:43.515 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:55:43 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:55:43.077+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:55:43.515 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:55:43 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:55:43.077+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:55:43.515 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:55:43 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:55:43.077+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:55:43.516 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:55:43 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:55:43.077+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:55:43.815 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:55:43 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:55:43.513+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T21:55:43.816 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:55:43 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:55:43.513+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T21:55:43.816 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:55:43 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:55:43.513+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T21:55:43.816 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:55:43 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:55:43.514+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:55:43.816 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:55:43 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:55:43.606+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:55:43.816 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:55:43 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:55:43.606+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:55:43.816 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:55:43 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:55:43.606+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:55:43.817 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:55:43 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:55:43.608+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:55:44.315 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:55:44 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:55:44.057+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:55:44.315 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:55:44 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:55:44.057+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:55:44.316 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:55:44 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:55:44.057+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:55:44.316 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:55:44 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:55:44.058+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:55:44.815 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:55:44 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:55:44.627+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:55:44.815 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:55:44 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:55:44.627+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:55:44.816 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:55:44 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:55:44.627+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:55:44.816 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:55:44 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:55:44.628+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:55:44.816 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:55:44 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:55:44.564+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T21:55:44.816 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:55:44 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:55:44.564+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T21:55:44.816 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:55:44 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:55:44.564+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T21:55:44.816 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:55:44 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:55:44.565+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:55:45.565 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:55:45 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:55:45.094+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:55:45.565 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:55:45 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:55:45.094+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:55:45.566 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:55:45 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:55:45.094+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:55:45.566 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:55:45 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:55:45.095+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:55:46.065 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:55:45 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:55:45.664+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:55:46.066 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:55:45 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:55:45.664+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:55:46.066 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:55:45 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:55:45.664+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:55:46.066 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:55:45 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:55:45.665+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:55:46.066 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:55:45 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:55:45.563+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T21:55:46.066 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:55:45 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:55:45.563+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T21:55:46.066 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:55:45 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:55:45.563+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T21:55:46.066 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:55:45 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:55:45.565+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:55:46.501 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:55:46 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:55:46.138+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:55:46.502 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:55:46 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:55:46.138+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:55:46.502 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:55:46 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:55:46.138+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:55:46.502 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:55:46 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:55:46.138+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:55:46.815 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Dec 16 21:55:46 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-mon-smithi045[80759]: 2023-12-16T21:55:46.499+0000 7f45a1f8f700 -1 mon.smithi045@0(probing) e2 get_health_metrics reporting 14858 slow ops, oldest is monmgrreport(gid 14214, 0 checks, 1 progress events) 2023-12-16T21:55:46.816 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:55:46 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:55:46.675+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:55:46.816 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:55:46 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:55:46.675+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:55:46.816 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:55:46 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:55:46.675+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:55:46.816 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:55:46 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:55:46.676+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:55:46.816 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:55:46 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:55:46.569+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T21:55:46.816 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:55:46 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:55:46.569+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T21:55:46.817 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:55:46 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:55:46.569+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T21:55:46.817 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:55:46 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:55:46.571+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:55:47.559 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:55:47 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:55:47.129+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:55:47.559 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:55:47 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:55:47.129+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:55:47.559 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:55:47 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:55:47.129+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:55:47.560 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:55:47 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:55:47.130+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:55:47.815 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:55:47 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:55:47.638+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:55:47.815 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:55:47 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:55:47.638+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:55:47.816 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:55:47 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:55:47.638+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:55:47.816 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:55:47 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:55:47.639+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:55:47.816 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:55:47 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:55:47.557+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T21:55:47.816 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:55:47 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:55:47.557+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T21:55:47.816 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:55:47 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:55:47.557+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T21:55:47.816 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:55:47 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:55:47.558+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:55:48.565 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:55:48 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:55:48.180+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:55:48.565 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:55:48 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:55:48.180+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:55:48.565 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:55:48 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:55:48.180+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:55:48.566 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:55:48 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:55:48.181+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:55:49.065 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:55:48 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:55:48.640+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:55:49.066 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:55:48 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:55:48.640+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:55:49.066 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:55:48 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:55:48.640+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:55:49.066 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:55:48 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:55:48.641+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:55:49.066 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:55:48 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:55:48.565+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T21:55:49.066 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:55:48 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:55:48.565+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T21:55:49.067 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:55:48 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:55:48.565+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T21:55:49.067 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:55:48 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:55:48.566+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:55:49.565 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:55:49 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:55:49.143+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:55:49.565 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:55:49 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:55:49.143+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:55:49.566 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:55:49 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:55:49.143+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:55:49.566 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:55:49 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:55:49.143+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:55:50.065 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:55:49 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:55:49.623+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:55:50.066 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:55:49 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:55:49.623+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:55:50.066 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:55:49 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:55:49.623+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:55:50.066 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:55:49 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:55:49.625+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:55:50.066 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:55:49 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:55:49.582+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T21:55:50.066 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:55:49 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:55:49.582+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T21:55:50.066 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:55:49 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:55:49.582+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T21:55:50.066 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:55:49 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:55:49.583+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:55:50.565 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:55:50 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:55:50.132+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:55:50.566 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:55:50 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:55:50.132+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:55:50.566 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:55:50 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:55:50.132+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:55:50.566 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:55:50 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:55:50.133+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:55:51.066 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:55:50 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:55:50.585+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:55:51.066 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:55:50 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:55:50.585+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:55:51.066 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:55:50 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:55:50.585+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:55:51.066 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:55:50 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:55:50.587+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:55:51.066 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:55:50 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:55:50.617+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T21:55:51.066 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:55:50 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:55:50.617+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T21:55:51.067 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:55:50 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:55:50.617+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T21:55:51.067 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:55:50 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:55:50.618+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:55:51.503 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:55:51 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:55:51.172+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:55:51.503 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:55:51 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:55:51.172+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:55:51.504 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:55:51 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:55:51.172+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:55:51.504 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:55:51 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:55:51.173+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:55:51.815 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Dec 16 21:55:51 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-mon-smithi045[80759]: 2023-12-16T21:55:51.501+0000 7f45a1f8f700 -1 mon.smithi045@0(probing) e2 get_health_metrics reporting 14955 slow ops, oldest is monmgrreport(gid 14214, 0 checks, 1 progress events) 2023-12-16T21:55:51.816 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:55:51 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:55:51.604+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:55:51.816 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:55:51 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:55:51.604+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:55:51.816 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:55:51 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:55:51.604+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:55:51.816 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:55:51 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:55:51.605+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:55:51.817 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:55:51 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:55:51.574+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T21:55:51.817 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:55:51 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:55:51.574+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T21:55:51.817 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:55:51 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:55:51.574+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T21:55:51.817 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:55:51 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:55:51.575+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:55:52.565 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:55:52 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:55:52.200+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:55:52.566 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:55:52 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:55:52.200+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:55:52.569 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:55:52 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:55:52.200+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:55:52.570 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:55:52 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:55:52.200+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:55:53.065 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:55:52 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:55:52.566+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:55:53.066 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:55:52 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:55:52.566+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:55:53.066 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:55:52 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:55:52.566+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:55:53.066 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:55:52 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:55:52.567+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:55:53.066 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:55:52 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:55:52.595+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T21:55:53.066 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:55:52 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:55:52.595+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T21:55:53.067 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:55:52 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:55:52.595+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T21:55:53.067 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:55:52 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:55:52.596+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:55:53.550 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:55:53 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:55:53.154+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:55:53.551 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:55:53 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:55:53.154+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:55:53.551 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:55:53 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:55:53.154+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:55:53.551 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:55:53 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:55:53.155+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:55:53.816 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:55:53 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:55:53.548+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:55:53.816 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:55:53 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:55:53.548+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:55:53.816 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:55:53 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:55:53.548+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:55:53.816 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:55:53 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:55:53.549+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:55:53.816 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:55:53 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:55:53.634+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T21:55:53.817 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:55:53 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:55:53.634+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T21:55:53.817 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:55:53 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:55:53.634+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T21:55:53.817 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:55:53 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:55:53.635+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:55:54.550 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:55:54 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:55:54.147+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:55:54.551 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:55:54 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:55:54.147+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:55:54.551 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:55:54 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:55:54.147+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:55:54.551 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:55:54 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:55:54.148+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:55:54.815 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:55:54 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:55:54.548+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:55:54.815 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:55:54 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:55:54.548+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:55:54.815 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:55:54 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:55:54.548+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:55:54.816 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:55:54 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:55:54.549+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:55:54.816 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:55:54 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:55:54.648+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T21:55:54.816 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:55:54 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:55:54.648+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T21:55:54.816 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:55:54 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:55:54.648+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T21:55:54.816 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:55:54 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:55:54.649+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:55:55.565 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:55:55 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:55:55.127+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:55:55.565 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:55:55 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:55:55.127+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:55:55.565 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:55:55 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:55:55.127+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:55:55.566 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:55:55 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:55:55.128+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:55:56.066 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:55:55 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:55:55.684+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T21:55:56.066 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:55:55 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:55:55.684+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T21:55:56.066 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:55:55 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:55:55.684+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T21:55:56.066 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:55:55 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:55:55.685+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:55:56.067 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:55:55 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:55:55.572+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:55:56.067 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:55:55 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:55:55.572+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:55:56.067 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:55:55 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:55:55.572+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:55:56.067 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:55:55 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:55:55.573+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:55:56.505 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:55:56 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:55:56.147+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:55:56.505 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:55:56 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:55:56.147+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:55:56.505 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:55:56 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:55:56.147+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:55:56.506 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:55:56 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:55:56.148+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:55:56.815 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Dec 16 21:55:56 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-mon-smithi045[80759]: 2023-12-16T21:55:56.503+0000 7f45a1f8f700 -1 mon.smithi045@0(probing) e2 get_health_metrics reporting 15048 slow ops, oldest is monmgrreport(gid 14214, 0 checks, 1 progress events) 2023-12-16T21:55:56.816 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:55:56 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:55:56.659+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T21:55:56.816 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:55:56 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:55:56.659+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T21:55:56.816 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:55:56 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:55:56.659+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T21:55:56.816 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:55:56 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:55:56.660+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:55:56.816 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:55:56 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:55:56.549+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:55:56.816 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:55:56 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:55:56.549+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:55:56.817 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:55:56 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:55:56.549+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:55:56.817 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:55:56 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:55:56.549+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:55:57.534 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:55:57 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:55:57.111+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:55:57.535 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:55:57 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:55:57.111+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:55:57.535 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:55:57 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:55:57.111+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:55:57.535 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:55:57 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:55:57.111+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:55:57.815 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:55:57 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:55:57.532+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:55:57.815 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:55:57 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:55:57.532+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:55:57.815 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:55:57 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:55:57.532+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:55:57.816 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:55:57 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:55:57.534+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:55:57.816 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:55:57 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:55:57.706+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T21:55:57.816 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:55:57 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:55:57.706+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T21:55:57.816 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:55:57 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:55:57.706+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T21:55:57.816 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:55:57 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:55:57.707+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:55:58.489 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:55:58 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:55:58.068+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:55:58.489 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:55:58 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:55:58.068+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:55:58.489 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:55:58 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:55:58.068+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:55:58.489 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:55:58 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:55:58.068+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:55:58.815 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:55:58 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:55:58.702+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T21:55:58.815 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:55:58 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:55:58.702+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T21:55:58.816 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:55:58 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:55:58.702+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T21:55:58.816 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:55:58 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:55:58.702+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:55:58.816 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:55:58 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:55:58.487+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:55:58.816 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:55:58 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:55:58.487+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:55:58.816 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:55:58 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:55:58.487+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:55:58.816 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:55:58 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:55:58.488+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:55:59.525 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:55:59 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:55:59.103+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:55:59.525 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:55:59 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:55:59.103+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:55:59.525 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:55:59 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:55:59.103+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:55:59.525 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:55:59 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:55:59.103+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:55:59.815 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:55:59 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:55:59.523+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:55:59.815 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:55:59 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:55:59.523+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:55:59.816 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:55:59 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:55:59.523+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:55:59.816 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:55:59 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:55:59.524+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:55:59.816 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:55:59 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:55:59.702+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T21:55:59.816 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:55:59 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:55:59.702+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T21:55:59.816 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:55:59 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:55:59.702+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T21:55:59.816 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:55:59 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:55:59.703+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:56:00.155 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:56:00 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:56:00.088+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:56:00.156 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:56:00 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:56:00.088+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:56:00.156 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:56:00 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:56:00.088+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:56:00.156 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:56:00 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:56:00.089+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:56:00.815 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:56:00 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:56:00.703+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T21:56:00.816 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:56:00 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:56:00.703+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T21:56:00.816 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:56:00 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:56:00.703+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T21:56:00.816 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:56:00 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:56:00.704+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:56:00.816 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:56:00 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:56:00.557+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:56:00.816 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:56:00 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:56:00.557+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:56:00.816 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:56:00 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:56:00.557+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:56:00.817 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:56:00 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:56:00.558+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:56:01.506 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:56:01 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:56:01.083+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:56:01.507 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:56:01 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:56:01.083+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:56:01.507 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:56:01 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:56:01.083+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:56:01.507 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:56:01 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:56:01.084+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:56:01.815 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Dec 16 21:56:01 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-mon-smithi045[80759]: 2023-12-16T21:56:01.504+0000 7f45a1f8f700 -1 mon.smithi045@0(probing) e2 get_health_metrics reporting 15145 slow ops, oldest is monmgrreport(gid 14214, 0 checks, 1 progress events) 2023-12-16T21:56:01.815 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:56:01 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:56:01.567+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:56:01.816 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:56:01 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:56:01.567+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:56:01.816 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:56:01 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:56:01.567+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:56:01.816 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:56:01 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:56:01.568+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:56:01.816 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:56:01 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:56:01.725+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T21:56:01.816 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:56:01 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:56:01.725+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T21:56:01.816 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:56:01 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:56:01.725+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T21:56:01.816 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:56:01 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:56:01.726+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:56:02.315 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:56:02 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:56:02.039+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:56:02.315 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:56:02 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:56:02.039+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:56:02.316 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:56:02 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:56:02.039+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:56:02.316 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:56:02 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:56:02.039+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:56:02.815 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:56:02 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:56:02.742+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T21:56:02.815 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:56:02 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:56:02.742+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T21:56:02.816 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:56:02 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:56:02.742+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T21:56:02.816 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:56:02 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:56:02.743+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:56:02.816 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:56:02 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:56:02.524+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:56:02.816 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:56:02 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:56:02.524+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:56:02.816 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:56:02 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:56:02.524+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:56:02.816 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:56:02 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:56:02.525+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:56:03.315 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:56:03 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:56:03.053+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:56:03.315 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:56:03 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:56:03.053+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:56:03.315 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:56:03 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:56:03.053+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:56:03.316 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:56:03 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:56:03.054+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:56:03.815 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:56:03 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:56:03.533+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:56:03.815 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:56:03 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:56:03.533+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:56:03.816 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:56:03 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:56:03.533+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:56:03.816 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:56:03 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:56:03.534+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:56:03.816 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:56:03 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:56:03.701+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T21:56:03.816 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:56:03 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:56:03.701+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T21:56:03.816 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:56:03 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:56:03.701+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T21:56:03.816 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:56:03 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:56:03.702+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:56:04.541 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:56:04 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:56:04.081+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:56:04.541 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:56:04 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:56:04.081+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:56:04.541 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:56:04 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:56:04.081+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:56:04.541 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:56:04 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:56:04.082+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:56:04.815 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:56:04 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:56:04.731+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T21:56:04.815 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:56:04 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:56:04.731+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T21:56:04.816 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:56:04 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:56:04.731+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T21:56:04.816 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:56:04 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:56:04.732+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:56:04.816 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:56:04 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:56:04.539+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:56:04.816 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:56:04 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:56:04.539+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:56:04.816 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:56:04 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:56:04.539+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:56:04.816 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:56:04 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:56:04.540+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:56:05.315 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:56:05 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:56:05.043+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:56:05.316 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:56:05 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:56:05.043+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:56:05.316 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:56:05 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:56:05.043+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:56:05.316 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:56:05 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:56:05.044+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:56:06.019 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:56:05 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:56:05.732+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T21:56:06.019 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:56:05 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:56:05.732+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T21:56:06.020 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:56:05 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:56:05.732+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T21:56:06.020 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:56:05 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:56:05.733+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:56:06.020 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:56:05 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:56:05.577+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:56:06.020 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:56:05 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:56:05.578+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:56:06.020 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:56:05 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:56:05.578+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:56:06.020 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:56:05 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:56:05.578+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:56:06.315 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:56:06 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:56:06.017+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:56:06.315 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:56:06 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:56:06.017+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:56:06.316 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:56:06 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:56:06.017+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:56:06.316 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:56:06 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:56:06.018+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:56:06.815 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Dec 16 21:56:06 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-mon-smithi045[80759]: 2023-12-16T21:56:06.506+0000 7f45a1f8f700 -1 mon.smithi045@0(probing) e2 get_health_metrics reporting 15238 slow ops, oldest is monmgrreport(gid 14214, 0 checks, 1 progress events) 2023-12-16T21:56:06.816 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:56:06 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:56:06.584+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:56:06.816 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:56:06 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:56:06.584+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:56:06.816 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:56:06 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:56:06.584+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:56:06.816 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:56:06 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:56:06.585+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:56:06.816 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:56:06 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:56:06.717+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T21:56:06.816 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:56:06 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:56:06.717+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T21:56:06.817 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:56:06 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:56:06.717+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T21:56:06.817 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:56:06 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:56:06.718+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:56:07.565 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:56:07 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:56:07.064+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:56:07.566 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:56:07 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:56:07.064+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:56:07.566 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:56:07 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:56:07.064+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:56:07.566 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:56:07 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:56:07.066+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:56:08.065 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:56:07 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:56:07.702+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T21:56:08.065 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:56:07 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:56:07.702+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T21:56:08.066 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:56:07 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:56:07.702+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T21:56:08.066 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:56:07 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:56:07.702+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:56:08.066 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:56:07 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:56:07.626+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:56:08.066 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:56:07 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:56:07.626+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:56:08.066 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:56:07 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:56:07.626+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:56:08.066 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:56:07 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:56:07.626+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:56:08.565 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:56:08 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:56:08.076+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:56:08.566 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:56:08 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:56:08.076+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:56:08.566 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:56:08 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:56:08.076+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:56:08.566 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:56:08 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:56:08.077+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:56:09.065 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:56:08 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:56:08.590+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:56:09.066 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:56:08 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:56:08.590+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:56:09.066 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:56:08 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:56:08.590+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:56:09.066 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:56:08 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:56:08.592+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:56:09.066 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:56:08 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:56:08.664+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T21:56:09.066 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:56:08 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:56:08.664+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T21:56:09.066 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:56:08 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:56:08.664+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T21:56:09.067 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:56:08 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:56:08.665+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:56:09.565 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:56:09 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:56:09.085+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:56:09.565 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:56:09 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:56:09.085+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:56:09.566 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:56:09 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:56:09.085+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:56:09.566 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:56:09 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:56:09.086+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:56:10.065 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:56:09 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:56:09.621+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T21:56:10.065 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:56:09 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:56:09.621+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T21:56:10.066 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:56:09 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:56:09.621+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T21:56:10.066 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:56:09 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:56:09.622+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:56:10.066 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:56:09 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:56:09.620+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:56:10.066 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:56:09 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:56:09.620+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:56:10.066 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:56:09 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:56:09.620+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:56:10.066 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:56:09 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:56:09.621+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:56:10.565 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:56:10 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:56:10.123+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:56:10.566 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:56:10 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:56:10.123+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:56:10.566 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:56:10 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:56:10.123+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:56:10.566 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:56:10 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:56:10.124+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:56:11.065 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:56:10 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:56:10.597+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T21:56:11.066 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:56:10 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:56:10.597+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T21:56:11.066 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:56:10 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:56:10.597+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T21:56:11.066 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:56:10 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:56:10.598+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:56:11.066 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:56:10 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:56:10.653+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:56:11.066 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:56:10 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:56:10.653+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:56:11.066 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:56:10 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:56:10.653+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:56:11.066 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:56:10 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:56:10.654+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:56:11.510 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:56:11 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:56:11.126+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:56:11.510 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:56:11 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:56:11.126+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:56:11.510 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:56:11 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:56:11.126+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:56:11.510 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:56:11 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:56:11.127+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:56:11.815 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Dec 16 21:56:11 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-mon-smithi045[80759]: 2023-12-16T21:56:11.508+0000 7f45a1f8f700 -1 mon.smithi045@0(probing) e2 get_health_metrics reporting 15333 slow ops, oldest is monmgrreport(gid 14214, 0 checks, 1 progress events) 2023-12-16T21:56:11.816 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:56:11 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:56:11.643+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T21:56:11.816 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:56:11 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:56:11.643+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T21:56:11.816 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:56:11 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:56:11.643+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T21:56:11.816 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:56:11 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:56:11.644+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:56:11.816 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:56:11 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:56:11.676+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:56:11.816 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:56:11 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:56:11.676+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:56:11.816 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:56:11 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:56:11.676+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:56:11.817 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:56:11 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:56:11.676+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:56:12.565 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:56:12 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:56:12.102+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:56:12.566 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:56:12 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:56:12.102+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:56:12.566 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:56:12 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:56:12.102+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:56:12.566 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:56:12 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:56:12.102+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:56:13.065 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:56:12 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:56:12.664+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:56:13.065 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:56:12 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:56:12.664+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:56:13.066 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:56:12 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:56:12.664+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:56:13.066 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:56:12 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:56:12.665+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:56:13.066 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:56:12 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:56:12.616+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T21:56:13.066 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:56:12 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:56:12.616+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T21:56:13.066 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:56:12 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:56:12.616+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T21:56:13.066 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:56:12 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:56:12.616+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:56:13.565 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:56:13 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:56:13.098+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:56:13.565 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:56:13 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:56:13.098+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:56:13.565 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:56:13 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:56:13.098+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:56:13.566 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:56:13 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:56:13.099+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:56:14.065 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:56:13 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:56:13.688+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:56:14.065 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:56:13 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:56:13.688+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:56:14.066 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:56:13 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:56:13.688+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:56:14.066 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:56:13 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:56:13.689+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:56:14.066 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:56:13 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:56:13.612+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T21:56:14.066 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:56:13 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:56:13.612+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T21:56:14.066 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:56:13 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:56:13.612+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T21:56:14.066 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:56:13 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:56:13.613+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:56:14.565 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:56:14 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:56:14.148+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:56:14.565 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:56:14 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:56:14.148+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:56:14.565 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:56:14 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:56:14.148+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:56:14.566 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:56:14 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:56:14.148+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:56:15.065 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:56:14 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:56:14.704+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:56:15.065 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:56:14 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:56:14.704+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:56:15.066 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:56:14 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:56:14.704+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:56:15.066 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:56:14 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:56:14.705+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:56:15.066 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:56:14 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:56:14.569+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T21:56:15.066 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:56:14 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:56:14.569+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T21:56:15.066 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:56:14 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:56:14.569+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T21:56:15.066 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:56:14 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:56:14.570+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:56:15.534 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:56:15 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:56:15.137+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:56:15.534 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:56:15 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:56:15.137+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:56:15.534 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:56:15 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:56:15.137+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:56:15.534 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:56:15 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:56:15.138+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:56:15.815 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:56:15 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:56:15.703+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:56:15.815 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:56:15 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:56:15.703+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:56:15.816 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:56:15 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:56:15.703+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:56:15.816 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:56:15 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:56:15.705+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:56:15.816 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:56:15 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:56:15.531+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T21:56:15.816 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:56:15 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:56:15.531+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T21:56:15.816 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:56:15 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:56:15.531+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T21:56:15.816 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:56:15 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:56:15.532+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:56:16.511 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:56:16 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:56:16.148+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:56:16.512 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:56:16 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:56:16.148+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:56:16.512 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:56:16 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:56:16.148+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:56:16.512 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:56:16 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:56:16.149+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:56:16.815 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Dec 16 21:56:16 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-mon-smithi045[80759]: 2023-12-16T21:56:16.509+0000 7f45a1f8f700 -1 mon.smithi045@0(probing) e2 get_health_metrics reporting 15420 slow ops, oldest is monmgrreport(gid 14214, 0 checks, 1 progress events) 2023-12-16T21:56:16.816 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:56:16 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:56:16.520+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T21:56:16.816 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:56:16 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:56:16.520+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T21:56:16.816 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:56:16 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:56:16.520+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T21:56:16.817 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:56:16 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:56:16.522+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:56:16.817 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:56:16 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:56:16.748+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:56:16.818 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:56:16 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:56:16.748+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:56:16.818 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:56:16 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:56:16.748+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:56:16.818 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:56:16 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:56:16.749+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:56:17.556 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:56:17 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:56:17.137+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:56:17.556 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:56:17 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:56:17.137+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:56:17.556 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:56:17 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:56:17.137+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:56:17.556 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:56:17 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:56:17.138+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:56:17.815 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:56:17 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:56:17.553+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T21:56:17.815 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:56:17 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:56:17.553+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T21:56:17.816 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:56:17 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:56:17.553+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T21:56:17.816 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:56:17 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:56:17.553+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:56:17.816 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:56:17 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:56:17.766+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:56:17.816 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:56:17 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:56:17.766+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:56:17.816 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:56:17 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:56:17.766+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:56:17.816 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:56:17 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:56:17.766+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:56:18.565 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:56:18 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:56:18.096+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:56:18.565 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:56:18 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:56:18.096+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:56:18.566 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:56:18 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:56:18.096+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:56:18.566 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:56:18 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:56:18.098+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:56:19.051 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:56:18 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:56:18.599+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T21:56:19.051 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:56:18 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:56:18.599+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T21:56:19.051 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:56:18 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:56:18.599+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T21:56:19.051 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:56:18 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:56:18.600+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:56:19.052 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:56:18 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:56:18.731+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:56:19.052 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:56:18 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:56:18.731+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:56:19.052 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:56:18 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:56:18.731+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:56:19.052 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:56:18 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:56:18.731+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:56:19.315 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:56:19 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:56:19.048+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:56:19.315 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:56:19 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:56:19.048+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:56:19.316 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:56:19 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:56:19.048+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:56:19.316 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:56:19 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:56:19.049+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:56:20.065 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:56:19 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:56:19.732+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:56:20.066 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:56:19 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:56:19.732+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:56:20.066 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:56:19 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:56:19.732+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:56:20.066 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:56:19 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:56:19.732+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:56:20.066 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:56:19 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:56:19.613+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T21:56:20.066 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:56:19 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:56:19.613+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T21:56:20.066 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:56:19 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:56:19.613+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T21:56:20.066 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:56:19 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:56:19.613+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:56:20.566 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:56:20 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:56:20.081+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:56:20.566 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:56:20 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:56:20.081+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:56:20.566 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:56:20 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:56:20.081+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:56:20.566 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:56:20 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:56:20.082+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:56:21.065 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:56:20 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:56:20.756+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:56:21.066 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:56:20 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:56:20.756+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:56:21.066 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:56:20 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:56:20.756+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:56:21.067 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:56:20 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:56:20.757+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:56:21.067 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:56:20 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:56:20.614+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T21:56:21.067 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:56:20 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:56:20.614+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T21:56:21.068 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:56:20 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:56:20.614+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T21:56:21.068 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:56:20 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:56:20.615+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:56:21.392 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:56:21 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:56:21.067+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:56:21.392 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:56:21 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:56:21.067+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:56:21.392 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:56:21 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:56:21.067+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:56:21.392 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:56:21 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:56:21.069+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:56:21.660 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Dec 16 21:56:21 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-mon-smithi045[80759]: 2023-12-16T21:56:21.510+0000 7f45a1f8f700 -1 mon.smithi045@0(probing) e2 get_health_metrics reporting 15509 slow ops, oldest is monmgrreport(gid 14214, 0 checks, 1 progress events) 2023-12-16T21:56:22.027 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:56:21 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:56:21.798+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:56:22.028 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:56:21 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:56:21.798+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:56:22.028 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:56:21 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:56:21.798+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:56:22.028 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:56:21 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:56:21.798+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:56:22.028 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:56:21 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:56:21.658+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T21:56:22.028 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:56:21 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:56:21.658+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T21:56:22.028 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:56:21 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:56:21.658+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T21:56:22.028 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:56:21 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:56:21.659+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:56:22.315 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:56:22 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:56:22.025+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:56:22.315 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:56:22 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:56:22.025+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:56:22.315 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:56:22 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:56:22.025+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:56:22.316 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:56:22 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:56:22.026+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:56:22.994 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:56:22 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:56:22.616+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T21:56:22.995 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:56:22 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:56:22.616+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T21:56:22.995 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:56:22 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:56:22.616+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T21:56:22.995 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:56:22 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:56:22.616+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:56:22.995 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:56:22 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:56:22.807+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:56:22.995 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:56:22 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:56:22.807+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:56:22.996 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:56:22 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:56:22.807+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:56:22.996 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:56:22 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:56:22.808+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:56:23.315 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:56:22 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:56:22.992+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:56:23.315 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:56:22 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:56:22.992+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:56:23.315 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:56:22 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:56:22.992+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:56:23.316 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:56:22 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:56:22.993+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:56:24.025 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:56:23 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:56:23.759+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:56:24.026 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:56:23 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:56:23.759+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:56:24.026 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:56:23 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:56:23.759+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:56:24.026 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:56:23 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:56:23.761+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:56:24.026 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:56:23 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:56:23.603+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T21:56:24.026 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:56:23 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:56:23.603+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T21:56:24.027 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:56:23 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:56:23.603+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T21:56:24.027 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:56:23 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:56:23.603+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:56:24.315 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:56:24 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:56:24.023+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:56:24.315 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:56:24 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:56:24.023+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:56:24.315 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:56:24 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:56:24.023+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:56:24.315 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:56:24 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:56:24.025+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:56:24.812 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:56:24 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:56:24.561+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T21:56:24.812 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:56:24 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:56:24.561+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T21:56:24.812 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:56:24 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:56:24.561+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T21:56:24.813 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:56:24 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:56:24.562+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:56:25.065 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:56:24 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:56:24.810+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:56:25.065 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:56:24 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:56:24.810+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:56:25.066 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:56:24 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:56:24.810+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:56:25.066 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:56:24 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:56:24.811+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:56:25.066 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:56:24 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:56:24.983+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:56:25.066 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:56:24 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:56:24.983+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:56:25.066 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:56:24 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:56:24.983+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:56:25.066 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:56:24 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:56:24.983+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:56:25.781 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:56:25 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:56:25.523+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T21:56:25.782 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:56:25 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:56:25.523+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T21:56:25.782 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:56:25 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:56:25.523+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T21:56:25.782 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:56:25 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:56:25.523+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:56:26.065 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:56:25 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:56:25.779+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:56:26.065 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:56:25 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:56:25.779+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:56:26.066 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:56:25 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:56:25.779+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:56:26.066 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:56:25 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:56:25.781+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:56:26.066 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:56:25 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:56:25.941+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:56:26.066 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:56:25 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:56:25.941+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:56:26.066 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:56:25 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:56:25.941+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:56:26.066 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:56:25 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:56:25.942+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:56:26.815 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Dec 16 21:56:26 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-mon-smithi045[80759]: 2023-12-16T21:56:26.512+0000 7f45a1f8f700 -1 mon.smithi045@0(probing) e2 get_health_metrics reporting 15595 slow ops, oldest is monmgrreport(gid 14214, 0 checks, 1 progress events) 2023-12-16T21:56:26.815 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:56:26 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:56:26.497+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T21:56:26.816 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:56:26 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:56:26.497+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T21:56:26.816 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:56:26 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:56:26.497+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T21:56:26.816 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:56:26 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:56:26.498+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:56:27.066 DEBUG:teuthology.orchestra.run:got remote process result: 124 2023-12-16T21:56:27.067 WARNING:tasks.kclient:Ordinary umount failed, forcing... 2023-12-16T21:56:27.068 DEBUG:teuthology.orchestra.run.smithi045:> set -ex 2023-12-16T21:56:27.068 DEBUG:teuthology.orchestra.run.smithi045:> dd if=/proc/self/mounts of=/dev/stdout 2023-12-16T21:56:27.126 DEBUG:teuthology.orchestra.run.smithi045:> set -ex 2023-12-16T21:56:27.127 DEBUG:teuthology.orchestra.run.smithi045:> dd if=/proc/self/mounts of=/dev/stdout 2023-12-16T21:56:27.185 DEBUG:teuthology.orchestra.run.smithi045:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-12-16T21:56:27.255 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:56:26 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:56:26.907+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:56:27.255 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:56:26 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:56:26.907+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:56:27.255 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:56:26 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:56:26.907+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:56:27.255 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:56:26 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:56:26.909+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:56:27.255 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:56:26 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:56:26.815+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:56:27.256 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:56:26 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:56:26.815+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:56:27.256 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:56:26 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:56:26.815+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:56:27.256 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:56:26 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:56:26.816+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:56:27.772 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:56:27 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:56:27.516+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T21:56:27.772 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:56:27 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:56:27.516+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T21:56:27.772 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:56:27 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:56:27.516+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T21:56:27.772 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:56:27 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:56:27.517+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:56:28.065 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:56:27 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:56:27.864+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:56:28.065 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:56:27 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:56:27.864+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:56:28.066 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:56:27 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:56:27.864+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:56:28.066 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:56:27 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:56:27.865+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:56:28.066 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:56:27 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:56:27.770+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:56:28.066 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:56:27 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:56:27.770+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:56:28.066 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:56:27 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:56:27.770+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:56:28.066 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:56:27 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:56:27.771+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:56:28.800 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:56:28 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:56:28.541+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T21:56:28.800 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:56:28 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:56:28.541+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T21:56:28.800 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:56:28 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:56:28.541+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T21:56:28.800 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:56:28 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:56:28.542+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:56:29.065 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:56:28 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:56:28.798+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:56:29.065 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:56:28 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:56:28.798+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:56:29.066 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:56:28 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:56:28.798+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:56:29.066 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:56:28 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:56:28.799+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:56:29.066 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:56:28 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:56:28.898+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:56:29.066 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:56:28 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:56:28.898+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:56:29.066 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:56:28 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:56:28.898+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:56:29.066 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:56:28 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:56:28.899+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:56:29.797 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:56:29 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:56:29.514+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T21:56:29.797 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:56:29 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:56:29.514+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T21:56:29.797 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:56:29 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:56:29.514+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T21:56:29.797 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:56:29 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:56:29.516+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:56:30.065 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:56:29 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:56:29.795+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:56:30.065 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:56:29 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:56:29.795+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:56:30.066 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:56:29 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:56:29.795+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:56:30.066 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:56:29 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:56:29.796+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:56:30.066 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:56:29 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:56:29.890+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:56:30.066 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:56:29 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:56:29.891+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:56:30.066 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:56:29 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:56:29.891+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:56:30.066 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:56:29 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:56:29.892+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:56:30.770 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:56:30 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:56:30.512+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T21:56:30.771 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:56:30 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:56:30.512+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T21:56:30.771 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:56:30 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:56:30.512+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T21:56:30.771 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:56:30 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:56:30.513+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:56:31.065 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:56:30 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:56:30.768+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:56:31.065 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:56:30 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:56:30.768+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:56:31.065 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:56:30 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:56:30.768+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:56:31.066 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:56:30 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:56:30.769+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:56:31.066 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:56:30 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:56:30.855+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:56:31.066 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:56:30 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:56:30.855+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:56:31.066 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:56:30 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:56:30.855+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:56:31.066 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:56:30 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:56:30.856+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:56:31.784 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Dec 16 21:56:31 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-mon-smithi045[80759]: 2023-12-16T21:56:31.514+0000 7f45a1f8f700 -1 mon.smithi045@0(probing) e2 get_health_metrics reporting 15685 slow ops, oldest is monmgrreport(gid 14214, 0 checks, 1 progress events) 2023-12-16T21:56:31.784 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:56:31 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:56:31.538+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T21:56:31.784 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:56:31 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:56:31.538+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T21:56:31.784 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:56:31 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:56:31.538+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T21:56:31.784 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:56:31 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:56:31.539+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:56:32.065 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:56:31 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:56:31.782+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:56:32.065 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:56:31 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:56:31.782+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:56:32.065 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:56:31 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:56:31.782+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:56:32.066 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:56:31 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:56:31.783+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:56:32.066 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:56:31 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:56:31.852+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:56:32.066 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:56:31 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:56:31.852+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:56:32.066 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:56:31 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:56:31.852+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:56:32.066 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:56:31 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:56:31.853+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:56:32.815 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:56:32 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:56:32.532+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T21:56:32.815 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:56:32 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:56:32.532+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T21:56:32.815 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:56:32 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:56:32.532+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T21:56:32.816 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:56:32 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:56:32.533+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:56:33.315 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:56:32 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:56:32.826+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:56:33.316 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:56:32 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:56:32.826+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:56:33.316 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:56:32 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:56:32.826+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:56:33.316 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:56:32 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:56:32.827+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:56:33.316 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:56:32 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:56:32.876+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:56:33.316 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:56:32 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:56:32.876+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:56:33.316 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:56:32 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:56:32.876+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:56:33.317 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:56:32 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:56:32.877+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:56:33.804 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:56:33 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:56:33.529+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T21:56:33.804 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:56:33 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:56:33.529+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T21:56:33.805 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:56:33 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:56:33.529+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T21:56:33.805 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:56:33 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:56:33.530+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:56:34.065 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:56:33 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:56:33.864+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:56:34.065 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:56:33 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:56:33.864+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:56:34.065 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:56:33 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:56:33.864+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:56:34.066 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:56:33 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:56:33.864+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:56:34.066 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:56:33 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:56:33.802+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:56:34.066 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:56:33 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:56:33.802+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:56:34.066 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:56:33 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:56:33.802+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:56:34.066 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:56:33 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:56:33.803+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:56:34.794 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:56:34 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:56:34.523+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T21:56:34.794 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:56:34 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:56:34.523+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T21:56:34.794 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:56:34 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:56:34.523+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T21:56:34.795 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:56:34 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:56:34.525+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:56:35.065 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:56:34 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:56:34.825+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:56:35.065 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:56:34 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:56:34.825+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:56:35.066 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:56:34 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:56:34.825+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:56:35.066 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:56:34 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:56:34.825+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:56:35.066 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:56:34 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:56:34.792+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:56:35.066 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:56:34 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:56:34.792+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:56:35.066 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:56:34 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:56:34.792+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:56:35.066 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:56:34 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:56:34.794+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:56:35.800 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:56:35 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:56:35.543+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T21:56:35.801 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:56:35 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:56:35.543+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T21:56:35.801 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:56:35 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:56:35.543+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T21:56:35.801 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:56:35 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:56:35.543+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:56:36.065 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:56:35 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:56:35.799+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:56:36.066 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:56:35 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:56:35.799+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:56:36.066 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:56:35 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:56:35.799+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:56:36.066 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:56:35 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:56:35.800+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:56:36.066 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:56:35 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:56:35.807+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:56:36.066 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:56:35 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:56:35.807+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:56:36.066 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:56:35 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:56:35.807+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:56:36.066 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:56:35 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:56:35.807+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:56:36.815 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Dec 16 21:56:36 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-mon-smithi045[80759]: 2023-12-16T21:56:36.516+0000 7f45a1f8f700 -1 mon.smithi045@0(probing) e2 get_health_metrics reporting 15772 slow ops, oldest is monmgrreport(gid 14214, 0 checks, 1 progress events) 2023-12-16T21:56:36.816 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:56:36 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:56:36.764+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:56:36.816 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:56:36 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:56:36.764+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:56:36.816 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:56:36 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:56:36.764+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:56:36.816 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:56:36 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:56:36.765+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:56:36.816 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:56:36 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:56:36.516+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T21:56:36.816 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:56:36 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:56:36.516+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T21:56:36.816 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:56:36 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:56:36.516+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T21:56:36.817 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:56:36 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:56:36.521+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:56:37.315 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:56:36 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:56:36.846+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:56:37.316 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:56:36 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:56:36.846+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:56:37.316 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:56:36 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:56:36.846+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:56:37.316 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:56:36 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:56:36.847+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:56:37.785 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:56:37 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:56:37.493+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T21:56:37.787 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:56:37 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:56:37.493+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T21:56:37.787 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:56:37 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:56:37.493+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T21:56:37.787 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:56:37 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:56:37.494+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:56:38.065 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:56:37 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:56:37.783+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:56:38.065 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:56:37 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:56:37.783+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:56:38.066 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:56:37 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:56:37.783+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:56:38.066 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:56:37 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:56:37.784+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:56:38.066 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:56:37 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:56:37.890+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:56:38.066 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:56:37 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:56:37.890+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:56:38.066 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:56:37 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:56:37.890+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:56:38.066 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:56:37 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:56:37.891+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:56:38.814 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:56:38 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:56:38.523+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T21:56:38.815 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:56:38 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:56:38.523+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T21:56:38.815 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:56:38 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:56:38.523+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T21:56:38.815 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:56:38 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:56:38.524+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:56:39.065 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:56:38 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:56:38.887+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:56:39.066 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:56:38 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:56:38.887+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:56:39.066 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:56:38 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:56:38.887+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:56:39.066 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:56:38 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:56:38.888+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:56:39.066 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:56:38 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:56:38.812+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:56:39.066 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:56:38 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:56:38.812+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:56:39.067 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:56:38 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:56:38.812+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:56:39.067 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:56:38 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:56:38.813+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:56:39.815 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:56:39 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:56:39.537+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T21:56:39.815 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:56:39 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:56:39.537+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T21:56:39.815 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:56:39 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:56:39.537+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T21:56:39.816 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:56:39 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:56:39.539+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:56:40.157 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:56:39 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:56:39.856+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:56:40.157 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:56:39 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:56:39.856+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:56:40.157 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:56:39 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:56:39.856+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:56:40.157 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:56:39 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:56:39.857+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:56:40.158 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:56:39 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:56:39.861+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:56:40.158 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:56:39 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:56:39.861+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:56:40.158 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:56:39 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:56:39.861+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:56:40.158 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:56:39 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:56:39.862+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:56:40.815 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:56:40 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:56:40.554+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T21:56:40.815 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:56:40 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:56:40.554+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T21:56:40.816 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:56:40 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:56:40.554+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T21:56:40.816 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:56:40 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:56:40.555+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:56:41.315 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:56:40 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:56:40.872+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:56:41.315 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:56:40 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:56:40.872+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:56:41.316 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:56:40 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:56:40.872+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:56:41.316 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:56:40 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:56:40.873+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:56:41.316 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:56:40 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:56:40.845+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:56:41.316 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:56:40 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:56:40.845+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:56:41.316 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:56:40 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:56:40.845+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:56:41.316 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:56:40 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:56:40.845+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:56:41.805 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Dec 16 21:56:41 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-mon-smithi045[80759]: 2023-12-16T21:56:41.517+0000 7f45a1f8f700 -1 mon.smithi045@0(probing) e2 get_health_metrics reporting 15862 slow ops, oldest is monmgrreport(gid 14214, 0 checks, 1 progress events) 2023-12-16T21:56:41.806 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:56:41 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:56:41.579+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T21:56:41.806 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:56:41 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:56:41.579+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T21:56:41.806 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:56:41 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:56:41.579+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T21:56:41.806 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:56:41 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:56:41.580+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:56:42.065 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:56:41 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:56:41.803+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:56:42.065 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:56:41 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:56:41.803+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:56:42.066 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:56:41 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:56:41.803+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:56:42.066 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:56:41 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:56:41.805+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:56:42.066 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:56:41 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:56:41.894+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:56:42.066 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:56:41 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:56:41.894+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:56:42.066 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:56:41 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:56:41.894+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:56:42.066 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:56:41 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:56:41.895+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:56:42.866 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:56:42 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:56:42.773+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:56:42.866 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:56:42 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:56:42.773+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:56:42.866 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:56:42 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:56:42.773+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:56:42.866 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:56:42 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:56:42.774+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:56:42.867 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:56:42 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:56:42.611+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T21:56:42.867 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:56:42 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:56:42.611+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T21:56:42.867 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:56:42 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:56:42.611+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T21:56:42.867 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:56:42 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:56:42.612+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:56:43.315 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:56:42 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:56:42.864+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:56:43.315 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:56:42 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:56:42.864+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:56:43.315 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:56:42 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:56:42.864+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:56:43.316 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:56:42 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:56:42.865+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:56:43.911 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:56:43 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:56:43.599+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T21:56:43.912 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:56:43 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:56:43.600+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T21:56:43.912 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:56:43 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:56:43.600+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T21:56:43.912 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:56:43 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:56:43.600+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:56:43.912 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:56:43 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:56:43.811+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:56:43.912 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:56:43 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:56:43.811+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:56:43.913 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:56:43 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:56:43.811+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:56:43.913 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:56:43 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:56:43.812+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:56:44.315 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:56:43 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:56:43.909+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:56:44.315 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:56:43 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:56:43.909+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:56:44.315 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:56:43 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:56:43.909+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:56:44.316 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:56:43 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:56:43.910+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:56:44.896 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:56:44 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:56:44.833+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:56:44.896 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:56:44 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:56:44.833+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:56:44.896 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:56:44 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:56:44.833+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:56:44.896 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:56:44 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:56:44.834+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:56:44.897 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:56:44 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:56:44.628+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T21:56:44.897 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:56:44 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:56:44.628+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T21:56:44.897 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:56:44 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:56:44.628+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T21:56:44.897 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:56:44 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:56:44.629+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:56:45.315 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:56:44 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:56:44.894+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:56:45.315 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:56:44 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:56:44.894+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:56:45.316 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:56:44 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:56:44.894+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:56:45.316 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:56:44 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:56:44.895+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:56:45.907 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:56:45 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:56:45.805+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:56:45.907 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:56:45 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:56:45.805+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:56:45.907 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:56:45 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:56:45.805+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:56:45.908 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:56:45 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:56:45.806+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:56:45.908 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:56:45 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:56:45.581+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T21:56:45.908 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:56:45 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:56:45.581+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T21:56:45.908 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:56:45 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:56:45.581+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T21:56:45.908 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:56:45 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:56:45.582+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:56:46.315 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:56:45 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:56:45.905+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:56:46.315 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:56:45 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:56:45.905+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:56:46.315 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:56:45 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:56:45.905+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:56:46.316 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:56:45 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:56:45.906+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:56:46.803 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Dec 16 21:56:46 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-mon-smithi045[80759]: 2023-12-16T21:56:46.519+0000 7f45a1f8f700 -1 mon.smithi045@0(probing) e2 get_health_metrics reporting 15949 slow ops, oldest is monmgrreport(gid 14214, 0 checks, 1 progress events) 2023-12-16T21:56:46.803 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:56:46 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:56:46.541+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T21:56:46.803 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:56:46 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:56:46.541+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T21:56:46.803 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:56:46 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:56:46.541+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T21:56:46.803 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:56:46 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:56:46.542+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:56:47.065 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:56:46 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:56:46.899+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:56:47.065 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:56:46 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:56:46.899+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:56:47.066 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:56:46 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:56:46.899+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:56:47.066 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:56:46 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:56:46.900+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:56:47.066 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:56:46 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:56:46.801+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:56:47.066 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:56:46 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:56:46.801+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:56:47.066 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:56:46 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:56:46.801+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:56:47.066 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:56:46 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:56:46.802+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:56:47.893 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:56:47 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:56:47.762+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:56:47.893 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:56:47 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:56:47.762+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:56:47.894 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:56:47 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:56:47.762+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:56:47.894 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:56:47 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:56:47.763+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:56:47.894 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:56:47 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:56:47.591+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T21:56:47.894 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:56:47 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:56:47.591+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T21:56:47.894 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:56:47 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:56:47.591+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T21:56:47.894 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:56:47 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:56:47.592+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:56:48.315 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:56:47 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:56:47.891+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:56:48.315 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:56:47 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:56:47.891+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:56:48.315 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:56:47 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:56:47.891+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:56:48.316 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:56:47 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:56:47.892+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:56:48.890 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:56:48 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:56:48.750+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:56:48.891 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:56:48 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:56:48.750+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:56:48.891 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:56:48 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:56:48.750+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:56:48.891 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:56:48 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:56:48.751+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:56:48.891 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:56:48 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:56:48.596+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T21:56:48.891 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:56:48 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:56:48.596+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T21:56:48.891 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:56:48 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:56:48.596+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T21:56:48.892 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:56:48 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:56:48.596+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:56:49.315 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:56:48 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:56:48.888+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:56:49.315 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:56:48 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:56:48.888+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:56:49.315 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:56:48 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:56:48.888+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:56:49.315 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:56:48 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:56:48.890+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:56:49.900 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:56:49 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:56:49.775+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:56:49.901 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:56:49 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:56:49.775+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:56:49.901 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:56:49 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:56:49.775+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:56:49.901 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:56:49 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:56:49.776+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:56:49.901 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:56:49 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:56:49.641+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T21:56:49.901 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:56:49 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:56:49.641+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T21:56:49.901 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:56:49 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:56:49.641+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T21:56:49.902 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:56:49 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:56:49.641+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:56:50.157 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:56:49 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:56:49.898+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:56:50.189 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:56:49 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:56:49.898+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:56:50.189 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:56:49 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:56:49.898+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:56:50.189 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:56:49 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:56:49.900+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:56:51.065 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:56:50 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:56:50.673+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T21:56:51.065 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:56:50 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:56:50.673+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T21:56:51.066 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:56:50 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:56:50.673+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T21:56:51.066 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:56:50 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:56:50.674+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:56:51.066 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:56:50 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:56:50.807+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:56:51.066 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:56:50 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:56:50.807+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:56:51.066 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:56:50 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:56:50.807+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:56:51.066 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:56:50 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:56:50.809+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:56:51.067 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:56:50 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:56:50.880+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:56:51.067 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:56:50 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:56:50.880+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:56:51.067 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:56:50 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:56:50.880+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:56:51.067 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:56:50 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:56:50.882+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:56:51.799 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:56:51 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:56:51.710+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T21:56:51.799 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:56:51 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:56:51.710+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T21:56:51.799 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:56:51 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:56:51.710+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T21:56:51.799 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:56:51 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:56:51.711+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:56:51.800 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Dec 16 21:56:51 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-mon-smithi045[80759]: 2023-12-16T21:56:51.521+0000 7f45a1f8f700 -1 mon.smithi045@0(probing) e2 get_health_metrics reporting 16040 slow ops, oldest is monmgrreport(gid 14214, 0 checks, 1 progress events) 2023-12-16T21:56:52.065 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:56:51 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:56:51.797+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:56:52.065 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:56:51 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:56:51.797+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:56:52.066 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:56:51 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:56:51.797+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:56:52.066 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:56:51 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:56:51.798+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:56:52.066 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:56:51 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:56:51.868+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:56:52.066 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:56:51 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:56:51.868+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:56:52.066 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:56:51 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:56:51.868+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:56:52.066 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:56:51 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:56:51.869+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:56:53.065 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:56:52 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:56:52.780+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:56:53.066 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:56:52 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:56:52.780+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:56:53.066 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:56:52 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:56:52.780+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:56:53.066 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:56:52 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:56:52.780+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:56:53.066 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:56:52 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:56:52.882+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:56:53.066 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:56:52 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:56:52.882+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:56:53.066 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:56:52 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:56:52.882+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:56:53.067 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:56:52 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:56:52.882+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:56:53.067 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:56:52 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:56:52.688+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T21:56:53.067 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:56:52 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:56:52.688+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T21:56:53.067 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:56:52 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:56:52.688+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T21:56:53.067 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:56:52 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:56:52.688+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:56:53.863 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:56:53 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:56:53.753+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:56:53.863 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:56:53 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:56:53.753+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:56:53.863 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:56:53 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:56:53.753+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:56:53.863 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:56:53 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:56:53.754+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:56:53.864 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:56:53 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:56:53.668+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T21:56:53.864 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:56:53 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:56:53.668+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T21:56:53.864 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:56:53 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:56:53.668+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T21:56:53.864 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:56:53 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:56:53.669+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:56:54.315 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:56:53 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:56:53.861+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:56:54.315 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:56:53 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:56:53.861+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:56:54.315 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:56:53 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:56:53.861+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:56:54.316 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:56:53 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:56:53.862+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:56:55.065 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:56:54 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:56:54.866+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:56:55.069 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:56:54 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:56:54.866+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:56:55.069 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:56:54 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:56:54.866+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:56:55.069 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:56:54 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:56:54.867+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:56:55.070 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:56:54 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:56:54.672+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T21:56:55.070 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:56:54 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:56:54.672+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T21:56:55.070 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:56:54 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:56:54.672+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T21:56:55.070 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:56:54 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:56:54.673+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:56:55.071 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:56:54 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:56:54.726+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:56:55.071 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:56:54 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:56:54.726+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:56:55.071 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:56:54 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:56:54.726+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:56:55.071 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:56:54 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:56:54.726+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:56:56.065 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:56:55 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:56:55.893+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:56:56.065 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:56:55 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:56:55.893+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:56:56.065 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:56:55 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:56:55.893+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:56:56.066 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:56:55 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:56:55.894+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:56:56.066 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:56:55 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:56:55.761+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:56:56.066 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:56:55 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:56:55.761+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:56:56.066 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:56:55 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:56:55.761+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:56:56.066 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:56:55 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:56:55.762+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:56:56.066 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:56:55 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:56:55.667+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T21:56:56.067 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:56:55 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:56:55.667+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T21:56:56.067 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:56:55 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:56:55.667+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T21:56:56.067 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:56:55 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:56:55.669+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:56:56.804 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Dec 16 21:56:56 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-mon-smithi045[80759]: 2023-12-16T21:56:56.522+0000 7f45a1f8f700 -1 mon.smithi045@0(probing) e2 get_health_metrics reporting 16127 slow ops, oldest is monmgrreport(gid 14214, 0 checks, 1 progress events) 2023-12-16T21:56:56.804 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:56:56 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:56:56.667+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T21:56:56.805 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:56:56 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:56:56.667+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T21:56:56.805 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:56:56 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:56:56.667+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T21:56:56.805 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:56:56 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:56:56.668+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:56:57.065 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:56:56 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:56:56.913+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:56:57.065 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:56:56 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:56:56.913+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:56:57.065 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:56:56 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:56:56.913+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:56:57.066 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:56:56 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:56:56.914+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:56:57.066 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:56:56 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:56:56.802+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:56:57.066 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:56:56 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:56:56.802+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:56:57.066 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:56:56 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:56:56.802+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:56:57.066 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:56:56 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:56:56.803+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:56:58.065 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:56:57 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:56:57.903+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:56:58.065 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:56:57 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:56:57.903+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:56:58.066 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:56:57 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:56:57.903+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:56:58.066 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:56:57 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:56:57.904+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:56:58.066 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:56:57 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:56:57.717+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T21:56:58.066 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:56:57 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:56:57.717+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T21:56:58.066 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:56:57 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:56:57.717+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T21:56:58.066 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:56:57 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:56:57.718+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:56:58.067 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:56:57 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:56:57.785+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:56:58.067 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:56:57 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:56:57.785+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:56:58.067 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:56:57 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:56:57.785+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:56:58.067 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:56:57 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:56:57.786+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:56:59.065 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:56:58 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:56:58.917+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:56:59.066 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:56:58 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:56:58.917+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:56:59.066 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:56:58 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:56:58.917+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:56:59.066 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:56:58 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:56:58.918+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:56:59.066 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:56:58 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:56:58.747+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T21:56:59.066 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:56:58 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:56:58.747+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T21:56:59.066 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:56:58 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:56:58.747+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T21:56:59.066 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:56:58 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:56:58.748+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:56:59.067 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:56:58 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:56:58.755+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:56:59.067 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:56:58 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:56:58.755+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:56:59.067 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:56:58 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:56:58.755+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:56:59.067 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:56:58 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:56:58.756+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:57:00.065 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:56:59 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:56:59.873+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:57:00.065 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:56:59 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:56:59.873+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:57:00.066 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:56:59 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:56:59.873+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:57:00.066 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:56:59 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:56:59.874+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:57:00.066 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:56:59 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:56:59.741+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T21:57:00.066 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:56:59 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:56:59.741+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T21:57:00.066 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:56:59 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:56:59.741+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T21:57:00.066 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:56:59 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:56:59.742+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:57:00.067 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:56:59 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:56:59.715+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:57:00.067 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:56:59 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:56:59.715+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:57:00.067 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:56:59 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:56:59.715+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:57:00.067 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:56:59 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:56:59.716+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:57:01.065 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:57:00 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:57:00.727+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:57:01.065 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:57:00 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:57:00.727+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:57:01.066 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:57:00 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:57:00.727+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:57:01.066 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:57:00 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:57:00.728+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:57:01.066 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:57:00 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:57:00.755+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T21:57:01.066 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:57:00 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:57:00.755+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T21:57:01.066 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:57:00 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:57:00.755+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T21:57:01.066 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:57:00 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:57:00.756+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:57:01.067 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:57:00 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:57:00.911+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:57:01.067 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:57:00 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:57:00.911+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:57:01.067 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:57:00 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:57:00.911+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:57:01.067 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:57:00 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:57:00.912+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:57:01.653 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Dec 16 21:57:01 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-mon-smithi045[80759]: 2023-12-16T21:57:01.524+0000 7f45a1f8f700 -1 mon.smithi045@0(probing) e2 get_health_metrics reporting 16217 slow ops, oldest is monmgrreport(gid 14214, 0 checks, 1 progress events) 2023-12-16T21:57:01.904 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:57:01 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:57:01.901+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:57:01.904 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:57:01 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:57:01.901+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:57:01.904 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:57:01 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:57:01.901+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:57:01.904 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:57:01 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:57:01.902+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:57:01.904 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:57:01 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:57:01.726+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T21:57:01.905 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:57:01 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:57:01.726+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T21:57:01.905 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:57:01 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:57:01.726+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T21:57:01.905 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:57:01 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:57:01.727+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:57:01.905 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:57:01 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:57:01.713+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:57:01.905 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:57:01 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:57:01.713+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:57:01.905 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:57:01 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:57:01.713+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:57:01.905 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:57:01 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:57:01.714+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:57:03.065 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:57:02 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:57:02.859+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:57:03.065 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:57:02 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:57:02.859+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:57:03.066 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:57:02 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:57:02.859+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:57:03.066 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:57:02 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:57:02.860+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:57:03.066 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:57:02 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:57:02.750+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T21:57:03.066 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:57:02 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:57:02.750+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T21:57:03.067 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:57:02 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:57:02.750+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T21:57:03.067 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:57:02 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:57:02.751+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:57:03.067 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:57:02 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:57:02.715+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:57:03.067 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:57:02 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:57:02.715+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:57:03.067 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:57:02 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:57:02.715+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:57:03.067 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:57:02 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:57:02.716+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:57:03.872 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:57:03 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:57:03.695+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:57:03.873 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:57:03 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:57:03.696+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:57:03.873 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:57:03 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:57:03.696+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:57:03.873 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:57:03 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:57:03.696+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:57:03.873 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:57:03 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:57:03.799+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T21:57:03.873 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:57:03 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:57:03.799+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T21:57:03.874 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:57:03 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:57:03.799+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T21:57:03.874 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:57:03 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:57:03.800+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:57:04.315 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:57:03 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:57:03.870+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:57:04.315 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:57:03 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:57:03.870+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:57:04.315 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:57:03 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:57:03.870+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:57:04.315 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:57:03 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:57:03.872+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:57:05.065 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:57:04 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:57:04.834+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:57:05.065 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:57:04 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:57:04.834+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:57:05.065 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:57:04 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:57:04.834+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:57:05.066 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:57:04 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:57:04.834+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:57:05.066 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:57:04 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:57:04.759+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T21:57:05.066 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:57:04 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:57:04.759+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T21:57:05.066 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:57:04 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:57:04.759+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T21:57:05.066 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:57:04 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:57:04.760+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:57:05.067 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:57:04 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:57:04.691+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:57:05.067 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:57:04 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:57:04.691+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:57:05.067 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:57:04 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:57:04.691+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:57:05.067 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:57:04 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:57:04.692+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:57:06.065 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:57:05 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:57:05.730+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:57:06.065 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:57:05 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:57:05.730+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:57:06.065 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:57:05 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:57:05.730+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:57:06.066 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:57:05 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:57:05.732+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:57:06.066 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:57:05 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:57:05.859+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:57:06.066 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:57:05 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:57:05.859+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:57:06.066 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:57:05 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:57:05.859+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:57:06.066 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:57:05 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:57:05.860+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:57:06.067 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:57:05 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:57:05.782+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T21:57:06.067 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:57:05 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:57:05.782+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T21:57:06.067 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:57:05 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:57:05.782+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T21:57:06.067 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:57:05 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:57:05.783+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:57:06.815 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Dec 16 21:57:06 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-mon-smithi045[80759]: 2023-12-16T21:57:06.526+0000 7f45a1f8f700 -1 mon.smithi045@0(probing) e2 get_health_metrics reporting 16304 slow ops, oldest is monmgrreport(gid 14214, 0 checks, 1 progress events) 2023-12-16T21:57:06.815 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:57:06 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:57:06.735+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:57:06.816 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:57:06 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:57:06.735+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:57:06.816 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:57:06 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:57:06.735+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:57:06.816 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:57:06 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:57:06.736+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:57:06.816 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:57:06 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:57:06.742+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T21:57:06.816 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:57:06 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:57:06.742+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T21:57:06.816 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:57:06 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:57:06.742+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T21:57:06.817 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:57:06 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:57:06.742+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:57:07.315 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:57:06 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:57:06.823+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:57:07.315 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:57:06 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:57:06.823+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:57:07.316 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:57:06 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:57:06.823+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:57:07.316 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:57:06 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:57:06.824+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:57:07.815 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:57:07 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:57:07.704+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T21:57:07.815 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:57:07 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:57:07.704+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T21:57:07.815 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:57:07 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:57:07.704+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T21:57:07.816 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:57:07 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:57:07.704+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:57:07.816 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:57:07 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:57:07.702+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:57:07.816 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:57:07 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:57:07.702+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:57:07.816 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:57:07 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:57:07.702+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:57:07.816 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:57:07 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:57:07.703+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:57:08.315 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:57:07 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:57:07.816+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:57:08.315 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:57:07 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:57:07.816+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:57:08.315 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:57:07 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:57:07.816+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:57:08.316 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:57:07 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:57:07.817+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:57:09.065 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:57:08 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:57:08.720+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T21:57:09.065 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:57:08 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:57:08.720+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T21:57:09.065 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:57:08 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:57:08.720+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T21:57:09.066 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:57:08 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:57:08.721+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:57:09.066 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:57:08 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:57:08.743+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:57:09.066 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:57:08 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:57:08.743+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:57:09.066 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:57:08 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:57:08.743+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:57:09.066 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:57:08 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:57:08.744+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:57:09.067 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:57:08 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:57:08.770+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:57:09.067 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:57:08 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:57:08.770+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:57:09.067 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:57:08 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:57:08.770+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:57:09.067 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:57:08 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:57:08.770+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:57:10.065 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:57:09 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:57:09.719+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:57:10.065 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:57:09 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:57:09.719+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:57:10.066 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:57:09 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:57:09.719+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:57:10.066 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:57:09 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:57:09.719+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:57:10.066 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:57:09 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:57:09.714+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T21:57:10.066 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:57:09 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:57:09.714+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T21:57:10.066 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:57:09 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:57:09.714+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T21:57:10.067 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:57:09 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:57:09.714+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:57:10.067 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:57:09 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:57:09.780+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:57:10.067 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:57:09 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:57:09.780+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:57:10.067 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:57:09 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:57:09.780+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:57:10.067 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:57:09 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:57:09.781+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:57:10.815 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:57:10 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:57:10.710+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T21:57:10.815 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:57:10 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:57:10.710+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T21:57:10.816 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:57:10 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:57:10.710+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T21:57:10.816 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:57:10 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:57:10.711+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:57:10.816 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:57:10 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:57:10.690+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:57:10.816 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:57:10 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:57:10.690+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:57:10.816 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:57:10 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:57:10.690+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:57:10.816 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:57:10 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:57:10.691+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:57:11.315 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:57:10 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:57:10.828+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:57:11.315 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:57:10 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:57:10.828+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:57:11.316 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:57:10 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:57:10.828+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:57:11.316 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:57:10 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:57:10.828+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:57:11.815 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Dec 16 21:57:11 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-mon-smithi045[80759]: 2023-12-16T21:57:11.528+0000 7f45a1f8f700 -1 mon.smithi045@0(probing) e2 get_health_metrics reporting 16395 slow ops, oldest is monmgrreport(gid 14214, 0 checks, 1 progress events) 2023-12-16T21:57:11.815 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:57:11 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:57:11.679+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:57:11.816 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:57:11 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:57:11.679+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:57:11.816 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:57:11 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:57:11.679+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:57:11.816 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:57:11 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:57:11.680+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:57:11.816 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:57:11 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:57:11.757+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T21:57:11.816 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:57:11 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:57:11.757+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T21:57:11.816 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:57:11 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:57:11.757+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T21:57:11.816 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:57:11 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:57:11.757+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:57:12.315 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:57:11 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:57:11.838+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:57:12.315 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:57:11 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:57:11.838+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:57:12.316 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:57:11 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:57:11.838+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:57:12.316 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:57:11 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:57:11.839+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:57:13.065 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:57:12 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:57:12.667+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:57:13.065 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:57:12 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:57:12.667+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:57:13.065 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:57:12 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:57:12.667+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:57:13.066 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:57:12 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:57:12.668+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:57:13.066 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:57:12 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:57:12.819+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:57:13.066 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:57:12 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:57:12.819+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:57:13.066 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:57:12 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:57:12.819+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:57:13.066 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:57:12 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:57:12.820+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:57:13.067 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:57:12 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:57:12.783+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T21:57:13.067 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:57:12 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:57:12.783+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T21:57:13.067 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:57:12 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:57:12.783+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T21:57:13.067 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:57:12 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:57:12.784+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:57:14.065 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:57:13 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:57:13.651+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:57:14.065 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:57:13 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:57:13.651+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:57:14.065 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:57:13 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:57:13.651+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:57:14.066 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:57:13 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:57:13.652+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:57:14.066 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:57:13 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:57:13.784+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T21:57:14.066 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:57:13 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:57:13.784+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T21:57:14.066 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:57:13 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:57:13.784+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T21:57:14.066 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:57:13 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:57:13.784+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:57:14.067 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:57:13 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:57:13.822+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:57:14.067 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:57:13 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:57:13.822+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:57:14.067 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:57:13 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:57:13.822+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:57:14.067 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:57:13 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:57:13.822+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:57:15.065 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:57:14 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:57:14.680+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:57:15.065 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:57:14 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:57:14.680+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:57:15.065 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:57:14 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:57:14.680+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:57:15.065 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:57:14 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:57:14.680+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:57:15.066 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:57:14 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:57:14.803+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:57:15.066 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:57:14 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:57:14.803+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:57:15.066 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:57:14 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:57:14.803+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:57:15.066 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:57:14 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:57:14.804+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:57:15.066 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:57:14 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:57:14.813+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T21:57:15.067 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:57:14 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:57:14.813+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T21:57:15.067 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:57:14 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:57:14.813+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T21:57:15.067 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:57:14 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:57:14.814+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:57:16.065 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:57:15 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:57:15.797+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:57:16.065 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:57:15 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:57:15.797+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:57:16.065 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:57:15 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:57:15.797+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:57:16.066 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:57:15 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:57:15.798+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:57:16.066 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:57:15 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:57:15.840+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T21:57:16.066 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:57:15 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:57:15.840+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T21:57:16.066 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:57:15 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:57:15.840+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T21:57:16.066 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:57:15 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:57:15.841+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:57:16.067 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:57:15 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:57:15.691+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:57:16.067 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:57:15 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:57:15.691+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:57:16.067 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:57:15 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:57:15.691+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:57:16.067 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:57:15 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:57:15.692+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:57:16.815 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Dec 16 21:57:16 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-mon-smithi045[80759]: 2023-12-16T21:57:16.530+0000 7f45a1f8f700 -1 mon.smithi045@0(probing) e2 get_health_metrics reporting 16481 slow ops, oldest is monmgrreport(gid 14214, 0 checks, 1 progress events) 2023-12-16T21:57:16.815 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:57:16 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:57:16.660+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:57:16.816 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:57:16 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:57:16.660+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:57:16.816 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:57:16 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:57:16.660+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:57:16.816 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:57:16 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:57:16.660+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:57:17.315 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:57:16 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:57:16.842+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:57:17.315 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:57:16 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:57:16.842+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:57:17.315 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:57:16 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:57:16.842+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:57:17.315 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:57:16 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:57:16.844+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:57:17.316 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:57:16 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:57:16.885+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T21:57:17.316 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:57:16 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:57:16.885+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T21:57:17.316 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:57:16 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:57:16.885+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T21:57:17.316 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:57:16 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:57:16.886+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:57:17.892 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:57:17 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:57:17.889+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:57:17.892 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:57:17 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:57:17.889+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:57:17.893 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:57:17 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:57:17.889+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:57:17.893 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:57:17 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:57:17.890+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:57:17.893 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:57:17 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:57:17.639+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:57:17.893 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:57:17 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:57:17.639+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:57:17.893 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:57:17 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:57:17.639+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:57:17.893 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:57:17 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:57:17.640+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:57:18.315 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:57:17 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:57:17.894+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T21:57:18.316 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:57:17 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:57:17.894+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T21:57:18.316 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:57:17 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:57:17.894+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T21:57:18.316 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:57:17 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:57:17.895+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:57:18.908 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:57:18 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:57:18.642+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:57:18.909 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:57:18 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:57:18.642+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:57:18.909 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:57:18 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:57:18.642+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:57:18.909 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:57:18 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:57:18.644+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:57:19.315 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:57:18 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:57:18.906+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T21:57:19.315 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:57:18 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:57:18.907+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T21:57:19.315 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:57:18 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:57:18.907+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T21:57:19.316 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:57:18 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:57:18.908+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:57:19.316 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:57:18 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:57:18.923+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:57:19.316 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:57:18 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:57:18.923+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:57:19.316 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:57:18 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:57:18.923+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:57:19.316 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:57:18 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:57:18.924+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:57:19.939 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:57:19 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:57:19.672+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:57:19.940 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:57:19 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:57:19.672+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:57:19.940 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:57:19 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:57:19.672+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:57:19.940 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:57:19 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:57:19.673+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:57:20.315 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:57:19 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:57:19.938+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T21:57:20.315 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:57:19 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:57:19.938+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T21:57:20.315 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:57:19 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:57:19.938+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T21:57:20.316 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:57:19 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:57:19.938+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:57:20.316 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:57:19 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:57:19.941+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:57:20.316 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:57:19 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:57:19.941+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:57:20.316 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:57:19 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:57:19.941+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:57:20.316 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:57:19 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:57:19.942+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:57:20.982 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:57:20 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:57:20.703+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:57:20.983 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:57:20 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:57:20.703+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:57:20.983 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:57:20 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:57:20.703+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:57:20.983 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:57:20 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:57:20.703+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:57:20.983 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:57:20 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:57:20.936+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:57:20.983 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:57:20 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:57:20.936+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:57:20.984 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:57:20 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:57:20.936+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:57:20.984 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:57:20 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:57:20.937+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:57:21.315 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:57:20 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:57:20.980+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T21:57:21.315 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:57:20 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:57:20.980+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T21:57:21.315 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:57:20 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:57:20.980+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T21:57:21.315 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:57:20 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:57:20.982+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:57:21.815 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Dec 16 21:57:21 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-mon-smithi045[80759]: 2023-12-16T21:57:21.531+0000 7f45a1f8f700 -1 mon.smithi045@0(probing) e2 get_health_metrics reporting 16571 slow ops, oldest is monmgrreport(gid 14214, 0 checks, 1 progress events) 2023-12-16T21:57:21.815 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:57:21 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:57:21.702+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:57:21.816 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:57:21 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:57:21.702+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:57:21.816 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:57:21 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:57:21.702+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:57:21.816 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:57:21 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:57:21.703+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:57:22.315 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:57:21 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:57:21.930+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:57:22.315 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:57:21 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:57:21.930+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:57:22.315 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:57:21 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:57:21.930+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:57:22.316 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:57:21 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:57:21.932+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:57:22.316 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:57:21 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:57:21.953+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T21:57:22.316 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:57:21 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:57:21.953+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T21:57:22.316 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:57:21 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:57:21.953+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T21:57:22.316 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:57:21 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:57:21.954+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:57:23.065 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:57:22 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:57:22.740+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:57:23.066 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:57:22 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:57:22.740+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:57:23.066 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:57:22 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:57:22.740+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:57:23.066 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:57:22 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:57:22.741+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:57:23.066 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:57:22 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:57:22.914+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:57:23.066 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:57:22 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:57:22.914+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:57:23.067 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:57:22 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:57:22.914+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:57:23.067 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:57:22 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:57:22.915+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:57:23.067 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:57:22 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:57:22.967+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T21:57:23.067 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:57:22 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:57:22.967+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T21:57:23.067 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:57:22 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:57:22.967+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T21:57:23.068 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:57:22 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:57:22.968+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:57:23.941 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:57:23 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:57:23.694+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:57:23.942 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:57:23 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:57:23.694+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:57:23.942 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:57:23 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:57:23.694+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:57:23.942 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:57:23 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:57:23.694+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:57:24.315 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:57:23 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:57:23.945+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T21:57:24.315 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:57:23 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:57:23.945+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T21:57:24.316 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:57:23 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:57:23.945+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T21:57:24.316 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:57:23 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:57:23.945+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:57:24.316 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:57:23 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:57:23.938+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:57:24.316 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:57:23 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:57:23.938+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:57:24.316 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:57:23 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:57:23.938+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:57:24.317 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:57:23 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:57:23.939+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:57:25.065 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:57:24 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:57:24.733+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:57:25.065 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:57:24 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:57:24.733+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:57:25.066 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:57:24 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:57:24.733+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:57:25.066 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:57:24 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:57:24.733+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:57:25.066 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:57:24 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:57:24.944+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:57:25.066 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:57:24 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:57:24.944+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:57:25.066 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:57:24 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:57:24.944+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:57:25.067 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:57:24 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:57:24.944+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:57:25.067 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:57:24 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:57:24.959+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T21:57:25.067 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:57:24 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:57:24.959+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T21:57:25.067 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:57:24 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:57:24.959+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T21:57:25.067 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:57:24 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:57:24.960+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:57:25.984 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:57:25 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:57:25.698+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:57:25.985 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:57:25 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:57:25.698+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:57:25.985 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:57:25 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:57:25.698+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:57:25.985 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:57:25 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:57:25.698+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:57:25.985 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:57:25 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:57:25.938+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:57:25.985 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:57:25 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:57:25.938+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:57:25.985 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:57:25 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:57:25.938+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:57:25.986 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:57:25 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:57:25.939+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:57:26.315 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:57:25 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:57:25.981+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T21:57:26.315 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:57:25 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:57:25.981+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T21:57:26.315 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:57:25 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:57:25.981+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T21:57:26.316 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:57:25 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:57:25.982+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:57:26.815 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Dec 16 21:57:26 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-mon-smithi045[80759]: 2023-12-16T21:57:26.532+0000 7f45a1f8f700 -1 mon.smithi045@0(probing) e2 get_health_metrics reporting 16658 slow ops, oldest is monmgrreport(gid 14214, 0 checks, 1 progress events) 2023-12-16T21:57:26.815 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:57:26 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:57:26.733+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:57:26.816 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:57:26 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:57:26.733+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:57:26.816 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:57:26 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:57:26.733+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:57:26.816 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:57:26 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:57:26.734+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:57:27.031 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:57:26 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:57:26.904+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:57:27.032 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:57:26 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:57:26.904+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:57:27.032 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:57:26 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:57:26.904+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:57:27.032 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:57:26 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:57:26.904+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:57:27.032 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:57:27 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:57:27.026+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T21:57:27.032 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:57:27 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:57:27.026+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T21:57:27.032 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:57:27 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:57:27.026+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T21:57:27.033 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:57:27 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:57:27.027+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:57:28.050 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:57:27 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:57:27.751+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:57:28.050 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:57:27 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:57:27.751+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:57:28.050 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:57:27 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:57:27.751+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:57:28.051 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:57:27 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:57:27.752+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:57:28.051 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:57:27 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:57:27.940+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:57:28.051 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:57:27 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:57:27.940+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:57:28.051 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:57:27 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:57:27.940+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:57:28.051 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:57:27 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:57:27.941+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:57:28.315 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:57:28 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:57:28.048+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T21:57:28.315 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:57:28 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:57:28.048+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T21:57:28.316 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:57:28 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:57:28.048+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T21:57:28.316 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:57:28 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:57:28.049+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:57:28.976 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:57:28 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:57:28.711+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:57:28.977 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:57:28 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:57:28.711+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:57:28.977 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:57:28 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:57:28.711+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:57:28.977 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:57:28 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:57:28.713+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:57:28.977 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:57:28 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:57:28.901+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:57:28.978 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:57:28 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:57:28.901+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:57:28.978 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:57:28 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:57:28.901+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:57:28.978 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:57:28 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:57:28.901+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:57:29.315 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:57:29 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:57:29.060+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T21:57:29.315 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:57:29 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:57:29.060+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T21:57:29.315 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:57:29 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:57:29.060+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T21:57:29.316 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:57:29 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:57:29.061+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:57:29.952 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:57:29 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:57:29.692+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:57:29.952 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:57:29 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:57:29.692+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:57:29.952 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:57:29 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:57:29.692+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:57:29.952 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:57:29 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:57:29.692+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:57:30.315 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:57:30 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:57:30.101+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T21:57:30.316 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:57:30 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:57:30.101+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T21:57:30.316 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:57:30 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:57:30.101+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T21:57:30.316 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:57:30 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:57:30.101+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:57:30.316 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:57:29 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:57:29.950+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:57:30.316 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:57:29 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:57:29.950+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:57:30.316 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:57:29 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:57:29.950+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:57:30.317 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:57:29 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:57:29.951+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:57:31.065 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:57:30 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:57:30.733+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:57:31.066 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:57:30 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:57:30.733+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:57:31.066 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:57:30 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:57:30.733+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:57:31.066 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:57:30 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:57:30.734+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:57:31.066 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:57:30 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:57:30.950+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:57:31.066 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:57:30 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:57:30.950+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:57:31.066 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:57:30 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:57:30.950+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:57:31.067 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:57:30 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:57:30.951+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:57:31.537 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:57:31 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:57:31.071+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T21:57:31.537 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:57:31 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:57:31.071+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T21:57:31.537 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:57:31 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:57:31.071+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T21:57:31.537 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:57:31 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:57:31.072+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:57:31.815 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Dec 16 21:57:31 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-mon-smithi045[80759]: 2023-12-16T21:57:31.535+0000 7f45a1f8f700 -1 mon.smithi045@0(probing) e2 get_health_metrics reporting 16749 slow ops, oldest is monmgrreport(gid 14214, 0 checks, 1 progress events) 2023-12-16T21:57:31.815 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:57:31 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:57:31.702+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:57:31.815 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:57:31 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:57:31.702+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:57:31.816 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:57:31 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:57:31.702+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:57:31.816 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:57:31 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:57:31.703+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:57:32.315 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:57:32 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:57:32.044+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T21:57:32.315 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:57:32 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:57:32.044+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T21:57:32.315 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:57:32 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:57:32.044+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T21:57:32.316 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:57:32 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:57:32.045+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:57:32.316 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:57:31 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:57:31.933+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:57:32.316 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:57:31 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:57:31.933+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:57:32.316 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:57:31 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:57:31.933+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:57:32.316 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:57:31 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:57:31.933+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:57:33.057 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:57:32 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:57:32.693+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:57:33.058 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:57:32 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:57:32.693+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:57:33.058 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:57:32 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:57:32.693+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:57:33.058 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:57:32 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:57:32.694+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:57:33.058 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:57:32 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:57:32.924+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:57:33.058 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:57:32 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:57:32.925+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:57:33.058 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:57:32 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:57:32.925+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:57:33.059 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:57:32 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:57:32.926+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:57:33.315 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:57:33 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:57:33.055+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T21:57:33.315 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:57:33 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:57:33.055+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T21:57:33.315 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:57:33 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:57:33.055+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T21:57:33.316 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:57:33 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:57:33.057+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:57:33.892 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:57:33 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:57:33.676+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:57:33.892 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:57:33 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:57:33.676+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:57:33.892 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:57:33 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:57:33.676+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:57:33.892 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:57:33 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:57:33.677+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:57:34.315 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:57:34 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:57:34.029+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T21:57:34.315 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:57:34 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:57:34.029+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T21:57:34.316 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:57:34 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:57:34.029+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T21:57:34.316 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:57:34 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:57:34.030+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:57:34.316 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:57:33 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:57:33.890+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:57:34.316 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:57:33 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:57:33.890+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:57:34.316 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:57:33 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:57:33.890+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:57:34.317 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:57:33 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:57:33.891+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:57:35.037 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:57:34 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:57:34.850+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:57:35.037 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:57:34 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:57:34.850+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:57:35.037 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:57:34 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:57:34.850+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:57:35.038 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:57:34 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:57:34.850+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:57:35.038 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:57:34 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:57:34.706+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:57:35.038 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:57:34 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:57:34.707+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:57:35.038 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:57:34 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:57:34.707+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:57:35.038 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:57:34 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:57:34.707+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:57:35.315 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:57:35 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:57:35.035+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T21:57:35.315 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:57:35 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:57:35.035+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T21:57:35.315 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:57:35 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:57:35.035+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T21:57:35.316 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:57:35 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:57:35.036+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:57:36.052 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:57:35 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:57:35.738+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:57:36.052 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:57:35 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:57:35.738+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:57:36.053 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:57:35 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:57:35.739+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:57:36.053 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:57:35 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:57:35.740+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:57:36.053 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:57:35 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:57:35.833+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:57:36.053 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:57:35 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:57:35.833+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:57:36.053 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:57:35 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:57:35.833+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:57:36.053 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:57:35 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:57:35.834+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:57:36.315 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:57:36 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:57:36.050+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T21:57:36.315 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:57:36 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:57:36.050+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T21:57:36.315 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:57:36 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:57:36.050+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T21:57:36.315 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:57:36 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:57:36.052+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:57:36.805 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Dec 16 21:57:36 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-mon-smithi045[80759]: 2023-12-16T21:57:36.537+0000 7f45a1f8f700 -1 mon.smithi045@0(probing) e2 get_health_metrics reporting 16836 slow ops, oldest is monmgrreport(gid 14214, 0 checks, 1 progress events) 2023-12-16T21:57:36.805 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:57:36 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:57:36.734+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:57:36.805 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:57:36 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:57:36.734+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:57:36.805 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:57:36 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:57:36.734+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:57:36.805 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:57:36 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:57:36.735+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:57:37.063 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:57:36 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:57:36.803+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:57:37.064 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:57:36 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:57:36.803+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:57:37.064 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:57:36 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:57:36.803+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:57:37.064 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:57:36 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:57:36.804+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:57:37.315 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:57:37 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:57:37.062+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T21:57:37.316 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:57:37 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:57:37.062+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T21:57:37.316 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:57:37 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:57:37.062+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T21:57:37.316 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:57:37 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:57:37.062+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:57:38.028 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:57:37 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:57:37.784+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:57:38.028 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:57:37 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:57:37.784+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:57:38.028 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:57:37 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:57:37.784+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:57:38.028 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:57:37 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:57:37.785+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:57:38.029 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:57:37 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:57:37.771+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:57:38.029 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:57:37 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:57:37.771+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:57:38.029 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:57:37 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:57:37.771+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:57:38.029 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:57:37 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:57:37.772+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:57:38.315 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:57:38 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:57:38.026+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T21:57:38.315 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:57:38 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:57:38.026+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T21:57:38.315 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:57:38 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:57:38.026+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T21:57:38.315 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:57:38 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:57:38.027+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:57:39.002 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:57:38 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:57:38.752+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:57:39.002 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:57:38 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:57:38.752+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:57:39.002 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:57:38 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:57:38.752+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:57:39.002 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:57:38 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:57:38.753+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:57:39.003 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:57:38 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:57:38.789+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:57:39.003 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:57:38 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:57:38.789+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:57:39.003 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:57:38 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:57:38.789+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:57:39.003 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:57:38 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:57:38.790+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:57:39.315 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:57:39 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:57:39.000+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T21:57:39.315 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:57:39 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:57:39.000+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T21:57:39.315 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:57:39 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:57:39.000+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T21:57:39.315 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:57:39 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:57:39.001+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:57:40.003 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:57:39 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:57:39.759+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:57:40.004 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:57:39 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:57:39.759+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:57:40.004 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:57:39 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:57:39.759+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:57:40.004 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:57:39 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:57:39.760+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:57:40.004 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:57:39 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:57:39.723+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:57:40.004 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:57:39 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:57:39.723+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:57:40.005 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:57:39 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:57:39.723+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:57:40.005 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:57:39 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:57:39.724+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:57:40.315 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:57:40 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:57:40.002+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T21:57:40.315 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:57:40 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:57:40.002+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T21:57:40.315 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:57:40 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:57:40.002+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T21:57:40.315 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:57:40 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:57:40.003+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:57:41.044 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:57:40 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:57:40.754+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:57:41.045 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:57:40 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:57:40.754+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:57:41.045 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:57:40 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:57:40.754+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:57:41.045 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:57:40 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:57:40.755+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:57:41.045 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:57:40 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:57:40.721+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:57:41.045 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:57:40 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:57:40.721+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:57:41.046 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:57:40 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:57:40.721+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:57:41.046 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:57:40 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:57:40.722+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:57:41.316 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:57:41 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:57:41.043+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T21:57:41.316 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:57:41 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:57:41.043+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T21:57:41.316 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:57:41 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:57:41.043+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T21:57:41.317 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:57:41 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:57:41.044+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:57:41.815 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Dec 16 21:57:41 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-mon-smithi045[80759]: 2023-12-16T21:57:41.539+0000 7f45a1f8f700 -1 mon.smithi045@0(probing) e2 get_health_metrics reporting 16927 slow ops, oldest is monmgrreport(gid 14214, 0 checks, 1 progress events) 2023-12-16T21:57:41.815 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:57:41 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:57:41.748+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:57:41.816 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:57:41 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:57:41.748+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:57:41.816 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:57:41 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:57:41.748+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:57:41.816 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:57:41 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:57:41.749+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:57:41.816 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:57:41 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:57:41.733+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:57:41.816 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:57:41 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:57:41.733+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:57:41.816 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:57:41 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:57:41.733+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:57:41.816 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:57:41 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:57:41.734+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:57:42.315 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:57:42 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:57:42.008+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T21:57:42.315 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:57:42 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:57:42.008+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T21:57:42.315 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:57:42 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:57:42.008+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T21:57:42.316 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:57:42 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:57:42.008+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:57:42.990 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:57:42 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:57:42.702+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:57:42.990 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:57:42 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:57:42.702+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:57:42.990 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:57:42 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:57:42.702+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:57:42.990 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:57:42 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:57:42.704+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:57:42.990 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:57:42 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:57:42.740+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:57:42.991 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:57:42 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:57:42.740+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:57:42.991 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:57:42 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:57:42.740+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:57:42.991 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:57:42 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:57:42.741+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:57:43.315 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:57:42 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:57:42.988+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T21:57:43.315 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:57:42 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:57:42.988+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T21:57:43.315 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:57:42 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:57:42.988+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T21:57:43.315 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:57:42 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:57:42.989+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:57:43.972 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:57:43 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:57:43.662+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:57:43.972 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:57:43 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:57:43.662+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:57:43.973 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:57:43 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:57:43.662+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:57:43.973 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:57:43 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:57:43.663+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:57:43.973 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:57:43 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:57:43.755+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:57:43.973 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:57:43 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:57:43.755+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:57:43.973 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:57:43 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:57:43.755+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:57:43.973 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:57:43 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:57:43.756+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:57:44.315 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:57:43 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:57:43.970+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T21:57:44.315 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:57:43 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:57:43.970+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T21:57:44.315 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:57:43 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:57:43.970+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T21:57:44.315 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:57:43 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:57:43.972+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:57:45.065 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:57:44 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:57:44.695+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:57:45.065 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:57:44 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:57:44.695+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:57:45.066 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:57:44 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:57:44.695+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:57:45.066 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:57:44 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:57:44.696+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:57:45.066 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:57:44 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:57:44.801+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:57:45.066 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:57:44 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:57:44.801+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:57:45.066 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:57:44 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:57:44.801+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:57:45.066 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:57:44 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:57:44.802+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:57:45.067 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:57:44 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:57:44.943+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T21:57:45.067 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:57:44 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:57:44.943+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T21:57:45.067 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:57:44 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:57:44.943+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T21:57:45.067 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:57:44 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:57:44.944+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:57:45.958 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:57:45 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:57:45.704+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:57:45.958 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:57:45 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:57:45.704+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:57:45.958 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:57:45 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:57:45.704+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:57:45.959 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:57:45 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:57:45.705+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:57:45.959 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:57:45 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:57:45.801+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:57:45.959 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:57:45 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:57:45.801+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:57:45.959 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:57:45 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:57:45.801+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:57:45.959 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:57:45 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:57:45.802+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:57:46.315 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:57:45 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:57:45.956+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T21:57:46.315 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:57:45 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:57:45.956+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T21:57:46.315 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:57:45 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:57:45.956+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T21:57:46.315 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:57:45 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:57:45.957+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:57:46.815 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Dec 16 21:57:46 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-mon-smithi045[80759]: 2023-12-16T21:57:46.541+0000 7f45a1f8f700 -1 mon.smithi045@0(probing) e2 get_health_metrics reporting 17015 slow ops, oldest is monmgrreport(gid 14214, 0 checks, 1 progress events) 2023-12-16T21:57:46.815 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:57:46 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:57:46.679+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:57:46.816 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:57:46 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:57:46.679+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:57:46.816 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:57:46 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:57:46.679+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:57:46.816 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:57:46 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:57:46.680+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:57:46.816 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:57:46 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:57:46.779+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:57:46.816 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:57:46 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:57:46.779+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:57:46.816 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:57:46 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:57:46.779+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:57:46.817 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:57:46 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:57:46.780+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:57:47.315 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:57:46 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:57:46.997+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T21:57:47.315 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:57:46 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:57:46.997+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T21:57:47.316 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:57:46 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:57:46.997+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T21:57:47.316 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:57:46 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:57:46.998+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:57:47.991 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:57:47 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:57:47.701+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:57:47.991 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:57:47 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:57:47.701+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:57:47.991 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:57:47 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:57:47.701+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:57:47.991 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:57:47 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:57:47.702+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:57:47.992 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:57:47 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:57:47.800+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:57:47.992 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:57:47 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:57:47.800+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:57:47.992 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:57:47 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:57:47.800+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:57:47.992 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:57:47 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:57:47.802+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:57:48.315 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:57:47 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:57:47.989+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T21:57:48.315 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:57:47 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:57:47.989+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T21:57:48.315 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:57:47 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:57:47.989+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T21:57:48.315 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:57:47 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:57:47.990+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:57:49.031 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:57:48 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:57:48.685+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:57:49.031 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:57:48 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:57:48.685+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:57:49.032 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:57:48 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:57:48.685+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:57:49.032 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:57:48 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:57:48.686+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:57:49.032 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:57:48 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:57:48.791+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:57:49.032 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:57:48 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:57:48.791+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:57:49.033 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:57:48 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:57:48.791+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:57:49.033 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:57:48 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:57:48.791+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:57:49.315 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:57:49 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:57:49.029+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T21:57:49.315 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:57:49 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:57:49.029+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T21:57:49.315 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:57:49 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:57:49.029+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T21:57:49.315 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:57:49 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:57:49.030+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:57:50.021 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:57:49 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:57:49.796+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:57:50.022 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:57:49 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:57:49.796+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:57:50.022 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:57:49 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:57:49.796+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:57:50.022 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:57:49 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:57:49.797+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:57:50.022 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:57:49 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:57:49.638+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:57:50.022 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:57:49 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:57:49.638+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:57:50.023 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:57:49 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:57:49.638+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:57:50.023 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:57:49 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:57:49.639+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:57:50.315 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:57:50 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:57:50.019+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T21:57:50.315 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:57:50 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:57:50.019+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T21:57:50.315 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:57:50 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:57:50.019+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T21:57:50.315 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:57:50 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:57:50.020+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:57:50.789 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:57:50 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:57:50.628+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:57:50.789 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:57:50 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:57:50.628+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:57:50.790 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:57:50 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:57:50.628+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:57:50.790 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:57:50 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:57:50.628+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:57:51.065 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:57:50 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:57:50.787+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:57:51.065 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:57:50 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:57:50.787+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:57:51.065 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:57:50 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:57:50.787+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:57:51.066 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:57:50 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:57:50.788+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:57:51.066 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:57:50 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:57:50.980+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T21:57:51.066 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:57:50 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:57:50.980+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T21:57:51.066 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:57:50 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:57:50.980+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T21:57:51.066 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:57:50 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:57:50.981+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:57:51.815 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Dec 16 21:57:51 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-mon-smithi045[80759]: 2023-12-16T21:57:51.543+0000 7f45a1f8f700 -1 mon.smithi045@0(probing) e2 get_health_metrics reporting 17107 slow ops, oldest is monmgrreport(gid 14214, 0 checks, 1 progress events) 2023-12-16T21:57:51.816 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:57:51 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:57:51.606+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:57:51.816 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:57:51 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:57:51.606+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:57:51.816 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:57:51 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:57:51.606+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:57:51.816 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:57:51 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:57:51.607+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:57:52.315 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:57:51 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:57:51.838+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:57:52.315 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:57:51 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:57:51.838+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:57:52.316 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:57:51 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:57:51.838+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:57:52.316 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:57:51 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:57:51.839+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:57:52.316 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:57:51 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:57:51.993+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T21:57:52.316 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:57:51 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:57:51.993+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T21:57:52.316 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:57:51 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:57:51.993+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T21:57:52.316 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:57:51 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:57:51.994+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:57:52.990 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:57:52 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:57:52.649+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:57:52.991 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:57:52 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:57:52.649+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:57:52.991 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:57:52 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:57:52.649+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:57:52.991 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:57:52 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:57:52.650+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:57:52.991 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:57:52 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:57:52.815+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:57:52.991 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:57:52 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:57:52.815+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:57:52.991 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:57:52 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:57:52.815+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:57:52.991 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:57:52 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:57:52.816+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:57:53.315 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:57:52 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:57:52.989+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T21:57:53.315 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:57:52 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:57:52.989+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T21:57:53.315 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:57:52 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:57:52.989+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T21:57:53.316 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:57:52 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:57:52.989+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:57:53.815 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:57:53 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:57:53.614+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:57:53.815 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:57:53 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:57:53.614+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:57:53.815 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:57:53 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:57:53.614+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:57:53.815 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:57:53 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:57:53.615+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:57:54.315 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:57:54 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:57:54.013+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T21:57:54.315 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:57:54 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:57:54.013+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T21:57:54.315 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:57:54 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:57:54.013+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T21:57:54.316 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:57:54 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:57:54.014+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:57:54.316 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:57:53 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:57:53.839+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:57:54.316 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:57:53 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:57:53.839+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:57:54.316 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:57:53 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:57:53.839+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:57:54.316 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:57:53 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:57:53.840+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:57:54.980 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:57:54 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:57:54.621+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:57:54.981 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:57:54 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:57:54.621+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:57:54.981 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:57:54 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:57:54.621+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:57:54.981 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:57:54 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:57:54.622+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:57:54.981 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:57:54 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:57:54.843+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:57:54.981 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:57:54 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:57:54.844+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:57:54.981 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:57:54 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:57:54.844+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:57:54.982 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:57:54 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:57:54.845+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:57:55.315 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:57:54 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:57:54.978+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T21:57:55.315 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:57:54 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:57:54.978+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T21:57:55.316 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:57:54 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:57:54.978+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T21:57:55.316 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:57:54 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:57:54.980+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:57:55.987 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:57:55 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:57:55.648+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:57:55.988 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:57:55 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:57:55.648+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:57:55.988 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:57:55 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:57:55.648+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:57:55.988 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:57:55 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:57:55.648+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:57:55.988 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:57:55 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:57:55.869+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:57:55.988 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:57:55 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:57:55.869+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:57:55.988 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:57:55 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:57:55.869+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:57:55.988 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:57:55 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:57:55.869+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:57:56.315 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:57:55 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:57:55.985+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T21:57:56.315 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:57:55 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:57:55.985+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T21:57:56.315 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:57:55 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:57:55.985+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T21:57:56.316 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:57:55 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:57:55.987+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:57:56.815 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Dec 16 21:57:56 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-mon-smithi045[80759]: 2023-12-16T21:57:56.545+0000 7f45a1f8f700 -1 mon.smithi045@0(probing) e2 get_health_metrics reporting 17195 slow ops, oldest is monmgrreport(gid 14214, 0 checks, 1 progress events) 2023-12-16T21:57:56.816 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:57:56 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:57:56.675+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:57:56.816 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:57:56 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:57:56.675+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:57:56.816 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:57:56 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:57:56.675+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:57:56.816 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:57:56 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:57:56.676+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:57:57.315 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:57:56 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:57:56.911+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:57:57.315 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:57:56 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:57:56.911+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:57:57.316 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:57:56 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:57:56.911+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:57:57.316 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:57:56 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:57:56.912+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:57:57.316 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:57:56 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:57:56.968+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T21:57:57.316 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:57:56 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:57:56.968+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T21:57:57.316 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:57:56 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:57:56.968+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T21:57:57.316 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:57:56 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:57:56.969+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:57:58.065 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:57:57 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:57:57.723+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:57:58.065 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:57:57 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:57:57.723+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:57:58.065 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:57:57 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:57:57.723+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:57:58.066 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:57:57 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:57:57.723+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:57:58.066 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:57:57 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:57:57.960+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:57:58.066 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:57:57 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:57:57.960+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:57:58.066 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:57:57 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:57:57.960+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:57:58.066 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:57:57 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:57:57.961+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:57:58.066 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:57:57 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:57:57.936+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T21:57:58.067 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:57:57 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:57:57.936+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T21:57:58.067 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:57:57 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:57:57.936+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T21:57:58.067 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:57:57 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:57:57.937+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:57:58.984 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:57:58 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:57:58.688+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:57:58.984 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:57:58 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:57:58.688+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:57:58.984 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:57:58 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:57:58.688+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:57:58.985 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:57:58 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:57:58.689+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:57:59.315 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:57:58 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:57:58.982+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T21:57:59.315 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:57:58 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:57:58.982+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T21:57:59.315 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:57:58 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:57:58.982+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T21:57:59.315 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:57:58 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:57:58.983+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:57:59.316 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:57:58 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:57:58.991+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:57:59.316 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:57:58 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:57:58.991+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:57:59.316 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:57:58 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:57:58.991+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:57:59.316 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:57:58 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:57:58.991+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:57:59.996 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:57:59 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:57:59.733+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:57:59.996 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:57:59 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:57:59.733+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:57:59.996 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:57:59 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:57:59.733+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:57:59.996 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:57:59 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:57:59.734+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:58:00.315 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:57:59 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:57:59.994+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:58:00.315 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:57:59 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:57:59.994+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:58:00.315 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:57:59 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:57:59.994+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:58:00.316 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:57:59 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:57:59.995+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:58:00.316 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:58:00 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:58:00.015+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T21:58:00.316 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:58:00 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:58:00.015+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T21:58:00.316 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:58:00 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:58:00.015+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T21:58:00.316 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:58:00 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:58:00.016+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:58:01.023 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:58:00 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:58:00.695+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:58:01.024 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:58:00 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:58:00.695+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:58:01.024 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:58:00 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:58:00.695+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:58:01.024 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:58:00 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:58:00.696+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:58:01.315 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:58:01 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:58:01.050+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T21:58:01.315 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:58:01 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:58:01.050+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T21:58:01.315 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:58:01 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:58:01.050+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T21:58:01.315 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:58:01 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:58:01.051+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:58:01.316 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:58:01 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:58:01.022+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:58:01.316 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:58:01 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:58:01.022+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:58:01.316 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:58:01 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:58:01.022+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:58:01.316 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:58:01 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:58:01.023+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:58:01.815 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Dec 16 21:58:01 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-mon-smithi045[80759]: 2023-12-16T21:58:01.547+0000 7f45a1f8f700 -1 mon.smithi045@0(probing) e2 get_health_metrics reporting 17286 slow ops, oldest is monmgrreport(gid 14214, 0 checks, 1 progress events) 2023-12-16T21:58:01.815 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:58:01 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:58:01.691+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:58:01.815 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:58:01 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:58:01.691+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:58:01.816 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:58:01 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:58:01.691+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:58:01.816 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:58:01 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:58:01.691+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:58:02.315 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:58:02 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:58:02.020+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:58:02.315 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:58:02 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:58:02.020+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:58:02.315 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:58:02 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:58:02.020+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:58:02.315 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:58:02 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:58:02.021+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:58:02.316 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:58:02 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:58:02.073+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T21:58:02.316 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:58:02 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:58:02.073+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T21:58:02.316 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:58:02 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:58:02.073+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T21:58:02.316 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:58:02 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:58:02.073+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:58:03.058 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:58:02 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:58:02.723+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:58:03.058 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:58:02 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:58:02.723+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:58:03.058 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:58:02 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:58:02.723+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:58:03.058 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:58:02 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:58:02.723+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:58:03.315 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:58:03 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:58:03.101+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T21:58:03.315 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:58:03 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:58:03.101+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T21:58:03.315 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:58:03 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:58:03.101+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T21:58:03.315 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:58:03 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:58:03.102+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:58:03.316 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:58:03 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:58:03.056+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:58:03.316 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:58:03 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:58:03.056+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:58:03.316 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:58:03 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:58:03.056+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:58:03.316 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:58:03 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:58:03.057+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:58:04.028 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:58:03 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:58:03.759+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:58:04.028 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:58:03 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:58:03.759+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:58:04.029 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:58:03 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:58:03.759+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:58:04.029 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:58:03 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:58:03.760+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:58:04.315 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:58:04 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:58:04.058+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T21:58:04.315 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:58:04 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:58:04.059+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T21:58:04.315 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:58:04 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:58:04.059+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T21:58:04.315 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:58:04 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:58:04.059+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:58:04.316 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:58:04 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:58:04.026+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:58:04.316 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:58:04 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:58:04.026+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:58:04.316 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:58:04 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:58:04.026+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:58:04.316 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:58:04 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:58:04.028+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:58:05.013 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:58:04 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:58:04.746+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:58:05.013 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:58:04 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:58:04.746+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:58:05.013 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:58:04 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:58:04.746+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:58:05.014 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:58:04 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:58:04.747+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:58:05.315 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:58:05 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:58:05.011+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:58:05.315 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:58:05 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:58:05.011+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:58:05.315 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:58:05 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:58:05.011+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:58:05.315 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:58:05 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:58:05.012+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:58:05.316 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:58:05 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:58:05.097+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T21:58:05.316 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:58:05 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:58:05.097+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T21:58:05.316 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:58:05 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:58:05.097+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T21:58:05.316 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:58:05 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:58:05.098+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:58:06.038 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:58:05 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:58:05.700+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:58:06.038 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:58:05 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:58:05.700+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:58:06.038 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:58:05 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:58:05.700+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:58:06.038 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:58:05 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:58:05.701+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:58:06.315 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:58:06 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:58:06.036+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:58:06.315 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:58:06 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:58:06.036+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:58:06.315 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:58:06 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:58:06.036+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:58:06.316 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:58:06 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:58:06.037+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:58:06.316 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:58:06 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:58:06.118+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T21:58:06.316 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:58:06 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:58:06.118+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T21:58:06.316 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:58:06 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:58:06.118+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T21:58:06.316 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:58:06 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:58:06.120+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:58:06.815 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Dec 16 21:58:06 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-mon-smithi045[80759]: 2023-12-16T21:58:06.548+0000 7f45a1f8f700 -1 mon.smithi045@0(probing) e2 get_health_metrics reporting 17374 slow ops, oldest is monmgrreport(gid 14214, 0 checks, 1 progress events) 2023-12-16T21:58:06.816 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:58:06 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:58:06.747+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:58:06.816 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:58:06 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:58:06.747+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:58:06.816 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:58:06 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:58:06.747+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:58:06.816 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:58:06 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:58:06.748+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:58:07.315 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:58:07 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:58:07.045+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:58:07.315 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:58:07 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:58:07.045+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:58:07.315 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:58:07 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:58:07.045+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:58:07.315 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:58:07 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:58:07.046+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:58:07.316 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:58:07 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:58:07.077+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T21:58:07.316 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:58:07 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:58:07.077+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T21:58:07.316 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:58:07 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:58:07.077+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T21:58:07.316 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:58:07 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:58:07.078+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:58:08.049 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:58:07 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:58:07.759+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:58:08.050 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:58:07 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:58:07.759+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:58:08.050 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:58:07 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:58:07.759+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:58:08.050 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:58:07 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:58:07.760+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:58:08.315 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:58:08 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:58:08.047+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:58:08.315 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:58:08 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:58:08.047+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:58:08.315 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:58:08 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:58:08.047+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:58:08.316 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:58:08 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:58:08.048+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:58:08.316 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:58:08 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:58:08.057+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T21:58:08.316 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:58:08 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:58:08.057+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T21:58:08.316 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:58:08 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:58:08.057+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T21:58:08.316 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:58:08 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:58:08.058+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:58:09.046 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:58:08 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:58:08.761+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:58:09.047 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:58:08 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:58:08.761+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:58:09.047 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:58:08 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:58:08.761+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:58:09.047 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:58:08 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:58:08.762+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:58:09.315 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:58:09 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:58:09.045+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:58:09.315 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:58:09 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:58:09.045+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:58:09.315 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:58:09 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:58:09.045+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:58:09.315 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:58:09 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:58:09.046+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:58:09.316 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:58:09 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:58:09.067+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T21:58:09.316 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:58:09 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:58:09.067+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T21:58:09.316 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:58:09 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:58:09.067+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T21:58:09.316 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:58:09 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:58:09.068+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:58:10.007 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:58:09 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:58:09.712+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:58:10.008 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:58:09 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:58:09.712+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:58:10.008 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:58:09 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:58:09.712+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:58:10.008 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:58:09 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:58:09.714+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:58:10.315 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:58:10 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:58:10.005+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:58:10.315 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:58:10 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:58:10.006+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:58:10.315 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:58:10 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:58:10.006+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:58:10.316 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:58:10 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:58:10.006+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:58:10.316 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:58:10 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:58:10.113+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T21:58:10.316 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:58:10 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:58:10.113+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T21:58:10.316 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:58:10 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:58:10.113+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T21:58:10.316 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:58:10 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:58:10.114+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:58:11.006 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:58:10 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:58:10.705+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:58:11.007 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:58:10 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:58:10.705+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:58:11.007 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:58:10 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:58:10.705+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:58:11.007 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:58:10 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:58:10.706+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:58:11.315 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:58:11 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:58:11.160+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T21:58:11.315 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:58:11 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:58:11.160+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T21:58:11.315 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:58:11 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:58:11.160+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T21:58:11.315 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:58:11 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:58:11.161+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:58:11.316 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:58:11 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:58:11.004+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:58:11.316 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:58:11 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:58:11.004+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:58:11.316 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:58:11 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:58:11.004+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:58:11.316 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:58:11 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:58:11.006+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:58:11.815 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Dec 16 21:58:11 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-mon-smithi045[80759]: 2023-12-16T21:58:11.550+0000 7f45a1f8f700 -1 mon.smithi045@0(probing) e2 get_health_metrics reporting 17466 slow ops, oldest is monmgrreport(gid 14214, 0 checks, 1 progress events) 2023-12-16T21:58:11.815 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:58:11 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:58:11.716+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:58:11.816 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:58:11 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:58:11.716+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:58:11.816 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:58:11 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:58:11.717+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:58:11.816 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:58:11 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:58:11.718+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:58:12.315 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:58:12 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:58:12.186+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T21:58:12.315 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:58:12 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:58:12.186+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T21:58:12.315 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:58:12 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:58:12.186+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T21:58:12.315 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:58:12 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:58:12.187+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:58:12.316 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:58:11 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:58:11.959+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:58:12.316 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:58:11 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:58:11.959+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:58:12.316 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:58:11 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:58:11.959+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:58:12.316 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:58:11 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:58:11.960+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:58:13.065 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:58:12 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:58:12.981+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:58:13.065 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:58:12 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:58:12.981+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:58:13.066 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:58:12 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:58:12.981+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:58:13.066 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:58:12 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:58:12.981+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:58:13.066 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:58:12 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:58:12.745+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:58:13.066 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:58:12 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:58:12.745+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:58:13.066 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:58:12 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:58:12.745+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:58:13.066 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:58:12 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:58:12.746+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:58:13.565 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:58:13 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:58:13.158+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T21:58:13.566 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:58:13 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:58:13.158+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T21:58:13.566 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:58:13 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:58:13.158+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T21:58:13.566 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:58:13 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:58:13.159+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:58:13.958 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:58:13 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:58:13.734+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:58:13.958 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:58:13 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:58:13.734+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:58:13.958 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:58:13 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:58:13.734+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:58:13.958 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:58:13 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:58:13.735+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:58:14.315 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:58:14 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:58:14.144+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T21:58:14.315 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:58:14 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:58:14.144+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T21:58:14.316 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:58:14 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:58:14.144+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T21:58:14.316 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:58:14 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:58:14.146+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:58:14.316 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:58:13 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:58:13.956+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:58:14.316 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:58:13 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:58:13.956+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:58:14.316 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:58:13 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:58:13.956+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:58:14.316 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:58:13 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:58:13.957+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:58:15.056 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:58:14 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:58:14.785+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:58:15.056 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:58:14 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:58:14.785+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:58:15.056 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:58:14 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:58:14.785+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:58:15.056 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:58:14 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:58:14.786+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:58:15.056 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:58:14 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:58:14.983+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:58:15.057 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:58:14 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:58:14.983+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:58:15.057 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:58:14 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:58:14.983+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:58:15.057 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:58:14 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:58:14.985+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:58:15.315 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:58:15 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:58:15.120+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T21:58:15.315 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:58:15 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:58:15.120+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T21:58:15.315 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:58:15 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:58:15.120+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T21:58:15.316 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:58:15 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:58:15.121+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:58:16.076 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:58:15 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:58:15.997+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:58:16.076 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:58:15 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:58:15.997+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:58:16.076 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:58:15 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:58:15.997+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:58:16.076 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:58:15 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:58:15.998+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:58:16.076 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:58:15 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:58:15.813+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:58:16.077 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:58:15 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:58:15.813+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:58:16.077 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:58:15 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:58:15.813+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:58:16.077 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:58:15 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:58:15.815+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:58:16.554 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:58:16 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:58:16.074+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T21:58:16.554 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:58:16 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:58:16.074+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T21:58:16.554 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:58:16 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:58:16.074+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T21:58:16.554 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:58:16 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:58:16.075+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:58:16.815 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Dec 16 21:58:16 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-mon-smithi045[80759]: 2023-12-16T21:58:16.552+0000 7f45a1f8f700 -1 mon.smithi045@0(probing) e2 get_health_metrics reporting 17554 slow ops, oldest is monmgrreport(gid 14214, 0 checks, 1 progress events) 2023-12-16T21:58:16.815 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:58:16 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:58:16.768+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:58:16.815 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:58:16 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:58:16.768+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:58:16.816 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:58:16 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:58:16.768+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:58:16.816 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:58:16 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:58:16.769+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:58:17.315 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:58:17 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:58:17.089+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T21:58:17.315 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:58:17 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:58:17.089+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T21:58:17.315 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:58:17 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:58:17.089+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T21:58:17.315 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:58:17 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:58:17.090+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:58:17.316 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:58:16 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:58:16.952+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:58:17.316 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:58:16 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:58:16.952+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:58:17.316 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:58:16 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:58:16.952+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:58:17.317 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:58:16 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:58:16.953+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:58:18.065 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:58:17 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:58:17.744+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:58:18.065 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:58:17 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:58:17.744+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:58:18.065 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:58:17 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:58:17.744+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:58:18.066 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:58:17 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:58:17.745+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:58:18.066 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:58:17 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:58:17.911+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:58:18.066 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:58:17 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:58:17.911+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:58:18.066 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:58:17 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:58:17.911+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:58:18.066 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:58:17 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:58:17.912+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:58:18.565 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:58:18 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:58:18.106+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T21:58:18.565 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:58:18 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:58:18.106+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T21:58:18.565 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:58:18 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:58:18.106+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T21:58:18.565 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:58:18 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:58:18.107+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:58:19.065 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:58:18 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:58:18.783+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:58:19.065 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:58:18 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:58:18.783+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:58:19.065 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:58:18 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:58:18.783+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:58:19.066 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:58:18 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:58:18.784+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:58:19.066 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:58:18 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:58:18.945+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:58:19.066 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:58:18 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:58:18.946+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:58:19.066 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:58:18 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:58:18.946+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:58:19.066 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:58:18 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:58:18.947+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:58:19.565 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:58:19 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:58:19.151+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T21:58:19.565 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:58:19 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:58:19.151+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T21:58:19.565 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:58:19 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:58:19.151+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T21:58:19.565 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:58:19 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:58:19.152+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:58:20.065 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:58:19 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:58:19.976+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:58:20.065 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:58:19 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:58:19.976+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:58:20.065 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:58:19 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:58:19.976+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:58:20.065 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:58:19 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:58:19.976+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:58:20.066 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:58:19 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:58:19.795+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:58:20.066 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:58:19 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:58:19.795+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:58:20.066 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:58:19 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:58:19.795+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:58:20.066 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:58:19 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:58:19.797+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:58:20.565 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:58:20 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:58:20.187+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T21:58:20.565 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:58:20 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:58:20.187+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T21:58:20.566 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:58:20 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:58:20.187+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T21:58:20.566 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:58:20 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:58:20.188+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:58:21.065 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:58:20 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:58:20.772+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:58:21.065 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:58:20 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:58:20.772+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:58:21.066 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:58:20 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:58:20.772+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:58:21.066 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:58:20 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:58:20.773+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:58:21.066 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:58:20 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:58:20.983+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:58:21.066 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:58:20 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:58:20.983+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:58:21.066 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:58:20 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:58:20.983+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:58:21.066 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:58:20 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:58:20.983+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:58:21.556 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:58:21 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:58:21.223+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T21:58:21.556 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:58:21 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:58:21.223+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T21:58:21.556 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:58:21 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:58:21.223+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T21:58:21.556 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:58:21 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:58:21.224+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:58:21.815 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Dec 16 21:58:21 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-mon-smithi045[80759]: 2023-12-16T21:58:21.554+0000 7f45a1f8f700 -1 mon.smithi045@0(probing) e2 get_health_metrics reporting 17644 slow ops, oldest is monmgrreport(gid 14214, 0 checks, 1 progress events) 2023-12-16T21:58:21.815 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:58:21 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:58:21.735+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:58:21.816 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:58:21 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:58:21.735+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:58:21.816 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:58:21 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:58:21.735+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:58:21.816 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:58:21 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:58:21.735+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:58:22.315 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:58:22 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:58:22.007+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:58:22.315 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:58:22 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:58:22.007+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:58:22.315 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:58:22 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:58:22.007+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:58:22.316 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:58:22 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:58:22.007+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:58:22.316 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:58:22 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:58:22.178+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T21:58:22.316 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:58:22 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:58:22.178+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T21:58:22.316 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:58:22 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:58:22.178+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T21:58:22.316 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:58:22 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:58:22.179+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:58:23.065 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:58:22 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:58:22.966+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:58:23.065 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:58:22 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:58:22.966+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:58:23.065 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:58:22 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:58:22.966+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:58:23.066 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:58:22 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:58:22.967+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:58:23.066 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:58:22 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:58:22.754+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:58:23.066 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:58:22 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:58:22.754+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:58:23.066 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:58:22 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:58:22.754+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:58:23.066 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:58:22 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:58:22.755+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:58:23.565 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:58:23 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:58:23.154+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T21:58:23.565 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:58:23 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:58:23.154+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T21:58:23.565 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:58:23 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:58:23.154+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T21:58:23.565 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:58:23 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:58:23.155+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:58:23.975 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:58:23 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:58:23.802+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:58:23.975 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:58:23 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:58:23.802+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:58:23.976 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:58:23 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:58:23.802+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:58:23.976 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:58:23 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:58:23.803+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:58:24.272 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:58:23 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:58:23.973+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:58:24.272 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:58:23 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:58:23.973+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:58:24.272 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:58:23 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:58:23.973+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:58:24.272 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:58:23 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:58:23.975+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:58:24.272 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:58:24 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:58:24.123+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T21:58:24.272 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:58:24 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:58:24.123+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T21:58:24.273 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:58:24 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:58:24.123+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T21:58:24.273 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:58:24 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:58:24.124+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:58:25.124 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:58:24 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:58:24.840+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:58:25.124 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:58:24 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:58:24.840+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:58:25.124 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:58:24 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:58:24.840+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:58:25.124 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:58:24 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:58:24.841+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:58:25.125 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:58:25 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:58:25.019+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:58:25.125 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:58:25 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:58:25.019+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:58:25.125 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:58:25 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:58:25.019+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:58:25.125 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:58:25 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:58:25.019+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:58:25.565 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:58:25 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:58:25.122+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T21:58:25.565 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:58:25 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:58:25.122+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T21:58:25.565 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:58:25 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:58:25.122+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T21:58:25.565 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:58:25 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:58:25.123+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:58:26.118 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:58:25 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:58:25.842+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:58:26.118 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:58:25 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:58:25.842+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:58:26.118 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:58:25 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:58:25.842+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:58:26.119 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:58:25 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:58:25.843+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:58:26.119 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:58:26 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:58:26.016+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:58:26.119 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:58:26 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:58:26.016+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:58:26.119 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:58:26 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:58:26.016+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:58:26.119 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:58:26 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:58:26.017+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:58:26.557 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:58:26 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:58:26.116+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T21:58:26.558 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:58:26 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:58:26.116+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T21:58:26.558 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:58:26 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:58:26.116+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T21:58:26.558 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:58:26 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:58:26.117+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:58:26.815 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Dec 16 21:58:26 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-mon-smithi045[80759]: 2023-12-16T21:58:26.556+0000 7f45a1f8f700 -1 mon.smithi045@0(probing) e2 get_health_metrics reporting 17733 slow ops, oldest is monmgrreport(gid 14214, 0 checks, 1 progress events) 2023-12-16T21:58:27.150 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:58:26 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:58:26.832+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:58:27.150 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:58:26 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:58:26.832+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:58:27.151 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:58:26 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:58:26.832+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:58:27.151 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:58:26 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:58:26.833+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:58:27.151 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:58:27 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:58:27.018+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:58:27.151 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:58:27 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:58:27.018+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:58:27.151 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:58:27 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:58:27.018+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:58:27.152 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:58:27 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:58:27.018+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:58:27.263 DEBUG:teuthology.orchestra.run:got remote process result: 124 2023-12-16T21:58:27.264 DEBUG:tasks.cephfs.mount:Force/lazy unmounting on client.0 2023-12-16T21:58:27.264 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-12-16T21:58:27.264 DEBUG:teuthology.orchestra.run.smithi045:> sudo umount --lazy --force /home/ubuntu/cephtest/mnt.0 2023-12-16T21:58:27.308 DEBUG:teuthology.orchestra.run:got remote process result: 2 2023-12-16T21:58:27.308 INFO:tasks.workunit.client.0.smithi045.stderr:tar: ltp-full-20091231/include/mk: Cannot mkdir: Input/output error 2023-12-16T21:58:27.308 INFO:tasks.workunit.client.0.smithi045.stderr:tar: ltp-full-20091231/include/mk/generic_trunk_target.inc: Cannot open: Input/output error 2023-12-16T21:58:27.309 INFO:tasks.workunit.client.0.smithi045.stderr:tar: ltp-full-20091231/include/mk/config.mk.in: Cannot open: Input/output error 2023-12-16T21:58:27.309 INFO:tasks.workunit.client.0.smithi045.stderr:tar: ltp-full-20091231/include/mk/config.mk.default: Cannot open: Input/output error 2023-12-16T21:58:27.309 INFO:tasks.workunit.client.0.smithi045.stderr:tar: ltp-full-20091231/include/mk/env_post.mk: Cannot open: Input/output error 2023-12-16T21:58:27.309 INFO:tasks.workunit.client.0.smithi045.stderr:tar: ltp-full-20091231/include/mk/lib.mk: Cannot open: Input/output error 2023-12-16T21:58:27.309 INFO:tasks.workunit.client.0.smithi045.stderr:tar: ltp-full-20091231/include/mk/generic_leaf_target.inc: Cannot open: Input/output error 2023-12-16T21:58:27.309 INFO:tasks.workunit.client.0.smithi045.stderr:tar: ltp-full-20091231/include/mk/automake.mk: Cannot open: Input/output error 2023-12-16T21:58:27.309 INFO:tasks.workunit.client.0.smithi045.stderr:tar: ltp-full-20091231/include/mk/testcases.mk: Cannot open: Input/output error 2023-12-16T21:58:27.309 INFO:tasks.workunit.client.0.smithi045.stderr:tar: ltp-full-20091231/include/mk/generic_leaf_target.mk: Cannot open: Input/output error 2023-12-16T21:58:27.309 INFO:tasks.workunit.client.0.smithi045.stderr:tar: ltp-full-20091231/include/mk/man.mk: Cannot open: Input/output error 2023-12-16T21:58:27.310 INFO:tasks.workunit.client.0.smithi045.stderr:tar: ltp-full-20091231/include/mk/generic_trunk_target.mk: Cannot open: Input/output error 2023-12-16T21:58:27.310 INFO:tasks.workunit.client.0.smithi045.stderr:tar: ltp-full-20091231/include/mk/functions.mk: Cannot open: Input/output error 2023-12-16T21:58:27.310 INFO:tasks.workunit.client.0.smithi045.stderr:tar: ltp-full-20091231/include/mk/env_pre.mk: Cannot open: Input/output error 2023-12-16T21:58:27.310 INFO:tasks.workunit.client.0.smithi045.stderr:tar: ltp-full-20091231/include/random_range.h: Cannot open: Input/output error 2023-12-16T21:58:27.310 INFO:tasks.workunit.client.0.smithi045.stderr:tar: ltp-full-20091231/include/open_flags.h: Cannot open: Input/output error 2023-12-16T21:58:27.310 INFO:tasks.workunit.client.0.smithi045.stderr:tar: ltp-full-20091231/include: Cannot utime: Input/output error 2023-12-16T21:58:27.310 INFO:tasks.workunit.client.0.smithi045.stderr:tar: ltp-full-20091231: Cannot utime: Input/output error 2023-12-16T21:58:27.310 INFO:tasks.workunit.client.0.smithi045.stderr:tar: Exiting with failure status due to previous errors 2023-12-16T21:58:27.311 INFO:tasks.workunit:Stopping ['suites/fsstress.sh'] on client.0... 2023-12-16T21:58:27.311 DEBUG:teuthology.orchestra.run.smithi045:> sudo rm -rf -- /home/ubuntu/cephtest/workunits.list.client.0 /home/ubuntu/cephtest/clone.client.0 2023-12-16T21:58:27.325 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi045.front.sepia.ceph.com 2023-12-16T21:58:27.325 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-12-16T21:58:27.325 DEBUG:teuthology.orchestra.run.smithi045:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.0) 2023-12-16T21:58:27.391 DEBUG:teuthology.orchestra.run.smithi169:> set -ex 2023-12-16T21:58:27.391 DEBUG:teuthology.orchestra.run.smithi169:> dd if=/proc/self/mounts of=/dev/stdout 2023-12-16T21:58:27.566 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:58:27 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:58:27.148+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T21:58:27.566 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:58:27 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:58:27.148+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T21:58:27.566 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:58:27 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:58:27.148+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T21:58:27.566 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:58:27 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:58:27.150+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:58:28.055 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:58:27 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:58:27.866+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:58:28.055 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:58:27 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:58:27.866+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:58:28.055 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:58:27 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:58:27.866+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:58:28.056 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:58:27 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:58:27.867+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:58:28.315 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:58:28 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:58:28.052+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:58:28.315 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:58:28 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:58:28.052+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:58:28.315 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:58:28 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:58:28.052+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:58:28.316 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:58:28 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:58:28.053+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:58:28.316 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:58:28 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:58:28.134+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T21:58:28.316 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:58:28 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:58:28.134+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T21:58:28.316 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:58:28 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:58:28.134+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T21:58:28.316 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:58:28 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:58:28.135+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:58:29.065 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:58:28 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:58:28.893+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:58:29.065 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:58:28 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:58:28.894+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:58:29.065 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:58:28 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:58:28.894+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:58:29.066 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:58:28 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:58:28.894+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:58:29.565 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:58:29 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:58:29.095+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T21:58:29.565 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:58:29 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:58:29.095+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T21:58:29.565 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:58:29 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:58:29.095+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T21:58:29.565 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:58:29 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:58:29.095+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:58:29.566 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:58:29 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:58:29.064+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:58:29.566 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:58:29 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:58:29.064+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:58:29.566 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:58:29 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:58:29.064+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:58:29.566 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:58:29 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:58:29.064+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:58:30.164 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:58:30 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:58:30.093+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T21:58:30.165 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:58:30 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:58:30.093+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T21:58:30.165 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:58:30 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:58:30.093+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T21:58:30.165 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:58:30 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:58:30.093+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:58:30.165 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:58:29 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:58:29.873+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:58:30.165 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:58:29 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:58:29.873+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:58:30.166 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:58:29 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:58:29.873+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:58:30.166 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:58:29 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:58:29.873+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:58:30.166 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:58:30 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:58:30.019+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:58:30.166 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:58:30 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:58:30.019+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:58:30.166 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:58:30 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:58:30.019+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:58:30.166 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:58:30 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:58:30.020+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:58:31.315 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:58:31 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:58:31.060+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:58:31.315 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:58:31 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:58:31.060+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:58:31.315 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:58:31 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:58:31.060+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:58:31.316 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:58:31 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:58:31.061+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:58:31.316 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:58:30 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:58:30.921+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:58:31.316 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:58:30 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:58:30.921+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:58:31.316 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:58:30 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:58:30.921+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:58:31.316 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:58:30 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:58:30.921+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:58:31.317 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:58:31 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:58:31.082+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T21:58:31.317 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:58:31 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:58:31.082+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T21:58:31.317 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:58:31 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:58:31.082+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T21:58:31.317 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:58:31 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:58:31.083+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:58:31.815 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Dec 16 21:58:31 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-mon-smithi045[80759]: 2023-12-16T21:58:31.557+0000 7f45a1f8f700 -1 mon.smithi045@0(probing) e2 get_health_metrics reporting 17823 slow ops, oldest is monmgrreport(gid 14214, 0 checks, 1 progress events) 2023-12-16T21:58:32.315 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:58:31 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:58:31.922+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:58:32.315 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:58:31 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:58:31.922+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:58:32.316 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:58:31 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:58:31.922+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:58:32.316 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:58:31 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:58:31.922+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:58:32.316 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:58:32 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:58:32.024+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:58:32.316 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:58:32 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:58:32.024+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:58:32.316 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:58:32 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:58:32.024+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:58:32.316 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:58:32 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:58:32.025+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:58:32.317 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:58:32 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:58:32.051+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T21:58:32.317 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:58:32 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:58:32.051+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T21:58:32.317 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:58:32 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:58:32.051+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T21:58:32.317 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:58:32 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:58:32.051+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:58:33.315 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:58:32 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:58:32.891+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:58:33.315 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:58:32 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:58:32.891+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:58:33.315 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:58:32 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:58:32.891+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:58:33.316 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:58:32 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:58:32.891+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:58:33.316 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:58:33 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:58:33.022+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T21:58:33.316 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:58:33 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:58:33.022+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T21:58:33.316 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:58:33 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:58:33.022+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T21:58:33.317 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:58:33 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:58:33.023+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:58:33.317 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:58:33 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:58:33.022+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:58:33.317 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:58:33 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:58:33.022+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:58:33.317 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:58:33 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:58:33.022+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:58:33.317 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:58:33 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:58:33.023+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:58:34.315 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:58:33 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:58:33.890+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:58:34.315 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:58:33 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:58:33.890+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:58:34.315 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:58:33 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:58:33.890+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:58:34.316 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:58:33 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:58:33.890+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:58:34.316 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:58:34 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:58:34.000+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:58:34.316 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:58:34 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:58:34.000+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:58:34.316 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:58:34 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:58:34.000+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:58:34.316 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:58:34 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:58:34.001+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:58:34.317 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:58:34 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:58:34.062+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T21:58:34.317 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:58:34 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:58:34.062+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T21:58:34.317 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:58:34 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:58:34.062+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T21:58:34.317 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:58:34 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:58:34.063+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:58:35.315 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:58:34 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:58:34.867+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:58:35.315 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:58:34 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:58:34.867+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:58:35.315 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:58:34 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:58:34.867+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:58:35.315 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:58:34 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:58:34.868+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:58:35.316 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:58:34 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:58:34.995+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:58:35.316 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:58:34 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:58:34.995+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:58:35.316 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:58:34 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:58:34.995+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:58:35.316 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:58:34 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:58:34.996+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:58:35.316 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:58:35 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:58:35.068+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T21:58:35.317 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:58:35 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:58:35.068+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T21:58:35.317 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:58:35 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:58:35.068+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T21:58:35.317 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:58:35 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:58:35.069+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:58:36.315 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:58:35 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:58:35.887+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:58:36.315 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:58:35 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:58:35.887+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:58:36.315 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:58:35 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:58:35.887+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:58:36.316 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:58:35 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:58:35.888+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:58:36.316 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:58:36 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:58:36.058+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T21:58:36.316 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:58:36 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:58:36.058+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T21:58:36.316 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:58:36 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:58:36.058+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T21:58:36.316 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:58:36 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:58:36.058+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:58:36.317 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:58:35 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:58:35.971+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:58:36.317 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:58:35 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:58:35.971+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:58:36.317 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:58:35 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:58:35.971+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:58:36.317 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:58:35 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:58:35.972+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:58:36.815 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Dec 16 21:58:36 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-mon-smithi045[80759]: 2023-12-16T21:58:36.560+0000 7f45a1f8f700 -1 mon.smithi045@0(probing) e2 get_health_metrics reporting 17912 slow ops, oldest is monmgrreport(gid 14214, 0 checks, 1 progress events) 2023-12-16T21:58:37.315 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:58:37 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:58:37.025+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T21:58:37.315 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:58:37 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:58:37.025+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T21:58:37.315 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:58:37 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:58:37.025+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T21:58:37.315 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:58:37 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:58:37.026+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:58:37.316 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:58:36 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:58:36.980+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:58:37.316 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:58:36 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:58:36.980+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:58:37.316 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:58:36 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:58:36.980+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:58:37.316 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:58:36 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:58:36.981+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:58:37.316 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:58:36 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:58:36.882+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:58:37.317 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:58:36 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:58:36.882+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:58:37.317 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:58:36 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:58:36.882+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:58:37.317 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:58:36 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:58:36.883+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:58:38.315 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:58:37 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:58:37.863+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:58:38.315 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:58:37 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:58:37.863+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:58:38.315 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:58:37 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:58:37.863+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:58:38.315 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:58:37 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:58:37.864+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:58:38.316 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:58:37 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:58:37.988+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:58:38.316 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:58:37 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:58:37.988+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:58:38.316 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:58:37 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:58:37.988+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:58:38.316 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:58:37 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:58:37.989+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:58:38.316 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:58:38 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:58:38.009+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T21:58:38.316 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:58:38 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:58:38.009+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T21:58:38.317 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:58:38 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:58:38.009+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T21:58:38.317 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:58:38 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:58:38.010+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:58:39.022 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:58:38 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:58:38.910+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:58:39.023 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:58:38 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:58:38.910+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:58:39.023 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:58:38 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:58:38.910+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:58:39.023 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:58:38 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:58:38.911+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:58:39.315 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:58:39 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:58:39.032+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:58:39.315 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:58:39 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:58:39.032+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:58:39.315 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:58:39 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:58:39.032+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:58:39.315 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:58:39 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:58:39.033+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:58:39.315 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:58:39 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:58:39.021+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T21:58:39.316 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:58:39 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:58:39.021+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T21:58:39.316 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:58:39 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:58:39.021+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T21:58:39.316 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:58:39 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:58:39.022+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:58:40.165 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:58:39 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:58:39.907+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:58:40.165 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:58:39 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:58:39.907+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:58:40.165 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:58:39 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:58:39.907+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:58:40.165 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:58:39 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:58:39.908+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:58:40.166 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:58:40 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:58:40.076+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:58:40.166 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:58:40 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:58:40.076+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:58:40.166 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:58:40 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:58:40.076+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:58:40.166 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:58:40 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:58:40.077+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:58:40.166 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:58:40 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:58:40.007+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T21:58:40.166 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:58:40 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:58:40.007+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T21:58:40.166 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:58:40 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:58:40.007+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T21:58:40.167 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:58:40 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:58:40.009+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:58:41.315 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:58:40 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:58:40.932+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:58:41.315 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:58:40 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:58:40.932+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:58:41.315 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:58:40 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:58:40.932+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:58:41.315 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:58:40 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:58:40.933+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:58:41.316 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:58:41 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:58:41.025+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T21:58:41.316 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:58:41 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:58:41.025+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T21:58:41.316 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:58:41 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:58:41.025+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T21:58:41.316 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:58:41 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:58:41.026+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:58:41.316 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:58:41 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:58:41.074+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:58:41.316 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:58:41 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:58:41.074+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:58:41.317 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:58:41 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:58:41.074+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:58:41.317 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:58:41 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:58:41.075+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:58:41.815 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Dec 16 21:58:41 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-mon-smithi045[80759]: 2023-12-16T21:58:41.562+0000 7f45a1f8f700 -1 mon.smithi045@0(probing) e2 get_health_metrics reporting 18002 slow ops, oldest is monmgrreport(gid 14214, 0 checks, 1 progress events) 2023-12-16T21:58:42.315 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:58:41 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:58:41.886+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:58:42.315 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:58:41 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:58:41.886+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:58:42.315 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:58:41 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:58:41.886+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:58:42.315 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:58:41 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:58:41.888+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:58:42.316 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:58:42 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:58:42.048+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:58:42.316 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:58:42 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:58:42.048+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:58:42.316 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:58:42 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:58:42.048+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:58:42.316 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:58:42 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:58:42.049+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:58:42.317 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:58:42 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:58:42.061+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T21:58:42.317 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:58:42 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:58:42.061+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T21:58:42.317 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:58:42 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:58:42.061+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T21:58:42.317 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:58:42 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:58:42.061+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:58:43.065 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:58:42 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:58:42.897+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:58:43.065 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:58:42 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:58:42.897+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:58:43.065 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:58:42 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:58:42.897+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:58:43.065 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:58:42 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:58:42.897+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:58:43.565 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:58:43 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:58:43.067+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:58:43.565 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:58:43 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:58:43.067+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:58:43.565 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:58:43 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:58:43.067+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:58:43.565 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:58:43 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:58:43.068+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:58:43.566 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:58:43 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:58:43.096+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T21:58:43.566 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:58:43 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:58:43.096+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T21:58:43.566 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:58:43 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:58:43.096+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T21:58:43.566 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:58:43 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:58:43.097+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:58:44.315 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:58:44 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:58:44.059+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:58:44.315 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:58:44 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:58:44.059+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:58:44.315 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:58:44 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:58:44.059+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:58:44.315 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:58:44 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:58:44.060+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:58:44.316 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:58:43 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:58:43.863+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:58:44.316 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:58:43 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:58:43.863+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:58:44.316 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:58:43 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:58:43.863+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:58:44.316 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:58:43 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:58:43.864+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:58:44.316 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:58:44 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:58:44.066+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T21:58:44.317 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:58:44 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:58:44.066+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T21:58:44.317 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:58:44 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:58:44.066+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T21:58:44.317 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:58:44 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:58:44.067+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:58:45.088 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:58:45 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:58:45.036+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T21:58:45.089 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:58:45 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:58:45.036+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T21:58:45.089 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:58:45 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:58:45.036+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T21:58:45.089 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:58:45 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:58:45.037+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:58:45.089 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:58:44 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:58:44.816+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:58:45.089 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:58:44 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:58:44.816+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:58:45.089 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:58:44 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:58:44.816+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:58:45.090 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:58:44 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:58:44.816+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:58:45.565 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:58:45 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:58:45.084+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:58:45.565 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:58:45 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:58:45.084+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:58:45.565 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:58:45 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:58:45.084+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:58:45.565 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:58:45 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:58:45.085+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:58:46.045 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:58:45 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:58:45.787+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:58:46.045 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:58:45 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:58:45.787+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:58:46.046 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:58:45 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:58:45.787+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:58:46.046 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:58:45 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:58:45.788+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:58:46.315 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:58:46 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:58:46.107+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:58:46.315 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:58:46 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:58:46.107+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:58:46.315 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:58:46 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:58:46.107+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:58:46.315 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:58:46 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:58:46.107+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:58:46.316 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:58:46 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:58:46.043+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T21:58:46.316 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:58:46 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:58:46.043+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T21:58:46.316 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:58:46 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:58:46.043+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T21:58:46.316 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:58:46 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:58:46.044+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:58:47.037 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Dec 16 21:58:46 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-mon-smithi045[80759]: 2023-12-16T21:58:46.564+0000 7f45a1f8f700 -1 mon.smithi045@0(probing) e2 get_health_metrics reporting 18091 slow ops, oldest is monmgrreport(gid 14214, 0 checks, 1 progress events) 2023-12-16T21:58:47.038 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:58:46 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:58:46.779+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:58:47.038 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:58:46 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:58:46.779+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:58:47.038 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:58:46 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:58:46.779+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:58:47.038 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:58:46 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:58:46.780+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:58:47.315 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:58:47 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:58:47.155+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:58:47.315 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:58:47 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:58:47.155+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:58:47.315 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:58:47 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:58:47.155+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:58:47.315 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:58:47 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:58:47.156+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:58:47.316 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:58:47 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:58:47.036+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T21:58:47.316 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:58:47 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:58:47.036+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T21:58:47.316 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:58:47 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:58:47.036+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T21:58:47.316 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:58:47 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:58:47.036+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:58:48.065 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:58:47 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:58:47.777+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:58:48.065 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:58:47 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:58:47.777+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:58:48.065 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:58:47 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:58:47.777+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:58:48.066 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:58:47 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:58:47.778+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:58:48.066 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:58:47 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:58:47.988+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T21:58:48.066 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:58:47 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:58:47.988+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T21:58:48.066 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:58:47 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:58:47.988+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T21:58:48.066 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:58:47 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:58:47.989+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:58:48.565 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:58:48 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:58:48.141+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:58:48.565 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:58:48 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:58:48.141+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:58:48.565 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:58:48 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:58:48.141+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:58:48.565 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:58:48 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:58:48.141+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:58:49.065 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:58:48 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:58:48.818+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:58:49.065 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:58:48 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:58:48.818+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:58:49.066 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:58:48 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:58:48.818+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:58:49.066 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:58:48 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:58:48.819+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:58:49.066 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:58:48 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:58:48.941+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T21:58:49.066 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:58:48 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:58:48.941+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T21:58:49.067 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:58:48 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:58:48.941+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T21:58:49.067 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:58:48 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:58:48.943+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:58:49.565 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:58:49 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:58:49.148+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:58:49.565 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:58:49 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:58:49.148+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:58:49.565 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:58:49 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:58:49.148+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:58:49.565 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:58:49 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:58:49.149+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:58:50.148 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:58:49 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:58:49.959+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T21:58:50.148 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:58:49 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:58:49.959+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T21:58:50.148 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:58:49 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:58:49.959+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T21:58:50.148 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:58:49 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:58:49.960+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:58:50.148 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:58:49 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:58:49.848+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:58:50.149 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:58:49 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:58:49.848+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:58:50.149 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:58:49 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:58:49.848+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:58:50.149 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:58:49 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:58:49.849+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:58:50.565 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:58:50 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:58:50.146+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:58:50.565 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:58:50 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:58:50.146+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:58:50.565 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:58:50 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:58:50.146+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:58:50.565 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:58:50 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:58:50.147+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:58:51.315 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:58:51 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:58:51.103+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:58:51.315 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:58:51 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:58:51.103+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:58:51.315 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:58:51 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:58:51.103+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:58:51.316 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:58:51 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:58:51.104+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:58:51.316 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:58:50 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:58:50.970+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T21:58:51.316 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:58:50 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:58:50.970+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T21:58:51.316 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:58:50 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:58:50.970+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T21:58:51.316 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:58:50 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:58:50.972+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:58:51.317 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:58:50 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:58:50.879+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:58:51.317 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:58:50 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:58:50.879+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:58:51.317 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:58:50 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:58:50.879+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:58:51.317 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:58:50 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:58:50.879+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:58:51.870 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Dec 16 21:58:51 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-mon-smithi045[80759]: 2023-12-16T21:58:51.566+0000 7f45a1f8f700 -1 mon.smithi045@0(probing) e2 get_health_metrics reporting 18182 slow ops, oldest is monmgrreport(gid 14214, 0 checks, 1 progress events) 2023-12-16T21:58:52.315 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:58:52 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:58:52.057+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:58:52.315 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:58:52 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:58:52.057+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:58:52.315 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:58:52 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:58:52.057+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:58:52.315 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:58:52 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:58:52.058+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:58:52.316 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:58:51 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:58:51.869+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:58:52.316 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:58:51 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:58:51.869+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:58:52.316 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:58:51 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:58:51.869+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:58:52.316 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:58:51 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:58:51.870+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:58:52.316 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:58:51 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:58:51.967+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T21:58:52.316 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:58:51 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:58:51.967+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T21:58:52.317 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:58:51 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:58:51.967+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T21:58:52.317 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:58:51 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:58:51.968+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:58:53.315 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:58:53 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:58:53.030+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:58:53.315 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:58:53 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:58:53.030+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:58:53.315 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:58:53 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:58:53.030+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:58:53.315 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:58:53 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:58:53.031+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:58:53.316 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:58:52 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:58:52.911+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:58:53.316 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:58:52 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:58:52.911+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:58:53.316 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:58:52 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:58:52.911+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:58:53.316 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:58:52 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:58:52.912+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:58:53.316 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:58:52 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:58:52.972+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T21:58:53.317 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:58:52 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:58:52.972+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T21:58:53.317 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:58:52 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:58:52.972+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T21:58:53.317 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:58:52 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:58:52.972+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:58:54.315 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:58:53 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:58:53.892+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:58:54.315 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:58:53 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:58:53.892+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:58:54.315 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:58:53 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:58:53.892+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:58:54.315 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:58:53 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:58:53.893+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:58:54.316 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:58:54 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:58:54.019+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:58:54.316 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:58:54 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:58:54.019+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:58:54.316 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:58:54 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:58:54.019+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:58:54.316 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:58:54 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:58:54.020+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:58:54.317 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:58:53 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:58:53.983+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T21:58:54.317 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:58:53 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:58:53.983+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T21:58:54.317 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:58:53 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:58:53.983+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T21:58:54.317 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:58:53 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:58:53.984+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:58:55.315 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:58:54 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:58:54.861+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:58:55.315 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:58:54 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:58:54.861+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:58:55.315 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:58:54 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:58:54.861+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:58:55.315 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:58:54 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:58:54.862+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:58:55.316 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:58:54 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:58:54.998+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:58:55.316 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:58:54 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:58:54.998+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:58:55.316 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:58:54 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:58:54.998+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:58:55.316 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:58:55 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:58:54.999+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:58:55.316 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:58:54 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:58:54.947+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T21:58:55.317 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:58:54 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:58:54.947+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T21:58:55.317 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:58:54 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:58:54.947+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T21:58:55.317 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:58:54 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:58:54.947+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:58:56.315 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:58:55 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:58:55.824+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:58:56.315 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:58:55 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:58:55.824+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:58:56.315 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:58:55 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:58:55.824+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:58:56.315 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:58:55 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:58:55.825+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:58:56.316 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:58:56 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:58:56.019+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:58:56.316 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:58:56 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:58:56.019+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:58:56.316 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:58:56 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:58:56.019+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:58:56.316 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:58:56 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:58:56.020+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:58:56.316 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:58:55 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:58:55.935+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T21:58:56.317 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:58:55 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:58:55.935+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T21:58:56.317 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:58:55 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:58:55.935+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T21:58:56.317 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:58:55 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:58:55.937+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:58:56.870 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Dec 16 21:58:56 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-mon-smithi045[80759]: 2023-12-16T21:58:56.568+0000 7f45a1f8f700 -1 mon.smithi045@0(probing) e2 get_health_metrics reporting 18271 slow ops, oldest is monmgrreport(gid 14214, 0 checks, 1 progress events) 2023-12-16T21:58:57.315 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:58:56 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:58:56.927+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T21:58:57.315 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:58:56 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:58:56.927+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T21:58:57.315 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:58:56 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:58:56.927+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T21:58:57.316 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:58:56 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:58:56.928+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:58:57.316 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:58:56 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:58:56.868+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:58:57.316 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:58:56 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:58:56.868+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:58:57.316 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:58:56 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:58:56.868+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:58:57.316 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:58:56 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:58:56.869+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:58:57.317 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:58:57 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:58:57.041+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:58:57.317 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:58:57 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:58:57.041+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:58:57.317 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:58:57 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:58:57.041+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:58:57.317 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:58:57 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:58:57.041+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:58:58.315 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:58:57 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:58:57.917+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T21:58:58.315 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:58:57 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:58:57.917+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T21:58:58.316 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:58:57 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:58:57.917+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T21:58:58.316 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:58:57 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:58:57.918+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:58:58.316 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:58:57 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:58:57.848+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:58:58.316 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:58:57 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:58:57.848+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:58:58.317 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:58:57 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:58:57.848+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:58:58.317 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:58:57 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:58:57.849+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:58:58.317 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:58:58 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:58:58.058+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:58:58.317 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:58:58 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:58:58.058+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:58:58.318 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:58:58 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:58:58.058+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:58:58.318 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:58:58 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:58:58.060+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:58:59.039 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:58:58 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:58:58.964+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T21:58:59.039 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:58:58 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:58:58.964+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T21:58:59.040 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:58:58 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:58:58.964+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T21:58:59.040 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:58:58 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:58:58.965+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:58:59.040 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:58:58 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:58:58.812+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:58:59.040 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:58:58 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:58:58.812+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:58:59.040 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:58:58 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:58:58.812+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:58:59.040 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:58:58 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:58:58.813+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:58:59.315 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:58:59 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:58:59.037+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:58:59.315 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:58:59 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:58:59.037+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:58:59.315 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:58:59 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:58:59.037+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:58:59.315 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:58:59 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:58:59.038+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:59:00.056 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:58:59 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:58:59.930+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T21:59:00.056 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:58:59 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:58:59.930+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T21:59:00.057 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:58:59 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:58:59.930+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T21:59:00.057 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:58:59 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:58:59.931+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:59:00.057 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:58:59 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:58:59.793+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:59:00.057 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:58:59 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:58:59.793+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:59:00.057 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:58:59 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:58:59.793+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:59:00.057 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:58:59 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:58:59.793+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:59:00.315 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:59:00 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:59:00.055+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:59:00.315 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:59:00 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:59:00.055+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:59:00.315 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:59:00 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:59:00.055+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:59:00.315 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:59:00 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:59:00.055+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:59:01.065 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:59:00 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:59:00.942+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T21:59:01.065 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:59:00 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:59:00.942+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T21:59:01.065 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:59:00 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:59:00.942+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T21:59:01.066 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:59:00 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:59:00.943+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:59:01.066 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:59:00 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:59:00.774+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:59:01.066 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:59:00 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:59:00.774+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:59:01.066 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:59:00 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:59:00.774+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:59:01.066 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:59:00 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:59:00.775+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:59:01.565 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:59:01 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:59:01.100+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:59:01.565 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:59:01 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:59:01.100+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:59:01.565 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:59:01 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:59:01.100+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:59:01.565 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:59:01 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:59:01.101+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:59:01.974 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Dec 16 21:59:01 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-mon-smithi045[80759]: 2023-12-16T21:59:01.570+0000 7f45a1f8f700 -1 mon.smithi045@0(probing) e2 get_health_metrics reporting 18361 slow ops, oldest is monmgrreport(gid 14214, 0 checks, 1 progress events) 2023-12-16T21:59:01.974 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:59:01 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:59:01.799+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:59:01.974 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:59:01 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:59:01.799+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:59:01.975 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:59:01 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:59:01.799+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:59:01.975 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:59:01 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:59:01.800+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:59:02.315 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:59:01 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:59:01.972+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T21:59:02.315 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:59:01 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:59:01.972+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T21:59:02.315 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:59:01 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:59:01.972+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T21:59:02.315 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:59:01 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:59:01.973+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:59:02.316 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:59:02 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:59:02.128+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:59:02.316 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:59:02 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:59:02.128+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:59:02.316 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:59:02 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:59:02.128+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:59:02.316 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:59:02 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:59:02.129+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:59:03.065 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:59:02 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:59:02.784+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:59:03.065 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:59:02 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:59:02.784+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:59:03.066 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:59:02 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:59:02.784+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:59:03.066 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:59:02 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:59:02.785+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:59:03.066 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:59:02 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:59:02.986+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T21:59:03.066 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:59:02 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:59:02.986+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T21:59:03.066 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:59:02 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:59:02.986+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T21:59:03.066 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:59:02 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:59:02.987+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:59:03.565 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:59:03 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:59:03.108+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:59:03.565 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:59:03 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:59:03.108+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:59:03.566 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:59:03 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:59:03.108+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:59:03.566 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:59:03 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:59:03.109+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:59:03.978 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:59:03 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:59:03.774+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:59:03.978 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:59:03 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:59:03.774+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:59:03.978 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:59:03 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:59:03.774+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:59:03.978 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:59:03 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:59:03.774+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:59:04.315 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:59:04 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:59:04.104+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:59:04.315 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:59:04 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:59:04.104+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:59:04.315 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:59:04 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:59:04.104+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:59:04.316 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:59:04 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:59:04.105+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:59:04.316 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:59:03 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:59:03.976+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T21:59:04.316 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:59:03 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:59:03.976+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T21:59:04.316 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:59:03 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:59:03.976+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T21:59:04.316 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:59:03 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:59:03.977+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:59:05.065 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:59:04 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:59:04.734+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:59:05.065 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:59:04 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:59:04.734+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:59:05.065 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:59:04 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:59:04.734+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:59:05.065 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:59:04 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:59:04.734+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:59:05.066 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:59:04 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:59:04.944+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T21:59:05.066 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:59:04 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:59:04.944+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T21:59:05.066 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:59:04 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:59:04.944+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T21:59:05.066 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:59:04 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:59:04.945+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:59:05.565 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:59:05 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:59:05.128+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:59:05.565 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:59:05 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:59:05.128+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:59:05.565 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:59:05 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:59:05.128+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:59:05.565 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:59:05 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:59:05.129+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:59:06.065 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:59:05 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:59:05.958+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T21:59:06.065 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:59:05 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:59:05.958+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T21:59:06.065 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:59:05 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:59:05.958+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T21:59:06.065 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:59:05 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:59:05.959+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:59:06.066 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:59:05 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:59:05.764+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:59:06.066 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:59:05 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:59:05.764+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:59:06.066 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:59:05 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:59:05.764+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:59:06.066 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:59:05 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:59:05.765+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:59:06.565 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:59:06 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:59:06.128+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:59:06.565 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:59:06 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:59:06.128+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:59:06.565 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:59:06 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:59:06.128+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:59:06.565 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:59:06 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:59:06.129+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:59:06.941 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Dec 16 21:59:06 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-mon-smithi045[80759]: 2023-12-16T21:59:06.572+0000 7f45a1f8f700 -1 mon.smithi045@0(probing) e2 get_health_metrics reporting 18450 slow ops, oldest is monmgrreport(gid 14214, 0 checks, 1 progress events) 2023-12-16T21:59:06.941 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:59:06 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:59:06.736+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:59:06.941 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:59:06 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:59:06.736+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:59:06.942 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:59:06 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:59:06.736+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:59:06.942 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:59:06 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:59:06.737+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:59:07.315 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:59:07 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:59:07.148+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:59:07.315 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:59:07 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:59:07.148+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:59:07.315 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:59:07 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:59:07.148+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:59:07.315 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:59:07 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:59:07.148+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:59:07.316 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:59:06 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:59:06.939+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T21:59:07.316 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:59:06 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:59:06.939+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T21:59:07.316 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:59:06 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:59:06.939+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T21:59:07.316 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:59:06 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:59:06.941+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:59:08.065 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:59:07 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:59:07.786+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:59:08.065 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:59:07 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:59:07.786+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:59:08.065 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:59:07 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:59:07.786+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:59:08.065 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:59:07 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:59:07.787+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:59:08.066 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:59:07 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:59:07.973+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T21:59:08.066 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:59:07 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:59:07.973+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T21:59:08.066 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:59:07 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:59:07.973+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T21:59:08.066 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:59:07 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:59:07.974+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:59:08.565 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:59:08 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:59:08.154+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:59:08.565 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:59:08 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:59:08.154+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:59:08.565 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:59:08 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:59:08.154+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:59:08.565 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:59:08 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:59:08.154+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:59:09.008 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:59:08 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:59:08.770+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:59:09.008 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:59:08 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:59:08.770+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:59:09.008 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:59:08 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:59:08.770+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:59:09.008 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:59:08 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:59:08.771+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:59:09.315 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:59:09 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:59:09.176+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:59:09.315 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:59:09 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:59:09.176+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:59:09.316 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:59:09 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:59:09.176+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:59:09.316 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:59:09 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:59:09.177+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:59:09.316 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:59:09 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:59:09.006+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T21:59:09.316 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:59:09 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:59:09.006+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T21:59:09.316 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:59:09 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:59:09.006+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T21:59:09.316 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:59:09 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:59:09.007+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:59:10.029 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:59:09 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:59:09.722+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:59:10.030 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:59:09 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:59:09.722+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:59:10.030 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:59:09 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:59:09.722+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:59:10.030 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:59:09 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:59:09.723+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:59:10.315 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:59:10 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:59:10.028+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T21:59:10.315 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:59:10 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:59:10.028+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T21:59:10.315 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:59:10 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:59:10.028+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T21:59:10.315 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:59:10 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:59:10.029+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:59:10.316 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:59:10 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:59:10.216+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:59:10.316 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:59:10 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:59:10.216+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:59:10.316 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:59:10 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:59:10.216+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:59:10.316 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:59:10 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:59:10.216+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:59:10.990 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:59:10 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:59:10.693+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:59:10.990 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:59:10 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:59:10.693+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:59:10.991 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:59:10 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:59:10.693+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:59:10.991 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:59:10 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:59:10.695+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:59:11.315 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:59:11 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:59:11.179+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:59:11.315 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:59:11 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:59:11.179+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:59:11.315 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:59:11 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:59:11.179+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:59:11.315 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:59:11 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:59:11.181+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:59:11.316 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:59:10 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:59:10.989+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T21:59:11.316 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:59:10 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:59:10.989+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T21:59:11.316 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:59:10 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:59:10.989+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T21:59:11.316 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:59:10 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:59:10.989+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:59:11.981 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Dec 16 21:59:11 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-mon-smithi045[80759]: 2023-12-16T21:59:11.574+0000 7f45a1f8f700 -1 mon.smithi045@0(probing) e2 get_health_metrics reporting 18541 slow ops, oldest is monmgrreport(gid 14214, 0 checks, 1 progress events) 2023-12-16T21:59:11.982 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:59:11 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:59:11.705+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:59:11.982 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:59:11 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:59:11.705+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:59:11.982 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:59:11 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:59:11.705+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:59:11.982 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:59:11 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:59:11.706+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:59:12.315 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:59:11 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:59:11.980+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T21:59:12.315 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:59:11 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:59:11.980+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T21:59:12.315 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:59:11 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:59:11.980+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T21:59:12.316 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:59:11 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:59:11.981+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:59:12.316 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:59:12 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:59:12.174+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:59:12.316 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:59:12 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:59:12.174+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:59:12.316 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:59:12 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:59:12.174+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:59:12.316 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:59:12 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:59:12.175+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:59:12.945 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:59:12 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:59:12.671+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:59:12.946 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:59:12 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:59:12.671+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:59:12.946 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:59:12 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:59:12.671+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:59:12.946 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:59:12 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:59:12.671+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:59:13.220 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:59:12 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:59:12.944+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T21:59:13.221 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:59:12 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:59:12.944+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T21:59:13.221 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:59:12 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:59:12.944+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T21:59:13.221 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:59:12 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:59:12.944+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:59:13.565 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:59:13 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:59:13.219+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:59:13.565 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:59:13 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:59:13.219+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:59:13.565 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:59:13 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:59:13.219+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:59:13.565 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:59:13 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:59:13.219+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:59:13.993 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:59:13 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:59:13.676+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:59:13.993 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:59:13 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:59:13.676+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:59:13.993 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:59:13 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:59:13.676+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:59:13.993 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:59:13 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:59:13.677+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:59:14.266 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:59:13 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:59:13.991+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T21:59:14.266 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:59:13 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:59:13.991+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T21:59:14.266 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:59:13 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:59:13.991+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T21:59:14.266 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:59:13 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:59:13.992+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:59:14.569 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:59:14 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:59:14.264+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:59:14.569 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:59:14 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:59:14.264+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:59:14.569 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:59:14 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:59:14.264+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:59:14.570 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:59:14 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:59:14.265+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:59:14.968 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:59:14 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:59:14.688+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:59:14.968 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:59:14 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:59:14.688+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:59:14.968 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:59:14 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:59:14.688+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:59:14.968 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:59:14 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:59:14.689+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:59:15.252 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:59:14 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:59:14.966+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T21:59:15.253 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:59:14 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:59:14.966+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T21:59:15.253 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:59:14 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:59:14.966+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T21:59:15.253 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:59:14 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:59:14.967+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:59:15.565 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:59:15 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:59:15.251+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:59:15.565 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:59:15 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:59:15.251+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:59:15.565 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:59:15 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:59:15.251+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:59:15.565 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:59:15 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:59:15.252+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:59:16.065 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:59:15 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:59:15.723+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:59:16.065 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:59:15 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:59:15.723+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:59:16.065 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:59:15 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:59:15.723+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:59:16.066 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:59:15 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:59:15.723+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:59:16.066 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:59:15 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:59:15.942+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T21:59:16.066 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:59:15 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:59:15.942+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T21:59:16.066 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:59:15 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:59:15.942+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T21:59:16.066 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:59:15 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:59:15.943+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:59:16.565 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:59:16 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:59:16.257+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:59:16.565 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:59:16 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:59:16.257+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:59:16.565 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:59:16 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:59:16.257+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:59:16.565 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:59:16 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:59:16.258+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:59:16.903 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Dec 16 21:59:16 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-mon-smithi045[80759]: 2023-12-16T21:59:16.576+0000 7f45a1f8f700 -1 mon.smithi045@0(probing) e2 get_health_metrics reporting 18630 slow ops, oldest is monmgrreport(gid 14214, 0 checks, 1 progress events) 2023-12-16T21:59:16.903 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:59:16 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:59:16.686+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:59:16.903 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:59:16 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:59:16.686+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:59:16.903 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:59:16 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:59:16.686+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:59:16.903 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:59:16 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:59:16.687+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:59:17.218 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:59:16 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:59:16.901+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T21:59:17.218 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:59:16 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:59:16.901+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T21:59:17.218 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:59:16 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:59:16.901+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T21:59:17.219 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:59:16 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:59:16.902+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:59:17.565 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:59:17 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:59:17.216+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:59:17.565 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:59:17 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:59:17.216+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:59:17.565 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:59:17 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:59:17.216+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:59:17.565 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:59:17 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:59:17.217+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:59:18.065 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:59:17 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:59:17.719+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:59:18.065 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:59:17 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:59:17.719+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:59:18.065 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:59:17 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:59:17.719+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:59:18.066 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:59:17 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:59:17.720+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:59:18.066 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:59:17 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:59:17.934+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T21:59:18.066 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:59:17 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:59:17.934+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T21:59:18.066 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:59:17 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:59:17.934+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T21:59:18.066 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:59:17 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:59:17.935+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:59:18.565 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:59:18 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:59:18.238+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:59:18.565 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:59:18 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:59:18.238+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:59:18.565 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:59:18 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:59:18.238+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:59:18.565 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:59:18 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:59:18.239+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:59:19.065 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:59:18 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:59:18.931+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T21:59:19.065 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:59:18 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:59:18.931+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T21:59:19.066 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:59:18 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:59:18.931+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T21:59:19.066 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:59:18 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:59:18.932+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:59:19.066 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:59:18 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:59:18.716+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:59:19.066 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:59:18 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:59:18.716+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:59:19.067 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:59:18 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:59:18.716+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:59:19.067 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:59:18 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:59:18.717+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:59:19.565 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:59:19 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:59:19.221+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:59:19.565 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:59:19 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:59:19.221+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:59:19.565 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:59:19 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:59:19.221+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:59:19.565 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:59:19 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:59:19.222+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:59:20.065 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:59:19 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:59:19.894+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T21:59:20.065 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:59:19 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:59:19.894+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T21:59:20.065 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:59:19 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:59:19.894+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T21:59:20.065 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:59:19 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:59:19.895+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:59:20.066 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:59:19 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:59:19.737+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:59:20.066 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:59:19 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:59:19.737+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:59:20.066 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:59:19 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:59:19.737+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:59:20.066 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:59:19 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:59:19.739+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:59:20.565 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:59:20 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:59:20.198+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:59:20.565 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:59:20 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:59:20.198+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:59:20.566 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:59:20 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:59:20.198+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:59:20.566 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:59:20 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:59:20.199+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:59:21.065 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:59:20 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:59:20.725+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:59:21.066 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:59:20 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:59:20.725+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:59:21.066 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:59:20 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:59:20.725+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:59:21.066 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:59:20 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:59:20.726+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:59:21.066 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:59:20 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:59:20.896+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T21:59:21.066 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:59:20 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:59:20.896+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T21:59:21.067 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:59:20 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:59:20.896+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T21:59:21.067 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:59:20 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:59:20.897+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:59:21.565 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:59:21 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:59:21.183+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:59:21.565 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:59:21 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:59:21.183+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:59:21.565 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:59:21 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:59:21.183+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:59:21.566 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:59:21 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:59:21.185+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:59:21.854 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Dec 16 21:59:21 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-mon-smithi045[80759]: 2023-12-16T21:59:21.578+0000 7f45a1f8f700 -1 mon.smithi045@0(probing) e2 get_health_metrics reporting 18720 slow ops, oldest is monmgrreport(gid 14214, 0 checks, 1 progress events) 2023-12-16T21:59:21.854 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:59:21 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:59:21.763+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:59:21.854 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:59:21 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:59:21.763+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:59:21.854 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:59:21 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:59:21.763+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:59:21.854 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:59:21 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:59:21.764+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:59:22.208 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:59:21 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:59:21.866+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T21:59:22.208 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:59:21 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:59:21.866+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T21:59:22.208 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:59:21 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:59:21.866+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T21:59:22.208 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:59:21 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:59:21.867+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:59:22.470 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:59:22 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:59:22.205+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:59:22.471 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:59:22 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:59:22.205+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:59:22.471 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:59:22 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:59:22.205+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:59:22.471 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:59:22 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:59:22.206+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:59:23.065 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:59:22 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:59:22.859+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T21:59:23.065 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:59:22 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:59:22.859+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T21:59:23.065 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:59:22 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:59:22.859+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T21:59:23.065 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:59:22 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:59:22.860+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:59:23.066 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:59:22 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:59:22.783+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:59:23.066 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:59:22 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:59:22.783+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:59:23.066 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:59:22 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:59:22.783+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:59:23.066 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:59:22 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:59:22.783+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:59:23.565 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:59:23 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:59:23.240+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:59:23.565 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:59:23 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:59:23.240+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:59:23.565 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:59:23 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:59:23.240+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:59:23.565 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:59:23 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:59:23.241+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:59:23.910 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:59:23 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:59:23.804+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:59:23.911 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:59:23 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:59:23.804+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:59:23.911 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:59:23 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:59:23.804+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:59:23.911 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:59:23 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:59:23.805+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:59:24.283 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:59:23 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:59:23.909+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T21:59:24.283 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:59:23 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:59:23.909+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T21:59:24.284 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:59:23 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:59:23.909+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T21:59:24.284 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:59:23 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:59:23.910+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:59:24.565 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:59:24 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:59:24.281+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:59:24.565 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:59:24 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:59:24.281+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:59:24.566 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:59:24 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:59:24.281+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:59:24.566 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:59:24 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:59:24.282+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:59:25.065 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:59:24 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:59:24.787+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:59:25.065 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:59:24 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:59:24.787+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:59:25.065 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:59:24 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:59:24.787+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:59:25.065 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:59:24 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:59:24.788+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:59:25.066 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:59:24 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:59:24.939+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T21:59:25.066 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:59:24 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:59:24.939+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T21:59:25.066 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:59:24 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:59:24.939+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T21:59:25.066 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:59:24 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:59:24.940+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:59:25.565 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:59:25 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:59:25.303+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:59:25.565 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:59:25 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:59:25.303+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:59:25.565 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:59:25 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:59:25.303+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:59:25.565 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:59:25 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:59:25.304+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:59:25.980 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:59:25 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:59:25.765+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:59:25.980 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:59:25 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:59:25.765+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:59:25.980 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:59:25 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:59:25.765+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:59:25.981 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:59:25 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:59:25.766+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:59:26.285 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:59:25 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:59:25.978+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T21:59:26.285 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:59:25 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:59:25.978+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T21:59:26.286 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:59:25 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:59:25.978+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T21:59:26.286 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:59:25 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:59:25.979+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:59:26.565 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:59:26 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:59:26.284+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:59:26.565 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:59:26 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:59:26.284+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:59:26.565 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:59:26 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:59:26.284+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:59:26.565 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:59:26 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:59:26.285+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:59:26.962 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Dec 16 21:59:26 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-mon-smithi045[80759]: 2023-12-16T21:59:26.580+0000 7f45a1f8f700 -1 mon.smithi045@0(probing) e2 get_health_metrics reporting 18809 slow ops, oldest is monmgrreport(gid 14214, 0 checks, 1 progress events) 2023-12-16T21:59:26.962 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:59:26 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:59:26.797+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:59:26.963 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:59:26 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:59:26.797+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:59:26.963 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:59:26 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:59:26.797+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:59:26.963 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:59:26 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:59:26.798+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:59:27.315 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:59:26 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:59:26.960+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T21:59:27.315 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:59:26 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:59:26.960+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T21:59:27.315 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:59:26 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:59:26.960+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T21:59:27.316 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:59:26 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:59:26.962+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:59:27.771 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:59:27 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:59:27.327+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:59:27.771 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:59:27 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:59:27.327+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:59:27.771 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:59:27 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:59:27.327+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:59:27.771 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:59:27 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:59:27.328+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:59:28.065 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:59:27 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:59:27.769+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:59:28.065 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:59:27 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:59:27.769+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:59:28.065 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:59:27 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:59:27.769+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:59:28.066 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:59:27 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:59:27.770+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:59:28.066 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:59:27 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:59:27.933+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T21:59:28.066 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:59:27 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:59:27.933+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T21:59:28.066 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:59:27 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:59:27.933+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T21:59:28.066 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:59:27 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:59:27.934+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:59:28.565 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:59:28 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:59:28.310+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:59:28.565 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:59:28 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:59:28.310+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:59:28.565 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:59:28 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:59:28.310+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:59:28.565 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:59:28 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:59:28.310+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:59:29.065 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:59:28 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:59:28.757+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:59:29.065 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:59:28 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:59:28.757+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:59:29.065 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:59:28 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:59:28.757+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:59:29.065 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:59:28 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:59:28.758+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:59:29.066 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:59:28 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:59:28.921+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T21:59:29.066 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:59:28 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:59:28.921+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T21:59:29.066 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:59:28 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:59:28.921+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T21:59:29.066 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:59:28 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:59:28.922+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:59:29.796 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:59:29 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:59:29.317+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:59:29.797 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:59:29 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:59:29.317+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:59:29.797 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:59:29 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:59:29.317+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:59:29.797 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:59:29 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:59:29.318+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:59:30.065 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:59:29 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:59:29.794+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:59:30.065 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:59:29 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:59:29.794+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:59:30.065 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:59:29 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:59:29.794+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:59:30.065 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:59:29 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:59:29.795+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:59:30.066 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:59:29 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:59:29.881+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T21:59:30.066 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:59:29 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:59:29.881+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T21:59:30.066 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:59:29 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:59:29.881+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T21:59:30.066 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:59:29 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:59:29.882+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:59:30.565 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:59:30 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:59:30.344+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:59:30.565 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:59:30 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:59:30.344+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:59:30.565 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:59:30 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:59:30.344+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:59:30.565 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:59:30 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:59:30.345+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:59:31.315 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:59:30 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:59:30.921+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T21:59:31.315 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:59:30 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:59:30.921+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T21:59:31.315 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:59:30 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:59:30.921+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T21:59:31.315 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:59:30 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:59:30.922+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:59:31.316 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:59:30 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:59:30.816+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:59:31.316 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:59:30 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:59:30.816+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:59:31.316 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:59:30 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:59:30.817+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:59:31.316 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:59:30 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:59:30.818+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:59:31.607 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Dec 16 21:59:31 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-mon-smithi045[80759]: 2023-12-16T21:59:31.582+0000 7f45a1f8f700 -1 mon.smithi045@0(probing) e2 get_health_metrics reporting 18900 slow ops, oldest is monmgrreport(gid 14214, 0 checks, 1 progress events) 2023-12-16T21:59:31.608 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:59:31 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:59:31.354+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:59:31.608 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:59:31 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:59:31.354+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:59:31.608 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:59:31 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:59:31.354+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:59:31.608 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:59:31 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:59:31.355+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:59:31.881 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:59:31 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:59:31.793+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:59:31.881 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:59:31 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:59:31.793+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:59:31.881 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:59:31 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:59:31.793+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:59:31.881 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:59:31 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:59:31.793+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:59:32.315 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:59:31 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:59:31.879+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T21:59:32.315 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:59:31 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:59:31.879+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T21:59:32.315 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:59:31 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:59:31.879+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T21:59:32.315 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:59:31 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:59:31.880+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:59:32.749 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:59:32 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:59:32.318+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:59:32.749 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:59:32 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:59:32.318+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:59:32.749 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:59:32 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:59:32.318+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:59:32.749 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:59:32 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:59:32.319+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:59:33.065 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:59:32 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:59:32.884+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T21:59:33.065 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:59:32 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:59:32.884+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T21:59:33.065 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:59:32 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:59:32.884+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T21:59:33.066 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:59:32 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:59:32.886+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:59:33.066 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:59:32 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:59:32.776+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:59:33.066 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:59:32 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:59:32.776+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:59:33.066 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:59:32 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:59:32.776+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:59:33.066 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:59:32 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:59:32.777+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:59:33.815 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:59:33 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:59:33.346+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:59:33.816 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:59:33 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:59:33.346+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:59:33.816 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:59:33 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:59:33.346+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:59:33.816 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:59:33 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:59:33.347+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:59:34.303 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:59:33 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:59:33.825+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:59:34.303 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:59:33 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:59:33.825+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:59:34.304 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:59:33 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:59:33.825+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:59:34.304 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:59:33 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:59:33.826+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:59:34.304 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:59:33 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:59:33.849+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T21:59:34.305 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:59:33 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:59:33.849+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T21:59:34.305 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:59:33 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:59:33.849+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T21:59:34.305 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:59:33 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:59:33.850+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:59:34.565 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:59:34 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:59:34.300+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:59:34.566 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:59:34 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:59:34.300+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:59:34.566 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:59:34 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:59:34.300+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:59:34.566 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:59:34 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:59:34.301+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:59:34.929 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:59:34 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:59:34.829+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T21:59:34.929 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:59:34 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:59:34.829+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T21:59:34.929 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:59:34 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:59:34.829+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T21:59:34.929 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:59:34 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:59:34.830+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:59:34.930 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:59:34 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:59:34.779+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:59:34.930 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:59:34 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:59:34.779+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:59:34.930 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:59:34 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:59:34.779+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:59:34.930 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:59:34 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:59:34.780+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:59:35.565 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:59:35 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:59:35.279+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:59:35.565 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:59:35 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:59:35.279+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:59:35.565 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:59:35 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:59:35.279+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:59:35.565 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:59:35 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:59:35.281+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:59:36.315 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:59:35 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:59:35.825+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T21:59:36.315 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:59:35 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:59:35.825+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T21:59:36.315 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:59:35 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:59:35.825+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T21:59:36.316 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:59:35 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:59:35.826+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:59:36.316 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:59:35 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:59:35.825+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:59:36.316 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:59:35 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:59:35.825+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:59:36.316 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:59:35 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:59:35.825+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:59:36.316 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:59:35 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:59:35.826+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:59:36.585 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:59:36 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:59:36.327+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:59:36.586 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:59:36 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:59:36.327+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:59:36.586 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:59:36 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:59:36.327+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:59:36.586 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:59:36 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:59:36.328+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:59:36.866 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Dec 16 21:59:36 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-mon-smithi045[80759]: 2023-12-16T21:59:36.583+0000 7f45a1f8f700 -1 mon.smithi045@0(probing) e2 get_health_metrics reporting 18989 slow ops, oldest is monmgrreport(gid 14214, 0 checks, 1 progress events) 2023-12-16T21:59:36.867 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:59:36 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:59:36.797+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T21:59:36.867 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:59:36 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:59:36.797+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T21:59:36.867 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:59:36 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:59:36.797+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T21:59:36.867 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:59:36 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:59:36.798+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:59:37.315 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:59:36 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:59:36.864+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:59:37.315 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:59:36 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:59:36.864+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:59:37.315 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:59:36 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:59:36.864+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:59:37.315 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:59:36 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:59:36.865+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:59:37.760 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:59:37 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:59:37.377+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:59:37.760 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:59:37 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:59:37.377+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:59:37.760 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:59:37 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:59:37.377+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:59:37.760 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:59:37 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:59:37.378+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:59:38.065 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:59:37 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:59:37.758+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T21:59:38.065 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:59:37 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:59:37.758+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T21:59:38.065 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:59:37 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:59:37.758+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T21:59:38.065 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:59:37 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:59:37.760+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:59:38.066 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:59:37 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:59:37.868+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:59:38.066 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:59:37 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:59:37.868+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:59:38.066 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:59:37 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:59:37.868+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:59:38.066 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:59:37 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:59:37.868+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:59:38.795 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:59:38 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:59:38.405+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:59:38.795 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:59:38 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:59:38.405+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:59:38.795 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:59:38 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:59:38.405+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:59:38.795 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:59:38 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:59:38.406+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:59:39.065 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:59:38 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:59:38.909+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:59:39.065 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:59:38 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:59:38.909+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:59:39.065 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:59:38 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:59:38.909+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:59:39.065 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:59:38 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:59:38.911+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:59:39.066 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:59:38 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:59:38.793+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T21:59:39.066 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:59:38 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:59:38.793+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T21:59:39.066 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:59:38 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:59:38.793+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T21:59:39.066 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:59:38 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:59:38.794+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:59:39.785 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:59:39 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:59:39.356+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:59:39.785 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:59:39 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:59:39.356+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:59:39.785 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:59:39 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:59:39.356+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:59:39.785 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:59:39 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:59:39.357+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:59:40.065 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:59:39 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:59:39.909+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:59:40.065 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:59:39 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:59:39.909+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:59:40.065 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:59:39 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:59:39.909+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:59:40.065 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:59:39 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:59:39.910+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:59:40.066 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:59:39 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:59:39.783+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T21:59:40.066 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:59:39 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:59:39.783+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T21:59:40.066 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:59:39 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:59:39.783+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T21:59:40.066 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:59:39 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:59:39.784+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:59:40.565 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:59:40 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:59:40.379+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:59:40.565 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:59:40 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:59:40.379+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:59:40.565 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:59:40 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:59:40.379+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:59:40.565 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:59:40 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:59:40.380+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:59:41.071 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:59:40 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:59:40.908+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:59:41.072 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:59:40 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:59:40.908+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:59:41.072 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:59:40 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:59:40.908+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:59:41.072 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:59:40 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:59:40.908+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:59:41.072 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:59:40 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:59:40.815+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T21:59:41.072 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:59:40 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:59:40.815+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T21:59:41.072 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:59:40 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:59:40.815+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T21:59:41.073 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:59:40 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:59:40.816+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:59:41.697 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Dec 16 21:59:41 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-mon-smithi045[80759]: 2023-12-16T21:59:41.586+0000 7f45a1f8f700 -1 mon.smithi045@0(probing) e2 get_health_metrics reporting 19079 slow ops, oldest is monmgrreport(gid 14214, 0 checks, 1 progress events) 2023-12-16T21:59:41.697 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:59:41 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:59:41.404+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:59:41.698 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:59:41 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:59:41.404+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:59:41.698 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:59:41 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:59:41.404+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:59:41.698 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:59:41 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:59:41.405+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:59:42.065 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:59:41 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:59:41.794+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T21:59:42.065 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:59:41 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:59:41.794+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T21:59:42.065 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:59:41 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:59:41.794+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T21:59:42.066 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:59:41 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:59:41.795+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:59:42.066 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:59:41 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:59:41.925+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:59:42.066 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:59:41 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:59:41.925+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:59:42.066 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:59:41 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:59:41.925+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:59:42.066 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:59:41 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:59:41.926+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:59:42.809 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:59:42 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:59:42.447+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:59:42.809 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:59:42 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:59:42.447+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:59:42.809 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:59:42 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:59:42.447+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:59:42.809 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:59:42 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:59:42.448+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:59:43.065 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:59:42 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:59:42.896+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:59:43.065 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:59:42 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:59:42.896+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:59:43.065 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:59:42 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:59:42.896+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:59:43.065 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:59:42 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:59:42.897+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:59:43.066 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:59:42 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:59:42.807+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T21:59:43.066 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:59:42 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:59:42.807+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T21:59:43.066 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:59:42 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:59:42.807+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T21:59:43.066 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:59:42 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:59:42.808+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:59:43.815 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:59:43 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:59:43.432+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:59:43.815 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:59:43 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:59:43.432+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:59:43.815 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:59:43 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:59:43.432+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:59:43.815 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:59:43 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:59:43.433+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:59:44.065 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:59:43 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:59:43.813+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T21:59:44.065 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:59:43 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:59:43.813+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T21:59:44.065 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:59:43 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:59:43.813+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T21:59:44.065 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:59:43 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:59:43.814+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:59:44.066 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:59:43 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:59:43.941+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:59:44.066 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:59:43 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:59:43.941+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:59:44.066 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:59:43 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:59:43.941+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:59:44.066 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:59:43 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:59:43.942+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:59:44.812 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:59:44 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:59:44.444+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:59:44.812 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:59:44 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:59:44.444+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:59:44.812 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:59:44 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:59:44.444+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:59:44.812 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:59:44 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:59:44.445+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:59:45.065 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:59:44 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:59:44.810+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T21:59:45.065 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:59:44 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:59:44.810+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T21:59:45.065 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:59:44 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:59:44.810+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T21:59:45.065 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:59:44 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:59:44.811+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:59:45.066 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:59:44 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:59:44.970+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:59:45.066 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:59:44 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:59:44.970+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:59:45.066 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:59:44 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:59:44.970+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:59:45.066 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:59:44 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:59:44.971+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:59:45.815 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:59:45 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:59:45.463+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:59:45.815 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:59:45 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:59:45.463+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:59:45.815 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:59:45 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:59:45.463+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:59:45.815 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:59:45 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:59:45.464+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:59:46.315 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:59:45 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:59:45.958+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:59:46.315 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:59:45 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:59:45.958+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:59:46.315 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:59:45 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:59:45.958+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:59:46.315 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:59:45 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:59:45.960+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:59:46.316 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:59:45 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:59:45.851+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T21:59:46.316 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:59:45 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:59:45.851+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T21:59:46.316 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:59:45 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:59:45.851+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T21:59:46.316 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:59:45 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:59:45.852+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:59:46.714 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:59:46 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:59:46.460+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:59:46.715 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:59:46 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:59:46.460+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:59:46.715 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:59:46 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:59:46.460+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:59:46.715 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:59:46 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:59:46.461+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:59:46.715 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Dec 16 21:59:46 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-mon-smithi045[80759]: 2023-12-16T21:59:46.588+0000 7f45a1f8f700 -1 mon.smithi045@0(probing) e2 get_health_metrics reporting 19168 slow ops, oldest is monmgrreport(gid 14214, 0 checks, 1 progress events) 2023-12-16T21:59:47.065 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:59:46 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:59:46.955+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:59:47.065 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:59:46 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:59:46.955+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:59:47.065 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:59:46 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:59:46.955+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:59:47.065 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:59:46 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:59:46.956+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:59:47.066 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:59:46 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:59:46.848+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T21:59:47.066 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:59:46 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:59:46.848+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T21:59:47.066 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:59:46 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:59:46.848+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T21:59:47.066 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:59:46 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:59:46.849+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:59:47.815 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:59:47 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:59:47.458+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:59:47.815 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:59:47 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:59:47.458+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:59:47.815 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:59:47 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:59:47.458+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:59:47.816 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:59:47 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:59:47.459+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:59:48.315 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:59:47 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:59:47.854+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T21:59:48.315 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:59:47 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:59:47.854+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T21:59:48.315 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:59:47 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:59:47.854+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T21:59:48.315 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:59:47 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:59:47.855+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:59:48.316 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:59:47 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:59:47.957+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:59:48.316 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:59:47 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:59:47.957+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:59:48.316 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:59:47 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:59:47.957+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:59:48.317 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:59:47 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:59:47.957+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:59:48.815 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:59:48 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:59:48.489+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:59:48.815 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:59:48 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:59:48.489+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:59:48.815 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:59:48 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:59:48.489+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:59:48.815 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:59:48 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:59:48.489+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:59:49.315 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:59:48 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:59:48.850+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T21:59:49.315 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:59:48 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:59:48.850+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T21:59:49.315 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:59:48 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:59:48.850+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T21:59:49.315 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:59:48 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:59:48.851+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:59:49.316 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:59:48 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:59:48.920+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:59:49.316 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:59:48 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:59:48.920+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:59:49.316 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:59:48 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:59:48.920+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:59:49.316 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:59:48 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:59:48.921+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:59:49.815 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:59:49 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:59:49.497+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:59:49.815 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:59:49 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:59:49.497+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:59:49.815 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:59:49 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:59:49.497+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:59:49.815 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:59:49 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:59:49.498+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:59:50.169 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:59:49 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:59:49.881+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T21:59:50.169 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:59:49 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:59:49.881+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T21:59:50.169 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:59:49 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:59:49.881+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T21:59:50.170 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:59:49 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:59:49.882+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:59:50.170 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:59:49 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:59:49.909+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:59:50.170 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:59:49 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:59:49.909+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:59:50.170 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:59:49 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:59:49.909+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:59:50.170 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:59:49 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:59:49.910+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:59:50.815 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:59:50 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:59:50.478+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:59:50.815 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:59:50 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:59:50.478+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:59:50.815 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:59:50 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:59:50.478+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:59:50.815 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:59:50 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:59:50.479+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:59:51.315 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:59:50 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:59:50.941+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:59:51.315 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:59:50 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:59:50.941+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:59:51.315 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:59:50 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:59:50.941+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:59:51.315 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:59:50 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:59:50.943+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:59:51.316 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:59:50 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:59:50.833+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T21:59:51.316 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:59:50 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:59:50.833+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T21:59:51.316 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:59:50 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:59:50.833+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T21:59:51.316 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:59:50 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:59:50.834+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:59:51.815 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Dec 16 21:59:51 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-mon-smithi045[80759]: 2023-12-16T21:59:51.590+0000 7f45a1f8f700 -1 mon.smithi045@0(probing) e2 get_health_metrics reporting 19259 slow ops, oldest is monmgrreport(gid 14214, 0 checks, 1 progress events) 2023-12-16T21:59:51.815 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:59:51 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:59:51.509+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:59:51.815 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:59:51 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:59:51.509+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:59:51.815 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:59:51 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:59:51.509+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:59:51.815 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:59:51 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:59:51.511+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:59:52.315 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:59:51 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:59:51.981+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:59:52.315 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:59:51 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:59:51.981+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:59:52.315 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:59:51 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:59:51.981+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:59:52.315 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:59:51 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:59:51.981+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:59:52.316 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:59:51 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:59:51.824+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T21:59:52.316 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:59:51 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:59:51.824+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T21:59:52.316 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:59:51 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:59:51.824+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T21:59:52.316 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:59:51 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:59:51.825+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:59:52.815 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:59:52 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:59:52.513+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:59:52.815 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:59:52 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:59:52.513+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:59:52.815 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:59:52 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:59:52.513+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:59:52.815 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:59:52 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:59:52.514+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:59:53.315 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:59:52 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:59:52.993+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:59:53.315 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:59:52 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:59:52.993+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:59:53.315 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:59:52 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:59:52.993+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:59:53.315 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:59:52 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:59:52.994+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:59:53.315 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:59:52 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:59:52.857+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T21:59:53.316 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:59:52 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:59:52.857+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T21:59:53.316 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:59:52 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:59:52.857+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T21:59:53.316 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:59:52 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:59:52.858+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:59:53.815 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:59:53 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:59:53.486+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:59:53.815 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:59:53 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:59:53.486+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:59:53.815 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:59:53 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:59:53.486+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:59:53.815 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:59:53 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:59:53.487+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:59:54.315 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:59:54 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:59:54.006+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:59:54.315 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:59:54 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:59:54.006+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:59:54.315 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:59:54 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:59:54.006+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:59:54.315 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:59:54 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:59:54.007+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:59:54.316 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:59:53 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:59:53.875+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T21:59:54.316 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:59:53 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:59:53.875+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T21:59:54.316 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:59:53 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:59:53.875+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T21:59:54.316 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:59:53 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:59:53.876+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:59:54.815 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:59:54 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:59:54.463+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:59:54.815 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:59:54 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:59:54.463+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:59:54.815 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:59:54 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:59:54.463+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:59:54.815 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:59:54 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:59:54.463+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:59:55.315 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:59:55 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:59:55.023+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:59:55.315 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:59:55 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:59:55.023+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:59:55.315 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:59:55 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:59:55.023+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:59:55.315 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:59:55 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:59:55.024+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:59:55.315 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:59:54 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:59:54.847+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T21:59:55.316 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:59:54 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:59:54.847+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T21:59:55.316 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:59:54 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:59:54.847+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T21:59:55.316 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:59:54 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:59:54.847+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:59:55.815 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:59:55 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:59:55.416+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:59:55.815 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:59:55 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:59:55.416+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:59:55.815 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:59:55 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:59:55.416+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:59:55.815 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:59:55 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:59:55.418+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:59:56.315 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:59:55 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:59:55.979+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:59:56.315 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:59:55 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:59:55.979+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:59:56.315 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:59:55 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:59:55.979+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:59:56.315 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:59:55 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:59:55.979+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:59:56.316 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:59:55 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:59:55.882+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T21:59:56.316 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:59:55 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:59:55.882+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T21:59:56.316 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:59:55 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:59:55.882+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T21:59:56.316 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:59:55 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:59:55.883+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:59:56.720 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Dec 16 21:59:56 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-mon-smithi045[80759]: 2023-12-16T21:59:56.592+0000 7f45a1f8f700 -1 mon.smithi045@0(probing) e2 get_health_metrics reporting 19348 slow ops, oldest is monmgrreport(gid 14214, 0 checks, 1 progress events) 2023-12-16T21:59:56.721 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:59:56 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:59:56.446+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:59:56.721 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:59:56 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:59:56.446+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:59:56.721 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:59:56 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:59:56.446+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:59:56.721 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:59:56 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:59:56.448+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:59:57.065 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:59:56 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:59:56.943+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:59:57.065 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:59:56 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:59:56.943+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:59:57.065 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:59:56 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:59:56.943+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:59:57.065 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:59:56 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:59:56.944+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:59:57.066 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:59:56 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:59:56.835+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T21:59:57.066 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:59:56 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:59:56.835+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T21:59:57.066 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:59:56 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:59:56.835+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T21:59:57.066 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:59:56 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:59:56.836+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:59:57.815 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:59:57 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:59:57.411+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:59:57.815 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:59:57 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:59:57.411+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:59:57.815 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:59:57 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:59:57.411+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:59:57.815 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:59:57 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:59:57.412+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:59:58.315 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:59:57 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:59:57.942+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:59:58.315 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:59:57 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:59:57.942+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:59:58.315 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:59:57 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:59:57.942+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:59:58.315 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:59:57 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:59:57.943+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:59:58.316 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:59:57 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:59:57.873+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T21:59:58.316 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:59:57 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:59:57.873+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T21:59:58.316 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:59:57 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:59:57.873+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T21:59:58.316 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:59:57 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:59:57.874+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:59:58.721 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:59:58 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:59:58.423+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:59:58.721 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:59:58 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:59:58.423+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:59:58.721 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:59:58 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:59:58.423+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:59:58.722 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:59:58 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:59:58.424+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:59:58.991 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:59:58 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:59:58.894+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T21:59:58.991 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:59:58 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:59:58.894+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T21:59:58.992 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:59:58 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:59:58.894+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T21:59:58.992 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:59:58 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:59:58.894+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:59:59.315 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:59:58 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:59:58.989+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:59:59.315 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:59:58 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:59:58.989+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:59:59.315 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:59:58 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:59:58.989+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T21:59:59.315 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:59:58 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:59:58.990+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T21:59:59.815 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:59:59 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:59:59.455+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:59:59.815 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:59:59 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:59:59.455+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:59:59.815 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:59:59 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:59:59.455+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T21:59:59.815 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 21:59:59 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T21:59:59.456+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:00:00.170 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:59:59 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:59:59.864+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T22:00:00.170 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:59:59 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:59:59.864+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T22:00:00.170 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:59:59 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:59:59.864+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T22:00:00.170 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 21:59:59 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T21:59:59.865+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:00:00.171 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:59:59 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:59:59.971+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T22:00:00.171 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:59:59 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:59:59.971+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T22:00:00.171 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:59:59 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:59:59.971+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T22:00:00.171 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 21:59:59 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T21:59:59.972+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:00:00.815 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:00:00 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:00:00.483+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T22:00:00.815 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:00:00 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:00:00.483+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T22:00:00.815 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:00:00 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:00:00.483+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T22:00:00.815 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:00:00 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:00:00.484+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:00:01.315 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:00:00 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:00:00.955+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T22:00:01.315 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:00:00 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:00:00.955+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T22:00:01.315 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:00:00 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:00:00.955+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T22:00:01.315 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:00:00 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:00:00.956+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:00:01.316 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:00:00 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:00:00.904+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T22:00:01.316 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:00:00 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:00:00.904+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T22:00:01.316 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:00:00 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:00:00.904+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T22:00:01.316 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:00:00 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:00:00.905+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:00:01.815 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Dec 16 22:00:01 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-mon-smithi045[80759]: 2023-12-16T22:00:01.594+0000 7f45a1f8f700 -1 mon.smithi045@0(probing) e2 get_health_metrics reporting 19438 slow ops, oldest is monmgrreport(gid 14214, 0 checks, 1 progress events) 2023-12-16T22:00:01.815 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:00:01 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:00:01.492+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T22:00:01.815 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:00:01 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:00:01.492+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T22:00:01.815 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:00:01 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:00:01.492+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T22:00:01.815 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:00:01 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:00:01.494+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:00:02.315 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:00:01 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:00:01.979+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T22:00:02.315 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:00:01 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:00:01.979+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T22:00:02.315 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:00:01 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:00:01.979+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T22:00:02.315 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:00:01 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:00:01.980+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:00:02.315 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:00:01 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:00:01.925+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T22:00:02.316 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:00:01 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:00:01.925+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T22:00:02.316 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:00:01 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:00:01.925+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T22:00:02.316 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:00:01 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:00:01.925+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:00:02.815 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:00:02 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:00:02.542+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T22:00:02.815 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:00:02 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:00:02.542+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T22:00:02.815 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:00:02 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:00:02.542+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T22:00:02.815 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:00:02 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:00:02.543+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:00:03.315 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:00:02 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:00:02.968+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T22:00:03.315 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:00:02 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:00:02.968+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T22:00:03.315 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:00:02 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:00:02.968+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T22:00:03.315 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:00:02 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:00:02.968+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:00:03.316 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:00:02 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:00:02.976+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T22:00:03.316 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:00:02 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:00:02.976+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T22:00:03.316 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:00:02 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:00:02.976+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T22:00:03.316 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:00:02 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:00:02.976+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:00:03.982 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:00:03 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:00:03.587+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T22:00:03.982 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:00:03 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:00:03.587+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T22:00:03.982 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:00:03 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:00:03.587+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T22:00:03.983 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:00:03 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:00:03.588+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:00:04.315 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:00:04 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:00:04.018+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T22:00:04.315 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:00:04 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:00:04.018+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T22:00:04.315 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:00:04 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:00:04.018+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T22:00:04.315 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:00:04 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:00:04.019+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:00:04.316 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:00:03 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:00:03.981+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T22:00:04.316 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:00:03 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:00:03.981+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T22:00:04.316 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:00:03 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:00:03.981+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T22:00:04.316 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:00:03 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:00:03.982+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:00:05.010 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:00:04 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:00:04.578+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T22:00:05.011 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:00:04 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:00:04.578+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T22:00:05.011 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:00:04 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:00:04.578+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T22:00:05.011 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:00:04 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:00:04.579+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:00:05.315 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:00:05 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:00:05.029+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T22:00:05.315 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:00:05 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:00:05.029+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T22:00:05.315 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:00:05 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:00:05.029+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T22:00:05.315 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:00:05 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:00:05.029+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:00:05.315 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:00:05 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:00:05.009+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T22:00:05.316 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:00:05 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:00:05.009+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T22:00:05.316 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:00:05 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:00:05.009+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T22:00:05.316 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:00:05 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:00:05.010+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:00:05.815 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:00:05 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:00:05.544+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T22:00:05.815 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:00:05 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:00:05.544+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T22:00:05.815 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:00:05 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:00:05.544+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:31.399437+0000 (oldest deadline 2023-12-16T21:49:51.399437+0000) 2023-12-16T22:00:05.815 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:00:05 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:00:05.544+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:00:06.315 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:00:06 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:00:06.020+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T22:00:06.315 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:00:06 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:00:06.020+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T22:00:06.315 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:00:06 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:00:06.020+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T22:00:06.315 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:00:06 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:00:06.021+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:00:06.316 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:00:06 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:00:06.030+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T22:00:06.316 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:00:06 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:00:06.030+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T22:00:06.316 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:00:06 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:00:06.030+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T22:00:06.316 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:00:06 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:00:06.030+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:00:06.815 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:00:06 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:00:06.529+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:00:06.815 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Dec 16 22:00:06 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-mon-smithi045[80759]: 2023-12-16T22:00:06.596+0000 7f45a1f8f700 -1 mon.smithi045@0(probing) e2 get_health_metrics reporting 19527 slow ops, oldest is monmgrreport(gid 14214, 0 checks, 1 progress events) 2023-12-16T22:00:07.315 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:00:07 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:00:07.007+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T22:00:07.315 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:00:07 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:00:07.007+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T22:00:07.315 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:00:07 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:00:07.007+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T22:00:07.315 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:00:07 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:00:07.008+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:00:07.315 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:00:07 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:00:07.041+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T22:00:07.316 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:00:07 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:00:07.041+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T22:00:07.316 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:00:07 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:00:07.041+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T22:00:07.316 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:00:07 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:00:07.042+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:00:07.963 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:00:07 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:00:07.566+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:00:08.315 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:00:08 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:00:08.031+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T22:00:08.315 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:00:08 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:00:08.031+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T22:00:08.315 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:00:08 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:00:08.031+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T22:00:08.315 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:00:08 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:00:08.032+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:00:08.316 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:00:07 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:00:07.961+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T22:00:08.316 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:00:07 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:00:07.961+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T22:00:08.316 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:00:07 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:00:07.961+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T22:00:08.316 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:00:07 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:00:07.962+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:00:08.815 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:00:08 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:00:08.552+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:00:09.315 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:00:09 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:00:09.018+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T22:00:09.315 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:00:09 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:00:09.018+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T22:00:09.315 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:00:09 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:00:09.018+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T22:00:09.315 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:00:09 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:00:09.019+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:00:09.316 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:00:08 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:00:08.930+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T22:00:09.316 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:00:08 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:00:08.930+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T22:00:09.316 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:00:08 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:00:08.930+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T22:00:09.316 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:00:08 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:00:08.931+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:00:09.814 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:00:09 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:00:09.552+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:00:10.315 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:00:09 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:00:09.929+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T22:00:10.315 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:00:09 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:00:09.929+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T22:00:10.315 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:00:09 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:00:09.929+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T22:00:10.315 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:00:09 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:00:09.930+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:00:10.316 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:00:10 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:00:10.006+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T22:00:10.316 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:00:10 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:00:10.006+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T22:00:10.316 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:00:10 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:00:10.006+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T22:00:10.316 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:00:10 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:00:10.007+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:00:10.815 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:00:10 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:00:10.514+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:00:11.315 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:00:10 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:00:10.952+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T22:00:11.315 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:00:10 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:00:10.952+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T22:00:11.315 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:00:10 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:00:10.952+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T22:00:11.315 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:00:10 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:00:10.953+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:00:11.316 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:00:10 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:00:10.982+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T22:00:11.316 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:00:10 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:00:10.982+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T22:00:11.316 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:00:10 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:00:10.982+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T22:00:11.316 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:00:10 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:00:10.983+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:00:11.793 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:00:11 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:00:11.541+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:00:11.793 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Dec 16 22:00:11 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-mon-smithi045[80759]: 2023-12-16T22:00:11.598+0000 7f45a1f8f700 -1 mon.smithi045@0(probing) e2 get_health_metrics reporting 19618 slow ops, oldest is monmgrreport(gid 14214, 0 checks, 1 progress events) 2023-12-16T22:00:12.065 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:00:12 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:00:12.015+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T22:00:12.065 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:00:12 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:00:12.015+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T22:00:12.065 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:00:12 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:00:12.015+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T22:00:12.065 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:00:12 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:00:12.016+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:00:12.066 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:00:12 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:00:12.000+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T22:00:12.066 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:00:12 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:00:12.000+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T22:00:12.066 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:00:12 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:00:12.000+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T22:00:12.066 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:00:12 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:00:12.002+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:00:12.993 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:00:12 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:00:12.575+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:00:13.315 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:00:13 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:00:13.029+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T22:00:13.315 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:00:13 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:00:13.029+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T22:00:13.315 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:00:13 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:00:13.029+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T22:00:13.315 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:00:13 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:00:13.030+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:00:13.316 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:00:12 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:00:12.992+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T22:00:13.316 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:00:12 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:00:12.992+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T22:00:13.316 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:00:12 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:00:12.992+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:38.189714+0000 (oldest deadline 2023-12-16T21:49:58.189714+0000) 2023-12-16T22:00:13.316 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:00:12 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:00:12.992+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:00:13.973 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:00:13 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:00:13.580+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:00:14.315 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:00:13 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:00:13.972+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:00:14.315 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:00:14 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:00:14.079+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T22:00:14.315 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:00:14 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:00:14.079+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T22:00:14.315 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:00:14 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:00:14.079+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T22:00:14.315 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:00:14 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:00:14.079+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:00:14.815 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:00:14 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:00:14.540+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:00:15.315 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:00:14 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:00:14.989+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:00:15.315 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:00:15 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:00:15.062+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T22:00:15.315 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:00:15 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:00:15.062+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T22:00:15.315 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:00:15 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:00:15.062+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T22:00:15.315 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:00:15 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:00:15.063+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:00:15.815 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:00:15 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:00:15.537+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:00:16.315 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:00:15 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:00:15.976+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:00:16.315 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:00:16 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:00:16.047+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T22:00:16.315 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:00:16 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:00:16.047+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T22:00:16.315 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:00:16 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:00:16.047+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T22:00:16.316 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:00:16 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:00:16.048+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:00:16.815 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:00:16 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:00:16.524+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:00:16.815 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Dec 16 22:00:16 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-mon-smithi045[80759]: 2023-12-16T22:00:16.600+0000 7f45a1f8f700 -1 mon.smithi045@0(probing) e2 get_health_metrics reporting 19707 slow ops, oldest is monmgrreport(gid 14214, 0 checks, 1 progress events) 2023-12-16T22:00:17.315 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:00:16 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:00:16.942+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:00:17.315 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:00:17 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:00:17.081+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T22:00:17.316 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:00:17 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:00:17.081+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T22:00:17.316 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:00:17 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:00:17.081+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T22:00:17.316 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:00:17 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:00:17.082+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:00:17.814 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:00:17 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:00:17.512+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:00:18.315 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:00:17 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:00:17.937+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:00:18.315 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:00:18 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:00:18.034+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T22:00:18.315 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:00:18 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:00:18.034+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T22:00:18.315 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:00:18 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:00:18.034+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T22:00:18.315 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:00:18 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:00:18.035+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:00:18.815 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:00:18 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:00:18.512+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:00:19.315 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:00:18 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:00:18.912+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:00:19.315 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:00:19 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:00:19.004+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T22:00:19.315 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:00:19 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:00:19.004+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T22:00:19.315 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:00:19 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:00:19.004+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T22:00:19.316 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:00:19 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:00:19.005+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:00:19.815 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:00:19 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:00:19.526+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:00:20.170 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:00:19 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:00:19.877+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:00:20.171 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:00:19 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:00:19.980+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T22:00:20.171 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:00:19 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:00:19.980+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T22:00:20.171 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:00:19 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:00:19.980+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T22:00:20.171 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:00:19 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:00:19.981+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:00:20.815 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:00:20 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:00:20.515+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:00:21.315 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:00:20 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:00:20.906+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:00:21.315 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:00:20 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:00:20.959+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T22:00:21.315 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:00:20 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:00:20.959+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T22:00:21.316 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:00:20 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:00:20.959+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T22:00:21.316 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:00:20 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:00:20.961+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:00:21.730 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Dec 16 22:00:21 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-mon-smithi045[80759]: 2023-12-16T22:00:21.602+0000 7f45a1f8f700 -1 mon.smithi045@0(probing) e2 get_health_metrics reporting 19797 slow ops, oldest is monmgrreport(gid 14214, 0 checks, 1 progress events) 2023-12-16T22:00:21.730 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:00:21 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:00:21.469+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:00:22.065 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:00:21 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:00:21.876+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:00:22.065 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:00:21 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:00:21.912+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T22:00:22.065 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:00:21 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:00:21.912+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T22:00:22.065 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:00:21 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:00:21.912+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T22:00:22.066 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:00:21 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:00:21.913+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:00:22.818 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:00:22 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:00:22.453+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:00:23.315 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:00:22 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:00:22.837+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:00:23.315 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:00:22 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:00:22.959+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T22:00:23.315 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:00:22 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:00:22.959+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T22:00:23.315 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:00:22 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:00:22.959+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T22:00:23.316 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:00:22 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:00:22.960+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:00:23.815 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:00:23 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:00:23.441+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:00:24.315 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:00:23 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:00:23.830+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:00:24.315 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:00:23 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:00:23.977+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T22:00:24.315 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:00:23 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:00:23.977+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T22:00:24.316 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:00:23 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:00:23.977+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T22:00:24.316 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:00:23 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:00:23.977+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:00:24.815 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:00:24 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:00:24.490+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:00:25.315 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:00:24 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:00:24.839+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:00:25.315 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:00:25 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:00:25.024+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T22:00:25.315 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:00:25 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:00:25.024+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T22:00:25.315 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:00:25 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:00:25.024+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T21:49:55.411372+0000 (oldest deadline 2023-12-16T21:50:15.411372+0000) 2023-12-16T22:00:25.316 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:00:25 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:00:25.025+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:00:25.815 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:00:25 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:00:25.511+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:00:26.315 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:00:26 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:00:26.021+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:00:26.315 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:00:25 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:00:25.823+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:00:26.815 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:00:26 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:00:26.541+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:00:26.816 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Dec 16 22:00:26 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-mon-smithi045[80759]: 2023-12-16T22:00:26.604+0000 7f45a1f8f700 -1 mon.smithi045@0(probing) e2 get_health_metrics reporting 19886 slow ops, oldest is monmgrreport(gid 14214, 0 checks, 1 progress events) 2023-12-16T22:00:27.315 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:00:26 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:00:26.828+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:00:27.315 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:00:26 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:00:26.989+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:00:27.815 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:00:27 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:00:27.559+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:00:28.315 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:00:27 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:00:27.848+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:00:28.315 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:00:28 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:00:28.015+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:00:28.883 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:00:28 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:00:28.575+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:00:29.315 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:00:28 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:00:28.881+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:00:29.315 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:00:28 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:00:28.987+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:00:29.816 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:00:29 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:00:29.561+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:00:30.171 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:00:29 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:00:29.959+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:00:30.171 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:00:29 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:00:29.865+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:00:30.910 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:00:30 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:00:30.602+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:00:30.911 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:00:30 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:00:30.602+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:00:30.911 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:00:30 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:00:30.602+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:00:30.911 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:00:30 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:00:30.603+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:00:31.315 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:00:30 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:00:30.909+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:00:31.315 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:00:30 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:00:30.953+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:00:31.906 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Dec 16 22:00:31 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-mon-smithi045[80759]: 2023-12-16T22:00:31.606+0000 7f45a1f8f700 -1 mon.smithi045@0(probing) e2 get_health_metrics reporting 19977 slow ops, oldest is monmgrreport(gid 14214, 0 checks, 1 progress events) 2023-12-16T22:00:31.907 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:00:31 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:00:31.591+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:00:31.907 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:00:31 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:00:31.591+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:00:31.907 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:00:31 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:00:31.591+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:00:31.907 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:00:31 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:00:31.592+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:00:32.315 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:00:31 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:00:31.933+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:00:32.315 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:00:31 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:00:31.905+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:00:32.896 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:00:32 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:00:32.594+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:00:32.896 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:00:32 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:00:32.594+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:00:32.896 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:00:32 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:00:32.594+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:00:32.897 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:00:32 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:00:32.595+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:00:33.295 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:00:32 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:00:32.905+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:00:33.296 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:00:32 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:00:32.894+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:00:33.854 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:00:33 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:00:33.622+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:00:33.854 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:00:33 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:00:33.622+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:00:33.854 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:00:33 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:00:33.622+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:00:33.854 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:00:33 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:00:33.623+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:00:34.315 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:00:33 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:00:33.852+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:00:34.315 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:00:33 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:00:33.873+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:00:34.877 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:00:34 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:00:34.813+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:00:34.877 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:00:34 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:00:34.613+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:00:34.877 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:00:34 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:00:34.613+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:00:34.877 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:00:34 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:00:34.613+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:00:34.877 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:00:34 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:00:34.614+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:00:35.315 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:00:34 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:00:34.875+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:00:35.858 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:00:35 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:00:35.594+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:00:35.859 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:00:35 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:00:35.594+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:00:35.859 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:00:35 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:00:35.594+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:00:35.859 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:00:35 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:00:35.595+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:00:36.315 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:00:35 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:00:35.857+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:00:36.315 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:00:35 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:00:35.925+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:00:36.619 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Dec 16 22:00:36 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-mon-smithi045[80759]: 2023-12-16T22:00:36.608+0000 7f45a1f8f700 -1 mon.smithi045@0(probing) e2 get_health_metrics reporting 20066 slow ops, oldest is monmgrreport(gid 14214, 0 checks, 1 progress events) 2023-12-16T22:00:36.619 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:00:36 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:00:36.547+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:00:36.619 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:00:36 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:00:36.547+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:00:36.620 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:00:36 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:00:36.547+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:00:36.620 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:00:36 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:00:36.549+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:00:36.937 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:00:36 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:00:36.858+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:00:37.315 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:00:36 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:00:36.935+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:00:37.315 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:00:36 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:00:36.935+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:00:37.315 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:00:36 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:00:36.935+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:00:37.315 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:00:36 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:00:36.936+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:00:37.815 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:00:37 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:00:37.550+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:00:37.815 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:00:37 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:00:37.550+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:00:37.815 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:00:37 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:00:37.550+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:00:37.815 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:00:37 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:00:37.551+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:00:38.315 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:00:37 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:00:37.882+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:00:38.315 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:00:37 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:00:37.917+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:00:38.315 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:00:37 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:00:37.917+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:00:38.315 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:00:37 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:00:37.917+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:00:38.316 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:00:37 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:00:37.918+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:00:38.815 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:00:38 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:00:38.510+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:00:38.815 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:00:38 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:00:38.510+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:00:38.816 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:00:38 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:00:38.510+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:00:38.816 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:00:38 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:00:38.511+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:00:39.315 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:00:38 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:00:38.867+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:00:39.315 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:00:38 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:00:38.929+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:00:39.315 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:00:38 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:00:38.929+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:00:39.316 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:00:38 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:00:38.929+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:00:39.316 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:00:38 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:00:38.930+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:00:39.815 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:00:39 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:00:39.526+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:00:39.815 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:00:39 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:00:39.526+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:00:39.815 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:00:39 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:00:39.526+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:00:39.815 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:00:39 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:00:39.527+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:00:40.172 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:00:39 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:00:39.889+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:00:40.172 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:00:39 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:00:39.923+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:00:40.173 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:00:39 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:00:39.923+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:00:40.173 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:00:39 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:00:39.923+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:00:40.173 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:00:39 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:00:39.924+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:00:40.815 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:00:40 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:00:40.531+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:00:40.815 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:00:40 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:00:40.531+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:00:40.815 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:00:40 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:00:40.531+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:00:40.816 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:00:40 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:00:40.532+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:00:41.315 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:00:40 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:00:40.891+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:00:41.315 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:00:40 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:00:40.876+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:00:41.315 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:00:40 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:00:40.876+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:00:41.315 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:00:40 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:00:40.876+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:00:41.315 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:00:40 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:00:40.877+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:00:41.866 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Dec 16 22:00:41 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-mon-smithi045[80759]: 2023-12-16T22:00:41.610+0000 7f45a1f8f700 -1 mon.smithi045@0(probing) e2 get_health_metrics reporting 20156 slow ops, oldest is monmgrreport(gid 14214, 0 checks, 1 progress events) 2023-12-16T22:00:41.866 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:00:41 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:00:41.569+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:00:41.866 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:00:41 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:00:41.569+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:00:41.867 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:00:41 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:00:41.569+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:00:41.867 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:00:41 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:00:41.570+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:00:42.315 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:00:41 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:00:41.864+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:00:42.315 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:00:41 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:00:41.885+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:00:42.315 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:00:41 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:00:41.885+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:00:42.316 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:00:41 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:00:41.885+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:00:42.316 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:00:41 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:00:41.885+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:00:42.815 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:00:42 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:00:42.579+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:00:42.815 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:00:42 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:00:42.579+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:00:42.815 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:00:42 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:00:42.579+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:00:42.815 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:00:42 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:00:42.581+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:00:43.315 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:00:42 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:00:42.821+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:00:43.315 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:00:42 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:00:42.839+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:00:43.315 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:00:42 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:00:42.839+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:00:43.315 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:00:42 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:00:42.839+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:00:43.315 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:00:42 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:00:42.841+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:00:43.860 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:00:43 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:00:43.835+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:00:43.860 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:00:43 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:00:43.628+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:00:43.860 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:00:43 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:00:43.628+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:00:43.861 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:00:43 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:00:43.628+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:00:43.861 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:00:43 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:00:43.629+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:00:44.315 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:00:43 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:00:43.858+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:00:44.315 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:00:43 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:00:43.858+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:00:44.315 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:00:43 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:00:43.858+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:00:44.315 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:00:43 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:00:43.860+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:00:45.065 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:00:44 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:00:44.808+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:00:45.065 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:00:44 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:00:44.837+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:00:45.065 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:00:44 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:00:44.837+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:00:45.065 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:00:44 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:00:44.837+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:00:45.066 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:00:44 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:00:44.838+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:00:45.066 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:00:44 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:00:44.677+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:00:45.066 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:00:44 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:00:44.677+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:00:45.066 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:00:44 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:00:44.677+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:00:45.066 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:00:44 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:00:44.678+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:00:46.065 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:00:45 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:00:45.781+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:00:46.065 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:00:45 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:00:45.819+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:00:46.065 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:00:45 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:00:45.819+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:00:46.066 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:00:45 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:00:45.819+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:00:46.066 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:00:45 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:00:45.819+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:00:46.066 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:00:45 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:00:45.674+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:00:46.066 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:00:45 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:00:45.674+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:00:46.066 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:00:45 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:00:45.674+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:00:46.066 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:00:45 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:00:45.675+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:00:47.065 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Dec 16 22:00:46 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-mon-smithi045[80759]: 2023-12-16T22:00:46.612+0000 7f45a1f8f700 -1 mon.smithi045@0(probing) e2 get_health_metrics reporting 20245 slow ops, oldest is monmgrreport(gid 14214, 0 checks, 1 progress events) 2023-12-16T22:00:47.065 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:00:46 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:00:46.798+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:00:47.066 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:00:46 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:00:46.724+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:00:47.066 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:00:46 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:00:46.724+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:00:47.066 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:00:46 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:00:46.724+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:00:47.066 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:00:46 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:00:46.724+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:00:47.066 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:00:46 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:00:46.810+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:00:47.067 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:00:46 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:00:46.810+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:00:47.067 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:00:46 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:00:46.810+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:00:47.067 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:00:46 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:00:46.811+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:00:48.065 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:00:47 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:00:47.796+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:00:48.065 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:00:47 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:00:47.743+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:00:48.065 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:00:47 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:00:47.743+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:00:48.066 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:00:47 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:00:47.743+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:00:48.066 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:00:47 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:00:47.745+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:00:48.066 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:00:47 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:00:47.825+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:00:48.066 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:00:47 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:00:47.825+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:00:48.066 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:00:47 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:00:47.825+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:00:48.066 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:00:47 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:00:47.826+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:00:49.065 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:00:48 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:00:48.710+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:00:49.065 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:00:48 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:00:48.710+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:00:49.065 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:00:48 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:00:48.710+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:00:49.065 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:00:48 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:00:48.711+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:00:49.066 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:00:48 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:00:48.759+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:00:49.066 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:00:48 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:00:48.759+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:00:49.066 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:00:48 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:00:48.759+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:00:49.066 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:00:48 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:00:48.760+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:00:49.066 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:00:48 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:00:48.843+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:00:49.066 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:00:48 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:00:48.843+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:00:49.067 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:00:48 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:00:48.843+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:00:49.067 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:00:48 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:00:48.844+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:00:50.065 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:00:49 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:00:49.756+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:00:50.065 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:00:49 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:00:49.756+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:00:50.065 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:00:49 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:00:49.756+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:00:50.065 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:00:49 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:00:49.757+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:00:50.066 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:00:49 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:00:49.847+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:00:50.066 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:00:49 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:00:49.847+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:00:50.066 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:00:49 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:00:49.847+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:00:50.066 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:00:49 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:00:49.848+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:00:50.066 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:00:49 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:00:49.749+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:00:50.066 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:00:49 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:00:49.749+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:00:50.067 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:00:49 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:00:49.749+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:00:50.067 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:00:49 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:00:49.750+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:00:51.065 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:00:50 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:00:50.855+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:00:51.066 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:00:50 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:00:50.855+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:00:51.066 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:00:50 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:00:50.855+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:00:51.066 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:00:50 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:00:50.856+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:00:51.067 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:00:50 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:00:50.717+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:00:51.067 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:00:50 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:00:50.717+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:00:51.067 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:00:50 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:00:50.717+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:00:51.067 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:00:50 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:00:50.718+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:00:51.068 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:00:50 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:00:50.793+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:00:51.068 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:00:50 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:00:50.793+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:00:51.068 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:00:50 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:00:50.793+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:00:51.068 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:00:50 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:00:50.794+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:00:52.065 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Dec 16 22:00:51 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-mon-smithi045[80759]: 2023-12-16T22:00:51.615+0000 7f45a1f8f700 -1 mon.smithi045@0(probing) e2 get_health_metrics reporting 20336 slow ops, oldest is monmgrreport(gid 14214, 0 checks, 1 progress events) 2023-12-16T22:00:52.065 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:00:51 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:00:51.754+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:00:52.065 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:00:51 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:00:51.754+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:00:52.065 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:00:51 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:00:51.754+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:00:52.066 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:00:51 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:00:51.755+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:00:52.066 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:00:51 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:00:51.805+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:00:52.066 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:00:51 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:00:51.805+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:00:52.066 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:00:51 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:00:51.805+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:00:52.066 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:00:51 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:00:51.806+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:00:52.066 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:00:51 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:00:51.820+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:00:52.067 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:00:51 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:00:51.820+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:00:52.067 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:00:51 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:00:51.820+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:00:52.067 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:00:51 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:00:51.821+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:00:53.065 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:00:52 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:00:52.851+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:00:53.065 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:00:52 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:00:52.851+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:00:53.065 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:00:52 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:00:52.851+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:00:53.065 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:00:52 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:00:52.851+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:00:53.066 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:00:52 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:00:52.784+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:00:53.066 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:00:52 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:00:52.784+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:00:53.066 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:00:52 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:00:52.784+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:00:53.066 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:00:52 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:00:52.785+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:00:53.066 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:00:52 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:00:52.796+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:00:53.066 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:00:52 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:00:52.796+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:00:53.066 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:00:52 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:00:52.796+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:00:53.067 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:00:52 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:00:52.797+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:00:54.065 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:00:53 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:00:53.834+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:00:54.065 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:00:53 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:00:53.834+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:00:54.065 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:00:53 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:00:53.834+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:00:54.065 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:00:53 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:00:53.836+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:00:54.066 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:00:53 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:00:53.835+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:00:54.066 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:00:53 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:00:53.835+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:00:54.066 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:00:53 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:00:53.835+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:00:54.066 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:00:53 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:00:53.836+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:00:54.066 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:00:53 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:00:53.796+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:00:54.066 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:00:53 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:00:53.796+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:00:54.066 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:00:53 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:00:53.796+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:00:54.067 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:00:53 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:00:53.797+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:00:55.065 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:00:54 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:00:54.817+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:00:55.065 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:00:54 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:00:54.817+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:00:55.065 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:00:54 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:00:54.817+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:00:55.065 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:00:54 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:00:54.817+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:00:55.066 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:00:54 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:00:54.860+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:00:55.066 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:00:54 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:00:54.860+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:00:55.066 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:00:54 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:00:54.860+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:00:55.066 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:00:54 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:00:54.862+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:00:55.066 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:00:54 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:00:54.820+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:00:55.066 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:00:54 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:00:54.820+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:00:55.067 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:00:54 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:00:54.820+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:00:55.067 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:00:54 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:00:54.821+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:00:56.065 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:00:55 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:00:55.798+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:00:56.065 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:00:55 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:00:55.798+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:00:56.065 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:00:55 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:00:55.798+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:00:56.065 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:00:55 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:00:55.800+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:00:56.066 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:00:55 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:00:55.848+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:00:56.066 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:00:55 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:00:55.848+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:00:56.066 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:00:55 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:00:55.848+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:00:56.066 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:00:55 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:00:55.849+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:00:56.066 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:00:55 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:00:55.831+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:00:56.066 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:00:55 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:00:55.831+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:00:56.067 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:00:55 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:00:55.831+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:00:56.067 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:00:55 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:00:55.832+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:00:57.065 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Dec 16 22:00:56 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-mon-smithi045[80759]: 2023-12-16T22:00:56.617+0000 7f45a1f8f700 -1 mon.smithi045@0(probing) e2 get_health_metrics reporting 20425 slow ops, oldest is monmgrreport(gid 14214, 0 checks, 1 progress events) 2023-12-16T22:00:57.065 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:00:56 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:00:56.812+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:00:57.065 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:00:56 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:00:56.812+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:00:57.065 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:00:56 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:00:56.812+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:00:57.066 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:00:56 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:00:56.813+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:00:57.066 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:00:56 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:00:56.814+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:00:57.066 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:00:56 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:00:56.814+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:00:57.066 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:00:56 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:00:56.814+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:00:57.066 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:00:56 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:00:56.815+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:00:57.067 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:00:56 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:00:56.828+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:00:57.067 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:00:56 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:00:56.828+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:00:57.067 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:00:56 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:00:56.828+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:00:57.067 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:00:56 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:00:56.829+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:00:58.315 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:00:57 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:00:57.823+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:00:58.315 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:00:57 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:00:57.823+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:00:58.315 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:00:57 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:00:57.823+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:00:58.315 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:00:57 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:00:57.824+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:00:58.316 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:00:57 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:00:57.835+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:00:58.316 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:00:57 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:00:57.835+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:00:58.316 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:00:57 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:00:57.835+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:00:58.316 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:00:57 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:00:57.835+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:00:58.316 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:00:57 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:00:57.829+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:00:58.317 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:00:57 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:00:57.829+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:00:58.317 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:00:57 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:00:57.829+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:00:58.317 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:00:57 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:00:57.830+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:00:59.065 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:00:58 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:00:58.805+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:00:59.065 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:00:58 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:00:58.805+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:00:59.065 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:00:58 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:00:58.805+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:00:59.065 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:00:58 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:00:58.806+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:00:59.066 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:00:58 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:00:58.836+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:00:59.066 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:00:58 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:00:58.836+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:00:59.066 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:00:58 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:00:58.836+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:00:59.066 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:00:58 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:00:58.838+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:00:59.066 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:00:58 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:00:58.833+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:00:59.066 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:00:58 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:00:58.833+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:00:59.067 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:00:58 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:00:58.833+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:00:59.067 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:00:58 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:00:58.834+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:01:00.065 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:00:59 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:00:59.869+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:01:00.065 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:00:59 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:00:59.869+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:01:00.065 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:00:59 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:00:59.869+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:01:00.065 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:00:59 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:00:59.869+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:01:00.066 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:00:59 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:00:59.795+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:01:00.066 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:00:59 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:00:59.795+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:01:00.066 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:00:59 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:00:59.795+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:01:00.066 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:00:59 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:00:59.796+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:01:00.066 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:00:59 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:00:59.792+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:01:00.067 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:00:59 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:00:59.792+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:01:00.067 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:00:59 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:00:59.792+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:01:00.067 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:00:59 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:00:59.793+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:01:01.029 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:01:00 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:01:00.769+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:01:01.029 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:01:00 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:01:00.769+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:01:01.029 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:01:00 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:01:00.769+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:01:01.029 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:01:00 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:01:00.770+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:01:01.030 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:01:00 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:01:00.829+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:01:01.030 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:01:00 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:01:00.829+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:01:01.030 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:01:00 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:01:00.829+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:01:01.030 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:01:00 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:01:00.830+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:01:01.030 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:01:00 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:01:00.804+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:01:01.030 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:01:00 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:01:00.804+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:01:01.031 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:01:00 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:01:00.804+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:01:01.031 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:01:00 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:01:00.805+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:01:02.065 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Dec 16 22:01:01 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-mon-smithi045[80759]: 2023-12-16T22:01:01.620+0000 7f45a1f8f700 -1 mon.smithi045@0(probing) e2 get_health_metrics reporting 20515 slow ops, oldest is monmgrreport(gid 14214, 0 checks, 1 progress events) 2023-12-16T22:01:02.065 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:01:01 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:01:01.842+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:01:02.065 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:01:01 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:01:01.842+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:01:02.065 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:01:01 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:01:01.842+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:01:02.065 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:01:01 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:01:01.843+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:01:02.066 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:01:01 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:01:01.756+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:01:02.066 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:01:01 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:01:01.756+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:01:02.066 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:01:01 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:01:01.756+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:01:02.066 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:01:01 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:01:01.756+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:01:02.066 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:01:01 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:01:01.806+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:01:02.067 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:01:01 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:01:01.806+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:01:02.067 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:01:01 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:01:01.806+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:01:02.067 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:01:01 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:01:01.807+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:01:03.065 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:01:02 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:01:02.796+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:01:03.065 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:01:02 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:01:02.796+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:01:03.065 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:01:02 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:01:02.796+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:01:03.065 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:01:02 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:01:02.797+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:01:03.066 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:01:02 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:01:02.839+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:01:03.066 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:01:02 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:01:02.839+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:01:03.066 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:01:02 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:01:02.839+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:01:03.066 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:01:02 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:01:02.840+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:01:03.066 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:01:02 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:01:02.869+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:01:03.066 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:01:02 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:01:02.869+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:01:03.067 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:01:02 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:01:02.869+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:01:03.067 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:01:02 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:01:02.869+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:01:03.882 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:01:03 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:01:03.811+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:01:03.882 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:01:03 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:01:03.811+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:01:03.882 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:01:03 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:01:03.811+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:01:03.883 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:01:03 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:01:03.812+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:01:03.883 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:01:03 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:01:03.771+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:01:03.883 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:01:03 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:01:03.771+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:01:03.883 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:01:03 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:01:03.771+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:01:03.884 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:01:03 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:01:03.772+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:01:04.315 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:01:03 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:01:03.880+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:01:04.315 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:01:03 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:01:03.880+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:01:04.315 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:01:03 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:01:03.880+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:01:04.315 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:01:03 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:01:03.882+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:01:05.065 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:01:04 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:01:04.794+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:01:05.065 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:01:04 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:01:04.794+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:01:05.065 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:01:04 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:01:04.794+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:01:05.065 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:01:04 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:01:04.795+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:01:05.066 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:01:04 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:01:04.891+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:01:05.066 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:01:04 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:01:04.891+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:01:05.066 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:01:04 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:01:04.891+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:01:05.066 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:01:04 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:01:04.891+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:01:05.066 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:01:04 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:01:04.736+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:01:05.066 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:01:04 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:01:04.736+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:01:05.067 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:01:04 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:01:04.736+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:01:05.067 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:01:04 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:01:04.737+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:01:06.065 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:01:05 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:01:05.817+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:01:06.065 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:01:05 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:01:05.817+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:01:06.065 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:01:05 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:01:05.817+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:01:06.066 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:01:05 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:01:05.818+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:01:06.066 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:01:05 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:01:05.923+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:01:06.066 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:01:05 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:01:05.923+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:01:06.066 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:01:05 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:01:05.923+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:01:06.066 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:01:05 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:01:05.925+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:01:06.067 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:01:05 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:01:05.763+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:01:06.067 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:01:05 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:01:05.763+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:01:06.067 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:01:05 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:01:05.763+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:01:06.067 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:01:05 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:01:05.764+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:01:06.955 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:01:06 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:01:06.793+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:01:06.955 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:01:06 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:01:06.793+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:01:06.956 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:01:06 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:01:06.793+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:01:06.956 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:01:06 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:01:06.794+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:01:06.956 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:01:06 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:01:06.725+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:01:06.956 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:01:06 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:01:06.725+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:01:06.956 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:01:06 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:01:06.725+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:01:06.957 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:01:06 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:01:06.725+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:01:06.957 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Dec 16 22:01:06 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-mon-smithi045[80759]: 2023-12-16T22:01:06.622+0000 7f45a1f8f700 -1 mon.smithi045@0(probing) e2 get_health_metrics reporting 20604 slow ops, oldest is monmgrreport(gid 14214, 0 checks, 1 progress events) 2023-12-16T22:01:07.315 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:01:06 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:01:06.954+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:01:07.315 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:01:06 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:01:06.954+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:01:07.315 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:01:06 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:01:06.954+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:01:07.315 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:01:06 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:01:06.955+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:01:08.065 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:01:07 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:01:07.715+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:01:08.065 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:01:07 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:01:07.715+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:01:08.065 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:01:07 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:01:07.715+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:01:08.065 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:01:07 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:01:07.716+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:01:08.066 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:01:07 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:01:07.761+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:01:08.066 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:01:07 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:01:07.761+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:01:08.066 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:01:07 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:01:07.761+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:01:08.066 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:01:07 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:01:07.763+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:01:08.067 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:01:07 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:01:07.907+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:01:08.067 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:01:07 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:01:07.907+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:01:08.067 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:01:07 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:01:07.907+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:01:08.067 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:01:07 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:01:07.908+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:01:09.065 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:01:08 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:01:08.703+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:01:09.065 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:01:08 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:01:08.703+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:01:09.065 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:01:08 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:01:08.703+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:01:09.065 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:01:08 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:01:08.704+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:01:09.066 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:01:08 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:01:08.891+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:01:09.066 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:01:08 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:01:08.891+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:01:09.066 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:01:08 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:01:08.891+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:01:09.066 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:01:08 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:01:08.892+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:01:09.067 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:01:08 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:01:08.776+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:01:09.067 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:01:08 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:01:08.776+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:01:09.067 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:01:08 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:01:08.776+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:01:09.067 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:01:08 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:01:08.777+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:01:10.065 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:01:09 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:01:09.700+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:01:10.065 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:01:09 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:01:09.700+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:01:10.065 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:01:09 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:01:09.700+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:01:10.066 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:01:09 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:01:09.701+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:01:10.066 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:01:09 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:01:09.773+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:01:10.066 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:01:09 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:01:09.773+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:01:10.066 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:01:09 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:01:09.773+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:01:10.067 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:01:09 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:01:09.775+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:01:10.067 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:01:09 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:01:09.844+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:01:10.067 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:01:09 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:01:09.844+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:01:10.067 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:01:09 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:01:09.844+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:01:10.067 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:01:09 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:01:09.845+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:01:11.065 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:01:10 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:01:10.659+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:01:11.065 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:01:10 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:01:10.659+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:01:11.065 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:01:10 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:01:10.659+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:01:11.066 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:01:10 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:01:10.660+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:01:11.066 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:01:10 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:01:10.755+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:01:11.066 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:01:10 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:01:10.755+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:01:11.066 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:01:10 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:01:10.755+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:01:11.066 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:01:10 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:01:10.756+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:01:11.067 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:01:10 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:01:10.816+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:01:11.067 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:01:10 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:01:10.816+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:01:11.067 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:01:10 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:01:10.816+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:01:11.067 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:01:10 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:01:10.816+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:01:12.065 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Dec 16 22:01:11 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-mon-smithi045[80759]: 2023-12-16T22:01:11.624+0000 7f45a1f8f700 -1 mon.smithi045@0(probing) e2 get_health_metrics reporting 20696 slow ops, oldest is monmgrreport(gid 14214, 0 checks, 1 progress events) 2023-12-16T22:01:12.065 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:01:11 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:01:11.627+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:01:12.065 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:01:11 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:01:11.627+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:01:12.065 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:01:11 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:01:11.627+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:01:12.065 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:01:11 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:01:11.628+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:01:12.066 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:01:11 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:01:11.788+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:01:12.066 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:01:11 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:01:11.788+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:01:12.066 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:01:11 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:01:11.788+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:01:12.066 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:01:11 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:01:11.789+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:01:12.066 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:01:11 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:01:11.825+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:01:12.067 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:01:11 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:01:11.825+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:01:12.067 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:01:11 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:01:11.825+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:01:12.067 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:01:11 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:01:11.826+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:01:13.065 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:01:12 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:01:12.832+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:01:13.065 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:01:12 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:01:12.832+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:01:13.065 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:01:12 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:01:12.832+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:01:13.065 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:01:12 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:01:12.833+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:01:13.066 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:01:12 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:01:12.831+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:01:13.066 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:01:12 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:01:12.831+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:01:13.066 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:01:12 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:01:12.831+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:01:13.066 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:01:12 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:01:12.831+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:01:13.066 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:01:12 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:01:12.628+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:01:13.067 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:01:12 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:01:12.628+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:01:13.067 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:01:12 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:01:12.628+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:01:13.067 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:01:12 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:01:12.629+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:01:14.065 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:01:13 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:01:13.811+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:01:14.065 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:01:13 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:01:13.811+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:01:14.065 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:01:13 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:01:13.811+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:01:14.065 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:01:13 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:01:13.812+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:01:14.066 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:01:13 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:01:13.786+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:01:14.066 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:01:13 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:01:13.786+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:01:14.066 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:01:13 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:01:13.786+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:01:14.066 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:01:13 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:01:13.787+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:01:14.066 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:01:13 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:01:13.679+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:01:14.067 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:01:13 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:01:13.679+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:01:14.067 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:01:13 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:01:13.679+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:01:14.067 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:01:13 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:01:13.679+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:01:15.065 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:01:14 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:01:14.856+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:01:15.065 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:01:14 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:01:14.856+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:01:15.065 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:01:14 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:01:14.856+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:01:15.065 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:01:14 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:01:14.857+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:01:15.066 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:01:14 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:01:14.784+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:01:15.066 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:01:14 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:01:14.784+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:01:15.066 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:01:14 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:01:14.784+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:01:15.066 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:01:14 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:01:14.785+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:01:15.066 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:01:14 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:01:14.698+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:01:15.066 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:01:14 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:01:14.698+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:01:15.067 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:01:14 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:01:14.698+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:01:15.067 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:01:14 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:01:14.699+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:01:16.065 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:01:15 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:01:15.699+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:01:16.065 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:01:15 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:01:15.699+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:01:16.065 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:01:15 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:01:15.699+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:01:16.065 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:01:15 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:01:15.700+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:01:16.066 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:01:15 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:01:15.887+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:01:16.066 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:01:15 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:01:15.887+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:01:16.066 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:01:15 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:01:15.887+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:01:16.066 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:01:15 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:01:15.888+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:01:16.066 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:01:15 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:01:15.753+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:01:16.066 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:01:15 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:01:15.753+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:01:16.067 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:01:15 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:01:15.753+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:01:16.067 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:01:15 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:01:15.754+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:01:16.905 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Dec 16 22:01:16 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-mon-smithi045[80759]: 2023-12-16T22:01:16.626+0000 7f45a1f8f700 -1 mon.smithi045@0(probing) e2 get_health_metrics reporting 20786 slow ops, oldest is monmgrreport(gid 14214, 0 checks, 1 progress events) 2023-12-16T22:01:16.905 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:01:16 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:01:16.789+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:01:16.905 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:01:16 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:01:16.789+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:01:16.906 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:01:16 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:01:16.789+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:01:16.906 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:01:16 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:01:16.791+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:01:16.906 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:01:16 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:01:16.698+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:01:16.906 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:01:16 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:01:16.698+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:01:16.906 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:01:16 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:01:16.698+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:01:16.906 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:01:16 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:01:16.699+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:01:17.314 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:01:16 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:01:16.903+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:01:17.315 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:01:16 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:01:16.903+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:01:17.315 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:01:16 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:01:16.903+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:01:17.315 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:01:16 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:01:16.904+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:01:18.065 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:01:17 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:01:17.872+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:01:18.065 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:01:17 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:01:17.872+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:01:18.065 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:01:17 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:01:17.872+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:01:18.065 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:01:17 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:01:17.872+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:01:18.066 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:01:17 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:01:17.830+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:01:18.066 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:01:17 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:01:17.830+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:01:18.066 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:01:17 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:01:17.830+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:01:18.066 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:01:17 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:01:17.831+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:01:18.066 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:01:17 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:01:17.671+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:01:18.067 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:01:17 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:01:17.671+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:01:18.067 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:01:17 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:01:17.671+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:01:18.067 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:01:17 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:01:17.671+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:01:19.065 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:01:18 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:01:18.863+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:01:19.065 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:01:18 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:01:18.863+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:01:19.065 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:01:18 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:01:18.863+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:01:19.065 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:01:18 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:01:18.864+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:01:19.066 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:01:18 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:01:18.706+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:01:19.066 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:01:18 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:01:18.706+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:01:19.066 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:01:18 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:01:18.706+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:01:19.066 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:01:18 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:01:18.707+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:01:19.066 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:01:18 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:01:18.904+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:01:19.066 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:01:18 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:01:18.904+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:01:19.067 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:01:18 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:01:18.904+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:01:19.067 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:01:18 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:01:18.904+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:01:20.065 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:01:19 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:01:19.728+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:01:20.065 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:01:19 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:01:19.728+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:01:20.065 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:01:19 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:01:19.728+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:01:20.065 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:01:19 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:01:19.729+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:01:20.066 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:01:19 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:01:19.923+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:01:20.066 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:01:19 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:01:19.923+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:01:20.066 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:01:19 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:01:19.923+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:01:20.066 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:01:19 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:01:19.924+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:01:20.066 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:01:19 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:01:19.846+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:01:20.066 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:01:19 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:01:19.846+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:01:20.067 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:01:19 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:01:19.846+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:01:20.067 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:01:19 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:01:19.847+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:01:21.065 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:01:20 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:01:20.896+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:01:21.065 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:01:20 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:01:20.896+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:01:21.065 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:01:20 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:01:20.896+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:01:21.065 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:01:20 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:01:20.897+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:01:21.066 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:01:20 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:01:20.735+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:01:21.066 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:01:20 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:01:20.735+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:01:21.066 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:01:20 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:01:20.735+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:01:21.066 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:01:20 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:01:20.736+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:01:21.066 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:01:20 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:01:20.856+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:01:21.066 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:01:20 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:01:20.856+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:01:21.066 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:01:20 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:01:20.856+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:01:21.067 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:01:20 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:01:20.857+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:01:22.065 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Dec 16 22:01:21 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-mon-smithi045[80759]: 2023-12-16T22:01:21.628+0000 7f45a1f8f700 -1 mon.smithi045@0(probing) e2 get_health_metrics reporting 20877 slow ops, oldest is monmgrreport(gid 14214, 0 checks, 1 progress events) 2023-12-16T22:01:22.065 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:01:21 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:01:21.859+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:01:22.065 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:01:21 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:01:21.859+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:01:22.066 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:01:21 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:01:21.859+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:01:22.066 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:01:21 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:01:21.860+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:01:22.066 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:01:21 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:01:21.732+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:01:22.066 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:01:21 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:01:21.732+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:01:22.067 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:01:21 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:01:21.732+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:01:22.067 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:01:21 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:01:21.732+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:01:22.067 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:01:21 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:01:21.811+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:01:22.067 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:01:21 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:01:21.811+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:01:22.068 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:01:21 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:01:21.811+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:01:22.068 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:01:21 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:01:21.812+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:01:23.065 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:01:22 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:01:22.721+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:01:23.065 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:01:22 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:01:22.721+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:01:23.065 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:01:22 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:01:22.721+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:01:23.065 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:01:22 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:01:22.722+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:01:23.066 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:01:22 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:01:22.842+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:01:23.066 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:01:22 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:01:22.842+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:01:23.066 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:01:22 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:01:22.842+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:01:23.066 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:01:22 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:01:22.843+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:01:23.066 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:01:22 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:01:22.836+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:01:23.066 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:01:22 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:01:22.836+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:01:23.067 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:01:22 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:01:22.836+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:01:23.067 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:01:22 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:01:22.836+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:01:23.867 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:01:23 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:01:23.692+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:01:23.867 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:01:23 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:01:23.692+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:01:23.867 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:01:23 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:01:23.692+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:01:23.867 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:01:23 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:01:23.693+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:01:24.315 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:01:23 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:01:23.884+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:01:24.315 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:01:23 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:01:23.884+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:01:24.315 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:01:23 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:01:23.884+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:01:24.315 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:01:23 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:01:23.886+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:01:24.316 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:01:23 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:01:23.865+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:01:24.316 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:01:23 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:01:23.865+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:01:24.316 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:01:23 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:01:23.865+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:01:24.316 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:01:23 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:01:23.866+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:01:25.065 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:01:24 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:01:24.898+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:01:25.065 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:01:24 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:01:24.898+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:01:25.065 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:01:24 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:01:24.898+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:01:25.065 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:01:24 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:01:24.899+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:01:25.066 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:01:24 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:01:24.892+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:01:25.066 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:01:24 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:01:24.892+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:01:25.066 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:01:24 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:01:24.892+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:01:25.066 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:01:24 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:01:24.892+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:01:25.066 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:01:24 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:01:24.707+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:01:25.066 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:01:24 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:01:24.707+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:01:25.066 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:01:24 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:01:24.707+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:01:25.067 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:01:24 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:01:24.708+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:01:25.815 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:01:25 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:01:25.719+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:01:25.815 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:01:25 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:01:25.719+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:01:25.815 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:01:25 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:01:25.719+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:01:25.815 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:01:25 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:01:25.720+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:01:26.315 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:01:25 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:01:25.870+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:01:26.315 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:01:25 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:01:25.870+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:01:26.315 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:01:25 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:01:25.870+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:01:26.315 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:01:25 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:01:25.871+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:01:26.315 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:01:25 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:01:25.880+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:01:26.316 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:01:25 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:01:25.880+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:01:26.316 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:01:25 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:01:25.880+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:01:26.316 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:01:25 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:01:25.882+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:01:26.888 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Dec 16 22:01:26 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-mon-smithi045[80759]: 2023-12-16T22:01:26.632+0000 7f45a1f8f700 -1 mon.smithi045@0(probing) e2 get_health_metrics reporting 20966 slow ops, oldest is monmgrreport(gid 14214, 0 checks, 1 progress events) 2023-12-16T22:01:26.888 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:01:26 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:01:26.839+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:01:26.888 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:01:26 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:01:26.839+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:01:26.889 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:01:26 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:01:26.839+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:01:26.889 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:01:26 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:01:26.840+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:01:26.889 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:01:26 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:01:26.748+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:01:26.889 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:01:26 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:01:26.748+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:01:26.890 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:01:26 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:01:26.748+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:01:26.890 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:01:26 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:01:26.748+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:01:27.315 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:01:26 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:01:26.886+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:01:27.315 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:01:26 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:01:26.886+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:01:27.315 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:01:26 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:01:26.886+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:01:27.315 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:01:26 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:01:26.887+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:01:28.065 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:01:27 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:01:27.908+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:01:28.065 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:01:27 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:01:27.908+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:01:28.065 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:01:27 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:01:27.908+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:01:28.065 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:01:27 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:01:27.909+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:01:28.066 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:01:27 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:01:27.796+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:01:28.066 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:01:27 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:01:27.796+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:01:28.066 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:01:27 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:01:27.796+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:01:28.066 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:01:27 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:01:27.796+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:01:28.066 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:01:27 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:01:27.780+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:01:28.067 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:01:27 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:01:27.780+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:01:28.067 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:01:27 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:01:27.780+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:01:28.067 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:01:27 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:01:27.780+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:01:29.065 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:01:28 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:01:28.893+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:01:29.065 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:01:28 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:01:28.893+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:01:29.065 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:01:28 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:01:28.893+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:01:29.065 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:01:28 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:01:28.894+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:01:29.066 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:01:28 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:01:28.759+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:01:29.066 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:01:28 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:01:28.759+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:01:29.066 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:01:28 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:01:28.759+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:01:29.066 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:01:28 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:01:28.760+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:01:29.066 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:01:28 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:01:28.828+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:01:29.067 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:01:28 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:01:28.828+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:01:29.067 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:01:28 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:01:28.828+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:01:29.067 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:01:28 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:01:28.829+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:01:30.065 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:01:29 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:01:29.844+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:01:30.065 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:01:29 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:01:29.844+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:01:30.065 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:01:29 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:01:29.844+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:01:30.065 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:01:29 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:01:29.845+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:01:30.066 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:01:29 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:01:29.920+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:01:30.066 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:01:29 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:01:29.920+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:01:30.066 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:01:29 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:01:29.920+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:01:30.066 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:01:29 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:01:29.921+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:01:30.066 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:01:29 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:01:29.794+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:01:30.067 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:01:29 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:01:29.795+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:01:30.067 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:01:29 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:01:29.795+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:01:30.067 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:01:29 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:01:29.796+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:01:31.065 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:01:30 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:01:30.868+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:01:31.065 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:01:30 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:01:30.868+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:01:31.065 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:01:30 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:01:30.868+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:01:31.065 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:01:30 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:01:30.869+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:01:31.066 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:01:30 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:01:30.888+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:01:31.066 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:01:30 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:01:30.888+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:01:31.066 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:01:30 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:01:30.888+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:01:31.066 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:01:30 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:01:30.889+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:01:31.066 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:01:30 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:01:30.800+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:01:31.066 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:01:30 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:01:30.800+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:01:31.067 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:01:30 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:01:30.800+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:01:31.067 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:01:30 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:01:30.801+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:01:32.064 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Dec 16 22:01:31 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-mon-smithi045[80759]: 2023-12-16T22:01:31.634+0000 7f45a1f8f700 -1 mon.smithi045@0(probing) e2 get_health_metrics reporting 21059 slow ops, oldest is monmgrreport(gid 14214, 0 checks, 1 progress events) 2023-12-16T22:01:32.065 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:01:31 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:01:31.868+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:01:32.065 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:01:31 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:01:31.868+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:01:32.065 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:01:31 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:01:31.868+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:01:32.065 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:01:31 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:01:31.868+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:01:32.066 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:01:31 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:01:31.846+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:01:32.066 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:01:31 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:01:31.846+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:01:32.066 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:01:31 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:01:31.846+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:01:32.066 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:01:31 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:01:31.847+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:01:32.066 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:01:31 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:01:31.822+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:01:32.067 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:01:31 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:01:31.822+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:01:32.067 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:01:31 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:01:31.822+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:01:32.067 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:01:31 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:01:31.823+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:01:33.065 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:01:32 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:01:32.811+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:01:33.065 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:01:32 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:01:32.811+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:01:33.065 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:01:32 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:01:32.811+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:01:33.065 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:01:32 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:01:32.811+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:01:33.065 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:01:32 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:01:32.827+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:01:33.066 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:01:32 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:01:32.827+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:01:33.066 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:01:32 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:01:32.827+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:01:33.066 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:01:32 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:01:32.828+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:01:33.066 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:01:32 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:01:32.846+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:01:33.066 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:01:32 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:01:32.846+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:01:33.066 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:01:32 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:01:32.846+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:01:33.067 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:01:32 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:01:32.847+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:01:33.871 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:01:33 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:01:33.803+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:01:33.871 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:01:33 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:01:33.803+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:01:33.871 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:01:33 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:01:33.803+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:01:33.871 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:01:33 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:01:33.804+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:01:33.872 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:01:33 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:01:33.816+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:01:33.872 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:01:33 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:01:33.816+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:01:33.872 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:01:33 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:01:33.816+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:01:33.872 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:01:33 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:01:33.817+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:01:34.314 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:01:33 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:01:33.869+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:01:34.315 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:01:33 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:01:33.869+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:01:34.315 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:01:33 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:01:33.869+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:01:34.315 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:01:33 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:01:33.871+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:01:35.065 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:01:34 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:01:34.849+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:01:35.065 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:01:34 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:01:34.849+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:01:35.065 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:01:34 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:01:34.849+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:01:35.065 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:01:34 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:01:34.850+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:01:35.066 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:01:34 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:01:34.825+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:01:35.066 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:01:34 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:01:34.825+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:01:35.066 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:01:34 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:01:34.825+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:01:35.066 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:01:34 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:01:34.826+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:01:35.066 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:01:34 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:01:34.831+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:01:35.067 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:01:34 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:01:34.831+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:01:35.067 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:01:34 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:01:34.831+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:01:35.067 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:01:34 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:01:34.833+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:01:36.065 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:01:35 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:01:35.857+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:01:36.065 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:01:35 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:01:35.857+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:01:36.065 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:01:35 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:01:35.858+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:01:36.065 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:01:35 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:01:35.858+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:01:36.066 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:01:35 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:01:35.826+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:01:36.066 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:01:35 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:01:35.826+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:01:36.066 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:01:35 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:01:35.827+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:01:36.066 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:01:35 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:01:35.827+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:01:36.066 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:01:35 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:01:35.806+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:01:36.067 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:01:35 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:01:35.806+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:01:36.067 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:01:35 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:01:35.806+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:01:36.067 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:01:35 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:01:35.806+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:01:37.065 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Dec 16 22:01:36 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-mon-smithi045[80759]: 2023-12-16T22:01:36.636+0000 7f45a1f8f700 -1 mon.smithi045@0(probing) e2 get_health_metrics reporting 21148 slow ops, oldest is monmgrreport(gid 14214, 0 checks, 1 progress events) 2023-12-16T22:01:37.065 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:01:36 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:01:36.833+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:01:37.065 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:01:36 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:01:36.833+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:01:37.065 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:01:36 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:01:36.833+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:01:37.066 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:01:36 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:01:36.834+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:01:37.066 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:01:36 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:01:36.803+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:01:37.066 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:01:36 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:01:36.803+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:01:37.066 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:01:36 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:01:36.803+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:01:37.066 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:01:36 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:01:36.805+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:01:37.067 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:01:36 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:01:36.839+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:01:37.067 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:01:36 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:01:36.839+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:01:37.067 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:01:36 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:01:36.839+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:01:37.067 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:01:36 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:01:36.839+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:01:38.065 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:01:37 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:01:37.832+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:01:38.065 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:01:37 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:01:37.832+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:01:38.065 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:01:37 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:01:37.832+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:01:38.065 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:01:37 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:01:37.833+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:01:38.066 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:01:37 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:01:37.799+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:01:38.066 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:01:37 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:01:37.799+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:01:38.066 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:01:37 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:01:37.799+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:01:38.066 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:01:37 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:01:37.800+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:01:38.067 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:01:37 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:01:37.845+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:01:38.067 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:01:37 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:01:37.845+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:01:38.067 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:01:37 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:01:37.845+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:01:38.067 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:01:37 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:01:37.847+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:01:39.065 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:01:38 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:01:38.819+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:01:39.065 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:01:38 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:01:38.819+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:01:39.065 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:01:38 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:01:38.819+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:01:39.065 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:01:38 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:01:38.820+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:01:39.066 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:01:38 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:01:38.829+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:01:39.066 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:01:38 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:01:38.829+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:01:39.066 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:01:38 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:01:38.829+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:01:39.066 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:01:38 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:01:38.829+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:01:39.066 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:01:38 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:01:38.766+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:01:39.067 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:01:38 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:01:38.766+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:01:39.067 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:01:38 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:01:38.766+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:01:39.067 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:01:38 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:01:38.767+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:01:40.065 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:01:39 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:01:39.788+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:01:40.065 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:01:39 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:01:39.788+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:01:40.065 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:01:39 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:01:39.788+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:01:40.065 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:01:39 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:01:39.789+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:01:40.066 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:01:39 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:01:39.836+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:01:40.066 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:01:39 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:01:39.836+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:01:40.066 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:01:39 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:01:39.836+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:01:40.066 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:01:39 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:01:39.837+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:01:40.067 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:01:39 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:01:39.739+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:01:40.067 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:01:39 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:01:39.739+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:01:40.067 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:01:39 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:01:39.739+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:01:40.067 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:01:39 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:01:39.740+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:01:41.065 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:01:40 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:01:40.778+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:01:41.065 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:01:40 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:01:40.778+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:01:41.065 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:01:40 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:01:40.778+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:01:41.066 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:01:40 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:01:40.779+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:01:41.066 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:01:40 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:01:40.802+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:01:41.066 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:01:40 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:01:40.802+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:01:41.066 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:01:40 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:01:40.802+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:01:41.066 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:01:40 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:01:40.803+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:01:41.067 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:01:40 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:01:40.762+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:01:41.067 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:01:40 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:01:40.762+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:01:41.067 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:01:40 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:01:40.762+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:01:41.067 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:01:40 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:01:40.763+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:01:42.064 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Dec 16 22:01:41 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-mon-smithi045[80759]: 2023-12-16T22:01:41.638+0000 7f45a1f8f700 -1 mon.smithi045@0(probing) e2 get_health_metrics reporting 21240 slow ops, oldest is monmgrreport(gid 14214, 0 checks, 1 progress events) 2023-12-16T22:01:42.065 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:01:41 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:01:41.846+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:01:42.065 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:01:41 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:01:41.846+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:01:42.065 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:01:41 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:01:41.846+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:01:42.065 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:01:41 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:01:41.847+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:01:42.066 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:01:41 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:01:41.798+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:01:42.066 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:01:41 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:01:41.798+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:01:42.066 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:01:41 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:01:41.798+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:01:42.066 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:01:41 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:01:41.798+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:01:42.066 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:01:41 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:01:41.738+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:01:42.067 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:01:41 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:01:41.738+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:01:42.067 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:01:41 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:01:41.738+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:01:42.067 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:01:41 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:01:41.740+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:01:43.065 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:01:42 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:01:42.826+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:01:43.065 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:01:42 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:01:42.826+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:01:43.065 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:01:42 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:01:42.826+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:01:43.065 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:01:42 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:01:42.827+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:01:43.066 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:01:42 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:01:42.808+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:01:43.066 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:01:42 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:01:42.808+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:01:43.066 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:01:42 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:01:42.808+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:01:43.066 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:01:42 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:01:42.808+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:01:43.066 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:01:42 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:01:42.696+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:01:43.066 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:01:42 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:01:42.696+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:01:43.066 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:01:42 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:01:42.696+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:01:43.067 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:01:42 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:01:42.697+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:01:43.855 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:01:43 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:01:43.729+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:01:43.855 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:01:43 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:01:43.729+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:01:43.855 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:01:43 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:01:43.729+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:01:43.855 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:01:43 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:01:43.730+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:01:44.315 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:01:43 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:01:43.853+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:01:44.315 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:01:43 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:01:43.853+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:01:44.315 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:01:43 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:01:43.853+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:01:44.315 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:01:43 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:01:43.853+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:01:44.315 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:01:43 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:01:43.866+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:01:44.315 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:01:43 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:01:43.866+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:01:44.316 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:01:43 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:01:43.866+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:01:44.316 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:01:43 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:01:43.867+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:01:45.065 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:01:44 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:01:44.897+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:01:45.065 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:01:44 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:01:44.897+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:01:45.065 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:01:44 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:01:44.897+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:01:45.065 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:01:44 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:01:44.897+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:01:45.065 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:01:44 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:01:44.868+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:01:45.066 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:01:44 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:01:44.868+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:01:45.066 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:01:44 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:01:44.868+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:01:45.066 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:01:44 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:01:44.869+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:01:45.066 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:01:44 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:01:44.720+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:01:45.066 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:01:44 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:01:44.720+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:01:45.066 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:01:44 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:01:44.720+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:01:45.067 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:01:44 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:01:44.721+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:01:46.065 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:01:45 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:01:45.892+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:01:46.065 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:01:45 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:01:45.892+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:01:46.065 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:01:45 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:01:45.892+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:01:46.065 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:01:45 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:01:45.893+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:01:46.065 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:01:45 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:01:45.915+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:01:46.066 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:01:45 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:01:45.915+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:01:46.066 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:01:45 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:01:45.915+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:01:46.066 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:01:45 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:01:45.915+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:01:46.066 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:01:45 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:01:45.764+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:01:46.066 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:01:45 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:01:45.764+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:01:46.066 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:01:45 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:01:45.764+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:01:46.066 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:01:45 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:01:45.766+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:01:46.897 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Dec 16 22:01:46 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-mon-smithi045[80759]: 2023-12-16T22:01:46.639+0000 7f45a1f8f700 -1 mon.smithi045@0(probing) e2 get_health_metrics reporting 21329 slow ops, oldest is monmgrreport(gid 14214, 0 checks, 1 progress events) 2023-12-16T22:01:46.897 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:01:46 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:01:46.747+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:01:46.897 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:01:46 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:01:46.747+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:01:46.898 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:01:46 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:01:46.747+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:01:46.898 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:01:46 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:01:46.748+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:01:47.315 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:01:46 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:01:46.895+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:01:47.315 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:01:46 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:01:46.895+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:01:47.315 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:01:46 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:01:46.895+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:01:47.315 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:01:46 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:01:46.895+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:01:47.315 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:01:46 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:01:46.961+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:01:47.316 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:01:46 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:01:46.961+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:01:47.316 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:01:46 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:01:46.961+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:01:47.316 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:01:46 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:01:46.963+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:01:48.011 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:01:47 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:01:47.713+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:01:48.011 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:01:47 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:01:47.713+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:01:48.011 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:01:47 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:01:47.713+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:01:48.011 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:01:47 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:01:47.714+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:01:48.012 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:01:47 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:01:47.862+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:01:48.012 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:01:47 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:01:47.863+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:01:48.012 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:01:47 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:01:47.863+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:01:48.012 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:01:47 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:01:47.864+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:01:48.314 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:01:48 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:01:48.009+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:01:48.315 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:01:48 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:01:48.009+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:01:48.315 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:01:48 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:01:48.009+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:01:48.315 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:01:48 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:01:48.010+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:01:49.023 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:01:48 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:01:48.824+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:01:49.023 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:01:48 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:01:48.824+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:01:49.023 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:01:48 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:01:48.824+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:01:49.023 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:01:48 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:01:48.824+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:01:49.024 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:01:48 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:01:48.713+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:01:49.024 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:01:48 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:01:48.713+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:01:49.024 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:01:48 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:01:48.713+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:01:49.024 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:01:48 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:01:48.714+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:01:49.315 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:01:49 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:01:49.021+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:01:49.315 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:01:49 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:01:49.021+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:01:49.315 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:01:49 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:01:49.021+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:01:49.315 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:01:49 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:01:49.023+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:01:50.002 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:01:49 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:01:49.702+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:01:50.002 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:01:49 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:01:49.702+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:01:50.002 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:01:49 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:01:49.702+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:01:50.002 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:01:49 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:01:49.703+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:01:50.003 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:01:49 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:01:49.847+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:01:50.003 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:01:49 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:01:49.847+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:01:50.003 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:01:49 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:01:49.847+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:01:50.003 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:01:49 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:01:49.848+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:01:50.315 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:01:50 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:01:50.000+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:01:50.315 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:01:50 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:01:50.000+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:01:50.315 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:01:50 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:01:50.000+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:01:50.315 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:01:50 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:01:50.001+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:01:51.025 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:01:50 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:01:50.865+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:01:51.025 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:01:50 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:01:50.865+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:01:51.025 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:01:50 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:01:50.865+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:01:51.025 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:01:50 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:01:50.866+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:01:51.026 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:01:50 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:01:50.695+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:01:51.026 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:01:50 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:01:50.695+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:01:51.026 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:01:50 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:01:50.695+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:01:51.026 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:01:50 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:01:50.696+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:01:51.314 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:01:51 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:01:51.023+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:01:51.315 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:01:51 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:01:51.023+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:01:51.315 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:01:51 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:01:51.023+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:01:51.315 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:01:51 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:01:51.024+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:01:51.905 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Dec 16 22:01:51 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-mon-smithi045[80759]: 2023-12-16T22:01:51.643+0000 7f45a1f8f700 -1 mon.smithi045@0(probing) e2 get_health_metrics reporting 21422 slow ops, oldest is monmgrreport(gid 14214, 0 checks, 1 progress events) 2023-12-16T22:01:51.905 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:01:51 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:01:51.657+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:01:51.906 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:01:51 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:01:51.657+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:01:51.906 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:01:51 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:01:51.657+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:01:51.906 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:01:51 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:01:51.658+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:01:52.315 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:01:51 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:01:51.904+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:01:52.315 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:01:51 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:01:51.904+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:01:52.315 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:01:51 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:01:51.904+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:01:52.315 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:01:51 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:01:51.905+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:01:52.315 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:01:52 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:01:52.058+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:01:52.315 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:01:52 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:01:52.058+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:01:52.316 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:01:52 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:01:52.058+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:01:52.316 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:01:52 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:01:52.058+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:01:52.895 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:01:52 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:01:52.632+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:01:52.895 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:01:52 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:01:52.632+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:01:52.895 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:01:52 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:01:52.632+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:01:52.895 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:01:52 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:01:52.633+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:01:53.167 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:01:52 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:01:52.893+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:01:53.167 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:01:52 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:01:52.893+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:01:53.168 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:01:52 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:01:52.893+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:01:53.168 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:01:52 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:01:52.894+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:01:53.168 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:01:53 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:01:53.071+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:01:53.168 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:01:53 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:01:53.071+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:01:53.168 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:01:53 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:01:53.071+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:01:53.168 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:01:53 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:01:53.071+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:01:53.858 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:01:53 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:01:53.593+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:01:53.858 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:01:53 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:01:53.593+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:01:53.858 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:01:53 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:01:53.593+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:01:53.858 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:01:53 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:01:53.594+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:01:54.315 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:01:53 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:01:53.856+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:01:54.315 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:01:53 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:01:53.856+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:01:54.315 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:01:53 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:01:53.856+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:01:54.315 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:01:53 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:01:53.857+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:01:54.316 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:01:54 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:01:54.086+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:01:54.316 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:01:54 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:01:54.086+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:01:54.316 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:01:54 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:01:54.086+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:01:54.316 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:01:54 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:01:54.087+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:01:54.885 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:01:54 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:01:54.609+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:01:54.886 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:01:54 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:01:54.609+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:01:54.886 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:01:54 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:01:54.609+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:01:54.886 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:01:54 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:01:54.611+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:01:55.315 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:01:55 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:01:55.125+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:01:55.315 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:01:55 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:01:55.125+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:01:55.315 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:01:55 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:01:55.125+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:01:55.315 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:01:55 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:01:55.126+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:01:55.315 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:01:54 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:01:54.884+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:01:55.316 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:01:54 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:01:54.884+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:01:55.316 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:01:54 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:01:54.884+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:01:55.316 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:01:54 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:01:54.885+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:01:55.894 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:01:55 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:01:55.631+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:01:55.894 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:01:55 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:01:55.631+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:01:55.894 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:01:55 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:01:55.631+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:01:55.894 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:01:55 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:01:55.631+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:01:56.151 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:01:55 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:01:55.893+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:01:56.151 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:01:55 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:01:55.893+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:01:56.152 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:01:55 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:01:55.893+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:01:56.152 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:01:55 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:01:55.894+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:01:56.565 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:01:56 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:01:56.150+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:01:56.565 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:01:56 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:01:56.150+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:01:56.565 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:01:56 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:01:56.150+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:01:56.565 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:01:56 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:01:56.151+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:01:56.911 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Dec 16 22:01:56 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-mon-smithi045[80759]: 2023-12-16T22:01:56.645+0000 7f45a1f8f700 -1 mon.smithi045@0(probing) e2 get_health_metrics reporting 21511 slow ops, oldest is monmgrreport(gid 14214, 0 checks, 1 progress events) 2023-12-16T22:01:56.912 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:01:56 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:01:56.639+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:01:56.912 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:01:56 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:01:56.639+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:01:56.912 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:01:56 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:01:56.639+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:01:56.912 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:01:56 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:01:56.640+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:01:57.174 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:01:56 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:01:56.910+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:01:57.174 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:01:56 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:01:56.910+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:01:57.175 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:01:56 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:01:56.910+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:01:57.175 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:01:56 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:01:56.911+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:01:57.564 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:01:57 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:01:57.173+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:01:57.565 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:01:57 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:01:57.173+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:01:57.565 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:01:57 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:01:57.173+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:01:57.565 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:01:57 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:01:57.174+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:01:58.065 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:01:57 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:01:57.689+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:01:58.065 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:01:57 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:01:57.689+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:01:58.065 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:01:57 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:01:57.689+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:01:58.065 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:01:57 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:01:57.691+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:01:58.065 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:01:57 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:01:57.904+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:01:58.066 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:01:57 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:01:57.904+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:01:58.066 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:01:57 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:01:57.904+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:01:58.066 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:01:57 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:01:57.905+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:01:58.565 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:01:58 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:01:58.212+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:01:58.565 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:01:58 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:01:58.212+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:01:58.565 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:01:58 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:01:58.212+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:01:58.565 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:01:58 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:01:58.212+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:01:59.065 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:01:58 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:01:58.935+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:01:59.065 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:01:58 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:01:58.935+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:01:59.065 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:01:58 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:01:58.935+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:01:59.065 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:01:58 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:01:58.937+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:01:59.065 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:01:58 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:01:58.695+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:01:59.066 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:01:58 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:01:58.695+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:01:59.066 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:01:58 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:01:58.695+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:01:59.066 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:01:58 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:01:58.696+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:01:59.564 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:01:59 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:01:59.241+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:01:59.565 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:01:59 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:01:59.241+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:01:59.565 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:01:59 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:01:59.241+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:01:59.565 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:01:59 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:01:59.242+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:02:00.064 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:01:59 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:01:59.945+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:02:00.065 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:01:59 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:01:59.945+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:02:00.065 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:01:59 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:01:59.945+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:02:00.065 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:01:59 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:01:59.946+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:02:00.066 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:01:59 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:01:59.736+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:02:00.066 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:01:59 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:01:59.736+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:02:00.066 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:01:59 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:01:59.736+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:02:00.066 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:01:59 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:01:59.737+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:02:00.565 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:02:00 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:02:00.261+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:02:00.565 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:02:00 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:02:00.261+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:02:00.565 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:02:00 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:02:00.261+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:02:00.565 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:02:00 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:02:00.261+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:02:01.064 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:02:00 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:02:00.926+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:02:01.065 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:02:00 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:02:00.926+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:02:01.065 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:02:00 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:02:00.926+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:02:01.065 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:02:00 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:02:00.927+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:02:01.065 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:02:00 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:02:00.775+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:02:01.065 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:02:00 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:02:00.775+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:02:01.066 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:02:00 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:02:00.775+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:02:01.066 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:02:00 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:02:00.777+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:02:01.565 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:02:01 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:02:01.310+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:02:01.565 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:02:01 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:02:01.310+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:02:01.565 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:02:01 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:02:01.310+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:02:01.566 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:02:01 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:02:01.311+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:02:01.907 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Dec 16 22:02:01 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-mon-smithi045[80759]: 2023-12-16T22:02:01.647+0000 7f45a1f8f700 -1 mon.smithi045@0(probing) e2 get_health_metrics reporting 21603 slow ops, oldest is monmgrreport(gid 14214, 0 checks, 1 progress events) 2023-12-16T22:02:01.907 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:02:01 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:02:01.765+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:02:01.908 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:02:01 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:02:01.765+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:02:01.908 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:02:01 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:02:01.765+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:02:01.908 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:02:01 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:02:01.766+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:02:02.284 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:02:01 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:02:01.906+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:02:02.284 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:02:01 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:02:01.906+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:02:02.285 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:02:01 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:02:01.906+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:02:02.285 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:02:01 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:02:01.907+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:02:02.565 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:02:02 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:02:02.283+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:02:02.565 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:02:02 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:02:02.283+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:02:02.565 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:02:02 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:02:02.283+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:02:02.565 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:02:02 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:02:02.283+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:02:03.065 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:02:02 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:02:02.874+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:02:03.065 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:02:02 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:02:02.874+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:02:03.065 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:02:02 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:02:02.874+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:02:03.065 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:02:02 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:02:02.875+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:02:03.066 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:02:02 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:02:02.721+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:02:03.066 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:02:02 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:02:02.721+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:02:03.066 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:02:02 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:02:02.721+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:02:03.066 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:02:02 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:02:02.722+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:02:03.600 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:02:03 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:02:03.318+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:02:03.600 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:02:03 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:02:03.318+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:02:03.600 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:02:03 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:02:03.318+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:02:03.601 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:02:03 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:02:03.318+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:02:03.915 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:02:03 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:02:03.705+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:02:03.915 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:02:03 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:02:03.705+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:02:03.915 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:02:03 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:02:03.705+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:02:03.916 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:02:03 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:02:03.707+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:02:04.301 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:02:03 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:02:03.914+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:02:04.301 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:02:03 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:02:03.914+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:02:04.301 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:02:03 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:02:03.914+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:02:04.301 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:02:03 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:02:03.915+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:02:04.564 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:02:04 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:02:04.300+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:02:04.565 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:02:04 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:02:04.300+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:02:04.565 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:02:04 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:02:04.300+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:02:04.565 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:02:04 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:02:04.300+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:02:05.065 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:02:04 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:02:04.875+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:02:05.065 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:02:04 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:02:04.875+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:02:05.065 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:02:04 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:02:04.875+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:02:05.065 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:02:04 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:02:04.876+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:02:05.065 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:02:04 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:02:04.688+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:02:05.066 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:02:04 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:02:04.688+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:02:05.066 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:02:04 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:02:04.688+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:02:05.066 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:02:04 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:02:04.688+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:02:05.669 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:02:05 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:02:05.327+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:02:05.669 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:02:05 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:02:05.327+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:02:05.669 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:02:05 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:02:05.327+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:02:05.669 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:02:05 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:02:05.329+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:02:06.065 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:02:05 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:02:05.859+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:02:06.065 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:02:05 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:02:05.859+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:02:06.065 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:02:05 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:02:05.859+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:02:06.065 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:02:05 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:02:05.859+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:02:06.066 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:02:05 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:02:05.667+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:02:06.066 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:02:05 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:02:05.667+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:02:06.066 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:02:05 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:02:05.667+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:02:06.066 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:02:05 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:02:05.668+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:02:06.651 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:02:06 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:02:06.344+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:02:06.651 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:02:06 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:02:06.344+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:02:06.651 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:02:06 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:02:06.344+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:02:06.651 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:02:06 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:02:06.345+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:02:07.064 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Dec 16 22:02:06 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-mon-smithi045[80759]: 2023-12-16T22:02:06.649+0000 7f45a1f8f700 -1 mon.smithi045@0(probing) e2 get_health_metrics reporting 21689 slow ops, oldest is monmgrreport(gid 14214, 0 checks, 1 progress events) 2023-12-16T22:02:07.065 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:02:06 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:02:06.857+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:02:07.065 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:02:06 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:02:06.857+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:02:07.065 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:02:06 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:02:06.857+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:02:07.065 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:02:06 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:02:06.858+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:02:07.066 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:02:06 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:02:06.674+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:02:07.066 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:02:06 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:02:06.674+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:02:07.066 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:02:06 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:02:06.674+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:02:07.066 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:02:06 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:02:06.674+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:02:07.668 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:02:07 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:02:07.369+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:02:07.668 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:02:07 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:02:07.369+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:02:07.668 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:02:07 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:02:07.369+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:02:07.669 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:02:07 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:02:07.370+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:02:08.065 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:02:07 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:02:07.892+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:02:08.065 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:02:07 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:02:07.892+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:02:08.065 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:02:07 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:02:07.892+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:02:08.065 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:02:07 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:02:07.893+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:02:08.065 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:02:07 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:02:07.667+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:02:08.066 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:02:07 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:02:07.667+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:02:08.066 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:02:07 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:02:07.667+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:02:08.066 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:02:07 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:02:07.667+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:02:08.660 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:02:08 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:02:08.395+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:02:08.660 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:02:08 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:02:08.395+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:02:08.660 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:02:08 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:02:08.395+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:02:08.661 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:02:08 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:02:08.396+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:02:08.933 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:02:08 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:02:08.659+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:02:08.934 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:02:08 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:02:08.659+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:02:08.934 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:02:08 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:02:08.659+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:02:08.934 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:02:08 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:02:08.660+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:02:09.315 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:02:08 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:02:08.932+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:02:09.315 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:02:08 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:02:08.932+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:02:09.315 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:02:08 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:02:08.932+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:02:09.315 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:02:08 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:02:08.933+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:02:09.631 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:02:09 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:02:09.376+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:02:09.631 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:02:09 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:02:09.376+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:02:09.631 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:02:09 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:02:09.376+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:02:09.631 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:02:09 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:02:09.377+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:02:09.887 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:02:09 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:02:09.629+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:02:09.887 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:02:09 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:02:09.629+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:02:09.887 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:02:09 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:02:09.629+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:02:09.887 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:02:09 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:02:09.631+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:02:10.177 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:02:09 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:02:09.885+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:02:10.178 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:02:09 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:02:09.885+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:02:10.178 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:02:09 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:02:09.885+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:02:10.178 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:02:09 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:02:09.886+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:02:10.565 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:02:10 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:02:10.406+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:02:10.565 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:02:10 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:02:10.406+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:02:10.565 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:02:10 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:02:10.406+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:02:10.565 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:02:10 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:02:10.406+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:02:11.064 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:02:10 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:02:10.886+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:02:11.065 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:02:10 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:02:10.886+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:02:11.065 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:02:10 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:02:10.886+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:02:11.065 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:02:10 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:02:10.887+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:02:11.065 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:02:10 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:02:10.678+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:02:11.065 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:02:10 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:02:10.678+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:02:11.066 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:02:10 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:02:10.678+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:02:11.066 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:02:10 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:02:10.679+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:02:11.653 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:02:11 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:02:11.369+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:02:11.654 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:02:11 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:02:11.369+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:02:11.654 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:02:11 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:02:11.369+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:02:11.654 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:02:11 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:02:11.370+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:02:11.937 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Dec 16 22:02:11 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-mon-smithi045[80759]: 2023-12-16T22:02:11.652+0000 7f45a1f8f700 -1 mon.smithi045@0(probing) e2 get_health_metrics reporting 21779 slow ops, oldest is monmgrreport(gid 14214, 0 checks, 1 progress events) 2023-12-16T22:02:11.937 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:02:11 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:02:11.698+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:02:11.937 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:02:11 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:02:11.698+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:02:11.937 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:02:11 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:02:11.698+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:02:11.937 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:02:11 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:02:11.699+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:02:12.315 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:02:11 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:02:11.935+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:02:12.315 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:02:11 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:02:11.935+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:02:12.315 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:02:11 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:02:11.935+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:02:12.316 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:02:11 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:02:11.936+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:02:12.740 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:02:12 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:02:12.359+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:02:12.740 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:02:12 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:02:12.359+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:02:12.741 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:02:12 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:02:12.359+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:02:12.741 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:02:12 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:02:12.360+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:02:13.065 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:02:12 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:02:12.972+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:02:13.065 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:02:12 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:02:12.972+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:02:13.065 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:02:12 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:02:12.972+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:02:13.065 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:02:12 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:02:12.974+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:02:13.066 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:02:12 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:02:12.739+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:02:13.066 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:02:12 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:02:12.739+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:02:13.066 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:02:12 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:02:12.739+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:02:13.067 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:02:12 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:02:12.740+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:02:13.600 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:02:13 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:02:13.316+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:02:13.600 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:02:13 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:02:13.316+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:02:13.601 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:02:13 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:02:13.316+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:02:13.601 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:02:13 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:02:13.317+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:02:13.929 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:02:13 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:02:13.701+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:02:13.929 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:02:13 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:02:13.701+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:02:13.929 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:02:13 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:02:13.701+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:02:13.930 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:02:13 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:02:13.702+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:02:14.314 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:02:13 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:02:13.928+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:02:14.315 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:02:13 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:02:13.928+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:02:14.315 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:02:13 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:02:13.928+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:02:14.315 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:02:13 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:02:13.929+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:02:14.671 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:02:14 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:02:14.314+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:02:14.671 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:02:14 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:02:14.314+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:02:14.672 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:02:14 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:02:14.314+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:02:14.672 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:02:14 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:02:14.315+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:02:14.943 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:02:14 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:02:14.713+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:02:14.943 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:02:14 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:02:14.713+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:02:14.944 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:02:14 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:02:14.713+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:02:14.944 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:02:14 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:02:14.714+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:02:15.297 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:02:14 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:02:14.942+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:02:15.297 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:02:14 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:02:14.942+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:02:15.297 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:02:14 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:02:14.942+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:02:15.297 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:02:14 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:02:14.943+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:02:15.564 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:02:15 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:02:15.295+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:02:15.565 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:02:15 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:02:15.295+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:02:15.565 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:02:15 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:02:15.295+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:02:15.565 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:02:15 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:02:15.297+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:02:15.942 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:02:15 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:02:15.687+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:02:15.942 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:02:15 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:02:15.687+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:02:15.942 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:02:15 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:02:15.687+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:02:15.942 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:02:15 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:02:15.688+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:02:16.314 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:02:15 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:02:15.940+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:02:16.315 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:02:15 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:02:15.940+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:02:16.315 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:02:15 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:02:15.940+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:02:16.315 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:02:15 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:02:15.941+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:02:16.656 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:02:16 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:02:16.341+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:02:16.656 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:02:16 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:02:16.341+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:02:16.656 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:02:16 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:02:16.341+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:02:16.657 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:02:16 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:02:16.342+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:02:16.931 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Dec 16 22:02:16 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-mon-smithi045[80759]: 2023-12-16T22:02:16.654+0000 7f45a1f8f700 -1 mon.smithi045@0(probing) e2 get_health_metrics reporting 21862 slow ops, oldest is monmgrreport(gid 14214, 0 checks, 1 progress events) 2023-12-16T22:02:16.931 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:02:16 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:02:16.711+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:02:16.931 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:02:16 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:02:16.711+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:02:16.931 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:02:16 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:02:16.711+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:02:16.931 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:02:16 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:02:16.712+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:02:17.315 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:02:16 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:02:16.929+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:02:17.315 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:02:16 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:02:16.929+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:02:17.315 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:02:16 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:02:16.929+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:02:17.315 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:02:16 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:02:16.930+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:02:17.680 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:02:17 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:02:17.322+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:02:17.680 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:02:17 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:02:17.322+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:02:17.680 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:02:17 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:02:17.322+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:02:17.680 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:02:17 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:02:17.323+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:02:18.065 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:02:17 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:02:17.917+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:02:18.065 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:02:17 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:02:17.917+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:02:18.065 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:02:17 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:02:17.917+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:02:18.065 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:02:17 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:02:17.917+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:02:18.066 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:02:17 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:02:17.678+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:02:18.066 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:02:17 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:02:17.678+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:02:18.066 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:02:17 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:02:17.678+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:02:18.066 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:02:17 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:02:17.679+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:02:18.702 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:02:18 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:02:18.365+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:02:18.702 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:02:18 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:02:18.365+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:02:18.702 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:02:18 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:02:18.365+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:02:18.702 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:02:18 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:02:18.366+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:02:19.065 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:02:18 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:02:18.932+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:02:19.066 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:02:18 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:02:18.932+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:02:19.066 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:02:18 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:02:18.932+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:02:19.066 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:02:18 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:02:18.933+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:02:19.066 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:02:18 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:02:18.700+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:02:19.066 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:02:18 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:02:18.700+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:02:19.066 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:02:18 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:02:18.700+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:02:19.067 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:02:18 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:02:18.701+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:02:19.674 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:02:19 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:02:19.391+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:02:19.675 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:02:19 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:02:19.391+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:02:19.675 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:02:19 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:02:19.391+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:02:19.675 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:02:19 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:02:19.392+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:02:20.065 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:02:19 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:02:19.673+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:02:20.065 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:02:19 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:02:19.673+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:02:20.065 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:02:19 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:02:19.673+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:02:20.065 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:02:19 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:02:19.674+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:02:20.065 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:02:19 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:02:19.891+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:02:20.066 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:02:19 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:02:19.891+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:02:20.066 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:02:19 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:02:19.891+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:02:20.066 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:02:19 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:02:19.892+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:02:20.815 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:02:20 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:02:20.432+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:02:20.815 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:02:20 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:02:20.432+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:02:20.815 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:02:20 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:02:20.432+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:02:20.816 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:02:20 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:02:20.433+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:02:20.816 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:02:20 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:02:20.649+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:02:20.816 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:02:20 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:02:20.649+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:02:20.816 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:02:20 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:02:20.649+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:02:20.816 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:02:20 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:02:20.650+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:02:21.315 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:02:20 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:02:20.865+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:02:21.315 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:02:20 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:02:20.865+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:02:21.315 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:02:20 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:02:20.865+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:02:21.315 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:02:20 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:02:20.865+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:02:21.658 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:02:21 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:02:21.400+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:02:21.658 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:02:21 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:02:21.400+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:02:21.658 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:02:21 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:02:21.400+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:02:21.658 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:02:21 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:02:21.401+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:02:21.659 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:02:21 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:02:21.628+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:02:21.659 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:02:21 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:02:21.628+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:02:21.659 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:02:21 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:02:21.628+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:02:21.659 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:02:21 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:02:21.629+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:02:22.065 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Dec 16 22:02:21 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-mon-smithi045[80759]: 2023-12-16T22:02:21.656+0000 7f45a1f8f700 -1 mon.smithi045@0(probing) e2 get_health_metrics reporting 21948 slow ops, oldest is monmgrreport(gid 14214, 0 checks, 1 progress events) 2023-12-16T22:02:22.065 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:02:21 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:02:21.905+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:02:22.066 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:02:21 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:02:21.905+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:02:22.066 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:02:21 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:02:21.905+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:02:22.066 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:02:21 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:02:21.906+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:02:22.815 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:02:22 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:02:22.620+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:02:22.815 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:02:22 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:02:22.620+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:02:22.815 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:02:22 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:02:22.620+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:02:22.815 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:02:22 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:02:22.621+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:02:22.815 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:02:22 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:02:22.442+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:02:22.816 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:02:22 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:02:22.442+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:02:22.816 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:02:22 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:02:22.442+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:02:22.816 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:02:22 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:02:22.443+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:02:23.315 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:02:22 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:02:22.885+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:02:23.315 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:02:22 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:02:22.885+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:02:23.315 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:02:22 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:02:22.885+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:02:23.315 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:02:22 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:02:22.885+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:02:23.815 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:02:23 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:02:23.397+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:02:23.815 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:02:23 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:02:23.397+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:02:23.815 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:02:23 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:02:23.397+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:02:23.815 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:02:23 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:02:23.397+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:02:23.815 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:02:23 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:02:23.612+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:02:23.815 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:02:23 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:02:23.612+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:02:23.816 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:02:23 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:02:23.612+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:02:23.816 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:02:23 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:02:23.612+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:02:24.314 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:02:23 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:02:23.893+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:02:24.315 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:02:23 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:02:23.893+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:02:24.315 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:02:23 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:02:23.893+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:02:24.315 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:02:23 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:02:23.894+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:02:24.814 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:02:24 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:02:24.604+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:02:24.815 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:02:24 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:02:24.604+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:02:24.815 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:02:24 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:02:24.604+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:02:24.815 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:02:24 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:02:24.605+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:02:24.815 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:02:24 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:02:24.395+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:02:24.815 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:02:24 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:02:24.395+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:02:24.816 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:02:24 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:02:24.395+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:02:24.816 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:02:24 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:02:24.395+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:02:25.315 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:02:24 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:02:24.881+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:02:25.315 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:02:24 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:02:24.881+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:02:25.315 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:02:24 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:02:24.881+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:02:25.315 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:02:24 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:02:24.881+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:02:25.814 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:02:25 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:02:25.377+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:02:25.815 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:02:25 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:02:25.377+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:02:25.815 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:02:25 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:02:25.377+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:02:25.815 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:02:25 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:02:25.377+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:02:25.815 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:02:25 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:02:25.579+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:02:25.816 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:02:25 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:02:25.579+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:02:25.816 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:02:25 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:02:25.579+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:02:25.816 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:02:25 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:02:25.580+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:02:26.317 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:02:25 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:02:25.916+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:02:26.317 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:02:25 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:02:25.916+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:02:26.317 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:02:25 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:02:25.916+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:02:26.317 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:02:25 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:02:25.917+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:02:26.600 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:02:26 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:02:26.334+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:02:26.601 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:02:26 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:02:26.334+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:02:26.601 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:02:26 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:02:26.334+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:02:26.601 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:02:26 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:02:26.335+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:02:26.874 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Dec 16 22:02:26 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-mon-smithi045[80759]: 2023-12-16T22:02:26.659+0000 7f45a1f8f700 -1 mon.smithi045@0(probing) e2 get_health_metrics reporting 22028 slow ops, oldest is monmgrreport(gid 14214, 0 checks, 1 progress events) 2023-12-16T22:02:26.874 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:02:26 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:02:26.599+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:02:26.874 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:02:26 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:02:26.599+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:02:26.875 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:02:26 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:02:26.599+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:02:26.875 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:02:26 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:02:26.600+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:02:27.308 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:02:26 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:02:26.872+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:02:27.308 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:02:26 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:02:26.872+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:02:27.308 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:02:26 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:02:26.872+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:02:27.308 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:02:26 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:02:26.874+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:02:27.564 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:02:27 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:02:27.307+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:02:27.565 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:02:27 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:02:27.307+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:02:27.565 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:02:27 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:02:27.307+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:02:27.565 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:02:27 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:02:27.308+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:02:27.907 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:02:27 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:02:27.575+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:02:27.908 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:02:27 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:02:27.575+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:02:27.908 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:02:27 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:02:27.575+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:02:27.908 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:02:27 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:02:27.577+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:02:28.315 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:02:27 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:02:27.906+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:02:28.315 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:02:27 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:02:27.906+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:02:28.315 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:02:27 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:02:27.906+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:02:28.315 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:02:27 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:02:27.907+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:02:28.625 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:02:28 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:02:28.337+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:02:28.625 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:02:28 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:02:28.337+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:02:28.625 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:02:28 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:02:28.337+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:02:28.625 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:02:28 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:02:28.338+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:02:28.925 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:02:28 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:02:28.623+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:02:28.926 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:02:28 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:02:28.623+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:02:28.926 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:02:28 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:02:28.623+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:02:28.926 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:02:28 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:02:28.625+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:02:29.314 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:02:28 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:02:28.924+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:02:29.315 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:02:28 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:02:28.924+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:02:29.315 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:02:28 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:02:28.924+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:02:29.315 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:02:28 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:02:28.925+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:02:29.646 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:02:29 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:02:29.354+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:02:29.646 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:02:29 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:02:29.354+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:02:29.646 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:02:29 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:02:29.354+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:02:29.646 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:02:29 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:02:29.355+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:02:29.933 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:02:29 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:02:29.644+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:02:29.934 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:02:29 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:02:29.644+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:02:29.934 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:02:29 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:02:29.644+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:02:29.934 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:02:29 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:02:29.646+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:02:30.315 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:02:29 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:02:29.932+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:02:30.315 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:02:29 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:02:29.932+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:02:30.315 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:02:29 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:02:29.932+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:02:30.315 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:02:29 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:02:29.933+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:02:30.613 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:02:30 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:02:30.314+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:02:30.613 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:02:30 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:02:30.314+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:02:30.613 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:02:30 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:02:30.314+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:02:30.613 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:02:30 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:02:30.315+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:02:30.909 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:02:30 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:02:30.611+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:02:30.910 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:02:30 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:02:30.611+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:02:30.910 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:02:30 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:02:30.611+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:02:30.910 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:02:30 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:02:30.613+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:02:31.295 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:02:30 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:02:30.908+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:02:31.296 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:02:30 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:02:30.908+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:02:31.296 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:02:30 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:02:30.908+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:02:31.296 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:02:30 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:02:30.909+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:02:31.564 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:02:31 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:02:31.294+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:02:31.565 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:02:31 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:02:31.294+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:02:31.565 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:02:31 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:02:31.294+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:02:31.565 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:02:31 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:02:31.295+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:02:31.949 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Dec 16 22:02:31 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-mon-smithi045[80759]: 2023-12-16T22:02:31.661+0000 7f45a1f8f700 -1 mon.smithi045@0(probing) e2 get_health_metrics reporting 22115 slow ops, oldest is monmgrreport(gid 14214, 0 checks, 1 progress events) 2023-12-16T22:02:31.949 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:02:31 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:02:31.616+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:02:31.950 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:02:31 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:02:31.616+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:02:31.950 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:02:31 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:02:31.616+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:02:31.950 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:02:31 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:02:31.617+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:02:32.299 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:02:31 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:02:31.948+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:02:32.300 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:02:31 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:02:31.948+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:02:32.300 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:02:31 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:02:31.948+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:02:32.300 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:02:31 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:02:31.948+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:02:32.564 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:02:32 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:02:32.298+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:02:32.565 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:02:32 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:02:32.298+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:02:32.565 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:02:32 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:02:32.298+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:02:32.565 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:02:32 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:02:32.299+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:02:32.980 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:02:32 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:02:32.651+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:02:32.980 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:02:32 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:02:32.651+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:02:32.980 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:02:32 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:02:32.651+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:02:32.980 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:02:32 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:02:32.652+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:02:33.285 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:02:32 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:02:32.978+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:02:33.285 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:02:32 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:02:32.978+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:02:33.286 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:02:32 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:02:32.978+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:02:33.286 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:02:32 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:02:32.980+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:02:33.565 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:02:33 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:02:33.283+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:02:33.565 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:02:33 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:02:33.283+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:02:33.565 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:02:33 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:02:33.283+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:02:33.565 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:02:33 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:02:33.284+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:02:34.012 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:02:33 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:02:33.668+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:02:34.012 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:02:33 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:02:33.668+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:02:34.013 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:02:33 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:02:33.668+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:02:34.013 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:02:33 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:02:33.670+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:02:34.299 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:02:34 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:02:34.011+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:02:34.300 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:02:34 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:02:34.011+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:02:34.300 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:02:34 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:02:34.011+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:02:34.300 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:02:34 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:02:34.012+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:02:34.564 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:02:34 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:02:34.298+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:02:34.565 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:02:34 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:02:34.298+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:02:34.565 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:02:34 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:02:34.298+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:02:34.565 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:02:34 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:02:34.299+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:02:35.046 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:02:34 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:02:34.641+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:02:35.046 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:02:34 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:02:34.641+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:02:35.046 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:02:34 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:02:34.641+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:02:35.046 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:02:34 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:02:34.642+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:02:35.315 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:02:35 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:02:35.274+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:02:35.315 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:02:35 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:02:35.274+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:02:35.315 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:02:35 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:02:35.274+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:02:35.315 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:02:35 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:02:35.276+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:02:35.316 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:02:35 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:02:35.044+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:02:35.316 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:02:35 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:02:35.044+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:02:35.316 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:02:35 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:02:35.044+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:02:35.316 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:02:35 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:02:35.045+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:02:36.064 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:02:35 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:02:35.625+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:02:36.065 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:02:35 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:02:35.625+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:02:36.065 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:02:35 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:02:35.625+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:02:36.065 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:02:35 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:02:35.626+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:02:36.565 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:02:36 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:02:36.232+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:02:36.565 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:02:36 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:02:36.232+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:02:36.565 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:02:36 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:02:36.232+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:02:36.565 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:02:36 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:02:36.233+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:02:36.565 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:02:36 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:02:36.080+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:02:36.566 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:02:36 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:02:36.080+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:02:36.566 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:02:36 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:02:36.080+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:02:36.566 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:02:36 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:02:36.080+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:02:37.064 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Dec 16 22:02:36 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-mon-smithi045[80759]: 2023-12-16T22:02:36.664+0000 7f45a1f8f700 -1 mon.smithi045@0(probing) e2 get_health_metrics reporting 22198 slow ops, oldest is monmgrreport(gid 14214, 0 checks, 1 progress events) 2023-12-16T22:02:37.065 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:02:36 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:02:36.635+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:02:37.065 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:02:36 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:02:36.635+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:02:37.065 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:02:36 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:02:36.635+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:02:37.065 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:02:36 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:02:36.636+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:02:37.565 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:02:37 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:02:37.266+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:02:37.565 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:02:37 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:02:37.266+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:02:37.565 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:02:37 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:02:37.266+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:02:37.565 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:02:37 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:02:37.268+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:02:37.565 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:02:37 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:02:37.100+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:02:37.565 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:02:37 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:02:37.100+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:02:37.566 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:02:37 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:02:37.100+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:02:37.566 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:02:37 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:02:37.102+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:02:38.065 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:02:37 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:02:37.644+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:02:38.065 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:02:37 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:02:37.644+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:02:38.065 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:02:37 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:02:37.644+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:02:38.065 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:02:37 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:02:37.645+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:02:38.565 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:02:38 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:02:38.316+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:02:38.565 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:02:38 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:02:38.316+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:02:38.565 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:02:38 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:02:38.316+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:02:38.565 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:02:38 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:02:38.317+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:02:38.565 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:02:38 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:02:38.146+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:02:38.566 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:02:38 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:02:38.146+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:02:38.566 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:02:38 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:02:38.146+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:02:38.566 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:02:38 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:02:38.146+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:02:39.065 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:02:38 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:02:38.675+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:02:39.065 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:02:38 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:02:38.675+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:02:39.065 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:02:38 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:02:38.675+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:02:39.065 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:02:38 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:02:38.676+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:02:39.565 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:02:39 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:02:39.118+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:02:39.565 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:02:39 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:02:39.118+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:02:39.565 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:02:39 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:02:39.118+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:02:39.565 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:02:39 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:02:39.119+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:02:39.565 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:02:39 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:02:39.361+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:02:39.566 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:02:39 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:02:39.361+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:02:39.566 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:02:39 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:02:39.361+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:02:39.566 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:02:39 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:02:39.362+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:02:40.064 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:02:39 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:02:39.655+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:02:40.065 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:02:39 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:02:39.655+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:02:40.065 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:02:39 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:02:39.655+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:02:40.065 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:02:39 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:02:39.655+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:02:40.366 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:02:40 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:02:40.090+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:02:40.366 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:02:40 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:02:40.090+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:02:40.366 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:02:40 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:02:40.090+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:02:40.366 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:02:40 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:02:40.091+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:02:40.663 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:02:40 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:02:40.365+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:02:40.663 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:02:40 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:02:40.365+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:02:40.663 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:02:40 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:02:40.365+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:02:40.663 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:02:40 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:02:40.366+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:02:41.065 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:02:40 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:02:40.661+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:02:41.065 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:02:40 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:02:40.661+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:02:41.065 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:02:40 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:02:40.661+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:02:41.065 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:02:40 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:02:40.662+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:02:41.390 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:02:41 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:02:41.088+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:02:41.390 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:02:41 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:02:41.088+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:02:41.390 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:02:41 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:02:41.088+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:02:41.390 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:02:41 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:02:41.089+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:02:41.668 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:02:41 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:02:41.625+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:02:41.668 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:02:41 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:02:41.625+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:02:41.668 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:02:41 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:02:41.625+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:02:41.668 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:02:41 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:02:41.626+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:02:41.669 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:02:41 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:02:41.388+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:02:41.669 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:02:41 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:02:41.389+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:02:41.669 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:02:41 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:02:41.389+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:02:41.669 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:02:41 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:02:41.390+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:02:42.065 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Dec 16 22:02:41 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-mon-smithi045[80759]: 2023-12-16T22:02:41.666+0000 7f45a1f8f700 -1 mon.smithi045@0(probing) e2 get_health_metrics reporting 22287 slow ops, oldest is monmgrreport(gid 14214, 0 checks, 1 progress events) 2023-12-16T22:02:42.386 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:02:42 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:02:42.128+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:02:42.386 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:02:42 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:02:42.128+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:02:42.387 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:02:42 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:02:42.128+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:02:42.387 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:02:42 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:02:42.129+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:02:42.815 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:02:42 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:02:42.385+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:02:42.815 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:02:42 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:02:42.385+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:02:42.815 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:02:42 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:02:42.385+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:02:42.815 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:02:42 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:02:42.386+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:02:42.815 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:02:42 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:02:42.611+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:02:42.816 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:02:42 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:02:42.611+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:02:42.816 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:02:42 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:02:42.611+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:02:42.816 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:02:42 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:02:42.612+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:02:43.372 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:02:43 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:02:43.111+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:02:43.372 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:02:43 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:02:43.111+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:02:43.372 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:02:43 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:02:43.111+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:02:43.373 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:02:43 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:02:43.111+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:02:43.814 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:02:43 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:02:43.594+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:02:43.815 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:02:43 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:02:43.594+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:02:43.815 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:02:43 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:02:43.594+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:02:43.815 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:02:43 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:02:43.595+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:02:43.815 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:02:43 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:02:43.371+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:02:43.816 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:02:43 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:02:43.371+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:02:43.816 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:02:43 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:02:43.371+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:02:43.816 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:02:43 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:02:43.371+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:02:44.565 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:02:44 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:02:44.160+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:02:44.565 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:02:44 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:02:44.160+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:02:44.565 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:02:44 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:02:44.160+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:02:44.565 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:02:44 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:02:44.161+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:02:44.566 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:02:44 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:02:44.358+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:02:44.566 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:02:44 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:02:44.358+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:02:44.566 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:02:44 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:02:44.358+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:02:44.566 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:02:44 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:02:44.358+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:02:45.064 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:02:44 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:02:44.608+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:02:45.065 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:02:44 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:02:44.608+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:02:45.065 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:02:44 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:02:44.608+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:02:45.065 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:02:44 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:02:44.608+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:02:45.564 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:02:45 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:02:45.164+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:02:45.565 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:02:45 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:02:45.164+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:02:45.565 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:02:45 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:02:45.164+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:02:45.565 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:02:45 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:02:45.166+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:02:45.565 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:02:45 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:02:45.395+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:02:45.565 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:02:45 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:02:45.395+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:02:45.565 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:02:45 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:02:45.395+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:02:45.566 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:02:45 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:02:45.395+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:02:46.064 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:02:45 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:02:45.634+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:02:46.065 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:02:45 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:02:45.634+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:02:46.065 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:02:45 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:02:45.634+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:02:46.065 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:02:45 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:02:45.634+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:02:46.565 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:02:46 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:02:46.212+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:02:46.565 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:02:46 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:02:46.212+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:02:46.565 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:02:46 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:02:46.212+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:02:46.565 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:02:46 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:02:46.213+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:02:46.565 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:02:46 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:02:46.413+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:02:46.566 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:02:46 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:02:46.413+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:02:46.566 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:02:46 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:02:46.413+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:02:46.566 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:02:46 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:02:46.413+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:02:47.064 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Dec 16 22:02:46 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-mon-smithi045[80759]: 2023-12-16T22:02:46.667+0000 7f45a1f8f700 -1 mon.smithi045@0(probing) e2 get_health_metrics reporting 22373 slow ops, oldest is monmgrreport(gid 14214, 0 checks, 1 progress events) 2023-12-16T22:02:47.065 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:02:46 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:02:46.608+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:02:47.065 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:02:46 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:02:46.608+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:02:47.065 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:02:46 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:02:46.608+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:02:47.065 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:02:46 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:02:46.609+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:02:47.565 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:02:47 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:02:47.166+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:02:47.565 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:02:47 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:02:47.166+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:02:47.565 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:02:47 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:02:47.166+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:02:47.565 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:02:47 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:02:47.167+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:02:47.565 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:02:47 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:02:47.372+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:02:47.565 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:02:47 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:02:47.372+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:02:47.566 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:02:47 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:02:47.372+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:02:47.566 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:02:47 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:02:47.373+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:02:48.064 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:02:47 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:02:47.647+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:02:48.065 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:02:47 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:02:47.647+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:02:48.065 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:02:47 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:02:47.647+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:02:48.065 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:02:47 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:02:47.648+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:02:48.565 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:02:48 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:02:48.173+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:02:48.565 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:02:48 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:02:48.173+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:02:48.565 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:02:48 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:02:48.173+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:02:48.565 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:02:48 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:02:48.173+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:02:48.565 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:02:48 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:02:48.397+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:02:48.566 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:02:48 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:02:48.397+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:02:48.566 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:02:48 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:02:48.397+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:02:48.566 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:02:48 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:02:48.398+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:02:49.064 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:02:48 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:02:48.671+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:02:49.065 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:02:48 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:02:48.671+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:02:49.065 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:02:48 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:02:48.671+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:02:49.065 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:02:48 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:02:48.672+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:02:49.438 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:02:49 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:02:49.205+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:02:49.438 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:02:49 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:02:49.205+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:02:49.439 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:02:49 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:02:49.205+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:02:49.439 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:02:49 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:02:49.207+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:02:49.815 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:02:49 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:02:49.640+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:02:49.815 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:02:49 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:02:49.640+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:02:49.815 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:02:49 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:02:49.640+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:02:49.815 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:02:49 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:02:49.641+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:02:49.815 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:02:49 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:02:49.436+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:02:49.816 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:02:49 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:02:49.436+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:02:49.816 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:02:49 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:02:49.436+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:02:49.816 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:02:49 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:02:49.437+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:02:50.564 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:02:50 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:02:50.238+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:02:50.565 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:02:50 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:02:50.238+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:02:50.565 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:02:50 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:02:50.238+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:02:50.565 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:02:50 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:02:50.239+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:02:50.565 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:02:50 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:02:50.400+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:02:50.566 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:02:50 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:02:50.400+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:02:50.566 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:02:50 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:02:50.400+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:02:50.566 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:02:50 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:02:50.401+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:02:51.065 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:02:50 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:02:50.621+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:02:51.065 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:02:50 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:02:50.621+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:02:51.065 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:02:50 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:02:50.621+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:02:51.065 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:02:50 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:02:50.622+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:02:51.451 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:02:51 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:02:51.196+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:02:51.451 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:02:51 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:02:51.196+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:02:51.452 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:02:51 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:02:51.196+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:02:51.452 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:02:51 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:02:51.197+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:02:51.709 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:02:51 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:02:51.588+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:02:51.709 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:02:51 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:02:51.588+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:02:51.709 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:02:51 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:02:51.588+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:02:51.709 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:02:51 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:02:51.590+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:02:51.710 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Dec 16 22:02:51 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-mon-smithi045[80759]: 2023-12-16T22:02:51.670+0000 7f45a1f8f700 -1 mon.smithi045@0(probing) e2 get_health_metrics reporting 22465 slow ops, oldest is monmgrreport(gid 14214, 0 checks, 1 progress events) 2023-12-16T22:02:51.710 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:02:51 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:02:51.449+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:02:51.710 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:02:51 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:02:51.449+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:02:51.710 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:02:51 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:02:51.449+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:02:51.710 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:02:51 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:02:51.450+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:02:52.564 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:02:52 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:02:52.437+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:02:52.565 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:02:52 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:02:52.437+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:02:52.565 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:02:52 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:02:52.437+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:02:52.565 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:02:52 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:02:52.439+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:02:52.565 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:02:52 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:02:52.196+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:02:52.566 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:02:52 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:02:52.196+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:02:52.566 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:02:52 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:02:52.196+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:02:52.566 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:02:52 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:02:52.197+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:02:53.064 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:02:52 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:02:52.621+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:02:53.065 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:02:52 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:02:52.621+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:02:53.065 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:02:52 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:02:52.621+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:02:53.065 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:02:52 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:02:52.621+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:02:53.564 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:02:53 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:02:53.440+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:02:53.565 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:02:53 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:02:53.440+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:02:53.565 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:02:53 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:02:53.440+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:02:53.565 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:02:53 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:02:53.441+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:02:53.565 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:02:53 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:02:53.219+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:02:53.565 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:02:53 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:02:53.219+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:02:53.566 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:02:53 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:02:53.219+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:02:53.566 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:02:53 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:02:53.219+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:02:54.064 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:02:53 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:02:53.659+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:02:54.065 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:02:53 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:02:53.659+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:02:54.065 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:02:53 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:02:53.659+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:02:54.065 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:02:53 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:02:53.661+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:02:54.565 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:02:54 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:02:54.395+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:02:54.565 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:02:54 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:02:54.395+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:02:54.565 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:02:54 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:02:54.395+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:02:54.565 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:02:54 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:02:54.397+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:02:54.566 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:02:54 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:02:54.176+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:02:54.566 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:02:54 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:02:54.176+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:02:54.566 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:02:54 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:02:54.176+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:02:54.566 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:02:54 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:02:54.177+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:02:55.065 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:02:54 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:02:54.709+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:02:55.065 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:02:54 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:02:54.709+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:02:55.065 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:02:54 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:02:54.709+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:02:55.065 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:02:54 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:02:54.710+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:02:55.430 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:02:55 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:02:55.135+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:02:55.430 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:02:55 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:02:55.135+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:02:55.430 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:02:55 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:02:55.135+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:02:55.431 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:02:55 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:02:55.136+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:02:55.686 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:02:55 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:02:55.429+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:02:55.686 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:02:55 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:02:55.429+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:02:55.686 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:02:55 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:02:55.429+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:02:55.686 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:02:55 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:02:55.430+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:02:56.065 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:02:55 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:02:55.685+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:02:56.065 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:02:55 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:02:55.685+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:02:56.065 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:02:55 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:02:55.685+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:02:56.065 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:02:55 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:02:55.685+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:02:56.452 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:02:56 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:02:56.131+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:02:56.452 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:02:56 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:02:56.131+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:02:56.452 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:02:56 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:02:56.131+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:02:56.452 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:02:56 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:02:56.131+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:02:56.801 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Dec 16 22:02:56 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-mon-smithi045[80759]: 2023-12-16T22:02:56.673+0000 7f45a1f8f700 -1 mon.smithi045@0(probing) e2 get_health_metrics reporting 22554 slow ops, oldest is monmgrreport(gid 14214, 0 checks, 1 progress events) 2023-12-16T22:02:56.802 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:02:56 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:02:56.450+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:02:56.802 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:02:56 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:02:56.450+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:02:56.802 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:02:56 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:02:56.450+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:02:56.802 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:02:56 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:02:56.451+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:02:56.803 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:02:56 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:02:56.669+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:02:56.803 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:02:56 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:02:56.669+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:02:56.803 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:02:56 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:02:56.669+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:02:56.803 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:02:56 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:02:56.671+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:02:57.453 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:02:57 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:02:57.157+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:02:57.454 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:02:57 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:02:57.157+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:02:57.454 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:02:57 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:02:57.157+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:02:57.454 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:02:57 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:02:57.157+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:02:57.815 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:02:57 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:02:57.452+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:02:57.815 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:02:57 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:02:57.452+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:02:57.815 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:02:57 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:02:57.452+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:02:57.815 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:02:57 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:02:57.453+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:02:57.815 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:02:57 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:02:57.640+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:02:57.816 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:02:57 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:02:57.640+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:02:57.816 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:02:57 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:02:57.640+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:02:57.816 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:02:57 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:02:57.641+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:02:58.417 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:02:58 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:02:58.137+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:02:58.417 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:02:58 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:02:58.137+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:02:58.417 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:02:58 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:02:58.137+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:02:58.417 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:02:58 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:02:58.138+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:02:58.720 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:02:58 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:02:58.415+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:02:58.721 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:02:58 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:02:58.415+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:02:58.721 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:02:58 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:02:58.415+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:02:58.721 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:02:58 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:02:58.416+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:02:58.721 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:02:58 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:02:58.662+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:02:58.721 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:02:58 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:02:58.662+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:02:58.721 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:02:58 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:02:58.662+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:02:58.722 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:02:58 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:02:58.663+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:02:59.387 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:02:59 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:02:59.115+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:02:59.387 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:02:59 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:02:59.115+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:02:59.388 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:02:59 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:02:59.115+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:02:59.388 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:02:59 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:02:59.116+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:02:59.814 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:02:59 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:02:59.634+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:02:59.815 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:02:59 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:02:59.634+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:02:59.815 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:02:59 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:02:59.634+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:02:59.815 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:02:59 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:02:59.635+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:02:59.815 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:02:59 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:02:59.386+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:02:59.815 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:02:59 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:02:59.386+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:02:59.816 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:02:59 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:02:59.386+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:02:59.816 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:02:59 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:02:59.387+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:03:00.418 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:03:00 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:03:00.163+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:03:00.418 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:03:00 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:03:00.163+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:03:00.418 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:03:00 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:03:00.163+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:03:00.418 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:03:00 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:03:00.164+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:03:00.751 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:03:00 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:03:00.660+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:03:00.751 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:03:00 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:03:00.660+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:03:00.752 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:03:00 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:03:00.660+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:03:00.752 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:03:00 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:03:00.660+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:03:00.752 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:03:00 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:03:00.416+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:03:00.752 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:03:00 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:03:00.416+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:03:00.752 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:03:00 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:03:00.416+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:03:00.752 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:03:00 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:03:00.417+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:03:01.565 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:03:01 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:03:01.212+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:03:01.565 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:03:01 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:03:01.212+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:03:01.565 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:03:01 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:03:01.212+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:03:01.565 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:03:01 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:03:01.213+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:03:01.566 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:03:01 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:03:01.431+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:03:01.566 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:03:01 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:03:01.431+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:03:01.566 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:03:01 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:03:01.431+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:03:01.566 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:03:01 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:03:01.431+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:03:02.064 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Dec 16 22:03:01 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-mon-smithi045[80759]: 2023-12-16T22:03:01.676+0000 7f45a1f8f700 -1 mon.smithi045@0(probing) e2 get_health_metrics reporting 22646 slow ops, oldest is monmgrreport(gid 14214, 0 checks, 1 progress events) 2023-12-16T22:03:02.065 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:03:01 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:03:01.635+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:03:02.065 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:03:01 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:03:01.635+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:03:02.065 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:03:01 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:03:01.635+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:03:02.065 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:03:01 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:03:01.636+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:03:02.482 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:03:02 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:03:02.168+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:03:02.483 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:03:02 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:03:02.168+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:03:02.483 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:03:02 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:03:02.168+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:03:02.483 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:03:02 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:03:02.169+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:03:02.815 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:03:02 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:03:02.481+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:03:02.815 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:03:02 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:03:02.481+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:03:02.815 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:03:02 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:03:02.481+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:03:02.815 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:03:02 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:03:02.482+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:03:02.815 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:03:02 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:03:02.602+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:03:02.816 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:03:02 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:03:02.602+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:03:02.816 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:03:02 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:03:02.602+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:03:02.816 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:03:02 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:03:02.604+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:03:03.502 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:03:03 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:03:03.196+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:03:03.502 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:03:03 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:03:03.196+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:03:03.502 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:03:03 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:03:03.196+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:03:03.503 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:03:03 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:03:03.197+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:03:03.815 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:03:03 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:03:03.501+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:03:03.815 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:03:03 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:03:03.501+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:03:03.815 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:03:03 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:03:03.501+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:03:03.815 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:03:03 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:03:03.501+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:03:03.816 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:03:03 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:03:03.649+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:03:03.816 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:03:03 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:03:03.649+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:03:03.816 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:03:03 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:03:03.649+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:03:03.816 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:03:03 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:03:03.649+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:03:04.540 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:03:04 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:03:04.190+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:03:04.540 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:03:04 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:03:04.190+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:03:04.540 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:03:04 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:03:04.190+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:03:04.540 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:03:04 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:03:04.191+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:03:04.815 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:03:04 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:03:04.538+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:03:04.815 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:03:04 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:03:04.538+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:03:04.815 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:03:04 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:03:04.538+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:03:04.816 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:03:04 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:03:04.539+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:03:04.816 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:03:04 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:03:04.697+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:03:04.816 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:03:04 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:03:04.697+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:03:04.816 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:03:04 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:03:04.697+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:03:04.816 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:03:04 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:03:04.698+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:03:05.565 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:03:05 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:03:05.213+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:03:05.565 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:03:05 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:03:05.213+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:03:05.565 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:03:05 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:03:05.213+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:03:05.565 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:03:05 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:03:05.214+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:03:06.065 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:03:05 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:03:05.574+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:03:06.065 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:03:05 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:03:05.574+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:03:06.066 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:03:05 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:03:05.574+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:03:06.066 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:03:05 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:03:05.576+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:03:06.066 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:03:05 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:03:05.706+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:03:06.066 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:03:05 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:03:05.706+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:03:06.066 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:03:05 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:03:05.706+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:03:06.066 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:03:05 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:03:05.707+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:03:06.564 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:03:06 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:03:06.207+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:03:06.565 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:03:06 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:03:06.207+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:03:06.565 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:03:06 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:03:06.207+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:03:06.565 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:03:06 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:03:06.207+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:03:07.064 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Dec 16 22:03:06 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-mon-smithi045[80759]: 2023-12-16T22:03:06.678+0000 7f45a1f8f700 -1 mon.smithi045@0(probing) e2 get_health_metrics reporting 22735 slow ops, oldest is monmgrreport(gid 14214, 0 checks, 1 progress events) 2023-12-16T22:03:07.065 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:03:06 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:03:06.682+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:03:07.065 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:03:06 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:03:06.682+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:03:07.065 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:03:06 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:03:06.682+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:03:07.065 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:03:06 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:03:06.683+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:03:07.066 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:03:06 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:03:06.598+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:03:07.066 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:03:06 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:03:06.598+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:03:07.066 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:03:06 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:03:06.598+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:03:07.066 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:03:06 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:03:06.599+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:03:07.564 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:03:07 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:03:07.233+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:03:07.565 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:03:07 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:03:07.233+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:03:07.565 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:03:07 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:03:07.233+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:03:07.565 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:03:07 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:03:07.234+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:03:08.065 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:03:07 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:03:07.611+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:03:08.065 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:03:07 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:03:07.611+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:03:08.065 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:03:07 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:03:07.611+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:03:08.066 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:03:07 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:03:07.612+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:03:08.066 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:03:07 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:03:07.658+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:03:08.066 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:03:07 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:03:07.658+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:03:08.066 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:03:07 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:03:07.658+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:03:08.066 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:03:07 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:03:07.660+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:03:08.565 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:03:08 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:03:08.269+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:03:08.565 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:03:08 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:03:08.269+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:03:08.565 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:03:08 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:03:08.269+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:03:08.565 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:03:08 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:03:08.270+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:03:09.065 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:03:08 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:03:08.628+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:03:09.065 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:03:08 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:03:08.628+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:03:09.065 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:03:08 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:03:08.628+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:03:09.065 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:03:08 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:03:08.629+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:03:09.065 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:03:08 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:03:08.646+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:03:09.066 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:03:08 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:03:08.646+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:03:09.066 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:03:08 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:03:08.646+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:03:09.066 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:03:08 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:03:08.647+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:03:09.565 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:03:09 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:03:09.282+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:03:09.565 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:03:09 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:03:09.282+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:03:09.565 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:03:09 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:03:09.282+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:03:09.565 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:03:09 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:03:09.282+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:03:10.064 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:03:09 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:03:09.661+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:03:10.065 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:03:09 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:03:09.661+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:03:10.065 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:03:09 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:03:09.661+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:03:10.065 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:03:09 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:03:09.662+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:03:10.065 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:03:09 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:03:09.585+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:03:10.066 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:03:09 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:03:09.585+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:03:10.066 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:03:09 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:03:09.585+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:03:10.066 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:03:09 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:03:09.586+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:03:10.553 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:03:10 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:03:10.289+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:03:10.553 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:03:10 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:03:10.289+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:03:10.553 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:03:10 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:03:10.289+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:03:10.553 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:03:10 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:03:10.290+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:03:10.814 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:03:10 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:03:10.617+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:03:10.815 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:03:10 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:03:10.617+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:03:10.815 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:03:10 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:03:10.617+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:03:10.815 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:03:10 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:03:10.618+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:03:10.815 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:03:10 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:03:10.551+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:03:10.815 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:03:10 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:03:10.551+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:03:10.816 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:03:10 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:03:10.551+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:03:10.816 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:03:10 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:03:10.552+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:03:11.565 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:03:11 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:03:11.265+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:03:11.565 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:03:11 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:03:11.265+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:03:11.565 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:03:11 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:03:11.265+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:03:11.565 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:03:11 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:03:11.266+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:03:12.064 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Dec 16 22:03:11 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-mon-smithi045[80759]: 2023-12-16T22:03:11.681+0000 7f45a1f8f700 -1 mon.smithi045@0(probing) e2 get_health_metrics reporting 22828 slow ops, oldest is monmgrreport(gid 14214, 0 checks, 1 progress events) 2023-12-16T22:03:12.065 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:03:11 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:03:11.569+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:03:12.065 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:03:11 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:03:11.569+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:03:12.065 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:03:11 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:03:11.569+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:03:12.065 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:03:11 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:03:11.570+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:03:12.066 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:03:11 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:03:11.667+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:03:12.066 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:03:11 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:03:11.667+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:03:12.066 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:03:11 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:03:11.667+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:03:12.066 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:03:11 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:03:11.668+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:03:12.564 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:03:12 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:03:12.238+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:03:12.565 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:03:12 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:03:12.238+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:03:12.565 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:03:12 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:03:12.238+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:03:12.565 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:03:12 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:03:12.239+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:03:13.039 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:03:12 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:03:12.586+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:03:13.039 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:03:12 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:03:12.586+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:03:13.040 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:03:12 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:03:12.586+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:03:13.040 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:03:12 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:03:12.587+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:03:13.040 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:03:12 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:03:12.628+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:03:13.040 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:03:12 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:03:12.628+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:03:13.040 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:03:12 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:03:12.628+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:03:13.040 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:03:12 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:03:12.629+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:03:13.314 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:03:13 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:03:13.265+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:03:13.315 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:03:13 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:03:13.265+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:03:13.315 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:03:13 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:03:13.265+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:03:13.315 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:03:13 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:03:13.267+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:03:14.064 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:03:13 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:03:13.635+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:03:14.065 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:03:13 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:03:13.635+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:03:14.065 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:03:13 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:03:13.635+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:03:14.065 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:03:13 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:03:13.636+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:03:14.065 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:03:13 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:03:13.606+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:03:14.066 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:03:13 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:03:13.606+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:03:14.066 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:03:13 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:03:13.606+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:03:14.066 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:03:13 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:03:13.607+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:03:14.564 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:03:14 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:03:14.286+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:03:14.565 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:03:14 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:03:14.286+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:03:14.565 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:03:14 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:03:14.286+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:03:14.565 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:03:14 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:03:14.288+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:03:15.065 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:03:14 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:03:14.588+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:03:15.065 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:03:14 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:03:14.588+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:03:15.065 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:03:14 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:03:14.588+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:03:15.065 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:03:14 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:03:14.588+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:03:15.065 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:03:14 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:03:14.568+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:03:15.066 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:03:14 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:03:14.568+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:03:15.066 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:03:14 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:03:14.568+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:03:15.066 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:03:14 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:03:14.569+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:03:15.628 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:03:15 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:03:15.321+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:03:15.628 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:03:15 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:03:15.321+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:03:15.628 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:03:15 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:03:15.321+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:03:15.628 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:03:15 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:03:15.322+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:03:15.629 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:03:15 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:03:15.555+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:03:15.629 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:03:15 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:03:15.555+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:03:15.629 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:03:15 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:03:15.555+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:03:15.629 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:03:15 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:03:15.556+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:03:16.064 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:03:15 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:03:15.626+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:03:16.065 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:03:15 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:03:15.626+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:03:16.065 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:03:15 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:03:15.626+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:03:16.065 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:03:15 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:03:15.627+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:03:16.564 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:03:16 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:03:16.329+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:03:16.565 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:03:16 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:03:16.329+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:03:16.565 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:03:16 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:03:16.329+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:03:16.565 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:03:16 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:03:16.330+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:03:17.064 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Dec 16 22:03:16 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-mon-smithi045[80759]: 2023-12-16T22:03:16.683+0000 7f45a1f8f700 -1 mon.smithi045@0(probing) e2 get_health_metrics reporting 22917 slow ops, oldest is monmgrreport(gid 14214, 0 checks, 1 progress events) 2023-12-16T22:03:17.065 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:03:16 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:03:16.579+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:03:17.065 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:03:16 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:03:16.579+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:03:17.065 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:03:16 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:03:16.579+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:03:17.065 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:03:16 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:03:16.580+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:03:17.066 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:03:16 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:03:16.623+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:03:17.066 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:03:16 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:03:16.623+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:03:17.066 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:03:16 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:03:16.623+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:03:17.066 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:03:16 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:03:16.624+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:03:17.815 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:03:17 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:03:17.596+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:03:17.815 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:03:17 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:03:17.596+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:03:17.815 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:03:17 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:03:17.596+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:03:17.815 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:03:17 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:03:17.597+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:03:17.815 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:03:17 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:03:17.585+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:03:17.816 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:03:17 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:03:17.585+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:03:17.816 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:03:17 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:03:17.585+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:03:17.816 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:03:17 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:03:17.586+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:03:17.816 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:03:17 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:03:17.366+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:03:17.816 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:03:17 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:03:17.366+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:03:17.816 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:03:17 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:03:17.366+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:03:17.816 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:03:17 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:03:17.367+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:03:18.721 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:03:18 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:03:18.551+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:03:18.721 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:03:18 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:03:18.551+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:03:18.721 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:03:18 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:03:18.551+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:03:18.721 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:03:18 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:03:18.552+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:03:18.722 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:03:18 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:03:18.372+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:03:18.722 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:03:18 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:03:18.372+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:03:18.722 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:03:18 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:03:18.372+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:03:18.722 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:03:18 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:03:18.374+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:03:18.722 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:03:18 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:03:18.593+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:03:18.723 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:03:18 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:03:18.593+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:03:18.723 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:03:18 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:03:18.593+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:03:18.723 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:03:18 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:03:18.594+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:03:19.815 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:03:19 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:03:19.378+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:03:19.816 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:03:19 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:03:19.379+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:03:19.816 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:03:19 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:03:19.379+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:03:19.816 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:03:19 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:03:19.380+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:03:19.816 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:03:19 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:03:19.611+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:03:19.817 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:03:19 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:03:19.611+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:03:19.817 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:03:19 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:03:19.611+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:03:19.817 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:03:19 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:03:19.612+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:03:19.817 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:03:19 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:03:19.577+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:03:19.817 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:03:19 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:03:19.577+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:03:19.817 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:03:19 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:03:19.577+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:03:19.817 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:03:19 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:03:19.578+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:03:20.547 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:03:20 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:03:20.414+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:03:20.547 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:03:20 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:03:20.414+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:03:20.547 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:03:20 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:03:20.414+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:03:20.547 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:03:20 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:03:20.414+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:03:20.815 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:03:20 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:03:20.563+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:03:20.815 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:03:20 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:03:20.563+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:03:20.815 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:03:20 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:03:20.563+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:03:20.815 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:03:20 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:03:20.564+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:03:20.815 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:03:20 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:03:20.546+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:03:20.815 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:03:20 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:03:20.546+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:03:20.816 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:03:20 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:03:20.546+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:03:20.816 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:03:20 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:03:20.547+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:03:21.687 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:03:21 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:03:21.547+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:03:21.687 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:03:21 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:03:21.547+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:03:21.687 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:03:21 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:03:21.547+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:03:21.688 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:03:21 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:03:21.548+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:03:21.688 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:03:21 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:03:21.427+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:03:21.688 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:03:21 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:03:21.427+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:03:21.688 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:03:21 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:03:21.427+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:03:21.688 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:03:21 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:03:21.428+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:03:21.689 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:03:21 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:03:21.591+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:03:21.689 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:03:21 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:03:21.591+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:03:21.689 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:03:21 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:03:21.591+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:03:21.689 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:03:21 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:03:21.592+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:03:22.064 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Dec 16 22:03:21 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-mon-smithi045[80759]: 2023-12-16T22:03:21.686+0000 7f45a1f8f700 -1 mon.smithi045@0(probing) e2 get_health_metrics reporting 23009 slow ops, oldest is monmgrreport(gid 14214, 0 checks, 1 progress events) 2023-12-16T22:03:22.521 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:03:22 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:03:22.392+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:03:22.522 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:03:22 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:03:22.392+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:03:22.522 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:03:22 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:03:22.392+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:03:22.522 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:03:22 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:03:22.393+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:03:22.815 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:03:22 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:03:22.556+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:03:22.815 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:03:22 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:03:22.556+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:03:22.815 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:03:22 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:03:22.556+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:03:22.815 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:03:22 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:03:22.556+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:03:22.815 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:03:22 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:03:22.520+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:03:22.816 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:03:22 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:03:22.520+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:03:22.816 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:03:22 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:03:22.520+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:03:22.816 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:03:22 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:03:22.521+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:03:23.605 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:03:23 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:03:23.481+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:03:23.605 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:03:23 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:03:23.481+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:03:23.605 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:03:23 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:03:23.481+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:03:23.605 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:03:23 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:03:23.482+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:03:23.606 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:03:23 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:03:23.345+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:03:23.606 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:03:23 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:03:23.345+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:03:23.606 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:03:23 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:03:23.345+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:03:23.606 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:03:23 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:03:23.346+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:03:23.606 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:03:23 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:03:23.544+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:03:23.606 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:03:23 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:03:23.544+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:03:23.607 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:03:23 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:03:23.544+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:03:23.607 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:03:23 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:03:23.544+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:03:24.814 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:03:24 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:03:24.515+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:03:24.815 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:03:24 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:03:24.515+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:03:24.815 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:03:24 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:03:24.515+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:03:24.815 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:03:24 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:03:24.516+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:03:24.815 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:03:24 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:03:24.361+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:03:24.816 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:03:24 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:03:24.361+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:03:24.816 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:03:24 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:03:24.361+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:03:24.816 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:03:24 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:03:24.362+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:03:24.816 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:03:24 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:03:24.524+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:03:24.816 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:03:24 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:03:24.524+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:03:24.816 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:03:24 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:03:24.524+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:03:24.816 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:03:24 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:03:24.524+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:03:25.815 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:03:25 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:03:25.334+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:03:25.815 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:03:25 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:03:25.334+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:03:25.815 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:03:25 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:03:25.334+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:03:25.815 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:03:25 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:03:25.335+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:03:25.815 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:03:25 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:03:25.571+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:03:25.816 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:03:25 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:03:25.571+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:03:25.816 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:03:25 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:03:25.571+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:03:25.816 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:03:25 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:03:25.572+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:03:25.816 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:03:25 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:03:25.509+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:03:25.816 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:03:25 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:03:25.509+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:03:25.816 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:03:25 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:03:25.509+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:03:25.817 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:03:25 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:03:25.509+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:03:26.689 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:03:26 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:03:26.333+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:03:26.693 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:03:26 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:03:26.333+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:03:26.693 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:03:26 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:03:26.333+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:03:26.693 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:03:26 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:03:26.334+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:03:26.693 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:03:26 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:03:26.575+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:03:26.693 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:03:26 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:03:26.575+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:03:26.694 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:03:26 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:03:26.575+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:03:26.694 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:03:26 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:03:26.577+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:03:26.694 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:03:26 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:03:26.501+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:03:26.694 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:03:26 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:03:26.501+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:03:26.694 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:03:26 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:03:26.501+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:03:26.695 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:03:26 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:03:26.502+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:03:27.064 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Dec 16 22:03:26 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-mon-smithi045[80759]: 2023-12-16T22:03:26.688+0000 7f45a1f8f700 -1 mon.smithi045@0(probing) e2 get_health_metrics reporting 23098 slow ops, oldest is monmgrreport(gid 14214, 0 checks, 1 progress events) 2023-12-16T22:03:27.814 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:03:27 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:03:27.582+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:03:27.815 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:03:27 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:03:27.582+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:03:27.815 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:03:27 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:03:27.582+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:03:27.815 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:03:27 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:03:27.583+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:03:27.815 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:03:27 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:03:27.496+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:03:27.815 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:03:27 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:03:27.496+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:03:27.816 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:03:27 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:03:27.496+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:03:27.816 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:03:27 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:03:27.496+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:03:27.816 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:03:27 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:03:27.379+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:03:27.816 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:03:27 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:03:27.379+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:03:27.816 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:03:27 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:03:27.379+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:03:27.816 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:03:27 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:03:27.380+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:03:28.623 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:03:28 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:03:28.478+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:03:28.624 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:03:28 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:03:28.478+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:03:28.624 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:03:28 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:03:28.478+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:03:28.624 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:03:28 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:03:28.479+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:03:28.624 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:03:28 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:03:28.347+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:03:28.624 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:03:28 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:03:28.347+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:03:28.624 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:03:28 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:03:28.347+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:03:28.624 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:03:28 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:03:28.348+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:03:29.065 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:03:28 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:03:28.622+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:03:29.065 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:03:28 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:03:28.622+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:03:29.065 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:03:28 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:03:28.622+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:03:29.065 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:03:28 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:03:28.623+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:03:29.815 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:03:29 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:03:29.383+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:03:29.815 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:03:29 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:03:29.383+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:03:29.815 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:03:29 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:03:29.383+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:03:29.815 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:03:29 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:03:29.384+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:03:29.816 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:03:29 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:03:29.630+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:03:29.816 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:03:29 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:03:29.630+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:03:29.816 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:03:29 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:03:29.630+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:03:29.816 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:03:29 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:03:29.631+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:03:29.816 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:03:29 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:03:29.524+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:03:29.816 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:03:29 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:03:29.524+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:03:29.816 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:03:29 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:03:29.524+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:03:29.817 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:03:29 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:03:29.526+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:03:30.539 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:03:30 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:03:30.354+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:03:30.539 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:03:30 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:03:30.354+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:03:30.540 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:03:30 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:03:30.354+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:03:30.540 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:03:30 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:03:30.355+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:03:30.815 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:03:30 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:03:30.661+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:03:30.815 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:03:30 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:03:30.661+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:03:30.815 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:03:30 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:03:30.661+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:03:30.815 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:03:30 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:03:30.661+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:03:30.815 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:03:30 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:03:30.538+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:03:30.816 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:03:30 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:03:30.538+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:03:30.816 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:03:30 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:03:30.538+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:03:30.816 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:03:30 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:03:30.539+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:03:31.660 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:03:31 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:03:31.510+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:03:31.660 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:03:31 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:03:31.510+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:03:31.661 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:03:31 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:03:31.510+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:03:31.661 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:03:31 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:03:31.512+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:03:31.661 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:03:31 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:03:31.319+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:03:31.661 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:03:31 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:03:31.319+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:03:31.661 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:03:31 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:03:31.319+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:03:31.661 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:03:31 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:03:31.320+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:03:32.064 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Dec 16 22:03:31 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-mon-smithi045[80759]: 2023-12-16T22:03:31.690+0000 7f45a1f8f700 -1 mon.smithi045@0(probing) e2 get_health_metrics reporting 23191 slow ops, oldest is monmgrreport(gid 14214, 0 checks, 1 progress events) 2023-12-16T22:03:32.065 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:03:31 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:03:31.659+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:03:32.065 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:03:31 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:03:31.659+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:03:32.065 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:03:31 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:03:31.659+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:03:32.065 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:03:31 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:03:31.660+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:03:32.537 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:03:32 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:03:32.280+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:03:32.537 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:03:32 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:03:32.280+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:03:32.537 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:03:32 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:03:32.280+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:03:32.537 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:03:32 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:03:32.282+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:03:32.814 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:03:32 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:03:32.660+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:03:32.815 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:03:32 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:03:32.660+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:03:32.815 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:03:32 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:03:32.660+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:03:32.815 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:03:32 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:03:32.661+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:03:32.815 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:03:32 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:03:32.536+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:03:32.815 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:03:32 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:03:32.536+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:03:32.815 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:03:32 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:03:32.536+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:03:32.816 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:03:32 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:03:32.536+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:03:33.605 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:03:33 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:03:33.558+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:03:33.605 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:03:33 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:03:33.558+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:03:33.606 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:03:33 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:03:33.558+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:03:33.606 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:03:33 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:03:33.559+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:03:33.606 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:03:33 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:03:33.320+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:03:33.606 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:03:33 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:03:33.320+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:03:33.606 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:03:33 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:03:33.320+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:03:33.606 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:03:33 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:03:33.321+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:03:34.064 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:03:33 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:03:33.612+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:03:34.065 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:03:33 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:03:33.612+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:03:34.065 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:03:33 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:03:33.612+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:03:34.065 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:03:33 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:03:33.614+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:03:34.543 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:03:34 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:03:34.276+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:03:34.543 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:03:34 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:03:34.276+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:03:34.543 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:03:34 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:03:34.276+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:03:34.544 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:03:34 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:03:34.277+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:03:34.814 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:03:34 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:03:34.659+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:03:34.815 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:03:34 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:03:34.659+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:03:34.815 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:03:34 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:03:34.659+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:03:34.815 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:03:34 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:03:34.660+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:03:34.815 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:03:34 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:03:34.582+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:03:34.815 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:03:34 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:03:34.582+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:03:34.816 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:03:34 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:03:34.582+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:03:34.816 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:03:34 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:03:34.584+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:03:35.565 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:03:35 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:03:35.302+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:03:35.565 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:03:35 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:03:35.302+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:03:35.565 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:03:35 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:03:35.302+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:03:35.566 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:03:35 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:03:35.303+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:03:36.064 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:03:35 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:03:35.614+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:03:36.065 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:03:35 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:03:35.614+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:03:36.065 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:03:35 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:03:35.614+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:03:36.065 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:03:35 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:03:35.615+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:03:36.065 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:03:35 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:03:35.616+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:03:36.065 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:03:35 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:03:35.616+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:03:36.065 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:03:35 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:03:35.616+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:03:36.066 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:03:35 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:03:35.616+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:03:36.566 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:03:36 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:03:36.315+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:03:36.567 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:03:36 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:03:36.315+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:03:36.567 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:03:36 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:03:36.315+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:03:36.567 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:03:36 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:03:36.316+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:03:36.821 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:03:36 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:03:36.647+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:03:36.821 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:03:36 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:03:36.647+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:03:36.821 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:03:36 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:03:36.647+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:03:36.821 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:03:36 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:03:36.648+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:03:36.821 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:03:36 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:03:36.565+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:03:36.822 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:03:36 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:03:36.565+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:03:36.822 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:03:36 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:03:36.565+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:03:36.822 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:03:36 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:03:36.566+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:03:36.822 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Dec 16 22:03:36 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-mon-smithi045[80759]: 2023-12-16T22:03:36.693+0000 7f45a1f8f700 -1 mon.smithi045@0(probing) e2 get_health_metrics reporting 23280 slow ops, oldest is monmgrreport(gid 14214, 0 checks, 1 progress events) 2023-12-16T22:03:37.552 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:03:37 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:03:37.275+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:03:37.553 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:03:37 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:03:37.275+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:03:37.553 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:03:37 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:03:37.275+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:03:37.553 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:03:37 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:03:37.276+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:03:37.815 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:03:37 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:03:37.658+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:03:37.815 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:03:37 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:03:37.658+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:03:37.815 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:03:37 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:03:37.658+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:03:37.815 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:03:37 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:03:37.659+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:03:37.815 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:03:37 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:03:37.551+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:03:37.816 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:03:37 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:03:37.551+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:03:37.816 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:03:37 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:03:37.551+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:03:37.816 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:03:37 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:03:37.552+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:03:38.552 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:03:38 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:03:38.275+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:03:38.552 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:03:38 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:03:38.275+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:03:38.552 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:03:38 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:03:38.275+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:03:38.553 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:03:38 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:03:38.276+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:03:38.815 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:03:38 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:03:38.684+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:03:38.815 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:03:38 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:03:38.684+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:03:38.815 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:03:38 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:03:38.684+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:03:38.815 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:03:38 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:03:38.685+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:03:38.815 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:03:38 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:03:38.551+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:03:38.816 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:03:38 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:03:38.551+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:03:38.816 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:03:38 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:03:38.551+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:03:38.816 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:03:38 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:03:38.552+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:03:39.540 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:03:39 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:03:39.229+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:03:39.540 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:03:39 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:03:39.229+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:03:39.541 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:03:39 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:03:39.229+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:03:39.541 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:03:39 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:03:39.230+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:03:39.815 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:03:39 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:03:39.733+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:03:39.815 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:03:39 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:03:39.733+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:03:39.815 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:03:39 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:03:39.733+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:03:39.816 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:03:39 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:03:39.734+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:03:39.816 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:03:39 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:03:39.539+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:03:39.816 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:03:39 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:03:39.539+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:03:39.816 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:03:39 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:03:39.539+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:03:39.816 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:03:39 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:03:39.540+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:03:40.498 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:03:40 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:03:40.194+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:03:40.498 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:03:40 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:03:40.194+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:03:40.498 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:03:40 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:03:40.194+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:03:40.498 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:03:40 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:03:40.195+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:03:40.814 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:03:40 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:03:40.728+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:03:40.815 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:03:40 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:03:40.728+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:03:40.815 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:03:40 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:03:40.728+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:03:40.815 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:03:40 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:03:40.729+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:03:40.815 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:03:40 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:03:40.496+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:03:40.815 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:03:40 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:03:40.496+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:03:40.815 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:03:40 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:03:40.496+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:03:40.816 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:03:40 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:03:40.497+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:03:41.514 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:03:41 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:03:41.213+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:03:41.515 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:03:41 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:03:41.213+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:03:41.515 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:03:41 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:03:41.213+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:03:41.515 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:03:41 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:03:41.214+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:03:41.814 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Dec 16 22:03:41 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-mon-smithi045[80759]: 2023-12-16T22:03:41.695+0000 7f45a1f8f700 -1 mon.smithi045@0(probing) e2 get_health_metrics reporting 23372 slow ops, oldest is monmgrreport(gid 14214, 0 checks, 1 progress events) 2023-12-16T22:03:41.815 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:03:41 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:03:41.513+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:03:41.815 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:03:41 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:03:41.513+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:03:41.815 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:03:41 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:03:41.513+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:03:41.815 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:03:41 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:03:41.514+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:03:41.816 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:03:41 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:03:41.694+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:03:41.816 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:03:41 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:03:41.694+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:03:41.816 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:03:41 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:03:41.694+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:03:41.816 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:03:41 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:03:41.696+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:03:42.472 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:03:42 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:03:42.216+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:03:42.472 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:03:42 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:03:42.216+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:03:42.472 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:03:42 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:03:42.216+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:03:42.472 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:03:42 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:03:42.217+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:03:42.814 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:03:42 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:03:42.712+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:03:42.815 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:03:42 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:03:42.712+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:03:42.815 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:03:42 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:03:42.712+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:03:42.815 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:03:42 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:03:42.713+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:03:42.815 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:03:42 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:03:42.470+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:03:42.815 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:03:42 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:03:42.470+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:03:42.816 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:03:42 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:03:42.470+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:03:42.816 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:03:42 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:03:42.471+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:03:43.565 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:03:43 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:03:43.254+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:03:43.565 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:03:43 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:03:43.255+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:03:43.565 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:03:43 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:03:43.255+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:03:43.565 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:03:43 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:03:43.256+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:03:43.565 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:03:43 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:03:43.425+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:03:43.566 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:03:43 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:03:43.425+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:03:43.566 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:03:43 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:03:43.425+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:03:43.566 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:03:43 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:03:43.426+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:03:44.064 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:03:43 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:03:43.714+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:03:44.065 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:03:43 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:03:43.714+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:03:44.065 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:03:43 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:03:43.714+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:03:44.065 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:03:43 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:03:43.714+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:03:44.564 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:03:44 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:03:44.391+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:03:44.565 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:03:44 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:03:44.391+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:03:44.565 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:03:44 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:03:44.391+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:03:44.565 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:03:44 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:03:44.392+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:03:44.566 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:03:44 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:03:44.286+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:03:44.566 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:03:44 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:03:44.286+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:03:44.566 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:03:44 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:03:44.286+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:03:44.566 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:03:44 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:03:44.286+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:03:45.065 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:03:44 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:03:44.666+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:03:45.065 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:03:44 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:03:44.666+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:03:45.065 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:03:44 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:03:44.666+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:03:45.065 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:03:44 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:03:44.668+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:03:45.564 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:03:45 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:03:45.279+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:03:45.565 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:03:45 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:03:45.279+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:03:45.565 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:03:45 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:03:45.279+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:03:45.565 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:03:45 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:03:45.279+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:03:45.565 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:03:45 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:03:45.387+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:03:45.565 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:03:45 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:03:45.387+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:03:45.566 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:03:45 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:03:45.387+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:03:45.566 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:03:45 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:03:45.388+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:03:46.064 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:03:45 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:03:45.634+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:03:46.065 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:03:45 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:03:45.634+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:03:46.065 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:03:45 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:03:45.634+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:03:46.065 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:03:45 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:03:45.634+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:03:46.564 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:03:46 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:03:46.243+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:03:46.565 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:03:46 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:03:46.243+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:03:46.565 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:03:46 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:03:46.243+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:03:46.565 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:03:46 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:03:46.244+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:03:46.565 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:03:46 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:03:46.409+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:03:46.565 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:03:46 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:03:46.409+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:03:46.566 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:03:46 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:03:46.409+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:03:46.566 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:03:46 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:03:46.410+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:03:47.064 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Dec 16 22:03:46 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-mon-smithi045[80759]: 2023-12-16T22:03:46.698+0000 7f45a1f8f700 -1 mon.smithi045@0(probing) e2 get_health_metrics reporting 23461 slow ops, oldest is monmgrreport(gid 14214, 0 checks, 1 progress events) 2023-12-16T22:03:47.065 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:03:46 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:03:46.633+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:03:47.065 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:03:46 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:03:46.633+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:03:47.065 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:03:46 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:03:46.633+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:03:47.065 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:03:46 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:03:46.634+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:03:47.565 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:03:47 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:03:47.225+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:03:47.565 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:03:47 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:03:47.225+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:03:47.565 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:03:47 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:03:47.225+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:03:47.565 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:03:47 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:03:47.226+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:03:47.565 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:03:47 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:03:47.427+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:03:47.566 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:03:47 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:03:47.427+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:03:47.566 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:03:47 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:03:47.427+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:03:47.566 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:03:47 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:03:47.428+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:03:48.065 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:03:47 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:03:47.601+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:03:48.065 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:03:47 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:03:47.601+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:03:48.065 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:03:47 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:03:47.601+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:03:48.065 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:03:47 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:03:47.602+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:03:48.564 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:03:48 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:03:48.186+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:03:48.565 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:03:48 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:03:48.186+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:03:48.565 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:03:48 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:03:48.186+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:03:48.565 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:03:48 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:03:48.187+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:03:48.565 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:03:48 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:03:48.409+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:03:48.565 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:03:48 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:03:48.409+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:03:48.566 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:03:48 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:03:48.409+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:03:48.566 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:03:48 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:03:48.410+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:03:49.065 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:03:48 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:03:48.623+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:03:49.065 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:03:48 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:03:48.623+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:03:49.065 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:03:48 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:03:48.623+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:03:49.066 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:03:48 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:03:48.624+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:03:49.433 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:03:49 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:03:49.185+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:03:49.433 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:03:49 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:03:49.185+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:03:49.433 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:03:49 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:03:49.185+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:03:49.433 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:03:49 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:03:49.186+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:03:49.814 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:03:49 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:03:49.431+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:03:49.815 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:03:49 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:03:49.431+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:03:49.815 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:03:49 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:03:49.431+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:03:49.815 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:03:49 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:03:49.432+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:03:49.815 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:03:49 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:03:49.625+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:03:49.815 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:03:49 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:03:49.625+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:03:49.815 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:03:49 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:03:49.625+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:03:49.816 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:03:49 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:03:49.626+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:03:50.452 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:03:50 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:03:50.153+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:03:50.452 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:03:50 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:03:50.153+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:03:50.452 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:03:50 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:03:50.153+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:03:50.452 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:03:50 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:03:50.154+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:03:50.815 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:03:50 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:03:50.450+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:03:50.815 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:03:50 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:03:50.450+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:03:50.815 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:03:50 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:03:50.450+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:03:50.815 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:03:50 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:03:50.452+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:03:50.815 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:03:50 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:03:50.642+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:03:50.816 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:03:50 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:03:50.642+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:03:50.816 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:03:50 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:03:50.642+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:03:50.816 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:03:50 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:03:50.643+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:03:51.564 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:03:51 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:03:51.175+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:03:51.565 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:03:51 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:03:51.176+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:03:51.565 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:03:51 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:03:51.176+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:03:51.565 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:03:51 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:03:51.177+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:03:51.565 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:03:51 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:03:51.401+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:03:51.565 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:03:51 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:03:51.401+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:03:51.566 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:03:51 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:03:51.401+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:03:51.566 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:03:51 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:03:51.402+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:03:52.065 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Dec 16 22:03:51 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-mon-smithi045[80759]: 2023-12-16T22:03:51.699+0000 7f45a1f8f700 -1 mon.smithi045@0(probing) e2 get_health_metrics reporting 23554 slow ops, oldest is monmgrreport(gid 14214, 0 checks, 1 progress events) 2023-12-16T22:03:52.065 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:03:51 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:03:51.643+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:03:52.065 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:03:51 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:03:51.643+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:03:52.065 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:03:51 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:03:51.643+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:03:52.066 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:03:51 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:03:51.644+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:03:52.564 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:03:52 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:03:52.223+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:03:52.565 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:03:52 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:03:52.223+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:03:52.565 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:03:52 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:03:52.223+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:03:52.565 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:03:52 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:03:52.224+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:03:52.565 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:03:52 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:03:52.445+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:03:52.566 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:03:52 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:03:52.445+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:03:52.566 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:03:52 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:03:52.445+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:03:52.566 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:03:52 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:03:52.446+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:03:52.976 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:03:52 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:03:52.631+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:03:52.976 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:03:52 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:03:52.631+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:03:52.976 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:03:52 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:03:52.631+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:03:52.977 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:03:52 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:03:52.632+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:03:53.314 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:03:53 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:03:53.216+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:03:53.315 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:03:53 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:03:53.216+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:03:53.315 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:03:53 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:03:53.216+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:03:53.315 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:03:53 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:03:53.218+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:03:53.814 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:03:53 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:03:53.614+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:03:53.815 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:03:53 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:03:53.614+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:03:53.815 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:03:53 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:03:53.614+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:03:53.815 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:03:53 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:03:53.614+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:03:53.815 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:03:53 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:03:53.454+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:03:53.816 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:03:53 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:03:53.454+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:03:53.816 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:03:53 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:03:53.454+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:03:53.816 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:03:53 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:03:53.455+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:03:54.495 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:03:54 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:03:54.191+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:03:54.496 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:03:54 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:03:54.191+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:03:54.496 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:03:54 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:03:54.191+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:03:54.496 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:03:54 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:03:54.191+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:03:54.815 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:03:54 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:03:54.650+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:03:54.815 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:03:54 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:03:54.650+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:03:54.815 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:03:54 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:03:54.650+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:03:54.815 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:03:54 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:03:54.650+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:03:54.815 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:03:54 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:03:54.492+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:03:54.816 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:03:54 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:03:54.492+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:03:54.816 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:03:54 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:03:54.493+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:03:54.816 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:03:54 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:03:54.494+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:03:55.511 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:03:55 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:03:55.165+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:03:55.512 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:03:55 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:03:55.165+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:03:55.512 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:03:55 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:03:55.165+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:03:55.512 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:03:55 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:03:55.166+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:03:55.814 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:03:55 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:03:55.654+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:03:55.815 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:03:55 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:03:55.654+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:03:55.815 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:03:55 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:03:55.654+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:03:55.815 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:03:55 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:03:55.656+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:03:55.815 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:03:55 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:03:55.509+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:03:55.816 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:03:55 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:03:55.509+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:03:55.816 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:03:55 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:03:55.509+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:03:55.816 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:03:55 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:03:55.510+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:03:56.314 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:03:56 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:03:56.176+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:03:56.315 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:03:56 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:03:56.176+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:03:56.315 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:03:56 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:03:56.176+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:03:56.315 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:03:56 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:03:56.177+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:03:56.814 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Dec 16 22:03:56 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-mon-smithi045[80759]: 2023-12-16T22:03:56.703+0000 7f45a1f8f700 -1 mon.smithi045@0(probing) e2 get_health_metrics reporting 23643 slow ops, oldest is monmgrreport(gid 14214, 0 checks, 1 progress events) 2023-12-16T22:03:56.815 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:03:56 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:03:56.663+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:03:56.815 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:03:56 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:03:56.663+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:03:56.815 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:03:56 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:03:56.663+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:03:56.815 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:03:56 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:03:56.664+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:03:56.816 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:03:56 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:03:56.551+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:03:56.816 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:03:56 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:03:56.551+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:03:56.816 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:03:56 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:03:56.551+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:03:56.816 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:03:56 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:03:56.552+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:03:57.530 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:03:57 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:03:57.130+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:03:57.530 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:03:57 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:03:57.130+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:03:57.530 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:03:57 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:03:57.130+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:03:57.531 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:03:57 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:03:57.131+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:03:57.815 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:03:57 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:03:57.654+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:03:57.815 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:03:57 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:03:57.654+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:03:57.815 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:03:57 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:03:57.654+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:03:57.815 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:03:57 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:03:57.655+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:03:57.815 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:03:57 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:03:57.529+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:03:57.816 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:03:57 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:03:57.529+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:03:57.816 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:03:57 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:03:57.529+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:03:57.816 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:03:57 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:03:57.529+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:03:58.490 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:03:58 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:03:58.101+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:03:58.490 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:03:58 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:03:58.101+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:03:58.490 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:03:58 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:03:58.101+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:03:58.493 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:03:58 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:03:58.102+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:03:58.814 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:03:58 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:03:58.488+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:03:58.815 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:03:58 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:03:58.488+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:03:58.815 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:03:58 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:03:58.488+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:03:58.815 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:03:58 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:03:58.489+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:03:58.815 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:03:58 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:03:58.639+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:03:58.815 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:03:58 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:03:58.640+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:03:58.815 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:03:58 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:03:58.640+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:03:58.816 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:03:58 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:03:58.641+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:03:59.537 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:03:59 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:03:59.100+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:03:59.537 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:03:59 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:03:59.100+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:03:59.537 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:03:59 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:03:59.100+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:03:59.537 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:03:59 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:03:59.101+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:03:59.814 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:03:59 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:03:59.535+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:03:59.815 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:03:59 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:03:59.535+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:03:59.815 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:03:59 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:03:59.535+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:03:59.815 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:03:59 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:03:59.537+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:03:59.815 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:03:59 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:03:59.621+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:03:59.815 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:03:59 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:03:59.621+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:03:59.815 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:03:59 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:03:59.621+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:03:59.815 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:03:59 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:03:59.622+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:04:00.564 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:04:00 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:04:00.148+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:04:00.565 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:04:00 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:04:00.148+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:04:00.565 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:04:00 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:04:00.148+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:04:00.565 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:04:00 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:04:00.149+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:04:01.064 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:04:00 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:04:00.632+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:04:01.065 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:04:00 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:04:00.632+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:04:01.065 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:04:00 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:04:00.632+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:04:01.065 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:04:00 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:04:00.633+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:04:01.065 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:04:00 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:04:00.584+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:04:01.065 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:04:00 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:04:00.584+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:04:01.065 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:04:00 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:04:00.584+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:04:01.066 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:04:00 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:04:00.585+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:04:01.565 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:04:01 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:04:01.119+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:04:01.565 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:04:01 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:04:01.119+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:04:01.565 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:04:01 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:04:01.119+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:04:01.565 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:04:01 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:04:01.121+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:04:02.065 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Dec 16 22:04:01 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-mon-smithi045[80759]: 2023-12-16T22:04:01.706+0000 7f45a1f8f700 -1 mon.smithi045@0(probing) e2 get_health_metrics reporting 23735 slow ops, oldest is monmgrreport(gid 14214, 0 checks, 1 progress events) 2023-12-16T22:04:02.065 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:04:01 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:04:01.682+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:04:02.065 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:04:01 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:04:01.682+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:04:02.065 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:04:01 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:04:01.682+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:04:02.066 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:04:01 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:04:01.682+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:04:02.066 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:04:01 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:04:01.632+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:04:02.066 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:04:01 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:04:01.632+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:04:02.066 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:04:01 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:04:01.632+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:04:02.066 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:04:01 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:04:01.633+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:04:02.565 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:04:02 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:04:02.155+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:04:02.565 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:04:02 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:04:02.155+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:04:02.565 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:04:02 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:04:02.155+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:04:02.565 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:04:02 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:04:02.156+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:04:03.064 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:04:02 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:04:02.698+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:04:03.065 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:04:02 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:04:02.698+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:04:03.065 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:04:02 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:04:02.698+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:04:03.065 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:04:02 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:04:02.699+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:04:03.065 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:04:02 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:04:02.633+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:04:03.065 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:04:02 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:04:02.633+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:04:03.066 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:04:02 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:04:02.633+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:04:03.066 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:04:02 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:04:02.633+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:04:03.564 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:04:03 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:04:03.140+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:04:03.565 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:04:03 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:04:03.140+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:04:03.565 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:04:03 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:04:03.140+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:04:03.565 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:04:03 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:04:03.141+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:04:04.065 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:04:03 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:04:03.696+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:04:04.065 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:04:03 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:04:03.696+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:04:04.065 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:04:03 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:04:03.696+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:04:04.065 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:04:03 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:04:03.696+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:04:04.066 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:04:03 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:04:03.618+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:04:04.066 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:04:03 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:04:03.618+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:04:04.066 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:04:03 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:04:03.618+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:04:04.066 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:04:03 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:04:03.618+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:04:04.564 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:04:04 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:04:04.118+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:04:04.565 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:04:04 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:04:04.118+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:04:04.565 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:04:04 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:04:04.118+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:04:04.565 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:04:04 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:04:04.118+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:04:05.065 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:04:04 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:04:04.719+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:04:05.065 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:04:04 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:04:04.719+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:04:05.065 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:04:04 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:04:04.719+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:04:05.065 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:04:04 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:04:04.720+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:04:05.065 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:04:04 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:04:04.582+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:04:05.066 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:04:04 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:04:04.582+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:04:05.066 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:04:04 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:04:04.582+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:04:05.066 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:04:04 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:04:04.583+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:04:05.565 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:04:05 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:04:05.107+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:04:05.565 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:04:05 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:04:05.107+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:04:05.565 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:04:05 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:04:05.107+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:04:05.565 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:04:05 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:04:05.108+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:04:06.065 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:04:05 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:04:05.712+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:04:06.065 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:04:05 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:04:05.712+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:04:06.065 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:04:05 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:04:05.712+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:04:06.065 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:04:05 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:04:05.713+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:04:06.066 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:04:05 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:04:05.622+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:04:06.066 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:04:05 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:04:05.622+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:04:06.066 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:04:05 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:04:05.622+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:04:06.066 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:04:05 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:04:05.622+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:04:06.565 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:04:06 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:04:06.131+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:04:06.565 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:04:06 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:04:06.131+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:04:06.565 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:04:06 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:04:06.131+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:04:06.565 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:04:06 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:04:06.131+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:04:07.064 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Dec 16 22:04:06 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-mon-smithi045[80759]: 2023-12-16T22:04:06.708+0000 7f45a1f8f700 -1 mon.smithi045@0(probing) e2 get_health_metrics reporting 23824 slow ops, oldest is monmgrreport(gid 14214, 0 checks, 1 progress events) 2023-12-16T22:04:07.065 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:04:06 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:04:06.668+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:04:07.065 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:04:06 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:04:06.668+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:04:07.065 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:04:06 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:04:06.668+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:04:07.065 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:04:06 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:04:06.669+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:04:07.066 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:04:06 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:04:06.733+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:04:07.066 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:04:06 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:04:06.733+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:04:07.066 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:04:06 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:04:06.734+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:04:07.066 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:04:06 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:04:06.735+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:04:07.565 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:04:07 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:04:07.103+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:04:07.565 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:04:07 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:04:07.103+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:04:07.565 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:04:07 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:04:07.103+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:04:07.566 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:04:07 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:04:07.104+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:04:08.064 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:04:07 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:04:07.751+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:04:08.065 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:04:07 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:04:07.751+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:04:08.065 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:04:07 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:04:07.751+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:04:08.065 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:04:07 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:04:07.752+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:04:08.065 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:04:07 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:04:07.621+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:04:08.065 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:04:07 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:04:07.621+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:04:08.066 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:04:07 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:04:07.621+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:04:08.066 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:04:07 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:04:07.622+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:04:08.564 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:04:08 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:04:08.104+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:04:08.565 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:04:08 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:04:08.104+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:04:08.565 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:04:08 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:04:08.104+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:04:08.565 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:04:08 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:04:08.104+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:04:09.065 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:04:08 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:04:08.753+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:04:09.065 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:04:08 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:04:08.753+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:04:09.065 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:04:08 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:04:08.753+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:04:09.065 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:04:08 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:04:08.754+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:04:09.065 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:04:08 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:04:08.624+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:04:09.066 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:04:08 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:04:08.624+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:04:09.066 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:04:08 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:04:08.624+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:04:09.066 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:04:08 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:04:08.626+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:04:09.564 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:04:09 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:04:09.078+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:04:09.565 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:04:09 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:04:09.078+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:04:09.565 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:04:09 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:04:09.078+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:04:09.565 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:04:09 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:04:09.079+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:04:10.064 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:04:09 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:04:09.755+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:04:10.065 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:04:09 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:04:09.755+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:04:10.065 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:04:09 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:04:09.755+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:04:10.065 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:04:09 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:04:09.755+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:04:10.065 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:04:09 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:04:09.659+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:04:10.065 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:04:09 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:04:09.659+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:04:10.066 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:04:09 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:04:09.659+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:04:10.066 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:04:09 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:04:09.660+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:04:10.564 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:04:10 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:04:10.096+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:04:10.565 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:04:10 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:04:10.096+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:04:10.565 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:04:10 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:04:10.096+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:04:10.565 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:04:10 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:04:10.096+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:04:11.064 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:04:10 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:04:10.633+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:04:11.065 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:04:10 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:04:10.633+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:04:11.065 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:04:10 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:04:10.633+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:04:11.065 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:04:10 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:04:10.635+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:04:11.065 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:04:10 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:04:10.747+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:04:11.065 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:04:10 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:04:10.747+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:04:11.065 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:04:10 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:04:10.747+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:04:11.066 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:04:10 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:04:10.747+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:04:11.407 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:04:11 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:04:11.107+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:04:11.407 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:04:11 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:04:11.107+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:04:11.407 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:04:11 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:04:11.107+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:04:11.407 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:04:11 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:04:11.108+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:04:11.712 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:04:11 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:04:11.623+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:04:11.712 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:04:11 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:04:11.623+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:04:11.713 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:04:11 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:04:11.623+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:04:11.713 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:04:11 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:04:11.624+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:04:12.065 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Dec 16 22:04:11 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-mon-smithi045[80759]: 2023-12-16T22:04:11.711+0000 7f45a1f8f700 -1 mon.smithi045@0(probing) e2 get_health_metrics reporting 23917 slow ops, oldest is monmgrreport(gid 14214, 0 checks, 1 progress events) 2023-12-16T22:04:12.065 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:04:11 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:04:11.756+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:04:12.065 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:04:11 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:04:11.756+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:04:12.066 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:04:11 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:04:11.756+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:04:12.066 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:04:11 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:04:11.757+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:04:12.564 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:04:12 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:04:12.065+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:04:12.565 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:04:12 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:04:12.065+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:04:12.565 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:04:12 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:04:12.065+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:04:12.565 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:04:12 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:04:12.066+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:04:13.065 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:04:12 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:04:12.724+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:04:13.065 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:04:12 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:04:12.724+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:04:13.065 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:04:12 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:04:12.724+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:04:13.065 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:04:12 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:04:12.725+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:04:13.065 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:04:12 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:04:12.660+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:04:13.066 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:04:12 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:04:12.660+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:04:13.066 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:04:12 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:04:12.660+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:04:13.066 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:04:12 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:04:12.662+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:04:13.564 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:04:13 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:04:13.084+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:04:13.565 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:04:13 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:04:13.084+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:04:13.565 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:04:13 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:04:13.084+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:04:13.565 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:04:13 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:04:13.085+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:04:14.065 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:04:13 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:04:13.743+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:04:14.065 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:04:13 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:04:13.743+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:04:14.065 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:04:13 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:04:13.743+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:04:14.065 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:04:13 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:04:13.744+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:04:14.066 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:04:13 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:04:13.630+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:04:14.066 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:04:13 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:04:13.630+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:04:14.066 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:04:13 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:04:13.630+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:04:14.066 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:04:13 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:04:13.631+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:04:14.479 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:04:14 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:04:14.119+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:04:14.479 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:04:14 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:04:14.119+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:04:14.479 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:04:14 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:04:14.119+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:04:14.480 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:04:14 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:04:14.120+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:04:14.769 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:04:14 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:04:14.635+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:04:14.769 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:04:14 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:04:14.635+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:04:14.769 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:04:14 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:04:14.635+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:04:14.769 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:04:14 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:04:14.637+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:04:15.065 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:04:14 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:04:14.768+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:04:15.065 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:04:14 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:04:14.768+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:04:15.065 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:04:14 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:04:14.768+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:04:15.066 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:04:14 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:04:14.769+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:04:15.564 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:04:15 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:04:15.142+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:04:15.565 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:04:15 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:04:15.142+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:04:15.565 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:04:15 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:04:15.142+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:04:15.565 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:04:15 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:04:15.143+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:04:16.065 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:04:15 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:04:15.679+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:04:16.065 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:04:15 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:04:15.679+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:04:16.065 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:04:15 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:04:15.679+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:04:16.065 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:04:15 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:04:15.680+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:04:16.065 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:04:15 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:04:15.737+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:04:16.066 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:04:15 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:04:15.737+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:04:16.066 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:04:15 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:04:15.737+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:04:16.066 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:04:15 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:04:15.738+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:04:16.564 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:04:16 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:04:16.133+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:04:16.565 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:04:16 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:04:16.133+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:04:16.565 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:04:16 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:04:16.133+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:04:16.565 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:04:16 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:04:16.134+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:04:17.065 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Dec 16 22:04:16 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-mon-smithi045[80759]: 2023-12-16T22:04:16.714+0000 7f45a1f8f700 -1 mon.smithi045@0(probing) e2 get_health_metrics reporting 24006 slow ops, oldest is monmgrreport(gid 14214, 0 checks, 1 progress events) 2023-12-16T22:04:17.065 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:04:16 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:04:16.764+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:04:17.066 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:04:16 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:04:16.764+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:04:17.066 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:04:16 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:04:16.764+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:04:17.066 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:04:16 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:04:16.764+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:04:17.067 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:04:16 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:04:16.659+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:04:17.067 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:04:16 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:04:16.659+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:04:17.067 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:04:16 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:04:16.659+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:04:17.067 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:04:16 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:04:16.660+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:04:17.564 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:04:17 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:04:17.099+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:04:17.565 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:04:17 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:04:17.099+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:04:17.565 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:04:17 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:04:17.099+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:04:17.565 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:04:17 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:04:17.100+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:04:18.064 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:04:17 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:04:17.734+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:04:18.065 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:04:17 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:04:17.734+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:04:18.065 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:04:17 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:04:17.734+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:04:18.065 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:04:17 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:04:17.734+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:04:18.065 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:04:17 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:04:17.631+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:04:18.065 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:04:17 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:04:17.631+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:04:18.066 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:04:17 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:04:17.631+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:04:18.066 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:04:17 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:04:17.632+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:04:18.462 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:04:18 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:04:18.143+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:04:18.462 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:04:18 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:04:18.143+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:04:18.462 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:04:18 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:04:18.143+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:04:18.462 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:04:18 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:04:18.144+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:04:18.720 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:04:18 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:04:18.590+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:04:18.721 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:04:18 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:04:18.590+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:04:18.721 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:04:18 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:04:18.590+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:04:18.721 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:04:18 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:04:18.591+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:04:19.064 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:04:18 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:04:18.753+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:04:19.065 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:04:18 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:04:18.753+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:04:19.065 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:04:18 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:04:18.753+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:04:19.066 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:04:18 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:04:18.754+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:04:19.564 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:04:19 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:04:19.191+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:04:19.565 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:04:19 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:04:19.191+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:04:19.565 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:04:19 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:04:19.191+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:04:19.565 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:04:19 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:04:19.192+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:04:20.064 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:04:19 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:04:19.717+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:04:20.065 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:04:19 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:04:19.717+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:04:20.065 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:04:19 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:04:19.717+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:04:20.065 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:04:19 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:04:19.718+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:04:20.065 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:04:19 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:04:19.617+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:04:20.065 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:04:19 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:04:19.617+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:04:20.066 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:04:19 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:04:19.617+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:04:20.066 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:04:19 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:04:19.618+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:04:20.564 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:04:20 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:04:20.200+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:04:20.565 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:04:20 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:04:20.200+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:04:20.565 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:04:20 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:04:20.200+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:04:20.565 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:04:20 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:04:20.200+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:04:21.065 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:04:20 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:04:20.684+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:04:21.065 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:04:20 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:04:20.684+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:04:21.065 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:04:20 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:04:20.684+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:04:21.065 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:04:20 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:04:20.684+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:04:21.065 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:04:20 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:04:20.625+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:04:21.066 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:04:20 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:04:20.625+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:04:21.066 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:04:20 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:04:20.625+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:04:21.066 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:04:20 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:04:20.626+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:04:21.564 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:04:21 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:04:21.163+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:04:21.565 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:04:21 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:04:21.163+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:04:21.565 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:04:21 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:04:21.163+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:04:21.565 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:04:21 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:04:21.163+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:04:22.065 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Dec 16 22:04:21 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-mon-smithi045[80759]: 2023-12-16T22:04:21.716+0000 7f45a1f8f700 -1 mon.smithi045@0(probing) e2 get_health_metrics reporting 24098 slow ops, oldest is monmgrreport(gid 14214, 0 checks, 1 progress events) 2023-12-16T22:04:22.065 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:04:21 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:04:21.673+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:04:22.065 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:04:21 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:04:21.673+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:04:22.065 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:04:21 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:04:21.673+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:04:22.066 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:04:21 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:04:21.674+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:04:22.066 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:04:21 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:04:21.606+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:04:22.066 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:04:21 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:04:21.606+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:04:22.066 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:04:21 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:04:21.606+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:04:22.066 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:04:21 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:04:21.608+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:04:22.564 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:04:22 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:04:22.205+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:04:22.565 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:04:22 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:04:22.206+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:04:22.565 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:04:22 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:04:22.206+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:04:22.565 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:04:22 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:04:22.206+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:04:23.064 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:04:22 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:04:22.633+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:04:23.065 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:04:22 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:04:22.633+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:04:23.065 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:04:22 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:04:22.633+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:04:23.065 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:04:22 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:04:22.634+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:04:23.065 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:04:22 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:04:22.633+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:04:23.066 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:04:22 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:04:22.633+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:04:23.066 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:04:22 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:04:22.633+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:04:23.066 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:04:22 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:04:22.634+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:04:23.564 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:04:23 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:04:23.170+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:04:23.565 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:04:23 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:04:23.170+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:04:23.565 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:04:23 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:04:23.170+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:04:23.565 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:04:23 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:04:23.170+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:04:24.065 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:04:23 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:04:23.646+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:04:24.065 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:04:23 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:04:23.646+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:04:24.065 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:04:23 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:04:23.646+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:04:24.065 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:04:23 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:04:23.647+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:04:24.065 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:04:23 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:04:23.648+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:04:24.066 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:04:23 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:04:23.648+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:04:24.066 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:04:23 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:04:23.648+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:04:24.066 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:04:23 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:04:23.648+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:04:24.564 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:04:24 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:04:24.204+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:04:24.565 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:04:24 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:04:24.204+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:04:24.565 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:04:24 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:04:24.204+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:04:24.565 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:04:24 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:04:24.204+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:04:25.064 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:04:24 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:04:24.628+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:04:25.065 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:04:24 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:04:24.628+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:04:25.065 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:04:24 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:04:24.628+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:04:25.065 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:04:24 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:04:24.629+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:04:25.065 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:04:24 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:04:24.607+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:04:25.065 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:04:24 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:04:24.607+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:04:25.066 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:04:24 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:04:24.607+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:04:25.066 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:04:24 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:04:24.608+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:04:25.564 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:04:25 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:04:25.209+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:04:25.565 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:04:25 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:04:25.209+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:04:25.565 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:04:25 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:04:25.209+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:04:25.565 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:04:25 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:04:25.210+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:04:26.064 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:04:25 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:04:25.616+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:04:26.065 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:04:25 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:04:25.616+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:04:26.065 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:04:25 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:04:25.616+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:04:26.065 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:04:25 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:04:25.617+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:04:26.065 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:04:25 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:04:25.600+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:04:26.066 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:04:25 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:04:25.600+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:04:26.066 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:04:25 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:04:25.600+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:04:26.066 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:04:25 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:04:25.601+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:04:26.565 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:04:26 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:04:26.191+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:04:26.565 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:04:26 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:04:26.191+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:04:26.565 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:04:26 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:04:26.191+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:04:26.565 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:04:26 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:04:26.192+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:04:27.065 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Dec 16 22:04:26 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-mon-smithi045[80759]: 2023-12-16T22:04:26.719+0000 7f45a1f8f700 -1 mon.smithi045@0(probing) e2 get_health_metrics reporting 24187 slow ops, oldest is monmgrreport(gid 14214, 0 checks, 1 progress events) 2023-12-16T22:04:27.065 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:04:26 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:04:26.605+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:04:27.065 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:04:26 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:04:26.605+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:04:27.065 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:04:26 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:04:26.605+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:04:27.066 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:04:26 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:04:26.606+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:04:27.066 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:04:26 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:04:26.639+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:04:27.066 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:04:26 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:04:26.639+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:04:27.066 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:04:26 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:04:26.639+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:04:27.066 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:04:26 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:04:26.640+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:04:27.564 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:04:27 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:04:27.175+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:04:27.565 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:04:27 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:04:27.175+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:04:27.565 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:04:27 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:04:27.175+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:04:27.565 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:04:27 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:04:27.176+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:04:28.064 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:04:27 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:04:27.588+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:04:28.065 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:04:27 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:04:27.588+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:04:28.065 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:04:27 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:04:27.588+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:04:28.065 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:04:27 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:04:27.590+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:04:28.065 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:04:27 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:04:27.633+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:04:28.065 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:04:27 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:04:27.633+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:04:28.066 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:04:27 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:04:27.633+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:04:28.066 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:04:27 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:04:27.634+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:04:28.564 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:04:28 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:04:28.161+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:04:28.565 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:04:28 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:04:28.161+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:04:28.565 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:04:28 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:04:28.161+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:04:28.565 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:04:28 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:04:28.162+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:04:29.064 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:04:28 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:04:28.623+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:04:29.065 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:04:28 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:04:28.623+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:04:29.065 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:04:28 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:04:28.623+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:04:29.065 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:04:28 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:04:28.624+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:04:29.065 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:04:28 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:04:28.638+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:04:29.065 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:04:28 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:04:28.638+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:04:29.066 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:04:28 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:04:28.638+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:04:29.066 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:04:28 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:04:28.639+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:04:29.564 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:04:29 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:04:29.179+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:04:29.565 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:04:29 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:04:29.179+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:04:29.565 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:04:29 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:04:29.179+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:04:29.565 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:04:29 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:04:29.180+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:04:30.064 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:04:29 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:04:29.641+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:04:30.065 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:04:29 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:04:29.641+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:04:30.065 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:04:29 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:04:29.641+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:04:30.065 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:04:29 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:04:29.642+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:04:30.065 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:04:29 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:04:29.683+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:04:30.065 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:04:29 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:04:29.683+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:04:30.066 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:04:29 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:04:29.683+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:04:30.066 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:04:29 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:04:29.684+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:04:30.564 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:04:30 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:04:30.189+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:04:30.565 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:04:30 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:04:30.189+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:04:30.565 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:04:30 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:04:30.189+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:04:30.565 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:04:30 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:04:30.190+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:04:31.064 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:04:30 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:04:30.604+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:04:31.065 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:04:30 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:04:30.604+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:04:31.065 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:04:30 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:04:30.604+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:04:31.065 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:04:30 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:04:30.604+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:04:31.065 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:04:30 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:04:30.715+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:04:31.065 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:04:30 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:04:30.715+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:04:31.066 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:04:30 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:04:30.715+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:04:31.066 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:04:30 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:04:30.716+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:04:31.564 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:04:31 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:04:31.206+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:04:31.565 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:04:31 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:04:31.206+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:04:31.565 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:04:31 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:04:31.206+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:04:31.565 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:04:31 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:04:31.206+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:04:31.851 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Dec 16 22:04:31 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-mon-smithi045[80759]: 2023-12-16T22:04:31.722+0000 7f45a1f8f700 -1 mon.smithi045@0(probing) e2 get_health_metrics reporting 24280 slow ops, oldest is monmgrreport(gid 14214, 0 checks, 1 progress events) 2023-12-16T22:04:31.852 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:04:31 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:04:31.565+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:04:31.852 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:04:31 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:04:31.565+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:04:31.852 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:04:31 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:04:31.565+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:04:31.852 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:04:31 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:04:31.566+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:04:31.853 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:04:31 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:04:31.710+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:04:31.853 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:04:31 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:04:31.710+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:04:31.853 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:04:31 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:04:31.710+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:04:31.853 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:04:31 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:04:31.710+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:04:32.564 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:04:32 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:04:32.157+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:04:32.565 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:04:32 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:04:32.157+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:04:32.565 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:04:32 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:04:32.157+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:04:32.565 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:04:32 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:04:32.158+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:04:32.876 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:04:32 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:04:32.591+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:04:32.876 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:04:32 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:04:32.591+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:04:32.877 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:04:32 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:04:32.591+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:04:32.877 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:04:32 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:04:32.592+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:04:32.877 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:04:32 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:04:32.733+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:04:32.878 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:04:32 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:04:32.733+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:04:32.878 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:04:32 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:04:32.733+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:04:32.878 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:04:32 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:04:32.734+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:04:33.170 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:04:33 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:04:33.118+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:04:33.170 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:04:33 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:04:33.118+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:04:33.170 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:04:33 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:04:33.118+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:04:33.171 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:04:33 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:04:33.119+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:04:34.065 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:04:33 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:04:33.626+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:04:34.065 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:04:33 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:04:33.627+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:04:34.065 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:04:33 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:04:33.627+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:04:34.065 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:04:33 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:04:33.628+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:04:34.065 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:04:33 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:04:33.762+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:04:34.066 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:04:33 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:04:33.762+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:04:34.066 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:04:33 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:04:33.762+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:04:34.066 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:04:33 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:04:33.762+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:04:34.564 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:04:34 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:04:34.085+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:04:34.565 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:04:34 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:04:34.085+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:04:34.565 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:04:34 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:04:34.085+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:04:34.565 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:04:34 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:04:34.086+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:04:35.064 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:04:34 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:04:34.604+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:04:35.065 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:04:34 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:04:34.604+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:04:35.065 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:04:34 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:04:34.604+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:04:35.065 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:04:34 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:04:34.604+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:04:35.065 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:04:34 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:04:34.729+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:04:35.066 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:04:34 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:04:34.729+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:04:35.066 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:04:34 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:04:34.729+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:04:35.066 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:04:34 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:04:34.730+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:04:35.564 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:04:35 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:04:35.124+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:04:35.565 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:04:35 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:04:35.124+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:04:35.565 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:04:35 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:04:35.124+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:04:35.565 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:04:35 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:04:35.125+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:04:35.983 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:04:35 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:04:35.701+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:04:35.983 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:04:35 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:04:35.701+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:04:35.983 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:04:35 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:04:35.701+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:04:35.984 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:04:35 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:04:35.702+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:04:35.984 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:04:35 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:04:35.580+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:04:35.984 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:04:35 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:04:35.580+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:04:35.984 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:04:35 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:04:35.580+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:04:35.984 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:04:35 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:04:35.581+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:04:36.314 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:04:36 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:04:36.142+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:04:36.315 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:04:36 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:04:36.142+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:04:36.315 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:04:36 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:04:36.142+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:04:36.315 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:04:36 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:04:36.143+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:04:37.064 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Dec 16 22:04:36 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-mon-smithi045[80759]: 2023-12-16T22:04:36.725+0000 7f45a1f8f700 -1 mon.smithi045@0(probing) e2 get_health_metrics reporting 24369 slow ops, oldest is monmgrreport(gid 14214, 0 checks, 1 progress events) 2023-12-16T22:04:37.065 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:04:36 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:04:36.743+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:04:37.065 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:04:36 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:04:36.743+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:04:37.066 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:04:36 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:04:36.743+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:04:37.066 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:04:36 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:04:36.744+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:04:37.066 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:04:36 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:04:36.630+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:04:37.066 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:04:36 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:04:36.630+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:04:37.066 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:04:36 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:04:36.630+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:04:37.066 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:04:36 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:04:36.631+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:04:37.564 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:04:37 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:04:37.111+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:04:37.565 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:04:37 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:04:37.111+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:04:37.565 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:04:37 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:04:37.111+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:04:37.565 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:04:37 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:04:37.112+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:04:38.065 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:04:37 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:04:37.624+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:04:38.065 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:04:37 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:04:37.624+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:04:38.065 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:04:37 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:04:37.624+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:04:38.065 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:04:37 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:04:37.625+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:04:38.065 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:04:37 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:04:37.712+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:04:38.066 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:04:37 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:04:37.712+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:04:38.066 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:04:37 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:04:37.712+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:04:38.066 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:04:37 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:04:37.713+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:04:38.564 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:04:38 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:04:38.134+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:04:38.565 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:04:38 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:04:38.134+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:04:38.565 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:04:38 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:04:38.134+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:04:38.565 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:04:38 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:04:38.136+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:04:39.065 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:04:38 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:04:38.617+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:04:39.065 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:04:38 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:04:38.617+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:04:39.065 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:04:38 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:04:38.617+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:04:39.065 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:04:38 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:04:38.617+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:04:39.066 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:04:38 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:04:38.717+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:04:39.066 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:04:38 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:04:38.717+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:04:39.066 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:04:38 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:04:38.717+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:04:39.066 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:04:38 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:04:38.717+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:04:39.564 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:04:39 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:04:39.176+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:04:39.565 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:04:39 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:04:39.176+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:04:39.565 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:04:39 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:04:39.176+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:04:39.565 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:04:39 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:04:39.177+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:04:40.064 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:04:39 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:04:39.692+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:04:40.065 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:04:39 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:04:39.692+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:04:40.065 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:04:39 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:04:39.692+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:04:40.065 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:04:39 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:04:39.693+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:04:40.065 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:04:39 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:04:39.616+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:04:40.065 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:04:39 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:04:39.616+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:04:40.065 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:04:39 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:04:39.616+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:04:40.066 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:04:39 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:04:39.617+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:04:40.565 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:04:40 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:04:40.204+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:04:40.565 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:04:40 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:04:40.204+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:04:40.565 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:04:40 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:04:40.204+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:04:40.565 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:04:40 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:04:40.205+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:04:41.064 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:04:40 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:04:40.576+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:04:41.065 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:04:40 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:04:40.576+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:04:41.065 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:04:40 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:04:40.576+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:04:41.065 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:04:40 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:04:40.577+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:04:41.065 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:04:40 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:04:40.730+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:04:41.065 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:04:40 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:04:40.730+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:04:41.065 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:04:40 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:04:40.730+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:04:41.066 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:04:40 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:04:40.731+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:04:41.535 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:04:41 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:04:41.240+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:04:41.535 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:04:41 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:04:41.240+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:04:41.535 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:04:41 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:04:41.240+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:04:41.536 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:04:41 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:04:41.241+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:04:41.815 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Dec 16 22:04:41 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-mon-smithi045[80759]: 2023-12-16T22:04:41.728+0000 7f45a1f8f700 -1 mon.smithi045@0(probing) e2 get_health_metrics reporting 24461 slow ops, oldest is monmgrreport(gid 14214, 0 checks, 1 progress events) 2023-12-16T22:04:41.815 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:04:41 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:04:41.534+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:04:41.815 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:04:41 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:04:41.534+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:04:41.815 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:04:41 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:04:41.534+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:04:41.815 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:04:41 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:04:41.535+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:04:41.816 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:04:41 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:04:41.720+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:04:41.816 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:04:41 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:04:41.720+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:04:41.816 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:04:41 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:04:41.720+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:04:41.816 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:04:41 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:04:41.721+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:04:42.491 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:04:42 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:04:42.221+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:04:42.492 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:04:42 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:04:42.221+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:04:42.492 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:04:42 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:04:42.221+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:04:42.492 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:04:42 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:04:42.221+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:04:42.815 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:04:42 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:04:42.490+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:04:42.815 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:04:42 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:04:42.490+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:04:42.815 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:04:42 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:04:42.490+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:04:42.815 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:04:42 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:04:42.491+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:04:42.815 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:04:42 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:04:42.675+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:04:42.816 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:04:42 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:04:42.675+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:04:42.816 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:04:42 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:04:42.675+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:04:42.816 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:04:42 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:04:42.675+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:04:43.512 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:04:43 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:04:43.180+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:04:43.513 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:04:43 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:04:43.180+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:04:43.513 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:04:43 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:04:43.180+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:04:43.513 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:04:43 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:04:43.181+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:04:43.768 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:04:43 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:04:43.511+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:04:43.769 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:04:43 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:04:43.511+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:04:43.769 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:04:43 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:04:43.511+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:04:43.769 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:04:43 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:04:43.512+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:04:43.769 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:04:43 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:04:43.631+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:04:43.769 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:04:43 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:04:43.631+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:04:43.770 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:04:43 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:04:43.631+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:04:43.770 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:04:43 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:04:43.632+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:04:44.559 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:04:44 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:04:44.170+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:04:44.559 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:04:44 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:04:44.170+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:04:44.559 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:04:44 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:04:44.170+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:04:44.559 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:04:44 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:04:44.171+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:04:44.814 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:04:44 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:04:44.557+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:04:44.815 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:04:44 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:04:44.557+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:04:44.815 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:04:44 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:04:44.557+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:04:44.815 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:04:44 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:04:44.559+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:04:44.815 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:04:44 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:04:44.676+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:04:44.816 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:04:44 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:04:44.676+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:04:44.816 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:04:44 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:04:44.676+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:04:44.816 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:04:44 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:04:44.677+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:04:45.328 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:04:45 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:04:45.176+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:04:45.328 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:04:45 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:04:45.176+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:04:45.329 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:04:45 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:04:45.176+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:04:45.329 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:04:45 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:04:45.177+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:04:45.699 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:04:45 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:04:45.533+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:04:45.700 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:04:45 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:04:45.533+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:04:45.700 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:04:45 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:04:45.533+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:04:45.700 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:04:45 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:04:45.534+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:04:46.065 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:04:45 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:04:45.698+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:04:46.065 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:04:45 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:04:45.698+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:04:46.065 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:04:45 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:04:45.698+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:04:46.065 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:04:45 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:04:45.699+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:04:46.460 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:04:46 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:04:46.200+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:04:46.461 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:04:46 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:04:46.200+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:04:46.461 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:04:46 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:04:46.200+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:04:46.461 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:04:46 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:04:46.201+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:04:46.732 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:04:46 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:04:46.670+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:04:46.732 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:04:46 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:04:46.670+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:04:46.732 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:04:46 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:04:46.670+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:04:46.733 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:04:46 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:04:46.670+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:04:46.733 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:04:46 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:04:46.528+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:04:46.733 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:04:46 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:04:46.528+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:04:46.733 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:04:46 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:04:46.528+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:04:46.733 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:04:46 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:04:46.529+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:04:47.064 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Dec 16 22:04:46 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-mon-smithi045[80759]: 2023-12-16T22:04:46.731+0000 7f45a1f8f700 -1 mon.smithi045@0(probing) e2 get_health_metrics reporting 24550 slow ops, oldest is monmgrreport(gid 14214, 0 checks, 1 progress events) 2023-12-16T22:04:47.560 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:04:47 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:04:47.158+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:04:47.560 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:04:47 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:04:47.158+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:04:47.560 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:04:47 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:04:47.158+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:04:47.560 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:04:47 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:04:47.159+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:04:47.815 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:04:47 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:04:47.559+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:04:47.815 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:04:47 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:04:47.559+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:04:47.815 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:04:47 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:04:47.559+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:04:47.815 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:04:47 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:04:47.560+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:04:47.815 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:04:47 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:04:47.634+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:04:47.816 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:04:47 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:04:47.634+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:04:47.816 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:04:47 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:04:47.634+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:04:47.816 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:04:47 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:04:47.635+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:04:48.564 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:04:48 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:04:48.128+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:04:48.565 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:04:48 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:04:48.128+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:04:48.565 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:04:48 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:04:48.128+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:04:48.565 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:04:48 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:04:48.128+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:04:49.064 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:04:48 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:04:48.571+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:04:49.065 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:04:48 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:04:48.571+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:04:49.065 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:04:48 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:04:48.571+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:04:49.065 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:04:48 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:04:48.571+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:04:49.065 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:04:48 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:04:48.623+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:04:49.065 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:04:48 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:04:48.623+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:04:49.066 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:04:48 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:04:48.623+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:04:49.066 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:04:48 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:04:48.624+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:04:49.564 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:04:49 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:04:49.176+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:04:49.565 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:04:49 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:04:49.176+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:04:49.565 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:04:49 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:04:49.176+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:04:49.565 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:04:49 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:04:49.177+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:04:50.065 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:04:49 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:04:49.583+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:04:50.065 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:04:49 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:04:49.583+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:04:50.065 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:04:49 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:04:49.583+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:04:50.065 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:04:49 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:04:49.583+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:04:50.065 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:04:49 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:04:49.590+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:04:50.066 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:04:49 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:04:49.590+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:04:50.066 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:04:49 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:04:49.590+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:04:50.066 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:04:49 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:04:49.591+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:04:50.543 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:04:50 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:04:50.220+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:04:50.543 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:04:50 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:04:50.220+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:04:50.543 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:04:50 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:04:50.220+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:04:50.543 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:04:50 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:04:50.220+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:04:50.815 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:04:50 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:04:50.612+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:04:50.815 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:04:50 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:04:50.612+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:04:50.815 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:04:50 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:04:50.612+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:04:50.816 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:04:50 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:04:50.613+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:04:50.816 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:04:50 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:04:50.542+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:04:50.816 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:04:50 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:04:50.542+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:04:50.816 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:04:50 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:04:50.542+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:04:50.817 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:04:50 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:04:50.542+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:04:51.519 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:04:51 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:04:51.185+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:04:51.520 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:04:51 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:04:51.185+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:04:51.520 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:04:51 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:04:51.185+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:04:51.520 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:04:51 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:04:51.186+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:04:51.814 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Dec 16 22:04:51 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-mon-smithi045[80759]: 2023-12-16T22:04:51.733+0000 7f45a1f8f700 -1 mon.smithi045@0(probing) e2 get_health_metrics reporting 24642 slow ops, oldest is monmgrreport(gid 14214, 0 checks, 1 progress events) 2023-12-16T22:04:51.815 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:04:51 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:04:51.658+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:04:51.815 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:04:51 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:04:51.658+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:04:51.815 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:04:51 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:04:51.658+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:04:51.815 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:04:51 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:04:51.659+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:04:51.816 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:04:51 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:04:51.518+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:04:51.816 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:04:51 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:04:51.518+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:04:51.816 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:04:51 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:04:51.518+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:04:51.816 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:04:51 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:04:51.519+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:04:52.535 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:04:52 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:04:52.140+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:04:52.535 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:04:52 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:04:52.140+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:04:52.535 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:04:52 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:04:52.140+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:04:52.535 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:04:52 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:04:52.141+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:04:52.814 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:04:52 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:04:52.611+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:04:52.815 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:04:52 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:04:52.611+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:04:52.815 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:04:52 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:04:52.611+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:04:52.815 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:04:52 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:04:52.612+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:04:52.815 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:04:52 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:04:52.534+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:04:52.815 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:04:52 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:04:52.534+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:04:52.815 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:04:52 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:04:52.534+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:04:52.816 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:04:52 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:04:52.534+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:04:53.532 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:04:53 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:04:53.120+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:04:53.532 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:04:53 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:04:53.120+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:04:53.533 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:04:53 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:04:53.120+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:04:53.533 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:04:53 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:04:53.121+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:04:53.814 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:04:53 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:04:53.647+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:04:53.815 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:04:53 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:04:53.647+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:04:53.815 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:04:53 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:04:53.647+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:04:53.815 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:04:53 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:04:53.648+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:04:53.815 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:04:53 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:04:53.531+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:04:53.815 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:04:53 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:04:53.531+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:04:53.816 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:04:53 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:04:53.531+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:04:53.816 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:04:53 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:04:53.532+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:04:54.415 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:04:54 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:04:54.109+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:04:54.415 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:04:54 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:04:54.109+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:04:54.415 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:04:54 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:04:54.109+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:04:54.415 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:04:54 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:04:54.110+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:04:54.681 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:04:54 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:04:54.552+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:04:54.682 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:04:54 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:04:54.552+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:04:54.682 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:04:54 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:04:54.552+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:04:54.682 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:04:54 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:04:54.553+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:04:55.065 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:04:54 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:04:54.680+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:04:55.065 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:04:54 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:04:54.680+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:04:55.065 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:04:54 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:04:54.680+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:04:55.065 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:04:54 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:04:54.682+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:04:55.550 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:04:55 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:04:55.108+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:04:55.550 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:04:55 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:04:55.108+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:04:55.550 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:04:55 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:04:55.108+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:04:55.551 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:04:55 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:04:55.108+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:04:55.815 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:04:55 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:04:55.663+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:04:55.815 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:04:55 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:04:55.663+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:04:55.815 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:04:55 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:04:55.663+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:04:55.815 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:04:55 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:04:55.664+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:04:55.816 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:04:55 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:04:55.549+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:04:55.816 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:04:55 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:04:55.549+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:04:55.816 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:04:55 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:04:55.549+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:04:55.816 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:04:55 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:04:55.550+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:04:56.510 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:04:56 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:04:56.139+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:04:56.510 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:04:56 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:04:56.139+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:04:56.510 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:04:56 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:04:56.139+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:04:56.510 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:04:56 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:04:56.140+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:04:56.814 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Dec 16 22:04:56 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-mon-smithi045[80759]: 2023-12-16T22:04:56.735+0000 7f45a1f8f700 -1 mon.smithi045@0(probing) e2 get_health_metrics reporting 24731 slow ops, oldest is monmgrreport(gid 14214, 0 checks, 1 progress events) 2023-12-16T22:04:56.815 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:04:56 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:04:56.628+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:04:56.815 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:04:56 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:04:56.628+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:04:56.815 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:04:56 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:04:56.628+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:04:56.815 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:04:56 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:04:56.629+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:04:56.815 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:04:56 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:04:56.573+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:04:56.816 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:04:56 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:04:56.573+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:04:56.816 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:04:56 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:04:56.573+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:04:56.816 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:04:56 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:04:56.575+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:04:57.564 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:04:57 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:04:57.140+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:04:57.565 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:04:57 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:04:57.140+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:04:57.565 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:04:57 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:04:57.140+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:04:57.565 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:04:57 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:04:57.141+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:04:58.064 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:04:57 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:04:57.574+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:04:58.065 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:04:57 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:04:57.574+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:04:58.065 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:04:57 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:04:57.574+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:04:58.065 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:04:57 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:04:57.575+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:04:58.065 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:04:57 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:04:57.664+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:04:58.065 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:04:57 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:04:57.664+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:04:58.066 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:04:57 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:04:57.664+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:04:58.066 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:04:57 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:04:57.665+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:04:58.548 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:04:58 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:04:58.113+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:04:58.548 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:04:58 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:04:58.113+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:04:58.548 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:04:58 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:04:58.113+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:04:58.548 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:04:58 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:04:58.114+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:04:58.815 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:04:58 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:04:58.667+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:04:58.815 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:04:58 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:04:58.667+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:04:58.815 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:04:58 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:04:58.667+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:04:58.815 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:04:58 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:04:58.667+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:04:58.815 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:04:58 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:04:58.547+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:04:58.816 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:04:58 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:04:58.547+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:04:58.816 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:04:58 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:04:58.547+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:04:58.816 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:04:58 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:04:58.547+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:04:59.522 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:04:59 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:04:59.111+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:04:59.523 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:04:59 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:04:59.111+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:04:59.523 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:04:59 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:04:59.111+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:04:59.523 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:04:59 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:04:59.111+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:04:59.814 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:04:59 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:04:59.520+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:04:59.815 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:04:59 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:04:59.520+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:04:59.815 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:04:59 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:04:59.520+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:04:59.815 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:04:59 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:04:59.521+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:04:59.815 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:04:59 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:04:59.699+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:04:59.816 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:04:59 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:04:59.699+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:04:59.816 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:04:59 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:04:59.699+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:04:59.816 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:04:59 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:04:59.699+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:05:00.500 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:05:00 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:05:00.072+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:05:00.501 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:05:00 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:05:00.072+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:05:00.501 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:05:00 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:05:00.072+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:05:00.501 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:05:00 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:05:00.073+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:05:00.815 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:05:00 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:05:00.498+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:05:00.815 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:05:00 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:05:00.498+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:05:00.815 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:05:00 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:05:00.498+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:05:00.815 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:05:00 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:05:00.499+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:05:00.815 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:05:00 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:05:00.739+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:05:00.816 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:05:00 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:05:00.739+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:05:00.816 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:05:00 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:05:00.739+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:05:00.816 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:05:00 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:05:00.740+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:05:01.459 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:05:01 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:05:01.080+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:05:01.459 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:05:01 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:05:01.080+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:05:01.459 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:05:01 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:05:01.080+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:05:01.459 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:05:01 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:05:01.080+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:05:01.716 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:05:01 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:05:01.457+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:05:01.716 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:05:01 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:05:01.457+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:05:01.716 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:05:01 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:05:01.457+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:05:01.716 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:05:01 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:05:01.458+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:05:02.037 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Dec 16 22:05:01 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-mon-smithi045[80759]: 2023-12-16T22:05:01.739+0000 7f45a1f8f700 -1 mon.smithi045@0(probing) e2 get_health_metrics reporting 24822 slow ops, oldest is monmgrreport(gid 14214, 0 checks, 1 progress events) 2023-12-16T22:05:02.038 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:05:01 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:05:01.723+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:05:02.038 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:05:01 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:05:01.723+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:05:02.038 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:05:01 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:05:01.723+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:05:02.038 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:05:01 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:05:01.723+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:05:02.314 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:05:02 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:05:02.036+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:05:02.315 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:05:02 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:05:02.036+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:05:02.315 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:05:02 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:05:02.036+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:05:02.315 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:05:02 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:05:02.037+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:05:02.773 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:05:02 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:05:02.440+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:05:02.773 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:05:02 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:05:02.440+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:05:02.773 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:05:02 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:05:02.440+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:05:02.774 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:05:02 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:05:02.441+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:05:03.065 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:05:02 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:05:02.772+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:05:03.065 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:05:02 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:05:02.772+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:05:03.065 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:05:02 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:05:02.772+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:05:03.065 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:05:02 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:05:02.773+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:05:03.448 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:05:03 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:05:03.086+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:05:03.448 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:05:03 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:05:03.086+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:05:03.448 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:05:03 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:05:03.086+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:05:03.449 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:05:03 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:05:03.087+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:05:03.814 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:05:03 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:05:03.446+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:05:03.815 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:05:03 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:05:03.446+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:05:03.815 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:05:03 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:05:03.446+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:05:03.815 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:05:03 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:05:03.448+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:05:04.096 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:05:03 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:05:03.813+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:05:04.096 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:05:03 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:05:03.813+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:05:04.097 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:05:03 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:05:03.813+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:05:04.097 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:05:03 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:05:03.814+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:05:04.474 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:05:04 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:05:04.095+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:05:04.474 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:05:04 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:05:04.095+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:05:04.474 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:05:04 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:05:04.095+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:05:04.474 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:05:04 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:05:04.096+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:05:04.814 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:05:04 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:05:04.473+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:05:04.815 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:05:04 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:05:04.473+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:05:04.815 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:05:04 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:05:04.473+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:05:04.815 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:05:04 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:05:04.474+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:05:05.315 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:05:04 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:05:04.827+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:05:05.315 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:05:04 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:05:04.827+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:05:05.315 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:05:04 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:05:04.827+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:05:05.315 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:05:04 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:05:04.828+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:05:05.315 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:05:05 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:05:05.067+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:05:05.315 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:05:05 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:05:05.067+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:05:05.316 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:05:05 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:05:05.067+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:05:05.316 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:05:05 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:05:05.068+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:05:05.782 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:05:05 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:05:05.506+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:05:05.782 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:05:05 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:05:05.506+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:05:05.783 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:05:05 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:05:05.506+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:05:05.783 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:05:05 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:05:05.507+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:05:06.065 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:05:05 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:05:05.854+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:05:06.065 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:05:05 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:05:05.855+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:05:06.065 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:05:05 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:05:05.855+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:05:06.065 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:05:05 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:05:05.855+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:05:06.491 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:05:06 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:05:06.100+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:05:06.491 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:05:06 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:05:06.100+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:05:06.491 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:05:06 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:05:06.100+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:05:06.491 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:05:06 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:05:06.101+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:05:06.743 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:05:06 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:05:06.489+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:05:06.743 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:05:06 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:05:06.489+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:05:06.743 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:05:06 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:05:06.489+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:05:06.743 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:05:06 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:05:06.490+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:05:07.064 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Dec 16 22:05:06 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-mon-smithi045[80759]: 2023-12-16T22:05:06.741+0000 7f45a1f8f700 -1 mon.smithi045@0(probing) e2 get_health_metrics reporting 24912 slow ops, oldest is monmgrreport(gid 14214, 0 checks, 1 progress events) 2023-12-16T22:05:07.065 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:05:06 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:05:06.889+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:05:07.065 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:05:06 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:05:06.889+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:05:07.065 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:05:06 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:05:06.889+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:05:07.065 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:05:06 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:05:06.890+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:05:07.523 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:05:07 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:05:07.079+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:05:07.523 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:05:07 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:05:07.079+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:05:07.523 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:05:07 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:05:07.079+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:05:07.523 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:05:07 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:05:07.080+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:05:07.814 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:05:07 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:05:07.521+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:05:07.815 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:05:07 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:05:07.521+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:05:07.815 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:05:07 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:05:07.521+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:05:07.815 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:05:07 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:05:07.522+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:05:08.314 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:05:07 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:05:07.888+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:05:08.315 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:05:07 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:05:07.888+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:05:08.315 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:05:07 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:05:07.888+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:05:08.315 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:05:07 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:05:07.889+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:05:08.315 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:05:08 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:05:08.116+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:05:08.315 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:05:08 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:05:08.116+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:05:08.316 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:05:08 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:05:08.116+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:05:08.316 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:05:08 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:05:08.117+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:05:08.814 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:05:08 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:05:08.544+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:05:08.815 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:05:08 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:05:08.544+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:05:08.815 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:05:08 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:05:08.544+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:05:08.815 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:05:08 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:05:08.544+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:05:09.314 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:05:08 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:05:08.930+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:05:09.315 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:05:08 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:05:08.930+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:05:09.315 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:05:08 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:05:08.930+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:05:09.315 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:05:08 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:05:08.930+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:05:09.315 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:05:09 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:05:09.068+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:05:09.315 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:05:09 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:05:09.068+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:05:09.315 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:05:09 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:05:09.068+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:05:09.316 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:05:09 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:05:09.069+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:05:09.814 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:05:09 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:05:09.507+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:05:09.815 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:05:09 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:05:09.507+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:05:09.815 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:05:09 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:05:09.507+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:05:09.815 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:05:09 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:05:09.508+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:05:10.187 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:05:10 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:05:10.074+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:05:10.187 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:05:10 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:05:10.074+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:05:10.188 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:05:10 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:05:10.074+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:05:10.188 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:05:10 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:05:10.076+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:05:10.188 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:05:09 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:05:09.901+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:05:10.188 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:05:09 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:05:09.901+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:05:10.188 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:05:09 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:05:09.901+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:05:10.188 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:05:09 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:05:09.902+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:05:10.814 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:05:10 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:05:10.500+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:05:10.815 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:05:10 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:05:10.500+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:05:10.815 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:05:10 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:05:10.500+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:05:10.815 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:05:10 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:05:10.500+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:05:11.314 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:05:10 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:05:10.919+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:05:11.315 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:05:10 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:05:10.919+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:05:11.315 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:05:10 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:05:10.919+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:05:11.315 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:05:10 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:05:10.920+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:05:11.315 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:05:11 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:05:11.085+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:05:11.316 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:05:11 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:05:11.085+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:05:11.316 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:05:11 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:05:11.085+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:05:11.316 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:05:11 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:05:11.086+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:05:11.716 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:05:11 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:05:11.460+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:05:11.717 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:05:11 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:05:11.460+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:05:11.717 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:05:11 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:05:11.460+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:05:11.717 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:05:11 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:05:11.461+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:05:11.969 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Dec 16 22:05:11 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-mon-smithi045[80759]: 2023-12-16T22:05:11.744+0000 7f45a1f8f700 -1 mon.smithi045@0(probing) e2 get_health_metrics reporting 25003 slow ops, oldest is monmgrreport(gid 14214, 0 checks, 1 progress events) 2023-12-16T22:05:12.315 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:05:11 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:05:11.968+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:05:12.315 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:05:11 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:05:11.968+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:05:12.315 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:05:11 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:05:11.968+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:05:12.315 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:05:11 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:05:11.969+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:05:12.315 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:05:12 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:05:12.065+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:05:12.316 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:05:12 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:05:12.065+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:05:12.316 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:05:12 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:05:12.065+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:05:12.316 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:05:12 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:05:12.066+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:05:12.814 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:05:12 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:05:12.504+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:05:12.815 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:05:12 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:05:12.504+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:05:12.815 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:05:12 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:05:12.504+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:05:12.815 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:05:12 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:05:12.505+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:05:13.315 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:05:12 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:05:12.934+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:05:13.315 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:05:12 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:05:12.934+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:05:13.315 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:05:12 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:05:12.934+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:05:13.315 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:05:12 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:05:12.935+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:05:13.316 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:05:13 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:05:13.079+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:05:13.316 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:05:13 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:05:13.079+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:05:13.316 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:05:13 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:05:13.079+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:05:13.316 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:05:13 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:05:13.080+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:05:13.814 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:05:13 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:05:13.512+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:05:13.815 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:05:13 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:05:13.512+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:05:13.815 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:05:13 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:05:13.512+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:05:13.815 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:05:13 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:05:13.513+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:05:14.315 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:05:13 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:05:13.898+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:05:14.315 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:05:13 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:05:13.898+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:05:14.315 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:05:13 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:05:13.898+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:05:14.316 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:05:13 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:05:13.899+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:05:14.316 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:05:14 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:05:14.106+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:05:14.316 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:05:14 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:05:14.106+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:05:14.316 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:05:14 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:05:14.106+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:05:14.317 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:05:14 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:05:14.107+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:05:14.815 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:05:14 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:05:14.504+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:05:14.815 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:05:14 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:05:14.504+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:05:14.815 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:05:14 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:05:14.504+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:05:14.815 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:05:14 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:05:14.505+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:05:15.314 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:05:14 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:05:14.924+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:05:15.315 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:05:14 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:05:14.924+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:05:15.315 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:05:14 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:05:14.924+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:05:15.315 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:05:14 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:05:14.925+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:05:15.315 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:05:15 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:05:15.141+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:05:15.315 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:05:15 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:05:15.141+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:05:15.315 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:05:15 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:05:15.141+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:05:15.315 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:05:15 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:05:15.142+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:05:15.815 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:05:15 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:05:15.487+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:05:15.815 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:05:15 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:05:15.487+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:05:15.815 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:05:15 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:05:15.487+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:05:15.815 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:05:15 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:05:15.488+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:05:16.314 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:05:16 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:05:16.119+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:05:16.315 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:05:16 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:05:16.119+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:05:16.315 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:05:16 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:05:16.119+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:05:16.315 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:05:16 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:05:16.120+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:05:16.315 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:05:15 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:05:15.950+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:05:16.315 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:05:15 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:05:15.950+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:05:16.315 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:05:15 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:05:15.950+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:05:16.316 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:05:15 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:05:15.951+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:05:16.749 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:05:16 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:05:16.464+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:05:16.749 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:05:16 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:05:16.464+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:05:16.749 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:05:16 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:05:16.464+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:05:16.749 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:05:16 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:05:16.466+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:05:17.065 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Dec 16 22:05:16 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-mon-smithi045[80759]: 2023-12-16T22:05:16.747+0000 7f45a1f8f700 -1 mon.smithi045@0(probing) e2 get_health_metrics reporting 25092 slow ops, oldest is monmgrreport(gid 14214, 0 checks, 1 progress events) 2023-12-16T22:05:17.065 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:05:16 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:05:16.954+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:05:17.065 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:05:16 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:05:16.954+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:05:17.065 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:05:16 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:05:16.954+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:05:17.065 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:05:16 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:05:16.955+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:05:17.459 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:05:17 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:05:17.138+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:05:17.460 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:05:17 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:05:17.138+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:05:17.460 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:05:17 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:05:17.138+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:05:17.460 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:05:17 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:05:17.139+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:05:17.815 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:05:17 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:05:17.458+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:05:17.815 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:05:17 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:05:17.458+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:05:17.815 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:05:17 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:05:17.458+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:05:17.815 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:05:17 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:05:17.459+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:05:18.314 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:05:17 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:05:17.950+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:05:18.315 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:05:17 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:05:17.950+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:05:18.315 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:05:17 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:05:17.950+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:05:18.315 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:05:17 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:05:17.951+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:05:18.315 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:05:18 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:05:18.108+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:05:18.315 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:05:18 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:05:18.108+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:05:18.315 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:05:18 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:05:18.108+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:05:18.315 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:05:18 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:05:18.110+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:05:18.720 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:05:18 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:05:18.443+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:05:18.720 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:05:18 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:05:18.443+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:05:18.720 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:05:18 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:05:18.443+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:05:18.721 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:05:18 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:05:18.444+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:05:19.061 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:05:18 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:05:18.934+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:05:19.061 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:05:18 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:05:18.934+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:05:19.062 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:05:18 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:05:18.934+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:05:19.062 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:05:18 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:05:18.935+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:05:19.314 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:05:19 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:05:19.060+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:05:19.315 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:05:19 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:05:19.060+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:05:19.315 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:05:19 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:05:19.060+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:05:19.315 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:05:19 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:05:19.061+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:05:19.815 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:05:19 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:05:19.488+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:05:19.815 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:05:19 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:05:19.488+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:05:19.815 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:05:19 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:05:19.488+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:05:19.815 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:05:19 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:05:19.489+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:05:20.315 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:05:20 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:05:20.102+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:05:20.315 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:05:20 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:05:20.102+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:05:20.315 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:05:20 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:05:20.102+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:05:20.315 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:05:20 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:05:20.103+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:05:20.316 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:05:19 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:05:19.950+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:05:20.316 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:05:19 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:05:19.950+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:05:20.316 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:05:19 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:05:19.950+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:05:20.316 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:05:19 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:05:19.951+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:05:20.814 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:05:20 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:05:20.440+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:05:20.815 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:05:20 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:05:20.440+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:05:20.815 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:05:20 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:05:20.440+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:05:20.815 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:05:20 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:05:20.440+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:05:21.315 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:05:20 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:05:20.930+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:05:21.315 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:05:20 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:05:20.930+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:05:21.315 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:05:20 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:05:20.930+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:05:21.315 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:05:20 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:05:20.931+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:05:21.315 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:05:21 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:05:21.087+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:05:21.316 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:05:21 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:05:21.087+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:05:21.316 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:05:21 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:05:21.087+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:05:21.316 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:05:21 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:05:21.088+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:05:21.717 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:05:21 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:05:21.407+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:05:21.718 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:05:21 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:05:21.407+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:05:21.718 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:05:21 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:05:21.407+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:05:21.718 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:05:21 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:05:21.407+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:05:22.060 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Dec 16 22:05:21 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-mon-smithi045[80759]: 2023-12-16T22:05:21.750+0000 7f45a1f8f700 -1 mon.smithi045@0(probing) e2 get_health_metrics reporting 25183 slow ops, oldest is monmgrreport(gid 14214, 0 checks, 1 progress events) 2023-12-16T22:05:22.061 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:05:21 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:05:21.958+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:05:22.061 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:05:21 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:05:21.958+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:05:22.061 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:05:21 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:05:21.958+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:05:22.061 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:05:21 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:05:21.959+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:05:22.315 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:05:22 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:05:22.059+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:05:22.315 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:05:22 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:05:22.059+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:05:22.315 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:05:22 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:05:22.059+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:05:22.315 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:05:22 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:05:22.060+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:05:22.814 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:05:22 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:05:22.359+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:05:22.815 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:05:22 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:05:22.359+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:05:22.815 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:05:22 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:05:22.359+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:05:22.815 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:05:22 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:05:22.360+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:05:23.315 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:05:23 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:05:23.004+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:05:23.315 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:05:23 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:05:23.004+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:05:23.315 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:05:23 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:05:23.004+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:05:23.315 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:05:23 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:05:23.005+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:05:23.315 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:05:23 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:05:23.082+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:05:23.315 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:05:23 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:05:23.082+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:05:23.316 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:05:23 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:05:23.082+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:05:23.316 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:05:23 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:05:23.083+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:05:23.814 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:05:23 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:05:23.404+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:05:23.815 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:05:23 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:05:23.404+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:05:23.815 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:05:23 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:05:23.404+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:05:23.815 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:05:23 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:05:23.405+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:05:24.314 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:05:24 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:05:24.090+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:05:24.315 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:05:24 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:05:24.090+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:05:24.315 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:05:24 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:05:24.090+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:05:24.315 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:05:24 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:05:24.090+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:05:24.315 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:05:24 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:05:24.029+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:05:24.315 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:05:24 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:05:24.029+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:05:24.316 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:05:24 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:05:24.029+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:05:24.316 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:05:24 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:05:24.029+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:05:24.815 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:05:24 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:05:24.372+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:05:24.815 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:05:24 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:05:24.372+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:05:24.815 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:05:24 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:05:24.372+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:05:24.815 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:05:24 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:05:24.373+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:05:25.314 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:05:25 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:05:25.043+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:05:25.315 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:05:25 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:05:25.043+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:05:25.315 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:05:25 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:05:25.043+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:05:25.315 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:05:25 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:05:25.044+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:05:25.315 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:05:25 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:05:25.067+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:05:25.315 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:05:25 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:05:25.067+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:05:25.316 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:05:25 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:05:25.067+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:05:25.316 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:05:25 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:05:25.069+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:05:25.815 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:05:25 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:05:25.383+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:05:25.815 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:05:25 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:05:25.383+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:05:25.815 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:05:25 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:05:25.383+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:05:25.815 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:05:25 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:05:25.383+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:05:26.314 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:05:26 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:05:26.001+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:05:26.315 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:05:26 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:05:26.001+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:05:26.315 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:05:26 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:05:26.001+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:05:26.315 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:05:26 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:05:26.002+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:05:26.315 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:05:26 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:05:26.102+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:05:26.315 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:05:26 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:05:26.102+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:05:26.316 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:05:26 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:05:26.102+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:05:26.316 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:05:26 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:05:26.103+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:05:26.754 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:05:26 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:05:26.364+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:05:26.754 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:05:26 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:05:26.364+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:05:26.754 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:05:26 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:05:26.364+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:05:26.754 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:05:26 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:05:26.365+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:05:27.064 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Dec 16 22:05:26 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-mon-smithi045[80759]: 2023-12-16T22:05:26.752+0000 7f45a1f8f700 -1 mon.smithi045@0(probing) e2 get_health_metrics reporting 25273 slow ops, oldest is monmgrreport(gid 14214, 0 checks, 1 progress events) 2023-12-16T22:05:27.065 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:05:26 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:05:26.977+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:05:27.065 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:05:26 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:05:26.977+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:05:27.065 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:05:26 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:05:26.977+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:05:27.065 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:05:26 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:05:26.978+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:05:27.388 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:05:27 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:05:27.386+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:05:27.388 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:05:27 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:05:27.386+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:05:27.389 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:05:27 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:05:27.386+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:05:27.389 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:05:27 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:05:27.136+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:05:27.389 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:05:27 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:05:27.137+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:05:27.389 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:05:27 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:05:27.137+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:05:27.389 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:05:27 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:05:27.138+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:05:27.814 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:05:27 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:05:27.387+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:05:28.314 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:05:28 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:05:28.157+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:05:28.315 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:05:28 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:05:28.157+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:05:28.315 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:05:28 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:05:28.157+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:05:28.315 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:05:28 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:05:28.159+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:05:28.315 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:05:27 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:05:27.977+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:05:28.316 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:05:27 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:05:27.977+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:05:28.316 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:05:27 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:05:27.977+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:05:28.316 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:05:27 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:05:27.978+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:05:28.720 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:05:28 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:05:28.339+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:05:28.720 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:05:28 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:05:28.339+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:05:28.720 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:05:28 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:05:28.339+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:05:28.721 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:05:28 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:05:28.340+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:05:29.064 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:05:28 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:05:28.961+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:05:29.065 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:05:28 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:05:28.961+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:05:29.065 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:05:28 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:05:28.961+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:05:29.065 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:05:28 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:05:28.962+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:05:29.564 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:05:29 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:05:29.186+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:05:29.565 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:05:29 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:05:29.186+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:05:29.565 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:05:29 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:05:29.186+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:05:29.565 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:05:29 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:05:29.186+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:05:29.565 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:05:29 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:05:29.364+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:05:29.565 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:05:29 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:05:29.364+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:05:29.565 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:05:29 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:05:29.364+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:05:29.565 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:05:29 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:05:29.365+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:05:30.315 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:05:30 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:05:30.151+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:05:30.315 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:05:30 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:05:30.151+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:05:30.315 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:05:30 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:05:30.151+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:05:30.316 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:05:30 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:05:30.152+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:05:30.316 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:05:29 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:05:29.988+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:05:30.316 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:05:29 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:05:29.988+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:05:30.317 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:05:29 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:05:29.988+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:05:30.317 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:05:29 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:05:29.989+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:05:30.814 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:05:30 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:05:30.392+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:05:30.815 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:05:30 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:05:30.392+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:05:30.815 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:05:30 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:05:30.392+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:05:30.815 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:05:30 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:05:30.393+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:05:31.279 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:05:31 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:05:31.016+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:05:31.279 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:05:31 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:05:31.016+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:05:31.280 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:05:31 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:05:31.016+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:05:31.280 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:05:31 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:05:31.017+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:05:31.280 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:05:31 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:05:31.198+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:05:31.280 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:05:31 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:05:31.198+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:05:31.280 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:05:31 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:05:31.198+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:05:31.280 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:05:31 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:05:31.199+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:05:31.564 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:05:31 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:05:31.427+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:05:31.565 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:05:31 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:05:31.427+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:05:31.565 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:05:31 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:05:31.427+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:05:31.565 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:05:31 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:05:31.427+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:05:31.975 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Dec 16 22:05:31 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-mon-smithi045[80759]: 2023-12-16T22:05:31.755+0000 7f45a1f8f700 -1 mon.smithi045@0(probing) e2 get_health_metrics reporting 25365 slow ops, oldest is monmgrreport(gid 14214, 0 checks, 1 progress events) 2023-12-16T22:05:32.314 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:05:31 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:05:31.974+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:05:32.315 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:05:31 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:05:31.974+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:05:32.315 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:05:31 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:05:31.974+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:05:32.315 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:05:31 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:05:31.975+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:05:32.315 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:05:32 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:05:32.172+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:05:32.315 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:05:32 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:05:32.172+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:05:32.315 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:05:32 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:05:32.172+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:05:32.316 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:05:32 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:05:32.172+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:05:32.814 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:05:32 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:05:32.423+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:05:32.815 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:05:32 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:05:32.423+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:05:32.815 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:05:32 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:05:32.423+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:05:32.815 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:05:32 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:05:32.423+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:05:33.314 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:05:33 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:05:33.018+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:05:33.315 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:05:33 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:05:33.018+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:05:33.315 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:05:33 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:05:33.018+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:05:33.315 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:05:33 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:05:33.019+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:05:33.315 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:05:33 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:05:33.137+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:05:33.315 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:05:33 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:05:33.137+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:05:33.315 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:05:33 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:05:33.137+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:05:33.316 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:05:33 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:05:33.138+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:05:33.814 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:05:33 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:05:33.416+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:05:33.815 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:05:33 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:05:33.416+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:05:33.815 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:05:33 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:05:33.416+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:05:33.815 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:05:33 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:05:33.417+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:05:34.315 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:05:33 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:05:33.990+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:05:34.315 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:05:33 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:05:33.990+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:05:34.315 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:05:33 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:05:33.990+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:05:34.315 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:05:33 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:05:33.991+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:05:34.315 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:05:34 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:05:34.116+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:05:34.315 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:05:34 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:05:34.116+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:05:34.316 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:05:34 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:05:34.116+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:05:34.316 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:05:34 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:05:34.117+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:05:34.814 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:05:34 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:05:34.380+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:05:34.815 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:05:34 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:05:34.380+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:05:34.815 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:05:34 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:05:34.380+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:05:34.815 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:05:34 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:05:34.381+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:05:35.315 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:05:35 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:05:35.028+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:05:35.315 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:05:35 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:05:35.028+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:05:35.315 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:05:35 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:05:35.028+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:05:35.315 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:05:35 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:05:35.029+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:05:35.315 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:05:35 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:05:35.140+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:05:35.316 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:05:35 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:05:35.140+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:05:35.316 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:05:35 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:05:35.140+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:05:35.316 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:05:35 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:05:35.140+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:05:35.814 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:05:35 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:05:35.335+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:05:35.815 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:05:35 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:05:35.335+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:05:35.815 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:05:35 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:05:35.335+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:05:35.815 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:05:35 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:05:35.336+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:05:36.296 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:05:35 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:05:35.985+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:05:36.297 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:05:35 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:05:35.985+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:05:36.297 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:05:35 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:05:35.985+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:05:36.297 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:05:35 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:05:35.986+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:05:36.297 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:05:36 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:05:36.128+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:05:36.297 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:05:36 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:05:36.128+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:05:36.297 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:05:36 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:05:36.128+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:05:36.298 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:05:36 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:05:36.129+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:05:36.564 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:05:36 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:05:36.294+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:05:36.565 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:05:36 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:05:36.294+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:05:36.565 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:05:36 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:05:36.294+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:05:36.565 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:05:36 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:05:36.295+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:05:37.065 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:05:36 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:05:36.987+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:05:37.065 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:05:36 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:05:36.987+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:05:37.065 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:05:36 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:05:36.987+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:05:37.065 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:05:36 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:05:36.988+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:05:37.066 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Dec 16 22:05:36 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-mon-smithi045[80759]: 2023-12-16T22:05:36.758+0000 7f45a1f8f700 -1 mon.smithi045@0(probing) e2 get_health_metrics reporting 25455 slow ops, oldest is monmgrreport(gid 14214, 0 checks, 1 progress events) 2023-12-16T22:05:37.564 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:05:37 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:05:37.155+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:05:37.565 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:05:37 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:05:37.155+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:05:37.565 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:05:37 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:05:37.155+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:05:37.565 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:05:37 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:05:37.156+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:05:37.565 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:05:37 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:05:37.330+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:05:37.565 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:05:37 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:05:37.330+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:05:37.565 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:05:37 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:05:37.330+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:05:37.566 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:05:37 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:05:37.330+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:05:38.314 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:05:37 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:05:37.975+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:05:38.315 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:05:37 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:05:37.975+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:05:38.315 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:05:37 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:05:37.975+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:05:38.315 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:05:37 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:05:37.976+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:05:38.315 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:05:38 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:05:38.194+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:05:38.315 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:05:38 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:05:38.194+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:05:38.316 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:05:38 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:05:38.194+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:05:38.316 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:05:38 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:05:38.194+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:05:38.720 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:05:38 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:05:38.345+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:05:38.720 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:05:38 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:05:38.345+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:05:38.720 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:05:38 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:05:38.345+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:05:38.721 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:05:38 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:05:38.345+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:05:39.315 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:05:39 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:05:39.013+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:05:39.315 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:05:39 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:05:39.013+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:05:39.315 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:05:39 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:05:39.013+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:05:39.315 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:05:39 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:05:39.014+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:05:39.315 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:05:39 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:05:39.200+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:05:39.316 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:05:39 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:05:39.200+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:05:39.316 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:05:39 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:05:39.200+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:05:39.316 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:05:39 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:05:39.201+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:05:39.814 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:05:39 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:05:39.322+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:05:39.815 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:05:39 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:05:39.322+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:05:39.815 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:05:39 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:05:39.322+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:05:39.815 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:05:39 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:05:39.323+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:05:40.314 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:05:40 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:05:40.015+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:05:40.315 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:05:40 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:05:40.015+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:05:40.315 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:05:40 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:05:40.015+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:05:40.315 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:05:40 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:05:40.016+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:05:40.315 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:05:40 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:05:40.193+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:05:40.315 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:05:40 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:05:40.193+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:05:40.315 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:05:40 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:05:40.193+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:05:40.316 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:05:40 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:05:40.194+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:05:40.815 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:05:40 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:05:40.337+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:05:40.815 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:05:40 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:05:40.337+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:05:40.815 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:05:40 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:05:40.337+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:05:40.815 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:05:40 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:05:40.337+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:05:41.314 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:05:40 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:05:40.995+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:05:41.315 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:05:40 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:05:40.995+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:05:41.315 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:05:40 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:05:40.995+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:05:41.315 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:05:40 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:05:40.996+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:05:41.315 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:05:41 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:05:41.223+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:05:41.315 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:05:41 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:05:41.223+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:05:41.315 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:05:41 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:05:41.223+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:05:41.316 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:05:41 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:05:41.224+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:05:41.718 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:05:41 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:05:41.357+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:05:41.719 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:05:41 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:05:41.357+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:05:41.719 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:05:41 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:05:41.357+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:05:41.719 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:05:41 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:05:41.358+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:05:41.987 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Dec 16 22:05:41 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-mon-smithi045[80759]: 2023-12-16T22:05:41.761+0000 7f45a1f8f700 -1 mon.smithi045@0(probing) e2 get_health_metrics reporting 25546 slow ops, oldest is monmgrreport(gid 14214, 0 checks, 1 progress events) 2023-12-16T22:05:42.314 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:05:41 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:05:41.985+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:05:42.315 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:05:41 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:05:41.985+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:05:42.315 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:05:41 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:05:41.985+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:05:42.315 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:05:41 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:05:41.986+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:05:42.315 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:05:42 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:05:42.215+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:05:42.315 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:05:42 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:05:42.215+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:05:42.315 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:05:42 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:05:42.215+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:05:42.315 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:05:42 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:05:42.216+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:05:42.814 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:05:42 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:05:42.386+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:05:42.815 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:05:42 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:05:42.386+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:05:42.815 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:05:42 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:05:42.386+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:05:42.815 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:05:42 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:05:42.387+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:05:43.314 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:05:43 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:05:43.008+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:05:43.315 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:05:43 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:05:43.008+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:05:43.315 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:05:43 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:05:43.008+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:05:43.315 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:05:43 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:05:43.008+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:05:43.315 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:05:43 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:05:43.240+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:05:43.315 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:05:43 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:05:43.240+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:05:43.316 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:05:43 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:05:43.240+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:05:43.316 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:05:43 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:05:43.241+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:05:43.615 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:05:43 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:05:43.357+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:05:43.617 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:05:43 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:05:43.357+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:05:43.617 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:05:43 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:05:43.357+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:05:43.617 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:05:43 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:05:43.358+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:05:44.314 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:05:44 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:05:44.206+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:05:44.315 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:05:44 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:05:44.206+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:05:44.315 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:05:44 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:05:44.206+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:05:44.315 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:05:44 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:05:44.207+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:05:44.315 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:05:44 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:05:44.040+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:05:44.315 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:05:44 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:05:44.040+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:05:44.315 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:05:44 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:05:44.040+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:05:44.315 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:05:44 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:05:44.040+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:05:44.814 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:05:44 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:05:44.355+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:05:44.815 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:05:44 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:05:44.355+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:05:44.815 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:05:44 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:05:44.355+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:05:44.815 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:05:44 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:05:44.356+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:05:45.314 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:05:45 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:05:45.003+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:05:45.315 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:05:45 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:05:45.003+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:05:45.315 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:05:45 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:05:45.003+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:05:45.315 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:05:45 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:05:45.004+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:05:45.315 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:05:45 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:05:45.181+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:05:45.315 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:05:45 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:05:45.181+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:05:45.316 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:05:45 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:05:45.181+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:05:45.316 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:05:45 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:05:45.182+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:05:45.814 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:05:45 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:05:45.388+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:05:45.815 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:05:45 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:05:45.388+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:05:45.815 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:05:45 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:05:45.388+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:05:45.815 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:05:45 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:05:45.389+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:05:46.314 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:05:46 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:05:46.005+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:05:46.315 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:05:46 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:05:46.005+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:05:46.315 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:05:46 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:05:46.005+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:05:46.315 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:05:46 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:05:46.006+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:05:46.315 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:05:46 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:05:46.184+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:05:46.315 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:05:46 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:05:46.184+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:05:46.315 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:05:46 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:05:46.184+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:05:46.315 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:05:46 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:05:46.185+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:05:46.639 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:05:46 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:05:46.388+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:05:46.639 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:05:46 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:05:46.388+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:05:46.639 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:05:46 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:05:46.388+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:05:46.639 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:05:46 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:05:46.388+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:05:46.892 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Dec 16 22:05:46 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-mon-smithi045[80759]: 2023-12-16T22:05:46.764+0000 7f45a1f8f700 -1 mon.smithi045@0(probing) e2 get_health_metrics reporting 25635 slow ops, oldest is monmgrreport(gid 14214, 0 checks, 1 progress events) 2023-12-16T22:05:47.233 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:05:46 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:05:46.963+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:05:47.233 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:05:46 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:05:46.963+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:05:47.233 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:05:46 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:05:46.963+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:05:47.234 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:05:46 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:05:46.964+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:05:47.564 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:05:47 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:05:47.231+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:05:47.565 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:05:47 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:05:47.231+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:05:47.565 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:05:47 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:05:47.231+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:05:47.565 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:05:47 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:05:47.233+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:05:47.565 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:05:47 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:05:47.387+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:05:47.565 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:05:47 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:05:47.387+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:05:47.565 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:05:47 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:05:47.387+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:05:47.566 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:05:47 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:05:47.388+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:05:48.219 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:05:47 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:05:47.943+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:05:48.220 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:05:47 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:05:47.943+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:05:48.220 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:05:47 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:05:47.943+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:05:48.220 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:05:47 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:05:47.944+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:05:48.564 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:05:48 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:05:48.218+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:05:48.565 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:05:48 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:05:48.218+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:05:48.565 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:05:48 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:05:48.218+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:05:48.565 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:05:48 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:05:48.219+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:05:48.565 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:05:48 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:05:48.386+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:05:48.565 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:05:48 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:05:48.386+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:05:48.565 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:05:48 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:05:48.386+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:05:48.566 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:05:48 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:05:48.386+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:05:49.064 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:05:48 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:05:48.944+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:05:49.065 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:05:48 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:05:48.944+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:05:49.065 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:05:48 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:05:48.944+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:05:49.065 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:05:48 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:05:48.944+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:05:49.404 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:05:49 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:05:49.230+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:05:49.404 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:05:49 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:05:49.230+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:05:49.404 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:05:49 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:05:49.230+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:05:49.404 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:05:49 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:05:49.231+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:05:49.814 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:05:49 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:05:49.403+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:05:49.815 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:05:49 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:05:49.403+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:05:49.815 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:05:49 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:05:49.403+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:05:49.815 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:05:49 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:05:49.404+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:05:50.253 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:05:49 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:05:49.987+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:05:50.253 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:05:49 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:05:49.987+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:05:50.254 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:05:49 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:05:49.987+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:05:50.254 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:05:49 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:05:49.988+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:05:50.564 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:05:50 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:05:50.252+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:05:50.565 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:05:50 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:05:50.252+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:05:50.565 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:05:50 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:05:50.252+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:05:50.565 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:05:50 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:05:50.253+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:05:50.565 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:05:50 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:05:50.374+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:05:50.565 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:05:50 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:05:50.374+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:05:50.566 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:05:50 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:05:50.374+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:05:50.566 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:05:50 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:05:50.375+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:05:51.314 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:05:51 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:05:51.204+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:05:51.315 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:05:51 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:05:51.204+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:05:51.315 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:05:51 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:05:51.204+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:05:51.315 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:05:51 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:05:51.205+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:05:51.315 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:05:50 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:05:50.995+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:05:51.315 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:05:50 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:05:50.995+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:05:51.315 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:05:50 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:05:50.995+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:05:51.316 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:05:50 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:05:50.996+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:05:51.719 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:05:51 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:05:51.381+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:05:51.720 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:05:51 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:05:51.381+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:05:51.720 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:05:51 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:05:51.381+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:05:51.720 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:05:51 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:05:51.382+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:05:52.023 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Dec 16 22:05:51 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-mon-smithi045[80759]: 2023-12-16T22:05:51.767+0000 7f45a1f8f700 -1 mon.smithi045@0(probing) e2 get_health_metrics reporting 25727 slow ops, oldest is monmgrreport(gid 14214, 0 checks, 1 progress events) 2023-12-16T22:05:52.314 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:05:52 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:05:52.249+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:05:52.315 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:05:52 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:05:52.249+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:05:52.315 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:05:52 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:05:52.249+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:05:52.315 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:05:52 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:05:52.250+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:05:52.315 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:05:52 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:05:52.022+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:05:52.315 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:05:52 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:05:52.022+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:05:52.316 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:05:52 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:05:52.022+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:05:52.316 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:05:52 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:05:52.023+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:05:52.783 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:05:52 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:05:52.393+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:05:52.783 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:05:52 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:05:52.393+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:05:52.783 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:05:52 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:05:52.393+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:05:52.783 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:05:52 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:05:52.394+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:05:53.064 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:05:52 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:05:52.980+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:05:53.065 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:05:52 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:05:52.980+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:05:53.065 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:05:52 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:05:52.980+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:05:53.065 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:05:52 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:05:52.981+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:05:53.564 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:05:53 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:05:53.253+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:05:53.565 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:05:53 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:05:53.253+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:05:53.565 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:05:53 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:05:53.253+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:05:53.565 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:05:53 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:05:53.254+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:05:53.565 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:05:53 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:05:53.359+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:05:53.565 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:05:53 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:05:53.359+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:05:53.566 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:05:53 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:05:53.359+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:05:53.566 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:05:53 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:05:53.360+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:05:54.302 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:05:54 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:05:54.011+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:05:54.303 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:05:54 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:05:54.011+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:05:54.303 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:05:54 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:05:54.011+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:05:54.303 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:05:54 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:05:54.012+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:05:54.565 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:05:54 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:05:54.301+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:05:54.565 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:05:54 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:05:54.301+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:05:54.565 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:05:54 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:05:54.301+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:05:54.565 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:05:54 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:05:54.302+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:05:54.566 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:05:54 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:05:54.356+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:05:54.566 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:05:54 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:05:54.356+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:05:54.566 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:05:54 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:05:54.356+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:05:54.566 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:05:54 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:05:54.356+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:05:55.314 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:05:55 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:05:55.061+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:05:55.315 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:05:55 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:05:55.061+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:05:55.315 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:05:55 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:05:55.061+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:05:55.315 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:05:55 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:05:55.062+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:05:55.315 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:05:55 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:05:55.273+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:05:55.315 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:05:55 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:05:55.273+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:05:55.316 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:05:55 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:05:55.273+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:05:55.316 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:05:55 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:05:55.274+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:05:55.316 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:05:55 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:05:55.310+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:05:55.316 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:05:55 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:05:55.310+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:05:55.316 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:05:55 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:05:55.310+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:05:55.316 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:05:55 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:05:55.310+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:05:56.317 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:05:56 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:05:56.023+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:05:56.317 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:05:56 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:05:56.023+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:05:56.317 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:05:56 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:05:56.023+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:05:56.318 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:05:56 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:05:56.024+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:05:56.771 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:05:56 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:05:56.351+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:05:56.771 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:05:56 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:05:56.351+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:05:56.771 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:05:56 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:05:56.351+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:05:56.771 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:05:56 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:05:56.352+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:05:56.772 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:05:56 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:05:56.322+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:05:56.772 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:05:56 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:05:56.322+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:05:56.772 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:05:56 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:05:56.322+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:05:56.772 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:05:56 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:05:56.322+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:05:57.064 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Dec 16 22:05:56 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-mon-smithi045[80759]: 2023-12-16T22:05:56.770+0000 7f45a1f8f700 -1 mon.smithi045@0(probing) e2 get_health_metrics reporting 25817 slow ops, oldest is monmgrreport(gid 14214, 0 checks, 1 progress events) 2023-12-16T22:05:57.065 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:05:56 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:05:56.996+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:05:57.065 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:05:56 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:05:56.996+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:05:57.065 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:05:56 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:05:56.996+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:05:57.065 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:05:56 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:05:56.997+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:05:57.564 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:05:57 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:05:57.281+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:05:57.565 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:05:57 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:05:57.281+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:05:57.565 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:05:57 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:05:57.281+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:05:57.565 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:05:57 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:05:57.282+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:05:57.565 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:05:57 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:05:57.347+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:05:57.565 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:05:57 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:05:57.347+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:05:57.565 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:05:57 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:05:57.347+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:05:57.566 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:05:57 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:05:57.348+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:05:58.293 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:05:57 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:05:57.979+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:05:58.293 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:05:57 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:05:57.979+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:05:58.293 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:05:57 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:05:57.979+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:05:58.293 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:05:57 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:05:57.980+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:05:58.564 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:05:58 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:05:58.291+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:05:58.565 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:05:58 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:05:58.292+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:05:58.565 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:05:58 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:05:58.292+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:05:58.565 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:05:58 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:05:58.292+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:05:58.565 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:05:58 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:05:58.365+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:05:58.565 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:05:58 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:05:58.365+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:05:58.566 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:05:58 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:05:58.365+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:05:58.566 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:05:58 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:05:58.366+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:05:59.278 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:05:58 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:05:58.971+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:05:59.278 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:05:58 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:05:58.971+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:05:59.278 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:05:58 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:05:58.971+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:05:59.278 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:05:58 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:05:58.972+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:05:59.564 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:05:59 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:05:59.277+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:05:59.565 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:05:59 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:05:59.277+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:05:59.565 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:05:59 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:05:59.277+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:05:59.565 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:05:59 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:05:59.278+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:05:59.565 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:05:59 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:05:59.383+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:05:59.565 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:05:59 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:05:59.383+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:05:59.566 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:05:59 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:05:59.383+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:05:59.566 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:05:59 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:05:59.383+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:06:00.314 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:06:00 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:06:00.234+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:06:00.315 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:06:00 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:06:00.234+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:06:00.315 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:06:00 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:06:00.234+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:06:00.315 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:06:00 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:06:00.234+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:06:00.315 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:06:00 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:06:00.011+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:06:00.315 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:06:00 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:06:00.011+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:06:00.315 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:06:00 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:06:00.011+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:06:00.316 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:06:00 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:06:00.012+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:06:00.814 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:06:00 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:06:00.405+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:06:00.815 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:06:00 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:06:00.405+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:06:00.815 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:06:00 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:06:00.405+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:06:00.815 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:06:00 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:06:00.406+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:06:01.314 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:06:01 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:06:01.237+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:06:01.315 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:06:01 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:06:01.237+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:06:01.315 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:06:01 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:06:01.237+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:06:01.315 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:06:01 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:06:01.238+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:06:01.315 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:06:01 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:06:00.999+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:06:01.315 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:06:01 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:06:00.999+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:06:01.315 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:06:01 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:06:00.999+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:06:01.316 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:06:01 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:06:01.000+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:06:01.721 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:06:01 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:06:01.448+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:06:01.721 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:06:01 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:06:01.448+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:06:01.721 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:06:01 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:06:01.448+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:06:01.721 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:06:01 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:06:01.449+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:06:02.000 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Dec 16 22:06:01 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-mon-smithi045[80759]: 2023-12-16T22:06:01.771+0000 7f45a1f8f700 -1 mon.smithi045@0(probing) e2 get_health_metrics reporting 25908 slow ops, oldest is monmgrreport(gid 14214, 0 checks, 1 progress events) 2023-12-16T22:06:02.000 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:06:01 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:06:01.960+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:06:02.000 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:06:01 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:06:01.960+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:06:02.001 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:06:01 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:06:01.960+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:06:02.001 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:06:01 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:06:01.961+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:06:02.314 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:06:02 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:06:02.215+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:06:02.315 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:06:02 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:06:02.215+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:06:02.315 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:06:02 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:06:02.215+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:06:02.315 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:06:02 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:06:02.216+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:06:02.814 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:06:02 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:06:02.407+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:06:02.815 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:06:02 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:06:02.408+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:06:02.815 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:06:02 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:06:02.408+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:06:02.815 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:06:02 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:06:02.408+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:06:03.261 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:06:02 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:06:02.912+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:06:03.261 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:06:02 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:06:02.912+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:06:03.261 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:06:02 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:06:02.912+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:06:03.261 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:06:02 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:06:02.913+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:06:03.564 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:06:03 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:06:03.259+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:06:03.565 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:06:03 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:06:03.259+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:06:03.565 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:06:03 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:06:03.259+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:06:03.565 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:06:03 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:06:03.260+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:06:03.565 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:06:03 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:06:03.418+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:06:03.565 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:06:03 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:06:03.418+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:06:03.565 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:06:03 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:06:03.418+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:06:03.566 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:06:03 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:06:03.418+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:06:04.238 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:06:03 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:06:03.932+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:06:04.238 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:06:03 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:06:03.932+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:06:04.239 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:06:03 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:06:03.932+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:06:04.239 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:06:03 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:06:03.933+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:06:04.564 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:06:04 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:06:04.236+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:06:04.565 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:06:04 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:06:04.236+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:06:04.565 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:06:04 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:06:04.236+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:06:04.565 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:06:04 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:06:04.237+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:06:04.565 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:06:04 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:06:04.448+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:06:04.565 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:06:04 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:06:04.448+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:06:04.565 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:06:04 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:06:04.448+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:06:04.566 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:06:04 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:06:04.449+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:06:05.219 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:06:04 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:06:04.968+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:06:05.219 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:06:04 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:06:04.968+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:06:05.220 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:06:04 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:06:04.968+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:06:05.220 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:06:04 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:06:04.968+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:06:05.564 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:06:05 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:06:05.416+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:06:05.565 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:06:05 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:06:05.416+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:06:05.565 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:06:05 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:06:05.416+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:06:05.565 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:06:05 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:06:05.416+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:06:05.565 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:06:05 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:06:05.218+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:06:05.565 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:06:05 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:06:05.218+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:06:05.565 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:06:05 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:06:05.218+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:06:05.566 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:06:05 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:06:05.218+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:06:06.242 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:06:05 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:06:05.953+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:06:06.242 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:06:05 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:06:05.953+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:06:06.242 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:06:05 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:06:05.953+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:06:06.242 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:06:05 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:06:05.954+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:06:06.564 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:06:06 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:06:06.240+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:06:06.565 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:06:06 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:06:06.240+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:06:06.565 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:06:06 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:06:06.240+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:06:06.565 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:06:06 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:06:06.240+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:06:06.565 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:06:06 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:06:06.391+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:06:06.565 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:06:06 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:06:06.391+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:06:06.565 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:06:06 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:06:06.391+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:06:06.566 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:06:06 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:06:06.392+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:06:07.064 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Dec 16 22:06:06 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-mon-smithi045[80759]: 2023-12-16T22:06:06.775+0000 7f45a1f8f700 -1 mon.smithi045@0(probing) e2 get_health_metrics reporting 25998 slow ops, oldest is monmgrreport(gid 14214, 0 checks, 1 progress events) 2023-12-16T22:06:07.065 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:06:06 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:06:06.959+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:06:07.065 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:06:06 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:06:06.959+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:06:07.065 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:06:06 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:06:06.959+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:06:07.065 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:06:06 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:06:06.959+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:06:07.564 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:06:07 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:06:07.226+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:06:07.565 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:06:07 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:06:07.226+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:06:07.565 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:06:07 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:06:07.226+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:06:07.565 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:06:07 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:06:07.227+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:06:07.565 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:06:07 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:06:07.374+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:06:07.565 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:06:07 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:06:07.374+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:06:07.565 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:06:07 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:06:07.374+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:06:07.565 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:06:07 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:06:07.375+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:06:08.314 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:06:08 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:06:08.007+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:06:08.315 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:06:08 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:06:08.007+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:06:08.315 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:06:08 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:06:08.007+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:06:08.315 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:06:08 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:06:08.009+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:06:08.315 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:06:08 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:06:08.213+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:06:08.315 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:06:08 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:06:08.213+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:06:08.315 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:06:08 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:06:08.213+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:06:08.316 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:06:08 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:06:08.214+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:06:08.720 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:06:08 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:06:08.422+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:06:08.720 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:06:08 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:06:08.422+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:06:08.721 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:06:08 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:06:08.422+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:06:08.721 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:06:08 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:06:08.423+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:06:09.314 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:06:09 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:06:09.050+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:06:09.315 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:06:09 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:06:09.050+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:06:09.315 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:06:09 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:06:09.050+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:06:09.315 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:06:09 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:06:09.052+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:06:09.315 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:06:09 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:06:09.236+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:06:09.315 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:06:09 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:06:09.236+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:06:09.315 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:06:09 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:06:09.236+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:06:09.315 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:06:09 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:06:09.237+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:06:09.814 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:06:09 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:06:09.395+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:06:09.815 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:06:09 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:06:09.395+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:06:09.815 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:06:09 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:06:09.395+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:06:09.815 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:06:09 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:06:09.396+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:06:10.314 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:06:10 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:06:10.256+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:06:10.315 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:06:10 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:06:10.256+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:06:10.315 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:06:10 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:06:10.256+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:06:10.315 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:06:10 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:06:10.257+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:06:10.315 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:06:10 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:06:10.030+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:06:10.315 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:06:10 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:06:10.030+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:06:10.315 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:06:10 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:06:10.030+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:06:10.316 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:06:10 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:06:10.031+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:06:10.814 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:06:10 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:06:10.350+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:06:10.815 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:06:10 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:06:10.350+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:06:10.815 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:06:10 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:06:10.350+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:06:10.815 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:06:10 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:06:10.351+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:06:11.314 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:06:11 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:06:11.007+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:06:11.315 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:06:11 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:06:11.007+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:06:11.315 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:06:11 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:06:11.007+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:06:11.315 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:06:11 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:06:11.008+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:06:11.315 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:06:11 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:06:11.229+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:06:11.315 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:06:11 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:06:11.229+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:06:11.315 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:06:11 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:06:11.229+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:06:11.316 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:06:11 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:06:11.230+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:06:11.721 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:06:11 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:06:11.365+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:06:11.722 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:06:11 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:06:11.365+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:06:11.722 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:06:11 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:06:11.365+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:06:11.722 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:06:11 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:06:11.366+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:06:12.003 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Dec 16 22:06:11 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-mon-smithi045[80759]: 2023-12-16T22:06:11.778+0000 7f45a1f8f700 -1 mon.smithi045@0(probing) e2 get_health_metrics reporting 26090 slow ops, oldest is monmgrreport(gid 14214, 0 checks, 1 progress events) 2023-12-16T22:06:12.314 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:06:12 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:06:12.002+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:06:12.315 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:06:12 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:06:12.002+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:06:12.315 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:06:12 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:06:12.002+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:06:12.315 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:06:12 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:06:12.003+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:06:12.315 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:06:12 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:06:12.233+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:06:12.315 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:06:12 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:06:12.233+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:06:12.315 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:06:12 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:06:12.233+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:06:12.315 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:06:12 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:06:12.234+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:06:12.814 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:06:12 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:06:12.343+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:06:12.815 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:06:12 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:06:12.343+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:06:12.815 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:06:12 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:06:12.343+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:06:12.815 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:06:12 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:06:12.344+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:06:13.310 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:06:13 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:06:13.047+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:06:13.310 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:06:13 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:06:13.047+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:06:13.310 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:06:13 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:06:13.047+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:06:13.311 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:06:13 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:06:13.048+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:06:13.311 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:06:13 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:06:13.253+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:06:13.311 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:06:13 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:06:13.253+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:06:13.311 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:06:13 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:06:13.253+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:06:13.311 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:06:13 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:06:13.254+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:06:13.564 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:06:13 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:06:13.309+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:06:13.564 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:06:13 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:06:13.309+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:06:13.565 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:06:13 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:06:13.309+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:06:13.565 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:06:13 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:06:13.310+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:06:14.564 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:06:14 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:06:14.078+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:06:14.565 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:06:14 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:06:14.078+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:06:14.565 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:06:14 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:06:14.078+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:06:14.565 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:06:14 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:06:14.079+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:06:14.565 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:06:14 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:06:14.273+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:06:14.565 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:06:14 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:06:14.273+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:06:14.566 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:06:14 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:06:14.273+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:06:14.566 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:06:14 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:06:14.274+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:06:14.566 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:06:14 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:06:14.264+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:06:14.566 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:06:14 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:06:14.264+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:06:14.566 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:06:14 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:06:14.264+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:06:14.566 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:06:14 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:06:14.266+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:06:15.564 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:06:15 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:06:15.105+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:06:15.565 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:06:15 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:06:15.105+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:06:15.565 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:06:15 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:06:15.105+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:06:15.565 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:06:15 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:06:15.106+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:06:15.565 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:06:15 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:06:15.283+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:06:15.565 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:06:15 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:06:15.283+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:06:15.566 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:06:15 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:06:15.283+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:06:15.566 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:06:15 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:06:15.284+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:06:15.566 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:06:15 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:06:15.313+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:06:15.566 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:06:15 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:06:15.313+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:06:15.566 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:06:15 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:06:15.313+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:06:15.566 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:06:15 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:06:15.314+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:06:16.564 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:06:16 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:06:16.084+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:06:16.565 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:06:16 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:06:16.084+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:06:16.565 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:06:16 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:06:16.084+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:06:16.565 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:06:16 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:06:16.085+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:06:16.565 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:06:16 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:06:16.306+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:06:16.565 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:06:16 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:06:16.306+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:06:16.566 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:06:16 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:06:16.306+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:06:16.566 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:06:16 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:06:16.307+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:06:16.566 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:06:16 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:06:16.268+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:06:16.566 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:06:16 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:06:16.268+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:06:16.566 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:06:16 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:06:16.268+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:06:16.566 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:06:16 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:06:16.268+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:06:17.042 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Dec 16 22:06:16 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-mon-smithi045[80759]: 2023-12-16T22:06:16.781+0000 7f45a1f8f700 -1 mon.smithi045@0(probing) e2 get_health_metrics reporting 26180 slow ops, oldest is monmgrreport(gid 14214, 0 checks, 1 progress events) 2023-12-16T22:06:17.314 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:06:17 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:06:17.269+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:06:17.315 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:06:17 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:06:17.269+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:06:17.315 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:06:17 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:06:17.269+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:06:17.315 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:06:17 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:06:17.270+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:06:17.315 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:06:17 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:06:17.041+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:06:17.315 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:06:17 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:06:17.041+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:06:17.316 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:06:17 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:06:17.041+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:06:17.316 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:06:17 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:06:17.042+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:06:17.316 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:06:17 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:06:17.249+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:06:17.316 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:06:17 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:06:17.249+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:06:17.316 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:06:17 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:06:17.249+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:06:17.317 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:06:17 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:06:17.250+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:06:18.314 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:06:18 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:06:18.050+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:06:18.315 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:06:18 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:06:18.050+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:06:18.315 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:06:18 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:06:18.050+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:06:18.315 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:06:18 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:06:18.051+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:06:18.315 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:06:18 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:06:18.213+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:06:18.316 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:06:18 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:06:18.213+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:06:18.316 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:06:18 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:06:18.213+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:06:18.316 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:06:18 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:06:18.214+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:06:18.316 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:06:18 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:06:18.230+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:06:18.316 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:06:18 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:06:18.230+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:06:18.316 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:06:18 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:06:18.230+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:06:18.316 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:06:18 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:06:18.230+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:06:19.314 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:06:19 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:06:19.036+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:06:19.315 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:06:19 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:06:19.036+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:06:19.315 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:06:19 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:06:19.036+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:06:19.315 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:06:19 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:06:19.037+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:06:19.315 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:06:19 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:06:19.197+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:06:19.315 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:06:19 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:06:19.197+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:06:19.315 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:06:19 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:06:19.197+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:06:19.316 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:06:19 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:06:19.198+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:06:19.316 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:06:19 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:06:19.242+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:06:19.316 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:06:19 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:06:19.242+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:06:19.316 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:06:19 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:06:19.242+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:06:19.316 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:06:19 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:06:19.242+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:06:20.431 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:06:20 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:06:20.292+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:06:20.431 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:06:20 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:06:20.292+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:06:20.431 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:06:20 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:06:20.292+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:06:20.432 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:06:20 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:06:20.293+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:06:20.432 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:06:20 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:06:20.087+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:06:20.432 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:06:20 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:06:20.087+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:06:20.432 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:06:20 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:06:20.087+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:06:20.432 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:06:20 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:06:20.088+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:06:20.432 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:06:20 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:06:20.211+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:06:20.433 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:06:20 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:06:20.211+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:06:20.433 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:06:20 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:06:20.211+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:06:20.433 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:06:20 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:06:20.212+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:06:21.564 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:06:21 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:06:21.116+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:06:21.565 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:06:21 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:06:21.116+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:06:21.565 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:06:21 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:06:21.116+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:06:21.565 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:06:21 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:06:21.117+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:06:21.565 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:06:21 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:06:21.263+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:06:21.566 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:06:21 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:06:21.263+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:06:21.566 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:06:21 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:06:21.263+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:06:21.566 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:06:21 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:06:21.264+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:06:21.566 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:06:21 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:06:21.203+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:06:21.566 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:06:21 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:06:21.203+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:06:21.566 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:06:21 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:06:21.203+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:06:21.566 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:06:21 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:06:21.204+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:06:22.064 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Dec 16 22:06:21 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-mon-smithi045[80759]: 2023-12-16T22:06:21.784+0000 7f45a1f8f700 -1 mon.smithi045@0(probing) e2 get_health_metrics reporting 26271 slow ops, oldest is monmgrreport(gid 14214, 0 checks, 1 progress events) 2023-12-16T22:06:22.564 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:06:22 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:06:22.075+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:06:22.565 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:06:22 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:06:22.076+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:06:22.565 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:06:22 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:06:22.076+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:06:22.565 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:06:22 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:06:22.076+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:06:22.565 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:06:22 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:06:22.239+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:06:22.565 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:06:22 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:06:22.239+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:06:22.566 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:06:22 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:06:22.239+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:06:22.566 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:06:22 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:06:22.240+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:06:22.566 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:06:22 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:06:22.247+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:06:22.566 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:06:22 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:06:22.247+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:06:22.566 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:06:22 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:06:22.247+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:06:22.566 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:06:22 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:06:22.248+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:06:23.292 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:06:23 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:06:23.040+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:06:23.292 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:06:23 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:06:23.040+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:06:23.292 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:06:23 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:06:23.040+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:06:23.292 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:06:23 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:06:23.041+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:06:23.293 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:06:23 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:06:23.242+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:06:23.293 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:06:23 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:06:23.242+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:06:23.293 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:06:23 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:06:23.242+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:06:23.293 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:06:23 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:06:23.243+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:06:23.293 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:06:23 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:06:23.289+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:06:23.293 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:06:23 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:06:23.289+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:06:23.293 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:06:23 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:06:23.289+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:06:23.294 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:06:23 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:06:23.291+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:06:24.314 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:06:24 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:06:24.247+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:06:24.315 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:06:24 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:06:24.247+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:06:24.315 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:06:24 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:06:24.247+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:06:24.315 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:06:24 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:06:24.248+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:06:24.315 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:06:24 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:06:24.049+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:06:24.315 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:06:24 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:06:24.049+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:06:24.315 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:06:24 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:06:24.049+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:06:24.316 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:06:24 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:06:24.050+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:06:24.815 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:06:24 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:06:24.339+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:06:24.815 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:06:24 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:06:24.339+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:06:24.815 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:06:24 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:06:24.339+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:06:24.815 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:06:24 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:06:24.340+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:06:25.291 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:06:25 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:06:25.023+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:06:25.291 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:06:25 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:06:25.023+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:06:25.291 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:06:25 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:06:25.023+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:06:25.291 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:06:25 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:06:25.024+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:06:25.564 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:06:25 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:06:25.290+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:06:25.565 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:06:25 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:06:25.290+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:06:25.565 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:06:25 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:06:25.290+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:06:25.565 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:06:25 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:06:25.291+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:06:25.565 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:06:25 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:06:25.349+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:06:25.565 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:06:25 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:06:25.349+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:06:25.565 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:06:25 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:06:25.349+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:06:25.566 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:06:25 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:06:25.350+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:06:26.314 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:06:26 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:06:26.044+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:06:26.315 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:06:26 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:06:26.044+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:06:26.315 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:06:26 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:06:26.044+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:06:26.315 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:06:26 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:06:26.045+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:06:26.575 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:06:26 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:06:26.313+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:06:26.576 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:06:26 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:06:26.313+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:06:26.576 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:06:26 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:06:26.313+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:06:26.576 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:06:26 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:06:26.315+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:06:26.576 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:06:26 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:06:26.374+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:06:26.576 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:06:26 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:06:26.374+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:06:26.576 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:06:26 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:06:26.374+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:06:26.576 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:06:26 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:06:26.375+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:06:26.914 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Dec 16 22:06:26 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-mon-smithi045[80759]: 2023-12-16T22:06:26.787+0000 7f45a1f8f700 -1 mon.smithi045@0(probing) e2 get_health_metrics reporting 26362 slow ops, oldest is monmgrreport(gid 14214, 0 checks, 1 progress events) 2023-12-16T22:06:27.315 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:06:27 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:06:27.008+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:06:27.315 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:06:27 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:06:27.008+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:06:27.315 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:06:27 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:06:27.008+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:06:27.315 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:06:27 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:06:27.009+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:06:27.815 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:06:27 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:06:27.362+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:06:27.815 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:06:27 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:06:27.362+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:06:27.815 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:06:27 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:06:27.362+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:06:27.815 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:06:27 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:06:27.363+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:06:27.816 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:06:27 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:06:27.337+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:06:27.816 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:06:27 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:06:27.337+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:06:27.816 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:06:27 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:06:27.337+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:06:27.817 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:06:27 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:06:27.337+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:06:28.314 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:06:27 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:06:27.977+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:06:28.315 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:06:27 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:06:27.977+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:06:28.315 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:06:27 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:06:27.977+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:06:28.315 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:06:27 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:06:27.977+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:06:28.720 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:06:28 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:06:28.326+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:06:28.721 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:06:28 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:06:28.326+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:06:28.721 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:06:28 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:06:28.326+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:06:28.721 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:06:28 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:06:28.327+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:06:28.721 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:06:28 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:06:28.383+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:06:28.721 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:06:28 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:06:28.383+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:06:28.721 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:06:28 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:06:28.383+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:06:28.722 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:06:28 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:06:28.383+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:06:29.064 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:06:28 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:06:28.934+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:06:29.065 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:06:28 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:06:28.934+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:06:29.065 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:06:28 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:06:28.934+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:06:29.065 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:06:28 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:06:28.935+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:06:29.564 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:06:29 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:06:29.360+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:06:29.565 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:06:29 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:06:29.360+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:06:29.565 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:06:29 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:06:29.360+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:06:29.565 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:06:29 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:06:29.361+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:06:29.565 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:06:29 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:06:29.304+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:06:29.565 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:06:29 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:06:29.304+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:06:29.566 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:06:29 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:06:29.304+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:06:29.566 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:06:29 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:06:29.305+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:06:30.286 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:06:29 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:06:29.961+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:06:30.287 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:06:29 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:06:29.961+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:06:30.287 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:06:29 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:06:29.961+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:06:30.287 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:06:29 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:06:29.963+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:06:30.565 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:06:30 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:06:30.387+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:06:30.566 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:06:30 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:06:30.387+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:06:30.566 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:06:30 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:06:30.387+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:06:30.566 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:06:30 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:06:30.388+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:06:30.566 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:06:30 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:06:30.285+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:06:30.566 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:06:30 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:06:30.285+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:06:30.567 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:06:30 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:06:30.285+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:06:30.567 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:06:30 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:06:30.286+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:06:31.315 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:06:30 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:06:30.947+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:06:31.315 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:06:30 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:06:30.947+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:06:31.315 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:06:30 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:06:30.947+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:06:31.315 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:06:30 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:06:30.948+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:06:31.723 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:06:31 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:06:31.412+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:06:31.723 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:06:31 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:06:31.412+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:06:31.723 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:06:31 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:06:31.412+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:06:31.723 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:06:31 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:06:31.413+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:06:31.723 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:06:31 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:06:31.318+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:06:31.724 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:06:31 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:06:31.318+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:06:31.724 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:06:31 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:06:31.318+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:06:31.724 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:06:31 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:06:31.319+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:06:32.064 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Dec 16 22:06:31 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-mon-smithi045[80759]: 2023-12-16T22:06:31.790+0000 7f45a1f8f700 -1 mon.smithi045@0(probing) e2 get_health_metrics reporting 26454 slow ops, oldest is monmgrreport(gid 14214, 0 checks, 1 progress events) 2023-12-16T22:06:32.065 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:06:31 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:06:31.913+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:06:32.065 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:06:31 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:06:31.913+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:06:32.065 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:06:31 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:06:31.913+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:06:32.065 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:06:31 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:06:31.914+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:06:32.719 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:06:32 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:06:32.445+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:06:32.719 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:06:32 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:06:32.445+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:06:32.719 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:06:32 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:06:32.445+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:06:32.720 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:06:32 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:06:32.445+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:06:32.720 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:06:32 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:06:32.365+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:06:32.720 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:06:32 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:06:32.365+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:06:32.720 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:06:32 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:06:32.365+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:06:32.720 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:06:32 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:06:32.366+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:06:33.064 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:06:32 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:06:32.947+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:06:33.065 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:06:32 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:06:32.947+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:06:33.065 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:06:32 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:06:32.947+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:06:33.065 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:06:32 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:06:32.949+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:06:33.814 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:06:33 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:06:33.384+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:06:33.815 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:06:33 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:06:33.384+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:06:33.815 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:06:33 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:06:33.384+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:06:33.815 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:06:33 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:06:33.384+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:06:33.815 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:06:33 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:06:33.416+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:06:33.815 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:06:33 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:06:33.416+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:06:33.816 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:06:33 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:06:33.416+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:06:33.816 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:06:33 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:06:33.417+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:06:34.315 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:06:33 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:06:33.906+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:06:34.315 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:06:33 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:06:33.906+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:06:34.315 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:06:33 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:06:33.906+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:06:34.316 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:06:33 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:06:33.907+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:06:34.814 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:06:34 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:06:34.408+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:06:34.815 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:06:34 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:06:34.409+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:06:34.815 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:06:34 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:06:34.409+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:06:34.815 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:06:34 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:06:34.409+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:06:34.815 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:06:34 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:06:34.415+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:06:34.815 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:06:34 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:06:34.415+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:06:34.815 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:06:34 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:06:34.415+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:06:34.816 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:06:34 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:06:34.415+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:06:35.314 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:06:34 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:06:34.923+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:06:35.315 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:06:34 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:06:34.923+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:06:35.315 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:06:34 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:06:34.923+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:06:35.315 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:06:34 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:06:34.924+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:06:35.791 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:06:35 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:06:35.453+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:06:35.791 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:06:35 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:06:35.453+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:06:35.791 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:06:35 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:06:35.453+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:06:35.792 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:06:35 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:06:35.454+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:06:35.792 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:06:35 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:06:35.428+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:06:35.792 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:06:35 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:06:35.428+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:06:35.792 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:06:35 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:06:35.428+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:06:35.792 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:06:35 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:06:35.429+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:06:36.064 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:06:35 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:06:35.969+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:06:36.065 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:06:35 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:06:35.969+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:06:36.065 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:06:35 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:06:35.969+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:06:36.065 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:06:35 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:06:35.969+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:06:36.794 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:06:36 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:06:36.398+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:06:36.794 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:06:36 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:06:36.398+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:06:36.794 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:06:36 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:06:36.398+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:06:36.794 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:06:36 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:06:36.398+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:06:36.795 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:06:36 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:06:36.416+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:06:36.795 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:06:36 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:06:36.416+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:06:36.795 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:06:36 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:06:36.416+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:06:36.795 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:06:36 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:06:36.417+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:06:37.064 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Dec 16 22:06:36 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-mon-smithi045[80759]: 2023-12-16T22:06:36.792+0000 7f45a1f8f700 -1 mon.smithi045@0(probing) e2 get_health_metrics reporting 26544 slow ops, oldest is monmgrreport(gid 14214, 0 checks, 1 progress events) 2023-12-16T22:06:37.065 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:06:36 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:06:36.982+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:06:37.065 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:06:36 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:06:36.982+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:06:37.065 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:06:36 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:06:36.982+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:06:37.065 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:06:36 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:06:36.983+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:06:37.814 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:06:37 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:06:37.363+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:06:37.815 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:06:37 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:06:37.363+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:06:37.815 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:06:37 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:06:37.363+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:06:37.815 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:06:37 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:06:37.364+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:06:37.815 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:06:37 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:06:37.451+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:06:37.816 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:06:37 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:06:37.451+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:06:37.816 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:06:37 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:06:37.451+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:06:37.816 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:06:37 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:06:37.452+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:06:38.314 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:06:37 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:06:37.991+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:06:38.315 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:06:37 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:06:37.991+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:06:38.315 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:06:37 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:06:37.991+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:06:38.315 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:06:37 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:06:37.992+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:06:38.720 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:06:38 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:06:38.353+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:06:38.720 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:06:38 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:06:38.353+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:06:38.721 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:06:38 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:06:38.353+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:06:38.721 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:06:38 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:06:38.353+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:06:38.721 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:06:38 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:06:38.491+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:06:38.721 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:06:38 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:06:38.491+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:06:38.721 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:06:38 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:06:38.491+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:06:38.721 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:06:38 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:06:38.492+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:06:39.315 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:06:39 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:06:39.034+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:06:39.315 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:06:39 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:06:39.034+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:06:39.315 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:06:39 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:06:39.034+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:06:39.315 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:06:39 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:06:39.034+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:06:39.814 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:06:39 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:06:39.364+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:06:39.815 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:06:39 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:06:39.364+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:06:39.815 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:06:39 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:06:39.364+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:06:39.815 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:06:39 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:06:39.365+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:06:39.815 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:06:39 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:06:39.502+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:06:39.815 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:06:39 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:06:39.502+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:06:39.815 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:06:39 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:06:39.502+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:06:39.815 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:06:39 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:06:39.503+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:06:40.329 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:06:40 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:06:40.078+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:06:40.330 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:06:40 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:06:40.078+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:06:40.330 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:06:40 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:06:40.078+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:06:40.330 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:06:40 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:06:40.079+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:06:40.817 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:06:40 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:06:40.480+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:06:40.818 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:06:40 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:06:40.480+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:06:40.818 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:06:40 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:06:40.480+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:06:40.818 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:06:40 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:06:40.482+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:06:40.818 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:06:40 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:06:40.328+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:06:40.818 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:06:40 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:06:40.328+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:06:40.818 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:06:40 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:06:40.328+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:06:40.819 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:06:40 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:06:40.330+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:06:41.527 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:06:41 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:06:41.312+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:06:41.527 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:06:41 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:06:41.312+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:06:41.528 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:06:41 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:06:41.312+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:06:41.528 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:06:41 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:06:41.314+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:06:41.528 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:06:41 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:06:41.128+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:06:41.528 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:06:41 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:06:41.128+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:06:41.528 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:06:41 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:06:41.128+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:06:41.528 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:06:41 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:06:41.129+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:06:41.797 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:06:41 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:06:41.526+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:06:41.797 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:06:41 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:06:41.526+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:06:41.797 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:06:41 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:06:41.526+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:06:41.797 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:06:41 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:06:41.527+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:06:42.064 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Dec 16 22:06:41 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-mon-smithi045[80759]: 2023-12-16T22:06:41.795+0000 7f45a1f8f700 -1 mon.smithi045@0(probing) e2 get_health_metrics reporting 26635 slow ops, oldest is monmgrreport(gid 14214, 0 checks, 1 progress events) 2023-12-16T22:06:42.513 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:06:42 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:06:42.306+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:06:42.513 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:06:42 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:06:42.306+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:06:42.513 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:06:42 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:06:42.306+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:06:42.513 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:06:42 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:06:42.307+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:06:42.514 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:06:42 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:06:42.098+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:06:42.514 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:06:42 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:06:42.098+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:06:42.514 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:06:42 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:06:42.098+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:06:42.514 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:06:42 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:06:42.100+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:06:42.814 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:06:42 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:06:42.512+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:06:42.815 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:06:42 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:06:42.512+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:06:42.815 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:06:42 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:06:42.512+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:06:42.815 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:06:42 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:06:42.512+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:06:43.556 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:06:43 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:06:43.072+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:06:43.556 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:06:43 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:06:43.072+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:06:43.556 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:06:43 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:06:43.072+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:06:43.556 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:06:43 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:06:43.072+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:06:43.557 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:06:43 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:06:43.318+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:06:43.557 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:06:43 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:06:43.318+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:06:43.557 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:06:43 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:06:43.318+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:06:43.557 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:06:43 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:06:43.318+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:06:43.814 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:06:43 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:06:43.555+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:06:43.814 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:06:43 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:06:43.555+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:06:43.815 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:06:43 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:06:43.555+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:06:43.815 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:06:43 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:06:43.556+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:06:44.565 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:06:44 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:06:44.078+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:06:44.565 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:06:44 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:06:44.078+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:06:44.565 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:06:44 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:06:44.078+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:06:44.566 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:06:44 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:06:44.079+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:06:44.566 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:06:44 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:06:44.320+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:06:44.566 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:06:44 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:06:44.320+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:06:44.567 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:06:44 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:06:44.320+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:06:44.567 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:06:44 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:06:44.320+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:06:45.007 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:06:44 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:06:44.587+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:06:45.007 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:06:44 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:06:44.587+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:06:45.007 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:06:44 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:06:44.587+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:06:45.007 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:06:44 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:06:44.587+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:06:45.314 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:06:45 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:06:45.086+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:06:45.314 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:06:45 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:06:45.086+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:06:45.315 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:06:45 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:06:45.086+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:06:45.315 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:06:45 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:06:45.087+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:06:45.814 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:06:45 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:06:45.576+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:06:45.814 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:06:45 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:06:45.576+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:06:45.815 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:06:45 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:06:45.576+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:06:45.815 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:06:45 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:06:45.577+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:06:45.815 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:06:45 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:06:45.332+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:06:45.815 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:06:45 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:06:45.332+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:06:45.815 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:06:45 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:06:45.332+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:06:45.815 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:06:45 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:06:45.333+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:06:46.564 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:06:46 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:06:46.127+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:06:46.565 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:06:46 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:06:46.127+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:06:46.565 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:06:46 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:06:46.127+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:06:46.565 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:06:46 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:06:46.128+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:06:46.565 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:06:46 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:06:46.369+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:06:46.566 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:06:46 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:06:46.369+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:06:46.566 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:06:46 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:06:46.369+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:06:46.566 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:06:46 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:06:46.369+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:06:46.924 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Dec 16 22:06:46 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-mon-smithi045[80759]: 2023-12-16T22:06:46.798+0000 7f45a1f8f700 -1 mon.smithi045@0(probing) e2 get_health_metrics reporting 26725 slow ops, oldest is monmgrreport(gid 14214, 0 checks, 1 progress events) 2023-12-16T22:06:46.924 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:06:46 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:06:46.613+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:06:46.925 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:06:46 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:06:46.613+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:06:46.925 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:06:46 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:06:46.613+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:06:46.925 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:06:46 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:06:46.613+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:06:47.314 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:06:47 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:06:47.167+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:06:47.314 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:06:47 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:06:47.167+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:06:47.315 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:06:47 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:06:47.167+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:06:47.315 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:06:47 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:06:47.168+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:06:47.814 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:06:47 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:06:47.570+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:06:47.815 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:06:47 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:06:47.570+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:06:47.815 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:06:47 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:06:47.570+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:06:47.815 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:06:47 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:06:47.571+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:06:47.815 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:06:47 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:06:47.348+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:06:47.815 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:06:47 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:06:47.348+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:06:47.815 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:06:47 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:06:47.348+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:06:47.816 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:06:47 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:06:47.349+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:06:48.564 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:06:48 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:06:48.171+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:06:48.565 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:06:48 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:06:48.171+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:06:48.565 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:06:48 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:06:48.171+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:06:48.565 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:06:48 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:06:48.173+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:06:48.565 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:06:48 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:06:48.356+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:06:48.565 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:06:48 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:06:48.356+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:06:48.566 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:06:48 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:06:48.356+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:06:48.566 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:06:48 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:06:48.357+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:06:49.064 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:06:48 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:06:48.580+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:06:49.065 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:06:48 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:06:48.580+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:06:49.065 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:06:48 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:06:48.580+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:06:49.065 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:06:48 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:06:48.581+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:06:49.370 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:06:49 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:06:49.137+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:06:49.371 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:06:49 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:06:49.137+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:06:49.371 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:06:49 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:06:49.137+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:06:49.371 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:06:49 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:06:49.137+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:06:49.815 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:06:49 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:06:49.560+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:06:49.815 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:06:49 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:06:49.560+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:06:49.815 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:06:49 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:06:49.560+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:06:49.815 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:06:49 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:06:49.561+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:06:49.816 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:06:49 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:06:49.369+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:06:49.816 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:06:49 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:06:49.369+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:06:49.816 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:06:49 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:06:49.369+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:06:49.816 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:06:49 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:06:49.370+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:06:50.410 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:06:50 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:06:50.110+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:06:50.411 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:06:50 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:06:50.110+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:06:50.411 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:06:50 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:06:50.110+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:06:50.411 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:06:50 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:06:50.112+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:06:50.814 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:06:50 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:06:50.409+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:06:50.815 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:06:50 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:06:50.409+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:06:50.815 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:06:50 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:06:50.409+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:06:50.815 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:06:50 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:06:50.410+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:06:50.815 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:06:50 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:06:50.527+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:06:50.815 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:06:50 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:06:50.527+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:06:50.815 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:06:50 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:06:50.527+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:06:50.816 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:06:50 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:06:50.528+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:06:51.437 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:06:51 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:06:51.075+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:06:51.438 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:06:51 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:06:51.075+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:06:51.438 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:06:51 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:06:51.075+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:06:51.438 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:06:51 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:06:51.076+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:06:51.725 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:06:51 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:06:51.559+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:06:51.725 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:06:51 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:06:51.559+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:06:51.725 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:06:51 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:06:51.559+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:06:51.725 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:06:51 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:06:51.560+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:06:51.726 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:06:51 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:06:51.436+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:06:51.726 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:06:51 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:06:51.436+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:06:51.726 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:06:51 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:06:51.436+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:06:51.726 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:06:51 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:06:51.437+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:06:52.053 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Dec 16 22:06:51 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-mon-smithi045[80759]: 2023-12-16T22:06:51.801+0000 7f45a1f8f700 -1 mon.smithi045@0(probing) e2 get_health_metrics reporting 26817 slow ops, oldest is monmgrreport(gid 14214, 0 checks, 1 progress events) 2023-12-16T22:06:52.314 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:06:52 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:06:52.052+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:06:52.314 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:06:52 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:06:52.052+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:06:52.315 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:06:52 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:06:52.052+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:06:52.315 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:06:52 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:06:52.053+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:06:52.814 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:06:52 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:06:52.388+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:06:52.815 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:06:52 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:06:52.388+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:06:52.815 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:06:52 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:06:52.388+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:06:52.815 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:06:52 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:06:52.389+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:06:52.815 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:06:52 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:06:52.584+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:06:52.815 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:06:52 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:06:52.584+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:06:52.815 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:06:52 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:06:52.584+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:06:52.816 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:06:52 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:06:52.585+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:06:53.314 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:06:53 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:06:53.024+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:06:53.315 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:06:53 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:06:53.024+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:06:53.315 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:06:53 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:06:53.024+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:06:53.315 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:06:53 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:06:53.025+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:06:53.618 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:06:53 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:06:53.360+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:06:53.619 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:06:53 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:06:53.360+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:06:53.619 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:06:53 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:06:53.360+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:06:53.619 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:06:53 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:06:53.360+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:06:54.064 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:06:53 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:06:53.620+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:06:54.065 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:06:53 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:06:53.620+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:06:54.065 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:06:53 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:06:53.620+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:06:54.065 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:06:53 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:06:53.621+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:06:54.374 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:06:54 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:06:54.070+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:06:54.375 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:06:54 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:06:54.070+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:06:54.375 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:06:54 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:06:54.070+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:06:54.375 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:06:54 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:06:54.071+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:06:54.814 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:06:54 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:06:54.373+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:06:54.815 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:06:54 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:06:54.373+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:06:54.815 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:06:54 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:06:54.373+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:06:54.815 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:06:54 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:06:54.374+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:06:54.816 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:06:54 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:06:54.585+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:06:54.816 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:06:54 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:06:54.585+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:06:54.816 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:06:54 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:06:54.585+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:06:54.816 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:06:54 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:06:54.586+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:06:55.315 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:06:55 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:06:55.038+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:06:55.315 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:06:55 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:06:55.038+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:06:55.315 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:06:55 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:06:55.038+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:06:55.315 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:06:55 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:06:55.039+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:06:55.814 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:06:55 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:06:55.413+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:06:55.815 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:06:55 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:06:55.413+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:06:55.815 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:06:55 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:06:55.413+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:06:55.815 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:06:55 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:06:55.414+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:06:55.815 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:06:55 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:06:55.600+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:06:55.815 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:06:55 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:06:55.600+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:06:55.815 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:06:55 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:06:55.600+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:06:55.816 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:06:55 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:06:55.601+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:06:56.314 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:06:56 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:06:56.046+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:06:56.314 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:06:56 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:06:56.046+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:06:56.315 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:06:56 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:06:56.046+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:06:56.315 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:06:56 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:06:56.047+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:06:56.625 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:06:56 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:06:56.369+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:06:56.625 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:06:56 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:06:56.369+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:06:56.625 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:06:56 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:06:56.369+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:06:56.625 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:06:56 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:06:56.370+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:06:56.929 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Dec 16 22:06:56 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-mon-smithi045[80759]: 2023-12-16T22:06:56.804+0000 7f45a1f8f700 -1 mon.smithi045@0(probing) e2 get_health_metrics reporting 26907 slow ops, oldest is monmgrreport(gid 14214, 0 checks, 1 progress events) 2023-12-16T22:06:56.929 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:06:56 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:06:56.624+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:06:56.929 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:06:56 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:06:56.624+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:06:56.930 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:06:56 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:06:56.624+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:06:56.930 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:06:56 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:06:56.625+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:06:57.314 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:06:56 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:06:56.999+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:06:57.314 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:06:57 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:06:56.999+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:06:57.315 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:06:57 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:06:56.999+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:06:57.315 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:06:57 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:06:56.999+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:06:57.601 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:06:57 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:06:57.334+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:06:57.601 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:06:57 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:06:57.334+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:06:57.601 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:06:57 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:06:57.334+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:06:57.602 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:06:57 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:06:57.335+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:06:58.010 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:06:57 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:06:57.600+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:06:58.010 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:06:57 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:06:57.600+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:06:58.010 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:06:57 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:06:57.600+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:06:58.010 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:06:57 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:06:57.601+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:06:58.314 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:06:58 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:06:58.009+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:06:58.315 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:06:58 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:06:58.009+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:06:58.315 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:06:58 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:06:58.009+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:06:58.315 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:06:58 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:06:58.009+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:06:58.646 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:06:58 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:06:58.339+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:06:58.646 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:06:58 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:06:58.339+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:06:58.646 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:06:58 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:06:58.339+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:06:58.646 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:06:58 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:06:58.340+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:06:59.008 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:06:58 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:06:58.645+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:06:59.009 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:06:58 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:06:58.645+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:06:59.009 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:06:58 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:06:58.645+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:06:59.009 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:06:58 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:06:58.646+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:06:59.314 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:06:59 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:06:59.007+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:06:59.314 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:06:59 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:06:59.007+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:06:59.315 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:06:59 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:06:59.007+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:06:59.315 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:06:59 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:06:59.007+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:06:59.814 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:06:59 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:06:59.613+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:06:59.814 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:06:59 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:06:59.613+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:06:59.815 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:06:59 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:06:59.613+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:06:59.815 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:06:59 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:06:59.615+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:06:59.815 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:06:59 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:06:59.383+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:06:59.815 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:06:59 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:06:59.383+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:06:59.815 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:06:59 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:06:59.383+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:06:59.815 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:06:59 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:06:59.384+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:07:00.314 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:07:00 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:07:00.004+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:07:00.315 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:07:00 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:07:00.004+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:07:00.315 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:07:00 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:07:00.004+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:07:00.315 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:07:00 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:07:00.005+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:07:00.652 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:07:00 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:07:00.381+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:07:00.652 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:07:00 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:07:00.381+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:07:00.652 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:07:00 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:07:00.381+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:07:00.653 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:07:00 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:07:00.382+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:07:01.007 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:07:00 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:07:00.651+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:07:01.008 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:07:00 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:07:00.651+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:07:01.008 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:07:00 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:07:00.651+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:07:01.008 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:07:00 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:07:00.652+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:07:01.314 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:07:01 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:07:01.006+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:07:01.315 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:07:01 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:07:01.006+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:07:01.315 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:07:01 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:07:01.006+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:07:01.315 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:07:01 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:07:01.007+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:07:01.689 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:07:01 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:07:01.410+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:07:01.689 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:07:01 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:07:01.410+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:07:01.689 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:07:01 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:07:01.410+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:07:01.689 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:07:01 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:07:01.411+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:07:02.015 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Dec 16 22:07:01 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-mon-smithi045[80759]: 2023-12-16T22:07:01.807+0000 7f45a1f8f700 -1 mon.smithi045@0(probing) e2 get_health_metrics reporting 26998 slow ops, oldest is monmgrreport(gid 14214, 0 checks, 1 progress events) 2023-12-16T22:07:02.015 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:07:01 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:07:01.688+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:07:02.016 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:07:01 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:07:01.688+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:07:02.016 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:07:01 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:07:01.688+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:07:02.016 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:07:01 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:07:01.688+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:07:02.314 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:07:02 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:07:02.014+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:07:02.315 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:07:02 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:07:02.014+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:07:02.315 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:07:02 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:07:02.014+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:07:02.315 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:07:02 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:07:02.015+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:07:02.703 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:07:02 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:07:02.435+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:07:02.704 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:07:02 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:07:02.435+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:07:02.704 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:07:02 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:07:02.435+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:07:02.704 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:07:02 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:07:02.436+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:07:02.987 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:07:02 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:07:02.702+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:07:02.988 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:07:02 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:07:02.702+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:07:02.988 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:07:02 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:07:02.702+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:07:02.988 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:07:02 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:07:02.703+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:07:03.314 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:07:02 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:07:02.986+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:07:03.315 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:07:02 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:07:02.986+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:07:03.315 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:07:02 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:07:02.986+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:07:03.315 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:07:02 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:07:02.987+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:07:03.733 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:07:03 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:07:03.459+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:07:03.734 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:07:03 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:07:03.459+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:07:03.734 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:07:03 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:07:03.459+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:07:03.734 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:07:03 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:07:03.459+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:07:04.010 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:07:03 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:07:03.732+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:07:04.010 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:07:03 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:07:03.732+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:07:04.010 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:07:03 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:07:03.732+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:07:04.011 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:07:03 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:07:03.733+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:07:04.314 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:07:04 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:07:04.009+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:07:04.315 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:07:04 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:07:04.009+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:07:04.315 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:07:04 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:07:04.009+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:07:04.315 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:07:04 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:07:04.010+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:07:04.739 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:07:04 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:07:04.413+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:07:04.740 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:07:04 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:07:04.413+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:07:04.740 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:07:04 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:07:04.413+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:07:04.740 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:07:04 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:07:04.414+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:07:05.064 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:07:04 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:07:04.964+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:07:05.065 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:07:04 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:07:04.964+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:07:05.065 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:07:04 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:07:04.964+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:07:05.065 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:07:04 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:07:04.965+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:07:05.065 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:07:04 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:07:04.738+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:07:05.065 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:07:04 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:07:04.738+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:07:05.066 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:07:04 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:07:04.738+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:07:05.066 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:07:04 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:07:04.739+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:07:05.697 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:07:05 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:07:05.414+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:07:05.697 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:07:05 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:07:05.414+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:07:05.697 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:07:05 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:07:05.414+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:07:05.698 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:07:05 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:07:05.415+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:07:05.957 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:07:05 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:07:05.696+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:07:05.958 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:07:05 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:07:05.696+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:07:05.958 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:07:05 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:07:05.696+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:07:05.958 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:07:05 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:07:05.697+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:07:06.314 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:07:05 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:07:05.956+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:07:06.315 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:07:05 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:07:05.956+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:07:06.315 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:07:05 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:07:05.956+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:07:06.315 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:07:05 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:07:05.957+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:07:06.739 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:07:06 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:07:06.367+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:07:06.739 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:07:06 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:07:06.367+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:07:06.739 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:07:06 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:07:06.367+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:07:06.739 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:07:06 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:07:06.368+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:07:07.064 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Dec 16 22:07:06 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-mon-smithi045[80759]: 2023-12-16T22:07:06.809+0000 7f45a1f8f700 -1 mon.smithi045@0(probing) e2 get_health_metrics reporting 27087 slow ops, oldest is monmgrreport(gid 14214, 0 checks, 1 progress events) 2023-12-16T22:07:07.065 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:07:06 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:07:06.926+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:07:07.065 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:07:06 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:07:06.926+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:07:07.065 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:07:06 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:07:06.926+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:07:07.065 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:07:06 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:07:06.927+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:07:07.065 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:07:06 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:07:06.736+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:07:07.066 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:07:06 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:07:06.736+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:07:07.066 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:07:06 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:07:06.736+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:07:07.066 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:07:06 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:07:06.737+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:07:07.782 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:07:07 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:07:07.415+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:07:07.782 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:07:07 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:07:07.415+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:07:07.782 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:07:07 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:07:07.415+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:07:07.782 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:07:07 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:07:07.416+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:07:08.064 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:07:07 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:07:07.933+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:07:08.065 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:07:07 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:07:07.933+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:07:08.065 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:07:07 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:07:07.933+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:07:08.065 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:07:07 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:07:07.933+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:07:08.065 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:07:07 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:07:07.780+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:07:08.065 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:07:07 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:07:07.780+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:07:08.065 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:07:07 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:07:07.780+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:07:08.066 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:07:07 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:07:07.780+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:07:08.720 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:07:08 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:07:08.395+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:07:08.720 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:07:08 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:07:08.395+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:07:08.720 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:07:08 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:07:08.395+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:07:08.721 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:07:08 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:07:08.396+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:07:09.064 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:07:08 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:07:08.911+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:07:09.065 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:07:08 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:07:08.911+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:07:09.065 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:07:08 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:07:08.911+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:07:09.065 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:07:08 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:07:08.913+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:07:09.066 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:07:08 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:07:08.801+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:07:09.066 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:07:08 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:07:08.801+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:07:09.066 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:07:08 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:07:08.801+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:07:09.066 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:07:08 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:07:08.802+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:07:09.772 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:07:09 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:07:09.412+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:07:09.773 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:07:09 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:07:09.412+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:07:09.773 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:07:09 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:07:09.412+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:07:09.773 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:07:09 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:07:09.413+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:07:10.064 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:07:09 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:07:09.770+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:07:10.065 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:07:09 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:07:09.770+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:07:10.065 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:07:09 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:07:09.770+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:07:10.065 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:07:09 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:07:09.771+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:07:10.065 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:07:09 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:07:09.930+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:07:10.065 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:07:09 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:07:09.930+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:07:10.065 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:07:09 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:07:09.930+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:07:10.066 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:07:09 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:07:09.931+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:07:10.759 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:07:10 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:07:10.454+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:07:10.760 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:07:10 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:07:10.454+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:07:10.760 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:07:10 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:07:10.454+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:07:10.760 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:07:10 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:07:10.456+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:07:11.064 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:07:10 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:07:10.757+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:07:11.065 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:07:10 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:07:10.757+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:07:11.065 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:07:10 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:07:10.757+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:07:11.065 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:07:10 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:07:10.758+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:07:11.065 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:07:10 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:07:10.909+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:07:11.065 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:07:10 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:07:10.909+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:07:11.066 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:07:10 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:07:10.909+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:07:11.066 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:07:10 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:07:10.910+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:07:11.713 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:07:11 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:07:11.414+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:07:11.713 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:07:11 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:07:11.414+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:07:11.713 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:07:11 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:07:11.414+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:07:11.713 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:07:11 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:07:11.415+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:07:12.064 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Dec 16 22:07:11 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-mon-smithi045[80759]: 2023-12-16T22:07:11.813+0000 7f45a1f8f700 -1 mon.smithi045@0(probing) e2 get_health_metrics reporting 27179 slow ops, oldest is monmgrreport(gid 14214, 0 checks, 1 progress events) 2023-12-16T22:07:12.065 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:07:11 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:07:11.711+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:07:12.065 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:07:11 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:07:11.711+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:07:12.065 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:07:11 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:07:11.711+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:07:12.065 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:07:11 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:07:11.713+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:07:12.065 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:07:11 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:07:11.919+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:07:12.066 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:07:11 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:07:11.919+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:07:12.066 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:07:11 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:07:11.919+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:07:12.066 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:07:11 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:07:11.920+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:07:12.699 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:07:12 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:07:12.423+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:07:12.699 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:07:12 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:07:12.423+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:07:12.699 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:07:12 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:07:12.423+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:07:12.699 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:07:12 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:07:12.424+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:07:13.064 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:07:12 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:07:12.940+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:07:13.065 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:07:12 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:07:12.940+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:07:13.065 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:07:12 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:07:12.940+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:07:13.065 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:07:12 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:07:12.940+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:07:13.065 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:07:12 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:07:12.698+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:07:13.065 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:07:12 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:07:12.698+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:07:13.066 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:07:12 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:07:12.698+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:07:13.066 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:07:12 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:07:12.698+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:07:13.677 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:07:13 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:07:13.407+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:07:13.678 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:07:13 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:07:13.407+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:07:13.678 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:07:13 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:07:13.407+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:07:13.678 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:07:13 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:07:13.408+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:07:14.064 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:07:13 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:07:13.676+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:07:14.065 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:07:13 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:07:13.676+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:07:14.065 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:07:13 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:07:13.676+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:07:14.065 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:07:13 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:07:13.677+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:07:14.065 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:07:13 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:07:13.925+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:07:14.065 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:07:13 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:07:13.925+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:07:14.066 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:07:13 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:07:13.925+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:07:14.066 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:07:13 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:07:13.926+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:07:14.720 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:07:14 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:07:14.375+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:07:14.720 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:07:14 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:07:14.375+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:07:14.721 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:07:14 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:07:14.375+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:07:14.721 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:07:14 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:07:14.376+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:07:15.064 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:07:14 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:07:14.719+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:07:15.065 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:07:14 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:07:14.719+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:07:15.065 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:07:14 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:07:14.719+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:07:15.065 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:07:14 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:07:14.720+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:07:15.065 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:07:14 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:07:14.931+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:07:15.065 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:07:14 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:07:14.931+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:07:15.066 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:07:14 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:07:14.931+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:07:15.066 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:07:14 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:07:14.932+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:07:15.727 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:07:15 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:07:15.415+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:07:15.727 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:07:15 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:07:15.415+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:07:15.727 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:07:15 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:07:15.415+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:07:15.727 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:07:15 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:07:15.416+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:07:16.064 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:07:15 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:07:15.937+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:07:16.065 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:07:15 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:07:15.937+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:07:16.065 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:07:15 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:07:15.937+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:07:16.065 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:07:15 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:07:15.938+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:07:16.065 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:07:15 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:07:15.766+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:07:16.066 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:07:15 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:07:15.766+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:07:16.066 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:07:15 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:07:15.766+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:07:16.066 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:07:15 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:07:15.767+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:07:16.755 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:07:16 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:07:16.405+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:07:16.755 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:07:16 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:07:16.406+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:07:16.755 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:07:16 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:07:16.406+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:07:16.755 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:07:16 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:07:16.406+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:07:17.064 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Dec 16 22:07:16 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-mon-smithi045[80759]: 2023-12-16T22:07:16.816+0000 7f45a1f8f700 -1 mon.smithi045@0(probing) e2 get_health_metrics reporting 27269 slow ops, oldest is monmgrreport(gid 14214, 0 checks, 1 progress events) 2023-12-16T22:07:17.065 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:07:16 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:07:16.955+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:07:17.065 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:07:16 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:07:16.955+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:07:17.065 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:07:16 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:07:16.955+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:07:17.065 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:07:16 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:07:16.957+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:07:17.065 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:07:16 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:07:16.753+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:07:17.066 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:07:16 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:07:16.753+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:07:17.066 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:07:16 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:07:16.753+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:07:17.066 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:07:16 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:07:16.754+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:07:17.715 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:07:17 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:07:17.397+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:07:17.715 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:07:17 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:07:17.397+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:07:17.715 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:07:17 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:07:17.398+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:07:17.715 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:07:17 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:07:17.399+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:07:18.064 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:07:17 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:07:17.951+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:07:18.065 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:07:17 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:07:17.951+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:07:18.065 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:07:17 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:07:17.951+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:07:18.065 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:07:17 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:07:17.952+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:07:18.065 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:07:17 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:07:17.714+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:07:18.065 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:07:17 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:07:17.714+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:07:18.065 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:07:17 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:07:17.714+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:07:18.066 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:07:17 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:07:17.715+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:07:18.721 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:07:18 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:07:18.440+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:07:18.721 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:07:18 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:07:18.440+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:07:18.721 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:07:18 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:07:18.440+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:07:18.722 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:07:18 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:07:18.441+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:07:19.064 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:07:18 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:07:18.947+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:07:19.065 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:07:18 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:07:18.947+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:07:19.065 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:07:18 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:07:18.947+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:07:19.065 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:07:18 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:07:18.948+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:07:19.065 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:07:18 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:07:18.738+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:07:19.065 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:07:18 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:07:18.738+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:07:19.065 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:07:18 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:07:18.738+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:07:19.066 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:07:18 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:07:18.739+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:07:19.767 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:07:19 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:07:19.469+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:07:19.768 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:07:19 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:07:19.469+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:07:19.768 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:07:19 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:07:19.469+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:07:19.768 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:07:19 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:07:19.470+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:07:20.064 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:07:19 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:07:19.766+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:07:20.065 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:07:19 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:07:19.766+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:07:20.065 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:07:19 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:07:19.766+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:07:20.065 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:07:19 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:07:19.767+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:07:20.066 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:07:19 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:07:19.969+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:07:20.066 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:07:19 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:07:19.969+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:07:20.066 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:07:19 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:07:19.969+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:07:20.067 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:07:19 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:07:19.969+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:07:20.814 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:07:20 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:07:20.754+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:07:20.815 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:07:20 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:07:20.754+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:07:20.815 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:07:20 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:07:20.754+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:07:20.815 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:07:20 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:07:20.754+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:07:20.815 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:07:20 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:07:20.520+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:07:20.815 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:07:20 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:07:20.520+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:07:20.815 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:07:20 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:07:20.520+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:07:20.816 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:07:20 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:07:20.521+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:07:21.314 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:07:20 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:07:20.996+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:07:21.315 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:07:20 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:07:20.996+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:07:21.315 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:07:20 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:07:20.996+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:07:21.315 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:07:20 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:07:20.998+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:07:21.778 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:07:21 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:07:21.506+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:07:21.778 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:07:21 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:07:21.506+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:07:21.778 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:07:21 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:07:21.506+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:07:21.778 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:07:21 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:07:21.507+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:07:22.064 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Dec 16 22:07:21 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-mon-smithi045[80759]: 2023-12-16T22:07:21.819+0000 7f45a1f8f700 -1 mon.smithi045@0(probing) e2 get_health_metrics reporting 27360 slow ops, oldest is monmgrreport(gid 14214, 0 checks, 1 progress events) 2023-12-16T22:07:22.065 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:07:21 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:07:21.776+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:07:22.065 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:07:21 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:07:21.776+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:07:22.065 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:07:21 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:07:21.776+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:07:22.065 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:07:21 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:07:21.778+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:07:22.065 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:07:21 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:07:21.994+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:07:22.066 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:07:21 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:07:21.994+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:07:22.066 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:07:21 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:07:21.994+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:07:22.066 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:07:21 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:07:21.994+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:07:22.771 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:07:22 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:07:22.503+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:07:22.771 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:07:22 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:07:22.503+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:07:22.771 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:07:22 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:07:22.503+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:07:22.771 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:07:22 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:07:22.504+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:07:23.065 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:07:23 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:07:23.005+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:07:23.065 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:07:23 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:07:23.006+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:07:23.065 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:07:23 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:07:23.006+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:07:23.066 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:07:23 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:07:23.006+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:07:23.066 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:07:22 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:07:22.770+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:07:23.066 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:07:22 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:07:22.770+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:07:23.067 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:07:22 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:07:22.770+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:07:23.067 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:07:22 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:07:22.771+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:07:23.814 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:07:23 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:07:23.507+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:07:23.815 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:07:23 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:07:23.507+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:07:23.815 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:07:23 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:07:23.507+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:07:23.815 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:07:23 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:07:23.507+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:07:23.815 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:07:23 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:07:23.743+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:07:23.815 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:07:23 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:07:23.743+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:07:23.815 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:07:23 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:07:23.743+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:07:23.815 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:07:23 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:07:23.744+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:07:24.314 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:07:24 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:07:23.999+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:07:24.314 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:07:24 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:07:24.000+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:07:24.315 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:07:24 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:07:24.000+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:07:24.315 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:07:24 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:07:24.000+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:07:24.814 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:07:24 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:07:24.520+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:07:24.815 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:07:24 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:07:24.520+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:07:24.815 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:07:24 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:07:24.520+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:07:24.815 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:07:24 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:07:24.521+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:07:24.815 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:07:24 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:07:24.741+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:07:24.815 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:07:24 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:07:24.741+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:07:24.815 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:07:24 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:07:24.741+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:07:24.816 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:07:24 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:07:24.742+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:07:25.314 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:07:24 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:07:24.952+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:07:25.315 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:07:24 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:07:24.952+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:07:25.315 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:07:24 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:07:24.952+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:07:25.315 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:07:24 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:07:24.953+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:07:25.814 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:07:25 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:07:25.706+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:07:25.814 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:07:25 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:07:25.706+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:07:25.815 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:07:25 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:07:25.706+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:07:25.815 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:07:25 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:07:25.707+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:07:25.815 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:07:25 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:07:25.528+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:07:25.815 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:07:25 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:07:25.528+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:07:25.815 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:07:25 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:07:25.528+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:07:25.815 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:07:25 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:07:25.529+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:07:26.314 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:07:25 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:07:25.959+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:07:26.315 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:07:25 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:07:25.959+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:07:26.315 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:07:25 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:07:25.959+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:07:26.315 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:07:25 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:07:25.960+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:07:26.814 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:07:26 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:07:26.749+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:07:26.815 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:07:26 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:07:26.749+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:07:26.815 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:07:26 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:07:26.749+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:07:26.815 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:07:26 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:07:26.750+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:07:26.815 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:07:26 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:07:26.505+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:07:26.815 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:07:26 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:07:26.505+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:07:26.815 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:07:26 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:07:26.505+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:07:26.815 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:07:26 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:07:26.506+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:07:27.314 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Dec 16 22:07:26 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-mon-smithi045[80759]: 2023-12-16T22:07:26.822+0000 7f45a1f8f700 -1 mon.smithi045@0(probing) e2 get_health_metrics reporting 27450 slow ops, oldest is monmgrreport(gid 14214, 0 checks, 1 progress events) 2023-12-16T22:07:27.315 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:07:27 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:07:27.006+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:07:27.315 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:07:27 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:07:27.006+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:07:27.315 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:07:27 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:07:27.006+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:07:27.315 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:07:27 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:07:27.008+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:07:27.776 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:07:27 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:07:27.476+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:07:27.776 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:07:27 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:07:27.476+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:07:27.776 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:07:27 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:07:27.476+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:07:27.776 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:07:27 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:07:27.477+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:07:28.058 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:07:27 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:07:27.774+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:07:28.058 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:07:27 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:07:27.774+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:07:28.059 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:07:27 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:07:27.774+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:07:28.059 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:07:27 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:07:27.776+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:07:28.314 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:07:28 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:07:28.056+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:07:28.315 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:07:28 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:07:28.056+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:07:28.315 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:07:28 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:07:28.056+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:07:28.315 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:07:28 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:07:28.058+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:07:28.720 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:07:28 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:07:28.466+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:07:28.720 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:07:28 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:07:28.466+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:07:28.720 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:07:28 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:07:28.466+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:07:28.720 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:07:28 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:07:28.466+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:07:29.029 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:07:28 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:07:28.797+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:07:29.029 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:07:28 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:07:28.797+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:07:29.029 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:07:28 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:07:28.797+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:07:29.029 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:07:28 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:07:28.798+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:07:29.314 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:07:29 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:07:29.028+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:07:29.315 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:07:29 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:07:29.028+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:07:29.315 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:07:29 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:07:29.028+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:07:29.315 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:07:29 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:07:29.028+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:07:29.814 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:07:29 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:07:29.760+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:07:29.815 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:07:29 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:07:29.760+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:07:29.815 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:07:29 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:07:29.760+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:07:29.815 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:07:29 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:07:29.762+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:07:29.815 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:07:29 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:07:29.512+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:07:29.815 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:07:29 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:07:29.512+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:07:29.815 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:07:29 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:07:29.512+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:07:29.816 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:07:29 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:07:29.514+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:07:30.314 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:07:29 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:07:29.986+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:07:30.315 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:07:29 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:07:29.986+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:07:30.315 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:07:29 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:07:29.986+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:07:30.315 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:07:29 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:07:29.987+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:07:30.777 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:07:30 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:07:30.505+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:07:30.777 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:07:30 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:07:30.505+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:07:30.777 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:07:30 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:07:30.505+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:07:30.777 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:07:30 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:07:30.506+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:07:31.064 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:07:30 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:07:30.775+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:07:31.064 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:07:30 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:07:30.775+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:07:31.065 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:07:30 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:07:30.775+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:07:31.065 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:07:30 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:07:30.776+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:07:31.065 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:07:30 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:07:30.988+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:07:31.065 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:07:30 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:07:30.988+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:07:31.065 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:07:30 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:07:30.988+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:07:31.065 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:07:30 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:07:30.989+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:07:31.792 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:07:31 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:07:31.539+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:07:31.792 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:07:31 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:07:31.539+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:07:31.792 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:07:31 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:07:31.539+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:07:31.792 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:07:31 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:07:31.540+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:07:32.066 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Dec 16 22:07:31 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-mon-smithi045[80759]: 2023-12-16T22:07:31.826+0000 7f45a1f8f700 -1 mon.smithi045@0(probing) e2 get_health_metrics reporting 27541 slow ops, oldest is monmgrreport(gid 14214, 0 checks, 1 progress events) 2023-12-16T22:07:32.066 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:07:31 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:07:31.790+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:07:32.066 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:07:31 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:07:31.790+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:07:32.066 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:07:31 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:07:31.790+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:07:32.067 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:07:31 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:07:31.792+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:07:32.067 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:07:31 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:07:31.980+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:07:32.067 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:07:31 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:07:31.980+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:07:32.068 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:07:31 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:07:31.980+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:07:32.068 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:07:31 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:07:31.980+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:07:32.786 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:07:32 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:07:32.498+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:07:32.787 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:07:32 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:07:32.498+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:07:32.787 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:07:32 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:07:32.498+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:07:32.787 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:07:32 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:07:32.498+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:07:33.064 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:07:33 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:07:33.009+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:07:33.065 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:07:33 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:07:33.009+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:07:33.065 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:07:33 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:07:33.009+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:07:33.065 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:07:33 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:07:33.010+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:07:33.065 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:07:32 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:07:32.785+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:07:33.065 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:07:32 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:07:32.785+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:07:33.065 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:07:32 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:07:32.785+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:07:33.066 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:07:32 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:07:32.787+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:07:33.814 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:07:33 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:07:33.545+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:07:33.815 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:07:33 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:07:33.545+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:07:33.815 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:07:33 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:07:33.545+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:07:33.815 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:07:33 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:07:33.546+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:07:33.815 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:07:33 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:07:33.764+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:07:33.815 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:07:33 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:07:33.764+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:07:33.815 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:07:33 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:07:33.764+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:07:33.815 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:07:33 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:07:33.765+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:07:34.314 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:07:33 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:07:33.972+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:07:34.315 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:07:33 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:07:33.972+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:07:34.315 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:07:33 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:07:33.972+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:07:34.315 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:07:33 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:07:33.974+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:07:34.998 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:07:34 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:07:34.592+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:07:34.998 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:07:34 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:07:34.592+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:07:34.998 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:07:34 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:07:34.592+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:07:34.998 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:07:34 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:07:34.593+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:07:34.998 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:07:34 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:07:34.767+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:07:34.999 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:07:34 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:07:34.767+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:07:34.999 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:07:34 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:07:34.767+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:07:34.999 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:07:34 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:07:34.768+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:07:35.314 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:07:34 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:07:34.996+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:07:35.314 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:07:34 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:07:34.996+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:07:35.315 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:07:34 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:07:34.996+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:07:35.315 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:07:34 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:07:34.998+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:07:35.984 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:07:35 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:07:35.598+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:07:35.984 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:07:35 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:07:35.598+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:07:35.984 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:07:35 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:07:35.598+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:07:35.984 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:07:35 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:07:35.600+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:07:35.985 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:07:35 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:07:35.805+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:07:35.985 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:07:35 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:07:35.805+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:07:35.985 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:07:35 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:07:35.805+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:07:35.985 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:07:35 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:07:35.806+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:07:36.314 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:07:35 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:07:35.983+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:07:36.314 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:07:35 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:07:35.983+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:07:36.315 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:07:35 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:07:35.983+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:07:36.315 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:07:35 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:07:35.984+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:07:36.831 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:07:36 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:07:36.579+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:07:36.831 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:07:36 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:07:36.579+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:07:36.831 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:07:36 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:07:36.579+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:07:36.831 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:07:36 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:07:36.580+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:07:37.315 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:07:36 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:07:36.852+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:07:37.315 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:07:36 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:07:36.852+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:07:37.315 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:07:36 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:07:36.852+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:07:37.315 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:07:36 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:07:36.853+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:07:37.316 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:07:36 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:07:36.945+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:07:37.316 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:07:36 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:07:36.945+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:07:37.316 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:07:36 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:07:36.945+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:07:37.316 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:07:36 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:07:36.946+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:07:37.316 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Dec 16 22:07:36 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-mon-smithi045[80759]: 2023-12-16T22:07:36.829+0000 7f45a1f8f700 -1 mon.smithi045@0(probing) e2 get_health_metrics reporting 27631 slow ops, oldest is monmgrreport(gid 14214, 0 checks, 1 progress events) 2023-12-16T22:07:37.936 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:07:37 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:07:37.827+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:07:37.936 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:07:37 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:07:37.827+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:07:37.936 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:07:37 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:07:37.827+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:07:37.936 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:07:37 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:07:37.828+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:07:37.937 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:07:37 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:07:37.584+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:07:37.937 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:07:37 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:07:37.584+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:07:37.937 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:07:37 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:07:37.584+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:07:37.937 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:07:37 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:07:37.586+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:07:38.314 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:07:37 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:07:37.934+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:07:38.315 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:07:37 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:07:37.934+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:07:38.315 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:07:37 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:07:37.934+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:07:38.315 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:07:37 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:07:37.936+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:07:38.943 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:07:38 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:07:38.568+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:07:38.943 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:07:38 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:07:38.568+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:07:38.943 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:07:38 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:07:38.568+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:07:38.943 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:07:38 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:07:38.568+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:07:38.944 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:07:38 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:07:38.793+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:07:38.944 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:07:38 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:07:38.793+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:07:38.944 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:07:38 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:07:38.793+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:07:38.944 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:07:38 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:07:38.794+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:07:39.314 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:07:38 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:07:38.942+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:07:39.314 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:07:38 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:07:38.942+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:07:39.315 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:07:38 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:07:38.942+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:07:39.315 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:07:38 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:07:38.943+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:07:39.917 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:07:39 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:07:39.584+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:07:39.917 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:07:39 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:07:39.584+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:07:39.917 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:07:39 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:07:39.584+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:07:39.917 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:07:39 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:07:39.585+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:07:39.918 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:07:39 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:07:39.776+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:07:39.918 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:07:39 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:07:39.776+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:07:39.918 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:07:39 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:07:39.776+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:07:39.918 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:07:39 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:07:39.777+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:07:40.198 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:07:39 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:07:39.916+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:07:40.198 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:07:39 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:07:39.916+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:07:40.198 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:07:39 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:07:39.916+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:07:40.198 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:07:39 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:07:39.917+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:07:40.941 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:07:40 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:07:40.733+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:07:40.941 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:07:40 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:07:40.733+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:07:40.941 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:07:40 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:07:40.733+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:07:40.941 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:07:40 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:07:40.734+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:07:40.941 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:07:40 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:07:40.627+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:07:40.942 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:07:40 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:07:40.627+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:07:40.942 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:07:40 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:07:40.627+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:07:40.942 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:07:40 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:07:40.627+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:07:41.315 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:07:40 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:07:40.939+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:07:41.315 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:07:40 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:07:40.939+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:07:41.315 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:07:40 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:07:40.939+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:07:41.315 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:07:40 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:07:40.941+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:07:41.905 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Dec 16 22:07:41 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-mon-smithi045[80759]: 2023-12-16T22:07:41.832+0000 7f45a1f8f700 -1 mon.smithi045@0(probing) e2 get_health_metrics reporting 27722 slow ops, oldest is monmgrreport(gid 14214, 0 checks, 1 progress events) 2023-12-16T22:07:41.905 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:07:41 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:07:41.650+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:07:41.905 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:07:41 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:07:41.650+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:07:41.906 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:07:41 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:07:41.650+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:07:41.906 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:07:41 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:07:41.651+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:07:41.906 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:07:41 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:07:41.691+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:07:41.906 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:07:41 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:07:41.691+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:07:41.906 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:07:41 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:07:41.691+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:07:41.906 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:07:41 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:07:41.693+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:07:42.314 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:07:41 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:07:41.904+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:07:42.314 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:07:41 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:07:41.904+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:07:42.315 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:07:41 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:07:41.904+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:07:42.315 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:07:41 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:07:41.904+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:07:42.904 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:07:42 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:07:42.606+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:07:42.904 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:07:42 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:07:42.606+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:07:42.904 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:07:42 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:07:42.606+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:07:42.905 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:07:42 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:07:42.607+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:07:42.905 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:07:42 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:07:42.724+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:07:42.905 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:07:42 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:07:42.724+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:07:42.905 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:07:42 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:07:42.724+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:07:42.905 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:07:42 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:07:42.725+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:07:43.314 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:07:42 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:07:42.903+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:07:43.314 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:07:42 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:07:42.903+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:07:43.315 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:07:42 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:07:42.903+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:07:43.315 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:07:42 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:07:42.904+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:07:43.891 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:07:43 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:07:43.675+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:07:43.892 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:07:43 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:07:43.675+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:07:43.892 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:07:43 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:07:43.675+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:07:43.892 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:07:43 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:07:43.676+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:07:43.892 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:07:43 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:07:43.628+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:07:43.892 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:07:43 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:07:43.628+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:07:43.893 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:07:43 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:07:43.628+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:07:43.893 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:07:43 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:07:43.628+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:07:44.314 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:07:43 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:07:43.890+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:07:44.315 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:07:43 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:07:43.890+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:07:44.315 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:07:43 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:07:43.890+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:07:44.315 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:07:43 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:07:43.891+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:07:44.893 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:07:44 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:07:44.687+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:07:44.893 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:07:44 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:07:44.687+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:07:44.893 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:07:44 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:07:44.687+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:07:44.893 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:07:44 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:07:44.689+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:07:44.894 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:07:44 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:07:44.596+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:07:44.894 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:07:44 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:07:44.596+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:07:44.894 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:07:44 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:07:44.596+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:07:44.894 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:07:44 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:07:44.597+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:07:45.314 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:07:44 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:07:44.891+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:07:45.315 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:07:44 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:07:44.891+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:07:45.315 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:07:44 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:07:44.891+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:07:45.315 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:07:44 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:07:44.892+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:07:45.934 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:07:45 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:07:45.722+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:07:45.934 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:07:45 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:07:45.722+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:07:45.934 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:07:45 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:07:45.722+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:07:45.934 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:07:45 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:07:45.723+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:07:45.935 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:07:45 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:07:45.637+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:07:45.935 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:07:45 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:07:45.637+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:07:45.935 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:07:45 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:07:45.637+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:07:45.935 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:07:45 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:07:45.638+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:07:46.314 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:07:45 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:07:45.931+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:07:46.314 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:07:45 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:07:45.932+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:07:46.315 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:07:45 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:07:45.932+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:07:46.315 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:07:45 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:07:45.933+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:07:46.715 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:07:46 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:07:46.655+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:07:46.715 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:07:46 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:07:46.655+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:07:46.715 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:07:46 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:07:46.655+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:07:46.715 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:07:46 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:07:46.657+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:07:47.064 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Dec 16 22:07:46 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-mon-smithi045[80759]: 2023-12-16T22:07:46.835+0000 7f45a1f8f700 -1 mon.smithi045@0(probing) e2 get_health_metrics reporting 27812 slow ops, oldest is monmgrreport(gid 14214, 0 checks, 1 progress events) 2023-12-16T22:07:47.065 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:07:46 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:07:46.938+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:07:47.065 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:07:46 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:07:46.938+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:07:47.065 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:07:46 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:07:46.938+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:07:47.065 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:07:46 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:07:46.939+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:07:47.065 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:07:46 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:07:46.714+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:07:47.066 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:07:46 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:07:46.714+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:07:47.066 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:07:46 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:07:46.714+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:07:47.066 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:07:46 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:07:46.715+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:07:47.959 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:07:47 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:07:47.720+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:07:47.959 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:07:47 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:07:47.720+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:07:47.959 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:07:47 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:07:47.720+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:07:47.959 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:07:47 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:07:47.720+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:07:47.959 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:07:47 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:07:47.635+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:07:47.960 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:07:47 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:07:47.635+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:07:47.960 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:07:47 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:07:47.635+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:07:47.960 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:07:47 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:07:47.636+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:07:48.314 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:07:47 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:07:47.957+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:07:48.314 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:07:47 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:07:47.957+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:07:48.315 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:07:47 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:07:47.957+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:07:48.315 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:07:47 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:07:47.958+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:07:48.996 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:07:48 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:07:48.717+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:07:48.997 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:07:48 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:07:48.717+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:07:48.997 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:07:48 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:07:48.717+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:07:48.997 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:07:48 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:07:48.718+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:07:48.997 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:07:48 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:07:48.685+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:07:48.997 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:07:48 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:07:48.685+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:07:48.998 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:07:48 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:07:48.685+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:07:48.998 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:07:48 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:07:48.685+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:07:49.314 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:07:48 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:07:48.995+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:07:49.315 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:07:48 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:07:48.995+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:07:49.315 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:07:48 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:07:48.995+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:07:49.315 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:07:48 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:07:48.997+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:07:49.986 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:07:49 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:07:49.720+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:07:49.986 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:07:49 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:07:49.720+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:07:49.986 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:07:49 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:07:49.720+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:07:49.986 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:07:49 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:07:49.721+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:07:49.986 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:07:49 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:07:49.650+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:07:49.986 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:07:49 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:07:49.650+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:07:49.987 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:07:49 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:07:49.650+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:07:49.987 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:07:49 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:07:49.651+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:07:50.314 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:07:49 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:07:49.984+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:07:50.315 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:07:49 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:07:49.984+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:07:50.315 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:07:49 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:07:49.984+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:07:50.315 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:07:49 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:07:49.985+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:07:50.943 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:07:50 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:07:50.717+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:07:50.943 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:07:50 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:07:50.717+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:07:50.943 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:07:50 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:07:50.717+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:07:50.944 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:07:50 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:07:50.718+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:07:50.944 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:07:50 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:07:50.623+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:07:50.944 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:07:50 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:07:50.623+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:07:50.944 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:07:50 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:07:50.623+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:07:50.944 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:07:50 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:07:50.624+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:07:51.314 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:07:50 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:07:50.942+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:07:51.315 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:07:50 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:07:50.942+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:07:51.315 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:07:50 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:07:50.942+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:07:51.315 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:07:50 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:07:50.943+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:07:51.864 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Dec 16 22:07:51 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-mon-smithi045[80759]: 2023-12-16T22:07:51.838+0000 7f45a1f8f700 -1 mon.smithi045@0(probing) e2 get_health_metrics reporting 27934 slow ops, oldest is monmgrreport(gid 14214, 0 checks, 1 progress events) 2023-12-16T22:07:51.864 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:07:51 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:07:51.612+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:07:51.865 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:07:51 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:07:51.612+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:07:51.865 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:07:51 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:07:51.612+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:07:51.865 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:07:51 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:07:51.613+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:07:51.865 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:07:51 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:07:51.698+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:07:51.865 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:07:51 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:07:51.698+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:07:51.865 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:07:51 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:07:51.698+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:07:51.866 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:07:51 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:07:51.699+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:07:52.314 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:07:51 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:07:51.973+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:07:52.314 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:07:51 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:07:51.973+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:07:52.315 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:07:51 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:07:51.973+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:07:52.315 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:07:51 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:07:51.974+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:07:53.021 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:07:52 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:07:52.568+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:07:53.021 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:07:52 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:07:52.568+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:07:53.022 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:07:52 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:07:52.568+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:07:53.022 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:07:52 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:07:52.569+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:07:53.022 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:07:52 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:07:52.712+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:07:53.022 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:07:52 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:07:52.712+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:07:53.022 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:07:52 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:07:52.712+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:07:53.022 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:07:52 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:07:52.712+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:07:53.314 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:07:53 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:07:53.020+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:07:53.314 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:07:53 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:07:53.020+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:07:53.315 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:07:53 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:07:53.020+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:07:53.315 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:07:53 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:07:53.021+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:07:53.814 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:07:53 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:07:53.690+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:07:53.815 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:07:53 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:07:53.690+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:07:53.815 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:07:53 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:07:53.690+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:07:53.815 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:07:53 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:07:53.691+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:07:53.815 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:07:53 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:07:53.539+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:07:53.815 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:07:53 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:07:53.539+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:07:53.815 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:07:53 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:07:53.539+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:07:53.815 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:07:53 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:07:53.540+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:07:54.314 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:07:54 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:07:54.062+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:07:54.315 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:07:54 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:07:54.062+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:07:54.315 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:07:54 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:07:54.062+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:07:54.315 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:07:54 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:07:54.062+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:07:54.814 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:07:54 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:07:54.724+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:07:54.815 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:07:54 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:07:54.724+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:07:54.815 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:07:54 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:07:54.724+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:07:54.815 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:07:54 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:07:54.724+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:07:54.815 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:07:54 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:07:54.498+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:07:54.815 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:07:54 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:07:54.498+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:07:54.815 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:07:54 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:07:54.498+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:07:54.816 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:07:54 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:07:54.499+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:07:55.314 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:07:55 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:07:55.052+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:07:55.315 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:07:55 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:07:55.052+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:07:55.315 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:07:55 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:07:55.052+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:07:55.315 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:07:55 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:07:55.052+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:07:55.814 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:07:55 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:07:55.677+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:07:55.815 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:07:55 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:07:55.677+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:07:55.815 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:07:55 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:07:55.677+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:07:55.815 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:07:55 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:07:55.678+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:07:55.815 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:07:55 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:07:55.486+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:07:55.815 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:07:55 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:07:55.486+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:07:55.816 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:07:55 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:07:55.486+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:07:55.816 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:07:55 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:07:55.486+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:07:56.314 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:07:56 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:07:56.016+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:07:56.315 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:07:56 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:07:56.016+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:07:56.315 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:07:56 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:07:56.016+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:07:56.315 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:07:56 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:07:56.017+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:07:56.814 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:07:56 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:07:56.510+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:07:56.815 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:07:56 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:07:56.510+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:07:56.815 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:07:56 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:07:56.510+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:07:56.815 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:07:56 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:07:56.511+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:07:56.815 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:07:56 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:07:56.637+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:07:56.815 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:07:56 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:07:56.637+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:07:56.815 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:07:56 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:07:56.637+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:07:56.815 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:07:56 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:07:56.638+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:07:57.314 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Dec 16 22:07:56 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-mon-smithi045[80759]: 2023-12-16T22:07:56.842+0000 7f45a1f8f700 -1 mon.smithi045@0(probing) e2 get_health_metrics reporting 28024 slow ops, oldest is monmgrreport(gid 14214, 0 checks, 1 progress events) 2023-12-16T22:07:57.315 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:07:56 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:07:56.978+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:07:57.315 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:07:56 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:07:56.978+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:07:57.315 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:07:56 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:07:56.978+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:07:57.315 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:07:56 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:07:56.979+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:07:57.814 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:07:57 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:07:57.622+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:07:57.815 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:07:57 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:07:57.622+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:07:57.815 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:07:57 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:07:57.622+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:07:57.815 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:07:57 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:07:57.623+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:07:57.815 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:07:57 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:07:57.477+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:07:57.815 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:07:57 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:07:57.477+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:07:57.816 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:07:57 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:07:57.477+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:07:57.816 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:07:57 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:07:57.478+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:07:58.314 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:07:57 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:07:57.993+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:07:58.315 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:07:57 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:07:57.993+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:07:58.315 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:07:57 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:07:57.993+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:07:58.315 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:07:57 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:07:57.994+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:07:58.814 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:07:58 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:07:58.645+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:07:58.815 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:07:58 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:07:58.645+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:07:58.815 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:07:58 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:07:58.645+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:07:58.815 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:07:58 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:07:58.646+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:07:58.815 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:07:58 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:07:58.503+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:07:58.816 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:07:58 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:07:58.503+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:07:58.816 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:07:58 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:07:58.503+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:07:58.816 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:07:58 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:07:58.504+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:07:59.314 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:07:58 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:07:58.952+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:07:59.315 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:07:58 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:07:58.952+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:07:59.315 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:07:58 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:07:58.952+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:07:59.315 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:07:58 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:07:58.952+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:07:59.815 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:07:59 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:07:59.624+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:07:59.815 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:07:59 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:07:59.624+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:07:59.815 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:07:59 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:07:59.624+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:07:59.815 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:07:59 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:07:59.624+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:07:59.816 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:07:59 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:07:59.484+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:07:59.816 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:07:59 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:07:59.484+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:07:59.816 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:07:59 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:07:59.484+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:07:59.817 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:07:59 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:07:59.484+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:08:00.314 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:07:59 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:07:59.976+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:08:00.315 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:07:59 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:07:59.976+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:08:00.315 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:07:59 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:07:59.976+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:08:00.315 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:07:59 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:07:59.976+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:08:00.814 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:08:00 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:08:00.474+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:08:00.815 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:08:00 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:08:00.474+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:08:00.815 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:08:00 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:08:00.474+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:08:00.815 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:08:00 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:08:00.474+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:08:00.815 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:08:00 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:08:00.621+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:08:00.815 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:08:00 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:08:00.621+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:08:00.816 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:08:00 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:08:00.621+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:08:00.816 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:08:00 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:08:00.622+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:08:01.314 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:08:01 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:08:01.008+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:08:01.314 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:08:01 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:08:01.008+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:08:01.315 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:08:01 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:08:01.008+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:08:01.315 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:08:01 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:08:01.008+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:08:01.807 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:08:01 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:08:01.501+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:08:01.807 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:08:01 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:08:01.501+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:08:01.807 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:08:01 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:08:01.501+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:08:01.808 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:08:01 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:08:01.502+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:08:01.808 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:08:01 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:08:01.628+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:08:01.808 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:08:01 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:08:01.628+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:08:01.808 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:08:01 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:08:01.628+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:08:01.808 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:08:01 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:08:01.628+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:08:02.064 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Dec 16 22:08:01 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-mon-smithi045[80759]: 2023-12-16T22:08:01.845+0000 7f45a1f8f700 -1 mon.smithi045@0(probing) e2 get_health_metrics reporting 28114 slow ops, oldest is monmgrreport(gid 14214, 0 checks, 1 progress events) 2023-12-16T22:08:02.065 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:08:02 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:08:02.039+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:08:02.065 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:08:02 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:08:02.039+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:08:02.065 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:08:02 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:08:02.039+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:08:02.065 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:08:02 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:08:02.040+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:08:02.814 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:08:02 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:08:02.518+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:08:02.815 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:08:02 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:08:02.518+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:08:02.815 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:08:02 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:08:02.518+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:08:02.815 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:08:02 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:08:02.519+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:08:02.815 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:08:02 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:08:02.658+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:08:02.815 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:08:02 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:08:02.658+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:08:02.815 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:08:02 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:08:02.658+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:08:02.815 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:08:02 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:08:02.659+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:08:03.315 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:08:02 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:08:02.996+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:08:03.316 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:08:02 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:08:02.996+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:08:03.316 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:08:02 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:08:02.996+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:08:03.316 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:08:02 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:08:02.997+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:08:03.815 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:08:03 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:08:03.610+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:08:03.815 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:08:03 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:08:03.610+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:08:03.815 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:08:03 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:08:03.610+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:08:03.816 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:08:03 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:08:03.611+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:08:03.816 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:08:03 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:08:03.521+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:08:03.816 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:08:03 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:08:03.521+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:08:03.816 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:08:03 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:08:03.521+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:08:03.817 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:08:03 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:08:03.522+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:08:04.314 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:08:03 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:08:03.997+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:08:04.314 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:08:03 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:08:03.997+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:08:04.315 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:08:03 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:08:03.997+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:08:04.315 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:08:03 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:08:03.998+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:08:04.814 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:08:04 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:08:04.514+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:08:04.815 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:08:04 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:08:04.514+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:08:04.815 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:08:04 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:08:04.514+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:08:04.815 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:08:04 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:08:04.515+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:08:04.815 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:08:04 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:08:04.588+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:08:04.815 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:08:04 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:08:04.588+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:08:04.815 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:08:04 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:08:04.588+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:08:04.816 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:08:04 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:08:04.588+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:08:05.314 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:08:04 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:08:04.988+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:08:05.315 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:08:04 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:08:04.988+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:08:05.315 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:08:04 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:08:04.988+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:08:05.315 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:08:04 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:08:04.990+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:08:05.814 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:08:05 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:08:05.551+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:08:05.815 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:08:05 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:08:05.551+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:08:05.815 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:08:05 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:08:05.551+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:08:05.815 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:08:05 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:08:05.552+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:08:05.815 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:08:05 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:08:05.563+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:08:05.815 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:08:05 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:08:05.563+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:08:05.815 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:08:05 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:08:05.563+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:08:05.816 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:08:05 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:08:05.564+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:08:06.314 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:08:06 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:08:06.021+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:08:06.315 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:08:06 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:08:06.021+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:08:06.315 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:08:06 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:08:06.021+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:08:06.315 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:08:06 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:08:06.021+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:08:06.814 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:08:06 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:08:06.507+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:08:06.815 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:08:06 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:08:06.507+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:08:06.815 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:08:06 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:08:06.507+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:08:06.815 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:08:06 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:08:06.508+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:08:06.815 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:08:06 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:08:06.530+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:08:06.815 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:08:06 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:08:06.530+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:08:06.816 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:08:06 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:08:06.530+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:08:06.816 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:08:06 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:08:06.530+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:08:07.314 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Dec 16 22:08:06 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-mon-smithi045[80759]: 2023-12-16T22:08:06.849+0000 7f45a1f8f700 -1 mon.smithi045@0(probing) e2 get_health_metrics reporting 28203 slow ops, oldest is monmgrreport(gid 14214, 0 checks, 1 progress events) 2023-12-16T22:08:07.315 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:08:07 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:08:07.067+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:08:07.315 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:08:07 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:08:07.067+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:08:07.315 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:08:07 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:08:07.067+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:08:07.315 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:08:07 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:08:07.068+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:08:07.814 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:08:07 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:08:07.531+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:08:07.815 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:08:07 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:08:07.531+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:08:07.815 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:08:07 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:08:07.531+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:08:07.815 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:08:07 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:08:07.532+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:08:07.815 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:08:07 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:08:07.524+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:08:07.815 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:08:07 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:08:07.524+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:08:07.816 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:08:07 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:08:07.524+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:08:07.816 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:08:07 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:08:07.524+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:08:08.314 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:08:08 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:08:08.051+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:08:08.315 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:08:08 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:08:08.051+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:08:08.315 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:08:08 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:08:08.051+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:08:08.315 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:08:08 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:08:08.052+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:08:08.814 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:08:08 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:08:08.572+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:08:08.815 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:08:08 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:08:08.572+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:08:08.815 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:08:08 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:08:08.572+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:08:08.815 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:08:08 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:08:08.573+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:08:08.815 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:08:08 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:08:08.493+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:08:08.815 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:08:08 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:08:08.493+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:08:08.815 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:08:08 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:08:08.493+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:08:08.815 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:08:08 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:08:08.494+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:08:09.314 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:08:09 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:08:09.010+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:08:09.315 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:08:09 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:08:09.010+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:08:09.315 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:08:09 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:08:09.010+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:08:09.315 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:08:09 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:08:09.011+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:08:09.814 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:08:09 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:08:09.543+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:08:09.815 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:08:09 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:08:09.543+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:08:09.815 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:08:09 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:08:09.543+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:08:09.815 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:08:09 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:08:09.544+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:08:09.815 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:08:09 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:08:09.508+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:08:09.815 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:08:09 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:08:09.508+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:08:09.815 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:08:09 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:08:09.508+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:08:09.815 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:08:09 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:08:09.509+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:08:10.314 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:08:09 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:08:09.968+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:08:10.314 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:08:09 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:08:09.968+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:08:10.315 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:08:09 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:08:09.968+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:08:10.315 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:08:09 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:08:09.968+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:08:10.814 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:08:10 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:08:10.534+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:08:10.815 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:08:10 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:08:10.534+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:08:10.815 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:08:10 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:08:10.534+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:08:10.815 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:08:10 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:08:10.535+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:08:10.815 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:08:10 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:08:10.498+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:08:10.815 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:08:10 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:08:10.498+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:08:10.816 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:08:10 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:08:10.498+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:08:10.816 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:08:10 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:08:10.499+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:08:11.314 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:08:10 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:08:10.972+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:08:11.315 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:08:10 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:08:10.972+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:08:11.315 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:08:10 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:08:10.972+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:08:11.315 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:08:10 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:08:10.973+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:08:11.814 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:08:11 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:08:11.526+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:08:11.815 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:08:11 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:08:11.527+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:08:11.815 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:08:11 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:08:11.527+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:08:11.815 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:08:11 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:08:11.527+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:08:11.815 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:08:11 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:08:11.545+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:08:11.815 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:08:11 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:08:11.545+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:08:11.815 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:08:11 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:08:11.545+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:08:11.816 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:08:11 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:08:11.546+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:08:12.314 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Dec 16 22:08:11 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-mon-smithi045[80759]: 2023-12-16T22:08:11.851+0000 7f45a1f8f700 -1 mon.smithi045@0(probing) e2 get_health_metrics reporting 28295 slow ops, oldest is monmgrreport(gid 14214, 0 checks, 1 progress events) 2023-12-16T22:08:12.315 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:08:11 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:08:11.982+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:08:12.315 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:08:11 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:08:11.982+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:08:12.315 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:08:11 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:08:11.982+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:08:12.315 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:08:11 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:08:11.982+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:08:12.814 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:08:12 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:08:12.525+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:08:12.815 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:08:12 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:08:12.525+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:08:12.815 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:08:12 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:08:12.525+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:08:12.815 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:08:12 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:08:12.526+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:08:12.815 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:08:12 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:08:12.498+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:08:12.815 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:08:12 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:08:12.498+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:08:12.815 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:08:12 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:08:12.498+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:08:12.815 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:08:12 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:08:12.499+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:08:13.314 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:08:12 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:08:12.996+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:08:13.315 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:08:12 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:08:12.996+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:08:13.315 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:08:12 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:08:12.996+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:08:13.315 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:08:12 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:08:12.996+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:08:13.814 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:08:13 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:08:13.557+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:08:13.815 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:08:13 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:08:13.557+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:08:13.815 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:08:13 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:08:13.557+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:08:13.815 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:08:13 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:08:13.557+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:08:13.815 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:08:13 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:08:13.520+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:08:13.815 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:08:13 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:08:13.520+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:08:13.815 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:08:13 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:08:13.520+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:08:13.815 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:08:13 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:08:13.521+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:08:14.314 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:08:14 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:08:14.003+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:08:14.315 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:08:14 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:08:14.003+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:08:14.315 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:08:14 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:08:14.003+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:08:14.315 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:08:14 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:08:14.004+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:08:14.814 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:08:14 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:08:14.525+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:08:14.815 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:08:14 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:08:14.525+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:08:14.815 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:08:14 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:08:14.525+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:08:14.815 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:08:14 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:08:14.525+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:08:14.815 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:08:14 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:08:14.522+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:08:14.815 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:08:14 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:08:14.522+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:08:14.815 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:08:14 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:08:14.522+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:08:14.816 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:08:14 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:08:14.523+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:08:15.315 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:08:15 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:08:15.006+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:08:15.315 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:08:15 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:08:15.006+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:08:15.315 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:08:15 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:08:15.006+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:08:15.315 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:08:15 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:08:15.007+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:08:15.814 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:08:15 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:08:15.513+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:08:15.815 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:08:15 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:08:15.513+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:08:15.815 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:08:15 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:08:15.513+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:08:15.815 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:08:15 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:08:15.514+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:08:15.815 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:08:15 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:08:15.535+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:08:15.815 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:08:15 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:08:15.535+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:08:15.816 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:08:15 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:08:15.535+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:08:15.816 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:08:15 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:08:15.535+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:08:16.314 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:08:16 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:08:16.035+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:08:16.315 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:08:16 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:08:16.035+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:08:16.315 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:08:16 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:08:16.035+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:08:16.315 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:08:16 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:08:16.036+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:08:16.814 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:08:16 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:08:16.544+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:08:16.815 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:08:16 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:08:16.544+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:08:16.815 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:08:16 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:08:16.544+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:08:16.815 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:08:16 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:08:16.544+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:08:16.815 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:08:16 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:08:16.485+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:08:16.815 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:08:16 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:08:16.485+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:08:16.815 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:08:16 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:08:16.485+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:08:16.815 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:08:16 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:08:16.485+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:08:17.314 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Dec 16 22:08:16 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-mon-smithi045[80759]: 2023-12-16T22:08:16.855+0000 7f45a1f8f700 -1 mon.smithi045@0(probing) e2 get_health_metrics reporting 28385 slow ops, oldest is monmgrreport(gid 14214, 0 checks, 1 progress events) 2023-12-16T22:08:17.315 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:08:17 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:08:17.030+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:08:17.315 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:08:17 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:08:17.030+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:08:17.315 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:08:17 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:08:17.030+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:08:17.315 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:08:17 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:08:17.031+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:08:17.814 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:08:17 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:08:17.533+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:08:17.815 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:08:17 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:08:17.533+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:08:17.815 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:08:17 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:08:17.533+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:08:17.815 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:08:17 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:08:17.534+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:08:17.815 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:08:17 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:08:17.505+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:08:17.815 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:08:17 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:08:17.505+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:08:17.815 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:08:17 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:08:17.505+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:08:17.816 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:08:17 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:08:17.505+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:08:18.314 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:08:17 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:08:17.982+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:08:18.315 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:08:17 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:08:17.982+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:08:18.315 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:08:17 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:08:17.982+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:08:18.315 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:08:17 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:08:17.983+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:08:18.814 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:08:18 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:08:18.567+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:08:18.815 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:08:18 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:08:18.567+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:08:18.815 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:08:18 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:08:18.567+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:08:18.815 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:08:18 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:08:18.568+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:08:18.815 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:08:18 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:08:18.513+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:08:18.816 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:08:18 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:08:18.513+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:08:18.816 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:08:18 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:08:18.514+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:08:18.816 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:08:18 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:08:18.514+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:08:19.314 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:08:18 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:08:18.977+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:08:19.315 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:08:18 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:08:18.977+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:08:19.315 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:08:18 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:08:18.977+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:08:19.315 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:08:18 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:08:18.978+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:08:19.814 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:08:19 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:08:19.548+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:08:19.815 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:08:19 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:08:19.548+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:08:19.815 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:08:19 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:08:19.548+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:08:19.815 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:08:19 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:08:19.549+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:08:19.815 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:08:19 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:08:19.505+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:08:19.815 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:08:19 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:08:19.505+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:08:19.816 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:08:19 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:08:19.505+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:08:19.816 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:08:19 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:08:19.506+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:08:20.314 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:08:19 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:08:19.998+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:08:20.315 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:08:19 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:08:19.998+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:08:20.315 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:08:19 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:08:19.998+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:08:20.315 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:08:20 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:08:19.999+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:08:20.814 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:08:20 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:08:20.522+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:08:20.815 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:08:20 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:08:20.522+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:08:20.815 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:08:20 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:08:20.522+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:08:20.815 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:08:20 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:08:20.523+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:08:20.815 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:08:20 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:08:20.552+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:08:20.816 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:08:20 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:08:20.552+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:08:20.816 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:08:20 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:08:20.552+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:08:20.816 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:08:20 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:08:20.554+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:08:21.314 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:08:21 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:08:21.032+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:08:21.315 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:08:21 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:08:21.032+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:08:21.315 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:08:21 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:08:21.032+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:08:21.315 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:08:21 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:08:21.033+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:08:21.814 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:08:21 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:08:21.531+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:08:21.815 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:08:21 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:08:21.531+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:08:21.815 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:08:21 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:08:21.531+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:08:21.815 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:08:21 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:08:21.532+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:08:21.815 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:08:21 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:08:21.548+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:08:21.815 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:08:21 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:08:21.548+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:08:21.815 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:08:21 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:08:21.548+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:08:21.816 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:08:21 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:08:21.548+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:08:22.314 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Dec 16 22:08:21 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-mon-smithi045[80759]: 2023-12-16T22:08:21.859+0000 7f45a1f8f700 -1 mon.smithi045@0(probing) e2 get_health_metrics reporting 28476 slow ops, oldest is monmgrreport(gid 14214, 0 checks, 1 progress events) 2023-12-16T22:08:22.315 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:08:22 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:08:22.044+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:08:22.315 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:08:22 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:08:22.044+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:08:22.315 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:08:22 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:08:22.044+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:08:22.315 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:08:22 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:08:22.044+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:08:22.814 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:08:22 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:08:22.513+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:08:22.815 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:08:22 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:08:22.513+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:08:22.815 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:08:22 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:08:22.513+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:08:22.815 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:08:22 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:08:22.514+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:08:22.815 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:08:22 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:08:22.553+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:08:22.815 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:08:22 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:08:22.553+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:08:22.815 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:08:22 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:08:22.553+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:08:22.815 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:08:22 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:08:22.553+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:08:23.311 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:08:23 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:08:22.999+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:08:23.311 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:08:23 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:08:22.999+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:08:23.311 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:08:23 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:08:22.999+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:08:23.311 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:08:23 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:08:23.000+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:08:23.564 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:08:23 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:08:23.527+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:08:23.565 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:08:23 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:08:23.527+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:08:23.565 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:08:23 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:08:23.527+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:08:23.565 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:08:23 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:08:23.528+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:08:23.565 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:08:23 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:08:23.533+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:08:23.565 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:08:23 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:08:23.533+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:08:23.565 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:08:23 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:08:23.533+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:08:23.565 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:08:23 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:08:23.534+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:08:24.314 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:08:23 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:08:23.970+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:08:24.314 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:08:23 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:08:23.970+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:08:24.315 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:08:23 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:08:23.970+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:08:24.315 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:08:23 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:08:23.971+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:08:24.814 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:08:24 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:08:24.560+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:08:24.814 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:08:24 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:08:24.560+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:08:24.815 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:08:24 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:08:24.560+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:08:24.815 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:08:24 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:08:24.561+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:08:24.815 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:08:24 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:08:24.551+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:08:24.815 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:08:24 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:08:24.551+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:08:24.815 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:08:24 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:08:24.551+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:08:24.815 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:08:24 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:08:24.552+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:08:25.314 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:08:24 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:08:24.958+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:08:25.314 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:08:24 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:08:24.958+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:08:25.315 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:08:24 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:08:24.958+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:08:25.315 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:08:24 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:08:24.960+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:08:25.814 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:08:25 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:08:25.507+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:08:25.815 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:08:25 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:08:25.507+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:08:25.815 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:08:25 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:08:25.507+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:08:25.815 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:08:25 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:08:25.508+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:08:25.815 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:08:25 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:08:25.544+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:08:25.815 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:08:25 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:08:25.544+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:08:25.815 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:08:25 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:08:25.544+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:08:25.815 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:08:25 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:08:25.545+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:08:26.314 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:08:25 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:08:25.957+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:08:26.314 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:08:25 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:08:25.957+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:08:26.314 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:08:25 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:08:25.957+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:08:26.315 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:08:25 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:08:25.958+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:08:26.814 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:08:26 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:08:26.550+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:08:26.815 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:08:26 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:08:26.550+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:08:26.815 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:08:26 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:08:26.550+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:08:26.815 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:08:26 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:08:26.551+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:08:26.815 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:08:26 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:08:26.500+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:08:26.815 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:08:26 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:08:26.500+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:08:26.815 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:08:26 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:08:26.500+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:08:26.816 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:08:26 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:08:26.501+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:08:27.314 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Dec 16 22:08:26 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-mon-smithi045[80759]: 2023-12-16T22:08:26.862+0000 7f45a1f8f700 -1 mon.smithi045@0(probing) e2 get_health_metrics reporting 28565 slow ops, oldest is monmgrreport(gid 14214, 0 checks, 1 progress events) 2023-12-16T22:08:27.315 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:08:26 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:08:26.951+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:08:27.315 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:08:26 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:08:26.951+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:08:27.315 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:08:26 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:08:26.951+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:08:27.315 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:08:26 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:08:26.952+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:08:27.814 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:08:27 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:08:27.515+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:08:27.815 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:08:27 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:08:27.515+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:08:27.815 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:08:27 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:08:27.515+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:08:27.815 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:08:27 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:08:27.516+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:08:27.815 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:08:27 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:08:27.468+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:08:27.815 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:08:27 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:08:27.469+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:08:27.815 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:08:27 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:08:27.469+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:08:27.815 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:08:27 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:08:27.470+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:08:28.314 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:08:27 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:08:27.917+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:08:28.314 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:08:27 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:08:27.917+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:08:28.315 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:08:27 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:08:27.917+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:08:28.315 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:08:27 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:08:27.918+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:08:28.720 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:08:28 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:08:28.421+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:08:28.721 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:08:28 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:08:28.421+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:08:28.721 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:08:28 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:08:28.421+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:08:28.721 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:08:28 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:08:28.422+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:08:28.721 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:08:28 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:08:28.496+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:08:28.721 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:08:28 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:08:28.496+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:08:28.721 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:08:28 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:08:28.496+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:08:28.722 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:08:28 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:08:28.497+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:08:29.064 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:08:28 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:08:28.904+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:08:29.065 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:08:28 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:08:28.904+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:08:29.065 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:08:28 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:08:28.904+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:08:29.065 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:08:28 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:08:28.905+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:08:29.814 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:08:29 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:08:29.475+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:08:29.815 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:08:29 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:08:29.475+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:08:29.815 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:08:29 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:08:29.475+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:08:29.815 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:08:29 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:08:29.476+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:08:29.815 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:08:29 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:08:29.377+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:08:29.815 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:08:29 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:08:29.377+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:08:29.815 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:08:29 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:08:29.377+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:08:29.816 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:08:29 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:08:29.378+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:08:30.203 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:08:29 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:08:29.882+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:08:30.203 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:08:29 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:08:29.882+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:08:30.203 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:08:29 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:08:29.882+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:08:30.203 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:08:29 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:08:29.883+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:08:30.564 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:08:30 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:08:30.437+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:08:30.565 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:08:30 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:08:30.437+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:08:30.565 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:08:30 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:08:30.437+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:08:30.565 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:08:30 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:08:30.439+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:08:30.565 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:08:30 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:08:30.407+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:08:30.565 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:08:30 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:08:30.407+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:08:30.566 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:08:30 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:08:30.407+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:08:30.566 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:08:30 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:08:30.408+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:08:31.314 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:08:30 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:08:30.870+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:08:31.316 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:08:30 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:08:30.870+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:08:31.316 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:08:30 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:08:30.870+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:08:31.316 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:08:30 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:08:30.871+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:08:31.732 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:08:31 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:08:31.417+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:08:31.732 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:08:31 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:08:31.417+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:08:31.733 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:08:31 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:08:31.417+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:08:31.733 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:08:31 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:08:31.418+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:08:31.733 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:08:31 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:08:31.452+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:08:31.733 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:08:31 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:08:31.452+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:08:31.733 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:08:31 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:08:31.452+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:08:31.734 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:08:31 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:08:31.453+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:08:31.993 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Dec 16 22:08:31 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-mon-smithi045[80759]: 2023-12-16T22:08:31.865+0000 7f45a1f8f700 -1 mon.smithi045@0(probing) e2 get_health_metrics reporting 28658 slow ops, oldest is monmgrreport(gid 14214, 0 checks, 1 progress events) 2023-12-16T22:08:31.993 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:08:31 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:08:31.889+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:08:31.994 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:08:31 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:08:31.889+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:08:31.994 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:08:31 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:08:31.889+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:08:31.994 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:08:31 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:08:31.890+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:08:32.814 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:08:32 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:08:32.439+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:08:32.815 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:08:32 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:08:32.439+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:08:32.815 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:08:32 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:08:32.439+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:08:32.815 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:08:32 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:08:32.440+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:08:32.815 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:08:32 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:08:32.452+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:08:32.815 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:08:32 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:08:32.452+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:08:32.815 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:08:32 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:08:32.452+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:08:32.815 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:08:32 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:08:32.453+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:08:33.314 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:08:32 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:08:32.901+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:08:33.315 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:08:32 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:08:32.901+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:08:33.315 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:08:32 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:08:32.901+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:08:33.315 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:08:32 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:08:32.902+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:08:33.814 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:08:33 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:08:33.487+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:08:33.815 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:08:33 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:08:33.487+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:08:33.815 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:08:33 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:08:33.487+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:08:33.815 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:08:33 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:08:33.487+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:08:33.816 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:08:33 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:08:33.482+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:08:33.816 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:08:33 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:08:33.482+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:08:33.816 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:08:33 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:08:33.482+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:08:33.817 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:08:33 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:08:33.484+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:08:34.314 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:08:33 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:08:33.883+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:08:34.315 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:08:33 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:08:33.883+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:08:34.315 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:08:33 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:08:33.883+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:08:34.315 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:08:33 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:08:33.883+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:08:34.814 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:08:34 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:08:34.529+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:08:34.814 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:08:34 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:08:34.529+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:08:34.815 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:08:34 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:08:34.529+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:08:34.815 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:08:34 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:08:34.530+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:08:34.815 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:08:34 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:08:34.480+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:08:34.815 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:08:34 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:08:34.480+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:08:34.815 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:08:34 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:08:34.480+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:08:34.815 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:08:34 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:08:34.482+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:08:35.314 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:08:34 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:08:34.863+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:08:35.315 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:08:34 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:08:34.863+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:08:35.315 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:08:34 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:08:34.863+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:08:35.315 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:08:34 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:08:34.864+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:08:35.815 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:08:35 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:08:35.580+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:08:35.815 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:08:35 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:08:35.580+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:08:35.815 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:08:35 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:08:35.580+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:08:35.815 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:08:35 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:08:35.581+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:08:35.816 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:08:35 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:08:35.531+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:08:35.816 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:08:35 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:08:35.531+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:08:35.816 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:08:35 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:08:35.531+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:08:35.816 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:08:35 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:08:35.532+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:08:36.314 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:08:35 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:08:35.888+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:08:36.315 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:08:35 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:08:35.888+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:08:36.315 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:08:35 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:08:35.888+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:08:36.315 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:08:35 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:08:35.889+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:08:36.814 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:08:36 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:08:36.583+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:08:36.815 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:08:36 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:08:36.583+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:08:36.815 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:08:36 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:08:36.583+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:08:36.815 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:08:36 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:08:36.584+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:08:36.815 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:08:36 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:08:36.531+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:08:36.815 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:08:36 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:08:36.531+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:08:36.815 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:08:36 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:08:36.531+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:08:36.816 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:08:36 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:08:36.531+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:08:37.314 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Dec 16 22:08:36 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-mon-smithi045[80759]: 2023-12-16T22:08:36.868+0000 7f45a1f8f700 -1 mon.smithi045@0(probing) e2 get_health_metrics reporting 28747 slow ops, oldest is monmgrreport(gid 14214, 0 checks, 1 progress events) 2023-12-16T22:08:37.315 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:08:36 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:08:36.871+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:08:37.315 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:08:36 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:08:36.871+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:08:37.315 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:08:36 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:08:36.871+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:08:37.315 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:08:36 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:08:36.872+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:08:37.814 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:08:37 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:08:37.514+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:08:37.815 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:08:37 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:08:37.514+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:08:37.815 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:08:37 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:08:37.514+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:08:37.815 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:08:37 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:08:37.515+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:08:37.815 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:08:37 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:08:37.616+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:08:37.815 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:08:37 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:08:37.616+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:08:37.815 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:08:37 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:08:37.616+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:08:37.815 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:08:37 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:08:37.617+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:08:38.314 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:08:37 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:08:37.858+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:08:38.314 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:08:37 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:08:37.858+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:08:38.315 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:08:37 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:08:37.858+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:08:38.315 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:08:37 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:08:37.859+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:08:38.814 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:08:38 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:08:38.561+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:08:38.815 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:08:38 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:08:38.561+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:08:38.815 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:08:38 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:08:38.561+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:08:38.815 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:08:38 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:08:38.562+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:08:38.815 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:08:38 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:08:38.621+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:08:38.815 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:08:38 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:08:38.621+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:08:38.816 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:08:38 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:08:38.621+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:08:38.816 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:08:38 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:08:38.621+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:08:39.314 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:08:38 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:08:38.864+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:08:39.315 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:08:38 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:08:38.864+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:08:39.315 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:08:38 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:08:38.864+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:08:39.315 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:08:38 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:08:38.865+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:08:39.814 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:08:39 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:08:39.523+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:08:39.815 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:08:39 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:08:39.523+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:08:39.815 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:08:39 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:08:39.523+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:08:39.815 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:08:39 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:08:39.524+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:08:39.815 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:08:39 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:08:39.628+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:08:39.815 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:08:39 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:08:39.628+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:08:39.815 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:08:39 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:08:39.628+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:08:39.816 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:08:39 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:08:39.629+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:08:40.203 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:08:39 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:08:39.915+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:08:40.204 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:08:39 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:08:39.915+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:08:40.204 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:08:39 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:08:39.915+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:08:40.204 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:08:39 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:08:39.916+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:08:40.814 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:08:40 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:08:40.637+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:08:40.815 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:08:40 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:08:40.637+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:08:40.815 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:08:40 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:08:40.637+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:08:40.815 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:08:40 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:08:40.638+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:08:40.815 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:08:40 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:08:40.539+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:08:40.815 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:08:40 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:08:40.539+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:08:40.815 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:08:40 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:08:40.539+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:08:40.816 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:08:40 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:08:40.540+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:08:41.314 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:08:40 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:08:40.899+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:08:41.315 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:08:40 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:08:40.899+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:08:41.315 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:08:40 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:08:40.899+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:08:41.315 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:08:40 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:08:40.900+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:08:41.873 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:08:41 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:08:41.578+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:08:41.873 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:08:41 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:08:41.579+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:08:41.873 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:08:41 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:08:41.579+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:08:41.873 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:08:41 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:08:41.579+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:08:41.874 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:08:41 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:08:41.597+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:08:41.874 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:08:41 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:08:41.597+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:08:41.874 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:08:41 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:08:41.597+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:08:41.874 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:08:41 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:08:41.598+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:08:42.314 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Dec 16 22:08:41 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-mon-smithi045[80759]: 2023-12-16T22:08:41.872+0000 7f45a1f8f700 -1 mon.smithi045@0(probing) e2 get_health_metrics reporting 28839 slow ops, oldest is monmgrreport(gid 14214, 0 checks, 1 progress events) 2023-12-16T22:08:42.315 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:08:41 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:08:41.905+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:08:42.315 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:08:41 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:08:41.905+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:08:42.315 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:08:41 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:08:41.905+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:08:42.315 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:08:41 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:08:41.906+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:08:42.862 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:08:42 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:08:42.625+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:08:42.863 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:08:42 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:08:42.625+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:08:42.863 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:08:42 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:08:42.625+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:08:42.863 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:08:42 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:08:42.626+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:08:42.863 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:08:42 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:08:42.594+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:08:42.863 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:08:42 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:08:42.594+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:08:42.863 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:08:42 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:08:42.594+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:08:42.864 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:08:42 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:08:42.595+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:08:43.314 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:08:42 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:08:42.861+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:08:43.314 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:08:42 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:08:42.861+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:08:43.315 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:08:42 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:08:42.861+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:08:43.315 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:08:42 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:08:42.862+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:08:43.913 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:08:43 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:08:43.643+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:08:43.913 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:08:43 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:08:43.643+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:08:43.914 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:08:43 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:08:43.643+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:08:43.914 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:08:43 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:08:43.645+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:08:43.914 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:08:43 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:08:43.606+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:08:43.914 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:08:43 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:08:43.606+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:08:43.914 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:08:43 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:08:43.606+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:08:43.915 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:08:43 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:08:43.607+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:08:44.315 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:08:43 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:08:43.912+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:08:44.315 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:08:43 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:08:43.912+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:08:44.315 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:08:43 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:08:43.912+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:08:44.315 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:08:43 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:08:43.913+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:08:44.954 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:08:44 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:08:44.619+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:08:44.954 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:08:44 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:08:44.619+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:08:44.955 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:08:44 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:08:44.619+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:08:44.955 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:08:44 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:08:44.620+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:08:44.955 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:08:44 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:08:44.573+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:08:44.955 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:08:44 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:08:44.573+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:08:44.955 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:08:44 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:08:44.573+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:08:44.955 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:08:44 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:08:44.574+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:08:45.314 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:08:44 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:08:44.953+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:08:45.315 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:08:44 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:08:44.953+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:08:45.315 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:08:44 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:08:44.953+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:08:45.315 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:08:44 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:08:44.954+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:08:45.815 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:08:45 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:08:45.592+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:08:45.815 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:08:45 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:08:45.592+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:08:45.815 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:08:45 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:08:45.592+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:08:45.815 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:08:45 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:08:45.592+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:08:45.816 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:08:45 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:08:45.554+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:08:45.816 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:08:45 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:08:45.554+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:08:45.816 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:08:45 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:08:45.554+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:08:45.816 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:08:45 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:08:45.555+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:08:46.314 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:08:45 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:08:45.987+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:08:46.314 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:08:45 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:08:45.987+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:08:46.315 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:08:45 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:08:45.987+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:08:46.315 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:08:45 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:08:45.988+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:08:46.814 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:08:46 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:08:46.597+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:08:46.815 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:08:46 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:08:46.597+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:08:46.815 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:08:46 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:08:46.597+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:08:46.815 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:08:46 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:08:46.598+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:08:46.815 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:08:46 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:08:46.562+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:08:46.815 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:08:46 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:08:46.562+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:08:46.815 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:08:46 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:08:46.562+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:08:46.815 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:08:46 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:08:46.563+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:08:47.315 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Dec 16 22:08:46 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-mon-smithi045[80759]: 2023-12-16T22:08:46.875+0000 7f45a1f8f700 -1 mon.smithi045@0(probing) e2 get_health_metrics reporting 28928 slow ops, oldest is monmgrreport(gid 14214, 0 checks, 1 progress events) 2023-12-16T22:08:47.316 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:08:47 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:08:47.022+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:08:47.316 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:08:47 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:08:47.022+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:08:47.316 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:08:47 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:08:47.022+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:08:47.316 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:08:47 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:08:47.024+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:08:48.002 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:08:47 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:08:47.616+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:08:48.003 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:08:47 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:08:47.616+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:08:48.003 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:08:47 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:08:47.616+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:08:48.003 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:08:47 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:08:47.616+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:08:48.003 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:08:47 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:08:47.590+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:08:48.003 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:08:47 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:08:47.590+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:08:48.003 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:08:47 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:08:47.590+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:08:48.004 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:08:47 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:08:47.592+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:08:48.314 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:08:48 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:08:48.001+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:08:48.315 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:08:48 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:08:48.001+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:08:48.315 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:08:48 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:08:48.001+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:08:48.315 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:08:48 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:08:48.002+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:08:48.995 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:08:48 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:08:48.595+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:08:48.996 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:08:48 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:08:48.595+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:08:48.996 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:08:48 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:08:48.595+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:08:48.996 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:08:48 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:08:48.596+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:08:48.996 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:08:48 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:08:48.657+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:08:48.996 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:08:48 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:08:48.657+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:08:48.996 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:08:48 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:08:48.657+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:08:48.997 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:08:48 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:08:48.658+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:08:49.314 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:08:48 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:08:48.994+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:08:49.314 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:08:48 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:08:48.994+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:08:49.315 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:08:48 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:08:48.994+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:08:49.315 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:08:48 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:08:48.995+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:08:49.989 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:08:49 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:08:49.648+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:08:49.990 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:08:49 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:08:49.648+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:08:49.990 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:08:49 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:08:49.648+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:08:49.990 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:08:49 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:08:49.648+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:08:49.990 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:08:49 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:08:49.620+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:08:49.990 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:08:49 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:08:49.620+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:08:49.990 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:08:49 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:08:49.620+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:08:49.991 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:08:49 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:08:49.621+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:08:50.314 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:08:49 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:08:49.988+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:08:50.315 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:08:49 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:08:49.988+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:08:50.315 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:08:49 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:08:49.988+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:08:50.315 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:08:49 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:08:49.989+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:08:50.948 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:08:50 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:08:50.603+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:08:50.949 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:08:50 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:08:50.603+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:08:50.949 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:08:50 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:08:50.603+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:08:50.949 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:08:50 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:08:50.604+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:08:50.949 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:08:50 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:08:50.685+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:08:50.949 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:08:50 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:08:50.685+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:08:50.949 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:08:50 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:08:50.685+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:08:50.950 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:08:50 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:08:50.686+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:08:51.314 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:08:50 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:08:50.947+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:08:51.314 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:08:50 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:08:50.947+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:08:51.315 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:08:50 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:08:50.947+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:08:51.315 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:08:50 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:08:50.948+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:08:51.879 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:08:51 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:08:51.567+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:08:51.879 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:08:51 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:08:51.567+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:08:51.880 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:08:51 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:08:51.567+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:08:51.880 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:08:51 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:08:51.568+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:08:51.880 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:08:51 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:08:51.663+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:08:51.880 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:08:51 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:08:51.663+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:08:51.880 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:08:51 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:08:51.663+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:08:51.880 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:08:51 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:08:51.665+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:08:52.314 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Dec 16 22:08:51 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-mon-smithi045[80759]: 2023-12-16T22:08:51.878+0000 7f45a1f8f700 -1 mon.smithi045@0(probing) e2 get_health_metrics reporting 29021 slow ops, oldest is monmgrreport(gid 14214, 0 checks, 1 progress events) 2023-12-16T22:08:52.315 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:08:51 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:08:51.967+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:08:52.315 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:08:51 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:08:51.967+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:08:52.315 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:08:51 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:08:51.967+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:08:52.315 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:08:51 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:08:51.968+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:08:52.974 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:08:52 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:08:52.660+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:08:52.974 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:08:52 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:08:52.660+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:08:52.974 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:08:52 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:08:52.660+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:08:52.974 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:08:52 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:08:52.661+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:08:52.974 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:08:52 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:08:52.567+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:08:52.975 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:08:52 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:08:52.567+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:08:52.975 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:08:52 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:08:52.567+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:08:52.975 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:08:52 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:08:52.568+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:08:53.314 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:08:52 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:08:52.973+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:08:53.314 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:08:52 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:08:52.973+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:08:53.315 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:08:52 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:08:52.973+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:08:53.315 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:08:52 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:08:52.974+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:08:53.939 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:08:53 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:08:53.582+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:08:53.940 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:08:53 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:08:53.582+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:08:53.940 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:08:53 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:08:53.582+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:08:53.940 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:08:53 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:08:53.583+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:08:53.940 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:08:53 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:08:53.639+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:08:53.940 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:08:53 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:08:53.639+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:08:53.940 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:08:53 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:08:53.639+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:08:53.940 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:08:53 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:08:53.640+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:08:54.314 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:08:53 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:08:53.938+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:08:54.315 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:08:53 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:08:53.938+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:08:54.315 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:08:53 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:08:53.938+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:08:54.315 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:08:53 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:08:53.939+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:08:54.930 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:08:54 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:08:54.593+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:08:54.930 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:08:54 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:08:54.593+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:08:54.931 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:08:54 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:08:54.593+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:08:54.931 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:08:54 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:08:54.594+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:08:54.931 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:08:54 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:08:54.622+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:08:54.931 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:08:54 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:08:54.622+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:08:54.931 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:08:54 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:08:54.622+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:08:54.932 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:08:54 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:08:54.623+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:08:55.314 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:08:54 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:08:54.929+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:08:55.314 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:08:54 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:08:54.929+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:08:55.315 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:08:54 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:08:54.929+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:08:55.315 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:08:54 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:08:54.930+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:08:55.959 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:08:55 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:08:55.622+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:08:55.960 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:08:55 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:08:55.622+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:08:55.960 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:08:55 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:08:55.622+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:08:55.960 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:08:55 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:08:55.623+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:08:55.960 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:08:55 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:08:55.643+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:08:55.960 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:08:55 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:08:55.643+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:08:55.961 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:08:55 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:08:55.643+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:08:55.961 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:08:55 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:08:55.644+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:08:56.314 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:08:55 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:08:55.958+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:08:56.314 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:08:55 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:08:55.958+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:08:56.314 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:08:55 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:08:55.958+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:08:56.315 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:08:55 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:08:55.959+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:08:56.907 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Dec 16 22:08:56 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-mon-smithi045[80759]: 2023-12-16T22:08:56.881+0000 7f45a1f8f700 -1 mon.smithi045@0(probing) e2 get_health_metrics reporting 29110 slow ops, oldest is monmgrreport(gid 14214, 0 checks, 1 progress events) 2023-12-16T22:08:56.908 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:08:56 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:08:56.661+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:08:56.908 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:08:56 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:08:56.661+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:08:56.908 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:08:56 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:08:56.661+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:08:56.909 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:08:56 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:08:56.662+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:08:56.909 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:08:56 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:08:56.656+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:08:56.909 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:08:56 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:08:56.656+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:08:56.909 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:08:56 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:08:56.656+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:08:56.909 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:08:56 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:08:56.656+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:08:57.314 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:08:56 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:08:56.955+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:08:57.314 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:08:56 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:08:56.955+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:08:57.315 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:08:56 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:08:56.955+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:08:57.315 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:08:56 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:08:56.956+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:08:57.963 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:08:57 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:08:57.618+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:08:57.964 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:08:57 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:08:57.618+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:08:57.964 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:08:57 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:08:57.618+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:08:57.964 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:08:57 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:08:57.619+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:08:57.964 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:08:57 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:08:57.648+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:08:57.965 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:08:57 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:08:57.648+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:08:57.965 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:08:57 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:08:57.648+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:08:57.965 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:08:57 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:08:57.649+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:08:58.314 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:08:57 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:08:57.962+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:08:58.314 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:08:57 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:08:57.962+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:08:58.315 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:08:57 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:08:57.962+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:08:58.315 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:08:57 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:08:57.962+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:08:58.998 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:08:58 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:08:58.632+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:08:58.999 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:08:58 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:08:58.632+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:08:58.999 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:08:58 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:08:58.632+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:08:58.999 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:08:58 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:08:58.632+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:08:58.999 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:08:58 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:08:58.653+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:08:58.999 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:08:58 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:08:58.653+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:08:58.999 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:08:58 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:08:58.653+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:08:59.000 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:08:58 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:08:58.654+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:08:59.314 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:08:58 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:08:58.997+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:08:59.315 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:08:58 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:08:58.997+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:08:59.315 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:08:58 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:08:58.997+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:08:59.315 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:08:58 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:08:58.998+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:09:00.029 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:08:59 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:08:59.671+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:09:00.029 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:08:59 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:08:59.671+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:09:00.029 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:08:59 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:08:59.671+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:09:00.029 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:08:59 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:08:59.672+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:09:00.029 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:08:59 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:08:59.631+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:09:00.030 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:08:59 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:08:59.631+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:09:00.030 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:08:59 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:08:59.631+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:09:00.030 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:08:59 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:08:59.632+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:09:00.314 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:09:00 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:09:00.027+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:09:00.314 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:09:00 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:09:00.027+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:09:00.315 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:09:00 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:09:00.027+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:09:00.315 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:09:00 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:09:00.028+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:09:01.002 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:09:00 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:09:00.664+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:09:01.002 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:09:00 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:09:00.664+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:09:01.002 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:09:00 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:09:00.664+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:09:01.003 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:09:00 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:09:00.665+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:09:01.003 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:09:00 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:09:00.624+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:09:01.003 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:09:00 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:09:00.624+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:09:01.003 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:09:00 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:09:00.624+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:09:01.003 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:09:00 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:09:00.625+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:09:01.314 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:09:01 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:09:01.001+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:09:01.314 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:09:01 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:09:01.001+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:09:01.315 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:09:01 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:09:01.001+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:09:01.315 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:09:01 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:09:01.002+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:09:01.886 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:09:01 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:09:01.632+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:09:01.886 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:09:01 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:09:01.632+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:09:01.886 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:09:01 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:09:01.632+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:09:01.886 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:09:01 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:09:01.633+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:09:01.887 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:09:01 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:09:01.595+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:09:01.887 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:09:01 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:09:01.595+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:09:01.887 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:09:01 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:09:01.595+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:09:01.887 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:09:01 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:09:01.596+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:09:02.314 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Dec 16 22:09:01 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-mon-smithi045[80759]: 2023-12-16T22:09:01.885+0000 7f45a1f8f700 -1 mon.smithi045@0(probing) e2 get_health_metrics reporting 29202 slow ops, oldest is monmgrreport(gid 14214, 0 checks, 1 progress events) 2023-12-16T22:09:02.315 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:09:02 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:09:02.026+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:09:02.315 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:09:02 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:09:02.026+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:09:02.315 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:09:02 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:09:02.026+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:09:02.315 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:09:02 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:09:02.027+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:09:02.998 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:09:02 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:09:02.596+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:09:02.998 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:09:02 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:09:02.596+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:09:02.998 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:09:02 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:09:02.596+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:09:02.999 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:09:02 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:09:02.597+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:09:02.999 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:09:02 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:09:02.592+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:09:02.999 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:09:02 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:09:02.592+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:09:02.999 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:09:02 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:09:02.592+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:09:02.999 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:09:02 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:09:02.592+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:09:03.314 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:09:02 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:09:02.997+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:09:03.315 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:09:02 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:09:02.997+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:09:03.315 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:09:02 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:09:02.997+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:09:03.315 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:09:02 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:09:02.998+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:09:03.963 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:09:03 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:09:03.608+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:09:03.963 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:09:03 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:09:03.608+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:09:03.963 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:09:03 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:09:03.608+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:09:03.963 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:09:03 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:09:03.609+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:09:03.963 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:09:03 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:09:03.578+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:09:03.963 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:09:03 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:09:03.578+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:09:03.964 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:09:03 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:09:03.578+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:09:03.964 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:09:03 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:09:03.579+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:09:04.318 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:09:03 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:09:03.962+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:09:04.319 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:09:03 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:09:03.962+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:09:04.319 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:09:03 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:09:03.962+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:09:04.319 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:09:03 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:09:03.963+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:09:04.814 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:09:04 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:09:04.626+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:09:04.815 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:09:04 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:09:04.626+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:09:04.815 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:09:04 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:09:04.626+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:09:04.815 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:09:04 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:09:04.627+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:09:04.815 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:09:04 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:09:04.531+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:09:04.815 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:09:04 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:09:04.531+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:09:04.815 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:09:04 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:09:04.531+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:09:04.815 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:09:04 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:09:04.532+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:09:05.314 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:09:04 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:09:04.985+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:09:05.315 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:09:04 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:09:04.985+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:09:05.315 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:09:04 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:09:04.985+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:09:05.315 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:09:04 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:09:04.986+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:09:05.814 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:09:05 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:09:05.617+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:09:05.814 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:09:05 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:09:05.617+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:09:05.815 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:09:05 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:09:05.617+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:09:05.815 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:09:05 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:09:05.618+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:09:05.815 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:09:05 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:09:05.517+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:09:05.815 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:09:05 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:09:05.517+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:09:05.815 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:09:05 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:09:05.517+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:09:05.815 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:09:05 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:09:05.518+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:09:06.314 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:09:06 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:09:06.028+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:09:06.315 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:09:06 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:09:06.028+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:09:06.315 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:09:06 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:09:06.028+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:09:06.315 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:09:06 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:09:06.029+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:09:06.635 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:09:06 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:09:06.538+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:09:06.636 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:09:06 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:09:06.538+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:09:06.636 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:09:06 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:09:06.538+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:09:06.636 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:09:06 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:09:06.539+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:09:06.889 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:09:06 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:09:06.634+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:09:06.889 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:09:06 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:09:06.634+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:09:06.889 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:09:06 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:09:06.634+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:09:06.890 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:09:06 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:09:06.635+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:09:07.314 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Dec 16 22:09:06 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-mon-smithi045[80759]: 2023-12-16T22:09:06.888+0000 7f45a1f8f700 -1 mon.smithi045@0(probing) e2 get_health_metrics reporting 29290 slow ops, oldest is monmgrreport(gid 14214, 0 checks, 1 progress events) 2023-12-16T22:09:07.315 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:09:07 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:09:07.056+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:09:07.315 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:09:07 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:09:07.056+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:09:07.315 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:09:07 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:09:07.056+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:09:07.315 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:09:07 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:09:07.057+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:09:07.814 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:09:07 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:09:07.591+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:09:07.815 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:09:07 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:09:07.591+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:09:07.815 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:09:07 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:09:07.591+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:09:07.815 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:09:07 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:09:07.592+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:09:07.815 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:09:07 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:09:07.549+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:09:07.815 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:09:07 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:09:07.550+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:09:07.815 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:09:07 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:09:07.550+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:09:07.816 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:09:07 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:09:07.550+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:09:08.564 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:09:08 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:09:08.099+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:09:08.565 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:09:08 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:09:08.099+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:09:08.565 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:09:08 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:09:08.099+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:09:08.565 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:09:08 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:09:08.100+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:09:09.052 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:09:08 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:09:08.601+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:09:09.053 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:09:08 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:09:08.601+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:09:09.053 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:09:08 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:09:08.601+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:09:09.053 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:09:08 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:09:08.601+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:09:09.053 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:09:08 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:09:08.586+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:09:09.053 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:09:08 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:09:08.586+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:09:09.053 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:09:08 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:09:08.586+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:09:09.054 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:09:08 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:09:08.587+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:09:09.314 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:09:09 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:09:09.051+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:09:09.314 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:09:09 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:09:09.051+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:09:09.315 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:09:09 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:09:09.051+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:09:09.315 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:09:09 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:09:09.052+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:09:10.007 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:09:09 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:09:09.650+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:09:10.007 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:09:09 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:09:09.650+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:09:10.007 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:09:09 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:09:09.650+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:09:10.008 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:09:09 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:09:09.651+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:09:10.008 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:09:09 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:09:09.601+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:09:10.008 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:09:09 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:09:09.601+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:09:10.008 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:09:09 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:09:09.601+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:09:10.008 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:09:09 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:09:09.602+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:09:10.314 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:09:10 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:09:10.006+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:09:10.314 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:09:10 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:09:10.006+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:09:10.315 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:09:10 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:09:10.006+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:09:10.315 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:09:10 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:09:10.007+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:09:11.049 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:09:10 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:09:10.607+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:09:11.049 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:09:10 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:09:10.607+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:09:11.049 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:09:10 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:09:10.607+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:09:11.049 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:09:10 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:09:10.608+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:09:11.050 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:09:10 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:09:10.629+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:09:11.050 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:09:10 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:09:10.629+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:09:11.050 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:09:10 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:09:10.629+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:09:11.050 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:09:10 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:09:10.630+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:09:11.314 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:09:11 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:09:11.048+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:09:11.315 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:09:11 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:09:11.048+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:09:11.315 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:09:11 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:09:11.048+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:09:11.315 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:09:11 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:09:11.049+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:09:11.894 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:09:11 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:09:11.673+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:09:11.894 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:09:11 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:09:11.673+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:09:11.894 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:09:11 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:09:11.673+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:09:11.894 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:09:11 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:09:11.674+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:09:11.895 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:09:11 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:09:11.606+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:09:11.895 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:09:11 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:09:11.606+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:09:11.895 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:09:11 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:09:11.606+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:09:11.895 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:09:11 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:09:11.607+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:09:12.314 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Dec 16 22:09:11 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-mon-smithi045[80759]: 2023-12-16T22:09:11.891+0000 7f45a1f8f700 -1 mon.smithi045@0(probing) e2 get_health_metrics reporting 29382 slow ops, oldest is monmgrreport(gid 14214, 0 checks, 1 progress events) 2023-12-16T22:09:12.315 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:09:12 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:09:12.077+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:09:12.315 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:09:12 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:09:12.077+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:09:12.315 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:09:12 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:09:12.077+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:09:12.315 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:09:12 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:09:12.078+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:09:12.814 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:09:12 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:09:12.639+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:09:12.815 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:09:12 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:09:12.639+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:09:12.815 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:09:12 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:09:12.639+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:09:12.815 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:09:12 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:09:12.640+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:09:12.815 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:09:12 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:09:12.616+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:09:12.815 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:09:12 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:09:12.617+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:09:12.815 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:09:12 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:09:12.617+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:09:12.815 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:09:12 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:09:12.618+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:09:13.564 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:09:13 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:09:13.065+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:09:13.564 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:09:13 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:09:13.065+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:09:13.565 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:09:13 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:09:13.065+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:09:13.565 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:09:13 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:09:13.066+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:09:14.054 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:09:13 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:09:13.602+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:09:14.055 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:09:13 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:09:13.602+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:09:14.055 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:09:13 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:09:13.602+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:09:14.055 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:09:13 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:09:13.602+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:09:14.055 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:09:13 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:09:13.614+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:09:14.055 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:09:13 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:09:13.614+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:09:14.055 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:09:13 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:09:13.614+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:09:14.056 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:09:13 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:09:13.614+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:09:14.314 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:09:14 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:09:14.053+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:09:14.315 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:09:14 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:09:14.053+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:09:14.315 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:09:14 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:09:14.053+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:09:14.315 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:09:14 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:09:14.054+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:09:15.064 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:09:14 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:09:14.655+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:09:15.065 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:09:14 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:09:14.655+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:09:15.065 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:09:14 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:09:14.655+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:09:15.065 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:09:14 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:09:14.656+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:09:15.065 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:09:14 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:09:14.635+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:09:15.065 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:09:14 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:09:14.635+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:09:15.065 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:09:14 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:09:14.635+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:09:15.066 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:09:14 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:09:14.636+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:09:15.535 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:09:15 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:09:15.081+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:09:15.535 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:09:15 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:09:15.081+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:09:15.535 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:09:15 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:09:15.081+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:09:15.535 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:09:15 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:09:15.082+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:09:15.814 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:09:15 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:09:15.656+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:09:15.815 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:09:15 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:09:15.656+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:09:15.815 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:09:15 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:09:15.656+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:09:15.815 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:09:15 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:09:15.657+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:09:15.815 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:09:15 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:09:15.634+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:09:15.815 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:09:15 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:09:15.634+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:09:15.815 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:09:15 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:09:15.634+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:09:15.815 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:09:15 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:09:15.635+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:09:16.314 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:09:16 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:09:16.043+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:09:16.315 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:09:16 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:09:16.043+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:09:16.315 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:09:16 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:09:16.043+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:09:16.315 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:09:16 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:09:16.044+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:09:17.022 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:09:16 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:09:16.682+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:09:17.022 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:09:16 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:09:16.682+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:09:17.023 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:09:16 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:09:16.682+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:09:17.023 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:09:16 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:09:16.683+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:09:17.023 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:09:16 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:09:16.672+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:09:17.023 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:09:16 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:09:16.672+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:09:17.023 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:09:16 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:09:16.672+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:09:17.023 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:09:16 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:09:16.673+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:09:17.024 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Dec 16 22:09:16 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-mon-smithi045[80759]: 2023-12-16T22:09:16.894+0000 7f45a1f8f700 -1 mon.smithi045@0(probing) e2 get_health_metrics reporting 29471 slow ops, oldest is monmgrreport(gid 14214, 0 checks, 1 progress events) 2023-12-16T22:09:17.314 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:09:17 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:09:17.071+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:09:17.314 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:09:17 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:09:17.071+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:09:17.315 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:09:17 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:09:17.071+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:09:17.315 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:09:17 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:09:17.072+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:09:18.064 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:09:17 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:09:17.624+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:09:18.064 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:09:17 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:09:17.624+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:09:18.065 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:09:17 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:09:17.624+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:09:18.065 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:09:17 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:09:17.626+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:09:18.065 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:09:17 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:09:17.690+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:09:18.065 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:09:17 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:09:17.690+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:09:18.065 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:09:17 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:09:17.690+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:09:18.066 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:09:17 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:09:17.691+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:09:18.564 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:09:18 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:09:18.121+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:09:18.564 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:09:18 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:09:18.121+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:09:18.564 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:09:18 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:09:18.121+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:09:18.565 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:09:18 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:09:18.121+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:09:19.064 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:09:18 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:09:18.659+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:09:19.065 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:09:18 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:09:18.659+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:09:19.065 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:09:18 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:09:18.659+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:09:19.065 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:09:18 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:09:18.660+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:09:19.065 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:09:18 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:09:18.638+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:09:19.065 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:09:18 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:09:18.638+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:09:19.066 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:09:18 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:09:18.638+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:09:19.066 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:09:18 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:09:18.639+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:09:19.564 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:09:19 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:09:19.121+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:09:19.564 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:09:19 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:09:19.121+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:09:19.565 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:09:19 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:09:19.121+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:09:19.565 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:09:19 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:09:19.121+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:09:20.064 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:09:19 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:09:19.692+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:09:20.065 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:09:19 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:09:19.692+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:09:20.065 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:09:19 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:09:19.692+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:09:20.065 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:09:19 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:09:19.692+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:09:20.065 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:09:19 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:09:19.594+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:09:20.065 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:09:19 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:09:19.594+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:09:20.065 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:09:19 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:09:19.594+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:09:20.065 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:09:19 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:09:19.594+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:09:20.564 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:09:20 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:09:20.117+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:09:20.565 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:09:20 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:09:20.117+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:09:20.565 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:09:20 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:09:20.117+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:09:20.565 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:09:20 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:09:20.117+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:09:21.065 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:09:20 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:09:20.731+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:09:21.065 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:09:20 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:09:20.731+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:09:21.065 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:09:20 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:09:20.731+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:09:21.065 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:09:20 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:09:20.732+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:09:21.066 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:09:20 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:09:20.609+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:09:21.066 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:09:20 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:09:20.609+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:09:21.066 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:09:20 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:09:20.609+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:09:21.066 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:09:20 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:09:20.609+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:09:21.564 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:09:21 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:09:21.115+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:09:21.564 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:09:21 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:09:21.115+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:09:21.565 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:09:21 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:09:21.115+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:09:21.565 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:09:21 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:09:21.115+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:09:21.899 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:09:21 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:09:21.628+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:09:21.900 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:09:21 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:09:21.628+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:09:21.900 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:09:21 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:09:21.628+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:09:21.900 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:09:21 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:09:21.630+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:09:21.900 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:09:21 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:09:21.684+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:09:21.900 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:09:21 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:09:21.684+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:09:21.900 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:09:21 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:09:21.684+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:09:21.900 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:09:21 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:09:21.685+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:09:22.315 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Dec 16 22:09:21 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-mon-smithi045[80759]: 2023-12-16T22:09:21.898+0000 7f45a1f8f700 -1 mon.smithi045@0(probing) e2 get_health_metrics reporting 29562 slow ops, oldest is monmgrreport(gid 14214, 0 checks, 1 progress events) 2023-12-16T22:09:22.315 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:09:22 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:09:22.114+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:09:22.316 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:09:22 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:09:22.114+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:09:22.316 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:09:22 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:09:22.114+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:09:22.316 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:09:22 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:09:22.115+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:09:23.064 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:09:22 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:09:22.658+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:09:23.065 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:09:22 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:09:22.658+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:09:23.065 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:09:22 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:09:22.658+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:09:23.065 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:09:22 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:09:22.658+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:09:23.066 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:09:22 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:09:22.627+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:09:23.066 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:09:22 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:09:22.627+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:09:23.066 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:09:22 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:09:22.627+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:09:23.067 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:09:22 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:09:22.628+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:09:23.565 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:09:23 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:09:23.122+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:09:23.565 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:09:23 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:09:23.122+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:09:23.565 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:09:23 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:09:23.122+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:09:23.565 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:09:23 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:09:23.123+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:09:24.065 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:09:23 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:09:23.659+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:09:24.065 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:09:23 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:09:23.659+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:09:24.065 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:09:23 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:09:23.659+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:09:24.065 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:09:23 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:09:23.660+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:09:24.065 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:09:23 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:09:23.639+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:09:24.066 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:09:23 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:09:23.639+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:09:24.066 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:09:23 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:09:23.639+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:09:24.066 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:09:23 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:09:23.640+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:09:24.564 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:09:24 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:09:24.149+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:09:24.565 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:09:24 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:09:24.149+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:09:24.565 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:09:24 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:09:24.149+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:09:24.565 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:09:24 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:09:24.150+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:09:25.064 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:09:24 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:09:24.615+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:09:25.065 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:09:24 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:09:24.615+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:09:25.065 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:09:24 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:09:24.615+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:09:25.065 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:09:24 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:09:24.617+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:09:25.065 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:09:24 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:09:24.618+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:09:25.065 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:09:24 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:09:24.618+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:09:25.066 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:09:24 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:09:24.618+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:09:25.066 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:09:24 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:09:24.619+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:09:25.564 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:09:25 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:09:25.143+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:09:25.565 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:09:25 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:09:25.143+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:09:25.565 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:09:25 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:09:25.143+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:09:25.565 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:09:25 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:09:25.144+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:09:26.064 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:09:25 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:09:25.641+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:09:26.065 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:09:25 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:09:25.641+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:09:26.065 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:09:25 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:09:25.641+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:09:26.065 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:09:25 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:09:25.643+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:09:26.065 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:09:25 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:09:25.588+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:09:26.066 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:09:25 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:09:25.588+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:09:26.066 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:09:25 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:09:25.588+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:09:26.066 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:09:25 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:09:25.589+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:09:26.564 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:09:26 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:09:26.118+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:09:26.564 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:09:26 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:09:26.118+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:09:26.565 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:09:26 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:09:26.118+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:09:26.565 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:09:26 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:09:26.119+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:09:26.903 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:09:26 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:09:26.586+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:09:26.903 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:09:26 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:09:26.586+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:09:26.903 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:09:26 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:09:26.586+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:09:26.903 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:09:26 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:09:26.587+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:09:26.903 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:09:26 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:09:26.627+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:09:26.904 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:09:26 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:09:26.627+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:09:26.904 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:09:26 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:09:26.627+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:09:26.904 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:09:26 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:09:26.628+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:09:27.314 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Dec 16 22:09:26 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-mon-smithi045[80759]: 2023-12-16T22:09:26.901+0000 7f45a1f8f700 -1 mon.smithi045@0(probing) e2 get_health_metrics reporting 29651 slow ops, oldest is monmgrreport(gid 14214, 0 checks, 1 progress events) 2023-12-16T22:09:27.315 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:09:27 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:09:27.151+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:09:27.315 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:09:27 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:09:27.151+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:09:27.315 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:09:27 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:09:27.151+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:09:27.316 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:09:27 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:09:27.151+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:09:27.814 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:09:27 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:09:27.543+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:09:27.815 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:09:27 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:09:27.543+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:09:27.815 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:09:27 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:09:27.543+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:09:27.815 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:09:27 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:09:27.544+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:09:27.815 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:09:27 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:09:27.600+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:09:27.815 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:09:27 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:09:27.600+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:09:27.815 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:09:27 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:09:27.600+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:09:27.815 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:09:27 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:09:27.601+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:09:28.512 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:09:28 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:09:28.153+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:09:28.512 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:09:28 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:09:28.153+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:09:28.512 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:09:28 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:09:28.153+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:09:28.512 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:09:28 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:09:28.154+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:09:28.814 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:09:28 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:09:28.511+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:09:28.815 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:09:28 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:09:28.511+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:09:28.815 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:09:28 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:09:28.511+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:09:28.815 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:09:28 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:09:28.512+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:09:28.815 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:09:28 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:09:28.576+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:09:28.815 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:09:28 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:09:28.576+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:09:28.815 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:09:28 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:09:28.576+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:09:28.816 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:09:28 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:09:28.577+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:09:29.496 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:09:29 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:09:29.192+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:09:29.496 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:09:29 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:09:29.192+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:09:29.496 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:09:29 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:09:29.192+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:09:29.496 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:09:29 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:09:29.192+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:09:29.814 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:09:29 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:09:29.611+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:09:29.814 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:09:29 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:09:29.611+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:09:29.815 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:09:29 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:09:29.611+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:09:29.815 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:09:29 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:09:29.612+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:09:29.815 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:09:29 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:09:29.495+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:09:29.815 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:09:29 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:09:29.495+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:09:29.815 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:09:29 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:09:29.495+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:09:29.815 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:09:29 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:09:29.495+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:09:30.481 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:09:30 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:09:30.226+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:09:30.482 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:09:30 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:09:30.226+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:09:30.482 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:09:30 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:09:30.226+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:09:30.482 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:09:30 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:09:30.227+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:09:30.815 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:09:30 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:09:30.657+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:09:30.815 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:09:30 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:09:30.657+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:09:30.815 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:09:30 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:09:30.657+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:09:30.815 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:09:30 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:09:30.658+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:09:30.816 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:09:30 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:09:30.480+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:09:30.816 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:09:30 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:09:30.480+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:09:30.816 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:09:30 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:09:30.480+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:09:30.816 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:09:30 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:09:30.481+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:09:31.564 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:09:31 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:09:31.507+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:09:31.564 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:09:31 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:09:31.507+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:09:31.565 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:09:31 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:09:31.507+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:09:31.565 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:09:31 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:09:31.509+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:09:31.565 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:09:31 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:09:31.275+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:09:31.565 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:09:31 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:09:31.275+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:09:31.565 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:09:31 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:09:31.275+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:09:31.565 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:09:31 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:09:31.275+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:09:31.921 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Dec 16 22:09:31 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-mon-smithi045[80759]: 2023-12-16T22:09:31.905+0000 7f45a1f8f700 -1 mon.smithi045@0(probing) e2 get_health_metrics reporting 29744 slow ops, oldest is monmgrreport(gid 14214, 0 checks, 1 progress events) 2023-12-16T22:09:31.922 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:09:31 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:09:31.664+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:09:31.922 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:09:31 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:09:31.664+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:09:31.922 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:09:31 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:09:31.664+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:09:31.922 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:09:31 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:09:31.665+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:09:32.564 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:09:32 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:09:32.293+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:09:32.565 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:09:32 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:09:32.293+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:09:32.565 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:09:32 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:09:32.293+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:09:32.565 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:09:32 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:09:32.295+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:09:32.565 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:09:32 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:09:32.535+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:09:32.565 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:09:32 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:09:32.535+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:09:32.565 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:09:32 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:09:32.535+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:09:32.565 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:09:32 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:09:32.536+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:09:33.064 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:09:32 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:09:32.637+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:09:33.064 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:09:32 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:09:32.637+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:09:33.065 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:09:32 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:09:32.637+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:09:33.065 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:09:32 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:09:32.638+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:09:33.623 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:09:33 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:09:33.322+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:09:33.623 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:09:33 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:09:33.322+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:09:33.623 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:09:33 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:09:33.322+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:09:33.623 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:09:33 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:09:33.322+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:09:33.623 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:09:33 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:09:33.494+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:09:33.624 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:09:33 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:09:33.494+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:09:33.624 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:09:33 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:09:33.494+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:09:33.624 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:09:33 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:09:33.495+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:09:33.967 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:09:33 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:09:33.621+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:09:33.967 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:09:33 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:09:33.621+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:09:33.967 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:09:33 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:09:33.621+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:09:33.967 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:09:33 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:09:33.622+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:09:34.598 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:09:34 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:09:34.329+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:09:34.598 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:09:34 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:09:34.329+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:09:34.599 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:09:34 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:09:34.329+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:09:34.599 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:09:34 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:09:34.329+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:09:34.599 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:09:34 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:09:34.456+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:09:34.599 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:09:34 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:09:34.456+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:09:34.599 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:09:34 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:09:34.456+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:09:34.599 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:09:34 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:09:34.458+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:09:35.064 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:09:34 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:09:34.597+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:09:35.065 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:09:34 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:09:34.597+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:09:35.065 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:09:34 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:09:34.597+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:09:35.065 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:09:34 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:09:34.598+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:09:35.564 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:09:35 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:09:35.484+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:09:35.564 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:09:35 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:09:35.484+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:09:35.565 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:09:35 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:09:35.484+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:09:35.565 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:09:35 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:09:35.484+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:09:35.565 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:09:35 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:09:35.292+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:09:35.565 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:09:35 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:09:35.292+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:09:35.565 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:09:35 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:09:35.292+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:09:35.565 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:09:35 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:09:35.293+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:09:36.065 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:09:35 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:09:35.600+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:09:36.065 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:09:35 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:09:35.600+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:09:36.065 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:09:35 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:09:35.600+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:09:36.065 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:09:35 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:09:35.600+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:09:36.564 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:09:36 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:09:36.469+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:09:36.565 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:09:36 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:09:36.469+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:09:36.565 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:09:36 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:09:36.469+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:09:36.565 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:09:36 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:09:36.469+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:09:36.565 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:09:36 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:09:36.264+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:09:36.565 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:09:36 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:09:36.264+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:09:36.566 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:09:36 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:09:36.264+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:09:36.566 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:09:36 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:09:36.264+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:09:36.910 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:09:36 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:09:36.609+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:09:36.910 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:09:36 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:09:36.609+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:09:36.910 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:09:36 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:09:36.609+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:09:36.910 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:09:36 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:09:36.609+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:09:37.217 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Dec 16 22:09:36 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-mon-smithi045[80759]: 2023-12-16T22:09:36.909+0000 7f45a1f8f700 -1 mon.smithi045@0(probing) e2 get_health_metrics reporting 29829 slow ops, oldest is monmgrreport(gid 14214, 0 checks, 1 progress events) 2023-12-16T22:09:37.480 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:09:37 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:09:37.216+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:09:37.481 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:09:37 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:09:37.216+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:09:37.481 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:09:37 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:09:37.216+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:09:37.481 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:09:37 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:09:37.217+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:09:37.814 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:09:37 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:09:37.479+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:09:37.814 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:09:37 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:09:37.479+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:09:37.815 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:09:37 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:09:37.479+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:09:37.815 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:09:37 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:09:37.481+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:09:37.815 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:09:37 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:09:37.626+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:09:37.815 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:09:37 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:09:37.626+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:09:37.815 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:09:37 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:09:37.626+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:09:37.815 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:09:37 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:09:37.627+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:09:38.564 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:09:38 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:09:38.249+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:09:38.565 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:09:38 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:09:38.249+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:09:38.565 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:09:38 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:09:38.249+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:09:38.565 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:09:38 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:09:38.249+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:09:38.565 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:09:38 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:09:38.489+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:09:38.565 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:09:38 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:09:38.489+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:09:38.566 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:09:38 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:09:38.489+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:09:38.566 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:09:38 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:09:38.490+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:09:39.065 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:09:38 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:09:38.638+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:09:39.065 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:09:38 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:09:38.638+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:09:39.065 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:09:38 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:09:38.638+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:09:39.065 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:09:38 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:09:38.639+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:09:39.564 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:09:39 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:09:39.270+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:09:39.565 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:09:39 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:09:39.270+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:09:39.565 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:09:39 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:09:39.270+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:09:39.565 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:09:39 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:09:39.271+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:09:39.565 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:09:39 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:09:39.517+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:09:39.565 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:09:39 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:09:39.517+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:09:39.566 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:09:39 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:09:39.517+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:09:39.566 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:09:39 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:09:39.519+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:09:40.064 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:09:39 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:09:39.657+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:09:40.065 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:09:39 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:09:39.657+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:09:40.065 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:09:39 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:09:39.657+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:09:40.065 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:09:39 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:09:39.659+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:09:40.559 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:09:40 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:09:40.225+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:09:40.560 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:09:40 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:09:40.225+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:09:40.560 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:09:40 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:09:40.225+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:09:40.560 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:09:40 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:09:40.225+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:09:40.814 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:09:40 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:09:40.558+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:09:40.815 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:09:40 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:09:40.558+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:09:40.815 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:09:40 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:09:40.558+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:09:40.815 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:09:40 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:09:40.559+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:09:40.815 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:09:40 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:09:40.706+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:09:40.815 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:09:40 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:09:40.706+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:09:40.815 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:09:40 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:09:40.706+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:09:40.816 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:09:40 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:09:40.707+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:09:41.564 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:09:41 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:09:41.242+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:09:41.564 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:09:41 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:09:41.242+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:09:41.565 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:09:41 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:09:41.242+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:09:41.565 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:09:41 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:09:41.243+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:09:41.914 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:09:41 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:09:41.664+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:09:41.914 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:09:41 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:09:41.664+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:09:41.914 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:09:41 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:09:41.664+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:09:41.914 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:09:41 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:09:41.665+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:09:41.915 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:09:41 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:09:41.604+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:09:41.915 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:09:41 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:09:41.604+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:09:41.915 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:09:41 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:09:41.604+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:09:41.916 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:09:41 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:09:41.605+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:09:42.195 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Dec 16 22:09:41 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-mon-smithi045[80759]: 2023-12-16T22:09:41.912+0000 7f45a1f8f700 -1 mon.smithi045@0(probing) e2 get_health_metrics reporting 29921 slow ops, oldest is monmgrreport(gid 14214, 0 checks, 1 progress events) 2023-12-16T22:09:42.564 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:09:42 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:09:42.195+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:09:42.565 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:09:42 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:09:42.195+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:09:42.565 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:09:42 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:09:42.195+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:09:42.565 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:09:42 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:09:42.195+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:09:43.064 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:09:42 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:09:42.634+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:09:43.065 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:09:42 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:09:42.634+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:09:43.065 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:09:42 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:09:42.634+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:09:43.065 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:09:42 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:09:42.635+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:09:43.065 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:09:42 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:09:42.628+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:09:43.065 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:09:42 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:09:42.628+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:09:43.065 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:09:42 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:09:42.628+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:09:43.066 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:09:42 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:09:42.629+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:09:43.564 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:09:43 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:09:43.238+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:09:43.565 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:09:43 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:09:43.238+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:09:43.565 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:09:43 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:09:43.238+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:09:43.565 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:09:43 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:09:43.239+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:09:43.920 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:09:43 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:09:43.621+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:09:43.921 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:09:43 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:09:43.621+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:09:43.921 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:09:43 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:09:43.621+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:09:43.921 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:09:43 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:09:43.622+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:09:43.921 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:09:43 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:09:43.594+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:09:43.921 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:09:43 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:09:43.594+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:09:43.922 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:09:43 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:09:43.594+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:09:43.922 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:09:43 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:09:43.595+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:09:44.494 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:09:44 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:09:44.215+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:09:44.494 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:09:44 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:09:44.215+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:09:44.495 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:09:44 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:09:44.215+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:09:44.495 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:09:44 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:09:44.217+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:09:44.814 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:09:44 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:09:44.607+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:09:44.815 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:09:44 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:09:44.607+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:09:44.815 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:09:44 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:09:44.607+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:09:44.815 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:09:44 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:09:44.609+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:09:44.815 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:09:44 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:09:44.627+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:09:44.815 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:09:44 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:09:44.627+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:09:44.815 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:09:44 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:09:44.627+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:09:44.816 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:09:44 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:09:44.627+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:09:45.565 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:09:45 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:09:45.257+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:09:45.565 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:09:45 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:09:45.257+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:09:45.565 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:09:45 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:09:45.257+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:09:45.566 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:09:45 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:09:45.257+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:09:46.064 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:09:45 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:09:45.610+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:09:46.064 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:09:45 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:09:45.610+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:09:46.065 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:09:45 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:09:45.610+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:09:46.065 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:09:45 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:09:45.611+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:09:46.065 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:09:45 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:09:45.616+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:09:46.065 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:09:45 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:09:45.616+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:09:46.065 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:09:45 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:09:45.616+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:09:46.066 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:09:45 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:09:45.616+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:09:46.564 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:09:46 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:09:46.277+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:09:46.565 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:09:46 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:09:46.277+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:09:46.565 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:09:46 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:09:46.277+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:09:46.565 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:09:46 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:09:46.279+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:09:46.918 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:09:46 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:09:46.664+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:09:46.919 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:09:46 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:09:46.664+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:09:46.919 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:09:46 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:09:46.664+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:09:46.919 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:09:46 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:09:46.665+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:09:46.919 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:09:46 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:09:46.612+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:09:46.919 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:09:46 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:09:46.612+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:09:46.920 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:09:46 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:09:46.612+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:09:46.920 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:09:46 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:09:46.613+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:09:47.251 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Dec 16 22:09:46 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-mon-smithi045[80759]: 2023-12-16T22:09:46.916+0000 7f45a1f8f700 -1 mon.smithi045@0(probing) e2 get_health_metrics reporting 30010 slow ops, oldest is monmgrreport(gid 14214, 0 checks, 1 progress events) 2023-12-16T22:09:47.564 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:09:47 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:09:47.250+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:09:47.564 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:09:47 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:09:47.250+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:09:47.565 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:09:47 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:09:47.250+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:09:47.565 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:09:47 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:09:47.251+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:09:48.064 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:09:47 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:09:47.619+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:09:48.064 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:09:47 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:09:47.619+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:09:48.065 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:09:47 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:09:47.619+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:09:48.065 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:09:47 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:09:47.620+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:09:48.065 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:09:47 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:09:47.574+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:09:48.065 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:09:47 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:09:47.574+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:09:48.065 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:09:47 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:09:47.574+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:09:48.065 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:09:47 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:09:47.575+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:09:48.564 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:09:48 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:09:48.285+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:09:48.564 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:09:48 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:09:48.285+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:09:48.565 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:09:48 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:09:48.285+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:09:48.565 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:09:48 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:09:48.286+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:09:49.064 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:09:48 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:09:48.591+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:09:49.065 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:09:48 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:09:48.591+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:09:49.065 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:09:48 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:09:48.591+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:09:49.065 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:09:48 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:09:48.591+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:09:49.066 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:09:48 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:09:48.601+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:09:49.066 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:09:48 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:09:48.601+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:09:49.066 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:09:48 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:09:48.601+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:09:49.066 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:09:48 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:09:48.602+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:09:49.559 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:09:49 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:09:49.295+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:09:49.559 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:09:49 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:09:49.295+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:09:49.559 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:09:49 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:09:49.295+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:09:49.560 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:09:49 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:09:49.296+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:09:49.814 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:09:49 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:09:49.627+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:09:49.814 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:09:49 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:09:49.627+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:09:49.815 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:09:49 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:09:49.627+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:09:49.815 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:09:49 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:09:49.628+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:09:49.815 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:09:49 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:09:49.558+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:09:49.815 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:09:49 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:09:49.558+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:09:49.815 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:09:49 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:09:49.558+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:09:49.815 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:09:49 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:09:49.559+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:09:50.557 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:09:50 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:09:50.252+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:09:50.557 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:09:50 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:09:50.253+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:09:50.557 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:09:50 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:09:50.253+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:09:50.557 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:09:50 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:09:50.253+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:09:50.814 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:09:50 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:09:50.670+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:09:50.814 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:09:50 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:09:50.670+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:09:50.815 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:09:50 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:09:50.670+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:09:50.815 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:09:50 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:09:50.671+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:09:50.815 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:09:50 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:09:50.556+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:09:50.815 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:09:50 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:09:50.556+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:09:50.815 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:09:50 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:09:50.556+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:09:50.816 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:09:50 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:09:50.557+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:09:51.531 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:09:51 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:09:51.267+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:09:51.531 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:09:51 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:09:51.267+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:09:51.531 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:09:51 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:09:51.267+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:09:51.531 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:09:51 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:09:51.268+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:09:51.814 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:09:51 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:09:51.658+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:09:51.815 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:09:51 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:09:51.658+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:09:51.815 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:09:51 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:09:51.658+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:09:51.815 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:09:51 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:09:51.659+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:09:51.815 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:09:51 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:09:51.530+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:09:51.815 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:09:51 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:09:51.530+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:09:51.815 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:09:51 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:09:51.530+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:09:51.815 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:09:51 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:09:51.531+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:09:52.286 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Dec 16 22:09:51 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-mon-smithi045[80759]: 2023-12-16T22:09:51.919+0000 7f45a1f8f700 -1 mon.smithi045@0(probing) e2 get_health_metrics reporting 30103 slow ops, oldest is monmgrreport(gid 14214, 0 checks, 1 progress events) 2023-12-16T22:09:52.564 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:09:52 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:09:52.285+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:09:52.565 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:09:52 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:09:52.285+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:09:52.565 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:09:52 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:09:52.285+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:09:52.565 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:09:52 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:09:52.286+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:09:52.565 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:09:52 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:09:52.527+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:09:52.565 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:09:52 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:09:52.527+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:09:52.565 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:09:52 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:09:52.527+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:09:52.566 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:09:52 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:09:52.528+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:09:53.064 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:09:52 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:09:52.653+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:09:53.065 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:09:52 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:09:52.653+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:09:53.065 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:09:52 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:09:52.653+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:09:53.065 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:09:52 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:09:52.654+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:09:53.564 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:09:53 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:09:53.297+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:09:53.564 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:09:53 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:09:53.297+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:09:53.565 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:09:53 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:09:53.297+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:09:53.565 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:09:53 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:09:53.298+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:09:53.565 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:09:53 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:09:53.486+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:09:53.565 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:09:53 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:09:53.486+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:09:53.565 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:09:53 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:09:53.486+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:09:53.565 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:09:53 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:09:53.487+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:09:54.064 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:09:53 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:09:53.701+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:09:54.065 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:09:53 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:09:53.701+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:09:54.065 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:09:53 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:09:53.701+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:09:54.065 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:09:53 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:09:53.702+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:09:54.722 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:09:54 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:09:54.341+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:09:54.723 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:09:54 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:09:54.341+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:09:54.723 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:09:54 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:09:54.341+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:09:54.723 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:09:54 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:09:54.342+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:09:54.723 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:09:54 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:09:54.524+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:09:54.723 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:09:54 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:09:54.524+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:09:54.723 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:09:54 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:09:54.524+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:09:54.724 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:09:54 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:09:54.525+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:09:55.064 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:09:54 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:09:54.721+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:09:55.064 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:09:54 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:09:54.721+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:09:55.065 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:09:54 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:09:54.721+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:09:55.065 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:09:54 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:09:54.722+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:09:55.714 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:09:55 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:09:55.341+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:09:55.715 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:09:55 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:09:55.341+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:09:55.715 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:09:55 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:09:55.341+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:09:55.715 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:09:55 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:09:55.342+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:09:55.715 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:09:55 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:09:55.481+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:09:55.715 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:09:55 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:09:55.481+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:09:55.715 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:09:55 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:09:55.481+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:09:55.716 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:09:55 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:09:55.482+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:09:56.064 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:09:55 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:09:55.713+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:09:56.064 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:09:55 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:09:55.713+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:09:56.065 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:09:55 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:09:55.713+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:09:56.065 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:09:55 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:09:55.714+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:09:56.721 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:09:56 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:09:56.347+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:09:56.721 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:09:56 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:09:56.347+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:09:56.721 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:09:56 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:09:56.347+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:09:56.722 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:09:56 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:09:56.348+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:09:56.722 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:09:56 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:09:56.500+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:09:56.722 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:09:56 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:09:56.500+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:09:56.722 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:09:56 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:09:56.500+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:09:56.722 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:09:56 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:09:56.502+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:09:57.048 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Dec 16 22:09:56 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-mon-smithi045[80759]: 2023-12-16T22:09:56.923+0000 7f45a1f8f700 -1 mon.smithi045@0(probing) e2 get_health_metrics reporting 30192 slow ops, oldest is monmgrreport(gid 14214, 0 checks, 1 progress events) 2023-12-16T22:09:57.049 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:09:56 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:09:56.720+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:09:57.049 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:09:56 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:09:56.720+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:09:57.049 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:09:56 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:09:56.720+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:09:57.049 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:09:56 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:09:56.720+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:09:57.723 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:09:57 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:09:57.365+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:09:57.723 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:09:57 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:09:57.365+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:09:57.723 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:09:57 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:09:57.365+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:09:57.723 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:09:57 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:09:57.366+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:09:57.723 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:09:57 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:09:57.461+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:09:57.724 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:09:57 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:09:57.461+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:09:57.724 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:09:57 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:09:57.462+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:09:57.724 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:09:57 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:09:57.462+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:09:58.064 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:09:57 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:09:57.722+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:09:58.064 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:09:57 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:09:57.722+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:09:58.065 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:09:57 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:09:57.722+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:09:58.065 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:09:57 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:09:57.722+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:09:58.721 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:09:58 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:09:58.445+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:09:58.721 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:09:58 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:09:58.445+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:09:58.721 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:09:58 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:09:58.445+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:09:58.721 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:09:58 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:09:58.446+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:09:58.722 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:09:58 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:09:58.361+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:09:58.722 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:09:58 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:09:58.361+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:09:58.722 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:09:58 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:09:58.361+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:09:58.722 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:09:58 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:09:58.361+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:09:59.064 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:09:58 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:09:58.725+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:09:59.064 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:09:58 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:09:58.725+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:09:59.065 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:09:58 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:09:58.725+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:09:59.065 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:09:58 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:09:58.726+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:09:59.775 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:09:59 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:09:59.337+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:09:59.775 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:09:59 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:09:59.337+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:09:59.775 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:09:59 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:09:59.337+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:09:59.776 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:09:59 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:09:59.338+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:09:59.776 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:09:59 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:09:59.427+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:09:59.776 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:09:59 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:09:59.427+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:09:59.776 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:09:59 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:09:59.427+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:09:59.776 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:09:59 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:09:59.428+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:10:00.064 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:09:59 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:09:59.774+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:10:00.065 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:09:59 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:09:59.774+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:10:00.065 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:09:59 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:09:59.774+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:10:00.065 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:09:59 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:09:59.775+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:10:00.468 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:10:00 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:10:00.335+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:10:00.468 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:10:00 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:10:00.335+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:10:00.468 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:10:00 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:10:00.335+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:10:00.469 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:10:00 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:10:00.336+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:10:00.741 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:10:00 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:10:00.467+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:10:00.742 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:10:00 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:10:00.467+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:10:00.742 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:10:00 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:10:00.467+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:10:00.742 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:10:00 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:10:00.467+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:10:01.060 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:10:00 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:10:00.740+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:10:01.061 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:10:00 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:10:00.740+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:10:01.061 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:10:00 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:10:00.740+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:10:01.062 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:10:00 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:10:00.741+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:10:01.314 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:10:01 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:10:01.293+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:10:01.315 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:10:01 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:10:01.293+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:10:01.315 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:10:01 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:10:01.293+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:10:01.315 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:10:01 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:10:01.294+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:10:01.764 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:10:01 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:10:01.499+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:10:01.765 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:10:01 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:10:01.499+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:10:01.765 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:10:01 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:10:01.499+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:10:01.765 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:10:01 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:10:01.500+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:10:02.058 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Dec 16 22:10:01 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-mon-smithi045[80759]: 2023-12-16T22:10:01.927+0000 7f45a1f8f700 -1 mon.smithi045@0(probing) e2 get_health_metrics reporting 30283 slow ops, oldest is monmgrreport(gid 14214, 0 checks, 1 progress events) 2023-12-16T22:10:02.058 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:10:01 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:10:01.763+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:10:02.058 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:10:01 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:10:01.763+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:10:02.059 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:10:01 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:10:01.764+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:10:02.059 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:10:01 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:10:01.764+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:10:02.314 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:10:02 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:10:02.250+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:10:02.314 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:10:02 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:10:02.250+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:10:02.314 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:10:02 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:10:02.250+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:10:02.315 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:10:02 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:10:02.251+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:10:02.814 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:10:02 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:10:02.737+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:10:02.815 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:10:02 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:10:02.737+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:10:02.815 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:10:02 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:10:02.737+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:10:02.815 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:10:02 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:10:02.738+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:10:02.815 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:10:02 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:10:02.528+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:10:02.815 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:10:02 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:10:02.528+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:10:02.816 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:10:02 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:10:02.528+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:10:02.816 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:10:02 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:10:02.529+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:10:03.552 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:10:03 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:10:03.232+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:10:03.552 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:10:03 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:10:03.232+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:10:03.553 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:10:03 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:10:03.232+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:10:03.553 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:10:03 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:10:03.233+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:10:03.814 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:10:03 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:10:03.731+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:10:03.815 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:10:03 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:10:03.731+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:10:03.815 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:10:03 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:10:03.731+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:10:03.815 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:10:03 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:10:03.732+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:10:03.815 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:10:03 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:10:03.551+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:10:03.816 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:10:03 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:10:03.551+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:10:03.816 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:10:03 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:10:03.551+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:10:03.816 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:10:03 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:10:03.552+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:10:04.530 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:10:04 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:10:04.248+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:10:04.530 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:10:04 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:10:04.248+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:10:04.531 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:10:04 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:10:04.248+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:10:04.531 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:10:04 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:10:04.249+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:10:04.814 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:10:04 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:10:04.690+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:10:04.814 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:10:04 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:10:04.690+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:10:04.815 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:10:04 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:10:04.690+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:10:04.815 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:10:04 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:10:04.690+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:10:04.815 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:10:04 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:10:04.529+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:10:04.815 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:10:04 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:10:04.529+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:10:04.815 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:10:04 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:10:04.529+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:10:04.815 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:10:04 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:10:04.530+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:10:05.526 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:10:05 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:10:05.252+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:10:05.526 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:10:05 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:10:05.252+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:10:05.526 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:10:05 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:10:05.252+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:10:05.526 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:10:05 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:10:05.253+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:10:05.814 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:10:05 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:10:05.733+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:10:05.815 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:10:05 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:10:05.733+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:10:05.815 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:10:05 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:10:05.733+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:10:05.815 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:10:05 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:10:05.733+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:10:05.816 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:10:05 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:10:05.525+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:10:05.816 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:10:05 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:10:05.525+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:10:05.816 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:10:05 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:10:05.525+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:10:05.817 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:10:05 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:10:05.526+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:10:06.564 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:10:06 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:10:06.483+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:10:06.565 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:10:06 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:10:06.483+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:10:06.565 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:10:06 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:10:06.483+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:10:06.565 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:10:06 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:10:06.484+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:10:06.565 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:10:06 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:10:06.289+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:10:06.565 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:10:06 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:10:06.289+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:10:06.565 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:10:06 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:10:06.289+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:10:06.566 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:10:06 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:10:06.290+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:10:06.944 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Dec 16 22:10:06 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-mon-smithi045[80759]: 2023-12-16T22:10:06.930+0000 7f45a1f8f700 -1 mon.smithi045@0(probing) e2 get_health_metrics reporting 30372 slow ops, oldest is monmgrreport(gid 14214, 0 checks, 1 progress events) 2023-12-16T22:10:06.945 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:10:06 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:10:06.692+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:10:06.945 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:10:06 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:10:06.692+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:10:06.945 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:10:06 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:10:06.692+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:10:06.945 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:10:06 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:10:06.693+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:10:07.735 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:10:07 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:10:07.510+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:10:07.735 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:10:07 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:10:07.510+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:10:07.735 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:10:07 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:10:07.510+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:10:07.735 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:10:07 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:10:07.511+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:10:07.736 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:10:07 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:10:07.340+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:10:07.736 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:10:07 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:10:07.340+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:10:07.736 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:10:07 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:10:07.340+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:10:07.736 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:10:07 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:10:07.341+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:10:08.064 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:10:07 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:10:07.733+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:10:08.064 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:10:07 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:10:07.733+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:10:08.065 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:10:07 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:10:07.733+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:10:08.065 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:10:07 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:10:07.735+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:10:08.721 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:10:08 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:10:08.332+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:10:08.721 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:10:08 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:10:08.332+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:10:08.721 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:10:08 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:10:08.332+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:10:08.721 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:10:08 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:10:08.332+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:10:08.722 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:10:08 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:10:08.539+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:10:08.722 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:10:08 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:10:08.539+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:10:08.722 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:10:08 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:10:08.539+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:10:08.722 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:10:08 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:10:08.540+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:10:09.064 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:10:08 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:10:08.773+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:10:09.065 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:10:08 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:10:08.773+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:10:09.065 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:10:08 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:10:08.773+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:10:09.065 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:10:08 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:10:08.774+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:10:09.558 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:10:09 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:10:09.283+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:10:09.558 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:10:09 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:10:09.283+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:10:09.559 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:10:09 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:10:09.283+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:10:09.559 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:10:09 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:10:09.284+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:10:09.814 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:10:09 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:10:09.557+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:10:09.815 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:10:09 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:10:09.557+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:10:09.815 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:10:09 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:10:09.557+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:10:09.815 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:10:09 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:10:09.558+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:10:09.815 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:10:09 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:10:09.735+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:10:09.815 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:10:09 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:10:09.736+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:10:09.815 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:10:09 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:10:09.736+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:10:09.815 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:10:09 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:10:09.736+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:10:10.541 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:10:10 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:10:10.263+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:10:10.541 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:10:10 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:10:10.263+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:10:10.541 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:10:10 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:10:10.263+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:10:10.542 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:10:10 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:10:10.264+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:10:10.814 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:10:10 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:10:10.540+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:10:10.814 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:10:10 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:10:10.540+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:10:10.815 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:10:10 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:10:10.540+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:10:10.815 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:10:10 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:10:10.541+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:10:10.815 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:10:10 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:10:10.729+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:10:10.815 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:10:10 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:10:10.729+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:10:10.815 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:10:10 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:10:10.729+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:10:10.815 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:10:10 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:10:10.730+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:10:11.564 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:10:11 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:10:11.227+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:10:11.564 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:10:11 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:10:11.227+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:10:11.565 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:10:11 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:10:11.227+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:10:11.565 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:10:11 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:10:11.228+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:10:11.935 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:10:11 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:10:11.701+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:10:11.935 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:10:11 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:10:11.701+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:10:11.935 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:10:11 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:10:11.701+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:10:11.935 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:10:11 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:10:11.702+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:10:11.936 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:10:11 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:10:11.582+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:10:11.936 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:10:11 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:10:11.582+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:10:11.936 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:10:11 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:10:11.582+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:10:11.936 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:10:11 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:10:11.583+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:10:12.259 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Dec 16 22:10:11 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-mon-smithi045[80759]: 2023-12-16T22:10:11.934+0000 7f45a1f8f700 -1 mon.smithi045@0(probing) e2 get_health_metrics reporting 30465 slow ops, oldest is monmgrreport(gid 14214, 0 checks, 1 progress events) 2023-12-16T22:10:12.564 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:10:12 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:10:12.258+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:10:12.564 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:10:12 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:10:12.258+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:10:12.564 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:10:12 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:10:12.258+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:10:12.565 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:10:12 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:10:12.259+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:10:13.064 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:10:12 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:10:12.664+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:10:13.065 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:10:12 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:10:12.664+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:10:13.065 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:10:12 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:10:12.664+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:10:13.065 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:10:12 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:10:12.664+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:10:13.065 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:10:12 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:10:12.581+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:10:13.065 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:10:12 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:10:12.581+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:10:13.065 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:10:12 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:10:12.581+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:10:13.066 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:10:12 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:10:12.581+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:10:13.564 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:10:13 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:10:13.250+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:10:13.564 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:10:13 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:10:13.250+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:10:13.565 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:10:13 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:10:13.250+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:10:13.565 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:10:13 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:10:13.251+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:10:13.903 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:10:13 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:10:13.617+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:10:13.903 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:10:13 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:10:13.617+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:10:13.903 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:10:13 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:10:13.617+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:10:13.903 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:10:13 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:10:13.618+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:10:13.904 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:10:13 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:10:13.585+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:10:13.904 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:10:13 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:10:13.585+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:10:13.904 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:10:13 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:10:13.585+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:10:13.904 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:10:13 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:10:13.585+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:10:14.565 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:10:14 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:10:14.225+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:10:14.565 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:10:14 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:10:14.225+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:10:14.565 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:10:14 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:10:14.225+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:10:14.565 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:10:14 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:10:14.226+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:10:15.065 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:10:14 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:10:14.597+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:10:15.065 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:10:14 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:10:14.597+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:10:15.065 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:10:14 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:10:14.597+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:10:15.065 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:10:14 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:10:14.597+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:10:15.066 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:10:14 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:10:14.612+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:10:15.066 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:10:14 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:10:14.612+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:10:15.066 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:10:14 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:10:14.612+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:10:15.066 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:10:14 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:10:14.613+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:10:15.564 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:10:15 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:10:15.205+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:10:15.565 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:10:15 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:10:15.205+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:10:15.565 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:10:15 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:10:15.205+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:10:15.565 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:10:15 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:10:15.207+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:10:16.065 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:10:15 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:10:15.568+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:10:16.065 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:10:15 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:10:15.568+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:10:16.065 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:10:15 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:10:15.568+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:10:16.065 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:10:15 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:10:15.569+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:10:16.066 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:10:15 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:10:15.624+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:10:16.066 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:10:15 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:10:15.624+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:10:16.066 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:10:15 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:10:15.624+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:10:16.066 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:10:15 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:10:15.625+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:10:16.564 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:10:16 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:10:16.231+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:10:16.564 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:10:16 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:10:16.231+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:10:16.565 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:10:16 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:10:16.231+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:10:16.565 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:10:16 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:10:16.231+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:10:16.939 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:10:16 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:10:16.602+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:10:16.939 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:10:16 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:10:16.602+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:10:16.939 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:10:16 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:10:16.602+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:10:16.939 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:10:16 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:10:16.603+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:10:16.940 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:10:16 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:10:16.605+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:10:16.940 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:10:16 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:10:16.605+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:10:16.940 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:10:16 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:10:16.605+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:10:16.940 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:10:16 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:10:16.606+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:10:17.225 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Dec 16 22:10:16 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-mon-smithi045[80759]: 2023-12-16T22:10:16.937+0000 7f45a1f8f700 -1 mon.smithi045@0(probing) e2 get_health_metrics reporting 30554 slow ops, oldest is monmgrreport(gid 14214, 0 checks, 1 progress events) 2023-12-16T22:10:17.562 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:10:17 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:10:17.224+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:10:17.562 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:10:17 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:10:17.225+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:10:17.562 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:10:17 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:10:17.225+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:10:17.563 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:10:17 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:10:17.225+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:10:17.814 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:10:17 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:10:17.561+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:10:17.815 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:10:17 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:10:17.561+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:10:17.815 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:10:17 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:10:17.561+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:10:17.815 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:10:17 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:10:17.562+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:10:17.815 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:10:17 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:10:17.594+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:10:17.815 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:10:17 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:10:17.594+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:10:17.815 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:10:17 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:10:17.594+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:10:17.815 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:10:17 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:10:17.595+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:10:18.564 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:10:18 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:10:18.190+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:10:18.565 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:10:18 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:10:18.190+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:10:18.565 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:10:18 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:10:18.190+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:10:18.565 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:10:18 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:10:18.191+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:10:19.065 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:10:18 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:10:18.582+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:10:19.065 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:10:18 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:10:18.582+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:10:19.065 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:10:18 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:10:18.582+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:10:19.065 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:10:18 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:10:18.583+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:10:19.065 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:10:18 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:10:18.642+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:10:19.066 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:10:18 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:10:18.642+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:10:19.066 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:10:18 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:10:18.642+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:10:19.066 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:10:18 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:10:18.643+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:10:19.564 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:10:19 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:10:19.159+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:10:19.565 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:10:19 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:10:19.159+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:10:19.565 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:10:19 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:10:19.159+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:10:19.565 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:10:19 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:10:19.160+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:10:20.065 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:10:19 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:10:19.636+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:10:20.065 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:10:19 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:10:19.636+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:10:20.065 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:10:19 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:10:19.636+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:10:20.066 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:10:19 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:10:19.638+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:10:20.066 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:10:19 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:10:19.577+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:10:20.066 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:10:19 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:10:19.577+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:10:20.067 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:10:19 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:10:19.577+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:10:20.067 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:10:19 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:10:19.578+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:10:20.564 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:10:20 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:10:20.197+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:10:20.565 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:10:20 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:10:20.197+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:10:20.565 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:10:20 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:10:20.197+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:10:20.565 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:10:20 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:10:20.198+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:10:21.064 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:10:20 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:10:20.616+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:10:21.065 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:10:20 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:10:20.616+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:10:21.065 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:10:20 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:10:20.616+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:10:21.065 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:10:20 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:10:20.617+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:10:21.065 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:10:20 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:10:20.671+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:10:21.065 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:10:20 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:10:20.671+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:10:21.066 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:10:20 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:10:20.671+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:10:21.066 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:10:20 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:10:20.671+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:10:21.564 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:10:21 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:10:21.195+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:10:21.564 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:10:21 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:10:21.195+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:10:21.565 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:10:21 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:10:21.195+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:10:21.565 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:10:21 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:10:21.197+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:10:21.942 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:10:21 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:10:21.671+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:10:21.942 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:10:21 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:10:21.671+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:10:21.942 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:10:21 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:10:21.671+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:10:21.943 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:10:21 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:10:21.672+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:10:21.943 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:10:21 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:10:21.651+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:10:21.943 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:10:21 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:10:21.651+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:10:21.943 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:10:21 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:10:21.651+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:10:21.943 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:10:21 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:10:21.652+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:10:22.235 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Dec 16 22:10:21 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-mon-smithi045[80759]: 2023-12-16T22:10:21.940+0000 7f45a1f8f700 -1 mon.smithi045@0(probing) e2 get_health_metrics reporting 30646 slow ops, oldest is monmgrreport(gid 14214, 0 checks, 1 progress events) 2023-12-16T22:10:22.564 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:10:22 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:10:22.233+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:10:22.565 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:10:22 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:10:22.233+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:10:22.565 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:10:22 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:10:22.233+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:10:22.565 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:10:22 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:10:22.234+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:10:23.064 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:10:22 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:10:22.694+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:10:23.065 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:10:22 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:10:22.694+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:10:23.065 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:10:22 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:10:22.694+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:10:23.065 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:10:22 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:10:22.695+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:10:23.065 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:10:22 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:10:22.643+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:10:23.065 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:10:22 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:10:22.643+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:10:23.066 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:10:22 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:10:22.643+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:10:23.066 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:10:22 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:10:22.644+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:10:23.564 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:10:23 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:10:23.192+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:10:23.565 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:10:23 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:10:23.192+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:10:23.565 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:10:23 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:10:23.192+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:10:23.565 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:10:23 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:10:23.192+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:10:24.064 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:10:23 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:10:23.686+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:10:24.065 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:10:23 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:10:23.686+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:10:24.065 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:10:23 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:10:23.686+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:10:24.065 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:10:23 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:10:23.687+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:10:24.065 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:10:23 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:10:23.602+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:10:24.065 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:10:23 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:10:23.602+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:10:24.065 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:10:23 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:10:23.602+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:10:24.066 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:10:23 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:10:23.603+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:10:24.564 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:10:24 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:10:24.226+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:10:24.565 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:10:24 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:10:24.226+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:10:24.565 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:10:24 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:10:24.226+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:10:24.565 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:10:24 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:10:24.227+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:10:25.064 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:10:24 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:10:24.604+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:10:25.065 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:10:24 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:10:24.604+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:10:25.065 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:10:24 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:10:24.604+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:10:25.065 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:10:24 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:10:24.604+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:10:25.065 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:10:24 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:10:24.728+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:10:25.065 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:10:24 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:10:24.728+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:10:25.065 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:10:24 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:10:24.728+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:10:25.065 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:10:24 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:10:24.728+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:10:25.564 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:10:25 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:10:25.196+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:10:25.565 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:10:25 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:10:25.196+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:10:25.565 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:10:25 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:10:25.196+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:10:25.565 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:10:25 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:10:25.196+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:10:26.064 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:10:25 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:10:25.711+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:10:26.065 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:10:25 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:10:25.711+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:10:26.065 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:10:25 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:10:25.711+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:10:26.065 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:10:25 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:10:25.712+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:10:26.065 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:10:25 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:10:25.639+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:10:26.065 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:10:25 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:10:25.639+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:10:26.066 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:10:25 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:10:25.639+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:10:26.066 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:10:25 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:10:25.641+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:10:26.564 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:10:26 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:10:26.224+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:10:26.564 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:10:26 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:10:26.224+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:10:26.565 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:10:26 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:10:26.224+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:10:26.565 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:10:26 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:10:26.224+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:10:26.946 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:10:26 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:10:26.625+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:10:26.946 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:10:26 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:10:26.625+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:10:26.946 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:10:26 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:10:26.625+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:10:26.946 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:10:26 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:10:26.626+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:10:26.947 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:10:26 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:10:26.754+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:10:26.947 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:10:26 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:10:26.754+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:10:26.947 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:10:26 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:10:26.754+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:10:26.947 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:10:26 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:10:26.755+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:10:27.314 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Dec 16 22:10:26 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-mon-smithi045[80759]: 2023-12-16T22:10:26.945+0000 7f45a1f8f700 -1 mon.smithi045@0(probing) e2 get_health_metrics reporting 30735 slow ops, oldest is monmgrreport(gid 14214, 0 checks, 1 progress events) 2023-12-16T22:10:27.315 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:10:27 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:10:27.177+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:10:27.315 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:10:27 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:10:27.177+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:10:27.315 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:10:27 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:10:27.177+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:10:27.315 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:10:27 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:10:27.177+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:10:28.064 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:10:27 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:10:27.578+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:10:28.065 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:10:27 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:10:27.578+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:10:28.065 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:10:27 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:10:27.578+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:10:28.065 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:10:27 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:10:27.579+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:10:28.069 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:10:27 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:10:27.775+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:10:28.069 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:10:27 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:10:27.775+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:10:28.069 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:10:27 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:10:27.775+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:10:28.069 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:10:27 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:10:27.775+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:10:28.564 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:10:28 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:10:28.190+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:10:28.564 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:10:28 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:10:28.190+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:10:28.565 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:10:28 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:10:28.190+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:10:28.565 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:10:28 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:10:28.191+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:10:29.064 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:10:28 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:10:28.769+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:10:29.065 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:10:28 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:10:28.769+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:10:29.065 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:10:28 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:10:28.769+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:10:29.065 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:10:28 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:10:28.771+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:10:29.065 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:10:28 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:10:28.565+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:10:29.065 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:10:28 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:10:28.565+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:10:29.065 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:10:28 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:10:28.565+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:10:29.066 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:10:28 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:10:28.566+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:10:29.560 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:10:29 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:10:29.167+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:10:29.560 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:10:29 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:10:29.167+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:10:29.561 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:10:29 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:10:29.167+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:10:29.561 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:10:29 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:10:29.168+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:10:29.814 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:10:29 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:10:29.559+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:10:29.815 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:10:29 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:10:29.559+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:10:29.815 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:10:29 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:10:29.559+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:10:29.815 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:10:29 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:10:29.560+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:10:29.815 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:10:29 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:10:29.788+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:10:29.815 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:10:29 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:10:29.788+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:10:29.816 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:10:29 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:10:29.788+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:10:29.816 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:10:29 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:10:29.789+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:10:30.564 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:10:30 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:10:30.184+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:10:30.565 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:10:30 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:10:30.184+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:10:30.565 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:10:30 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:10:30.184+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:10:30.565 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:10:30 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:10:30.185+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:10:31.064 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:10:30 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:10:30.578+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:10:31.065 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:10:30 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:10:30.578+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:10:31.065 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:10:30 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:10:30.578+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:10:31.065 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:10:30 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:10:30.579+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:10:31.065 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:10:30 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:10:30.767+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:10:31.066 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:10:30 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:10:30.767+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:10:31.066 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:10:30 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:10:30.767+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:10:31.066 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:10:30 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:10:30.768+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:10:31.564 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:10:31 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:10:31.144+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:10:31.565 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:10:31 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:10:31.144+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:10:31.565 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:10:31 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:10:31.144+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:10:31.565 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:10:31 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:10:31.145+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:10:31.953 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:10:31 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:10:31.764+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:10:31.953 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:10:31 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:10:31.764+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:10:31.953 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:10:31 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:10:31.764+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:10:31.953 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:10:31 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:10:31.765+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:10:31.954 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:10:31 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:10:31.569+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:10:31.954 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:10:31 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:10:31.569+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:10:31.954 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:10:31 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:10:31.569+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:10:31.954 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:10:31 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:10:31.569+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:10:32.314 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Dec 16 22:10:31 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-mon-smithi045[80759]: 2023-12-16T22:10:31.948+0000 7f45a1f8f700 -1 mon.smithi045@0(probing) e2 get_health_metrics reporting 30828 slow ops, oldest is monmgrreport(gid 14214, 0 checks, 1 progress events) 2023-12-16T22:10:32.314 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:10:32 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:10:32.146+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:10:32.315 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:10:32 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:10:32.146+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:10:32.315 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:10:32 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:10:32.146+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:10:32.315 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:10:32 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:10:32.147+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:10:33.064 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:10:32 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:10:32.610+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:10:33.065 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:10:32 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:10:32.610+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:10:33.065 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:10:32 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:10:32.610+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:10:33.065 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:10:32 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:10:32.611+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:10:33.065 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:10:32 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:10:32.795+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:10:33.065 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:10:32 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:10:32.795+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:10:33.066 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:10:32 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:10:32.795+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:10:33.066 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:10:32 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:10:32.797+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:10:33.564 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:10:33 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:10:33.099+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:10:33.564 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:10:33 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:10:33.099+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:10:33.565 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:10:33 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:10:33.099+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:10:33.565 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:10:33 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:10:33.099+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:10:34.064 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:10:33 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:10:33.621+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:10:34.065 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:10:33 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:10:33.621+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:10:34.065 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:10:33 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:10:33.621+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:10:34.065 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:10:33 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:10:33.622+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:10:34.065 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:10:33 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:10:33.828+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:10:34.066 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:10:33 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:10:33.828+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:10:34.066 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:10:33 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:10:33.828+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:10:34.066 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:10:33 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:10:33.829+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:10:34.564 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:10:34 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:10:34.122+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:10:34.565 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:10:34 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:10:34.122+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:10:34.565 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:10:34 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:10:34.122+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:10:34.565 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:10:34 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:10:34.123+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:10:35.064 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:10:34 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:10:34.587+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:10:35.065 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:10:34 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:10:34.587+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:10:35.065 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:10:34 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:10:34.587+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:10:35.065 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:10:34 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:10:34.587+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:10:35.065 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:10:34 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:10:34.832+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:10:35.066 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:10:34 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:10:34.832+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:10:35.066 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:10:34 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:10:34.832+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:10:35.066 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:10:34 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:10:34.833+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:10:35.407 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:10:35 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:10:35.086+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:10:35.407 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:10:35 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:10:35.086+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:10:35.408 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:10:35 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:10:35.086+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:10:35.408 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:10:35 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:10:35.087+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:10:35.815 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:10:35 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:10:35.610+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:10:35.815 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:10:35 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:10:35.610+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:10:35.815 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:10:35 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:10:35.610+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:10:35.815 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:10:35 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:10:35.611+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:10:36.314 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:10:35 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:10:35.835+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:10:36.315 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:10:35 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:10:35.835+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:10:36.315 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:10:35 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:10:35.835+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:10:36.315 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:10:35 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:10:35.836+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:10:36.315 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:10:36 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:10:36.042+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:10:36.315 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:10:36 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:10:36.042+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:10:36.316 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:10:36 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:10:36.042+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:10:36.316 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:10:36 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:10:36.043+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:10:36.953 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:10:36 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:10:36.853+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:10:36.954 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:10:36 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:10:36.853+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:10:36.954 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:10:36 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:10:36.853+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:10:36.954 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:10:36 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:10:36.854+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:10:36.954 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:10:36 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:10:36.611+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:10:36.954 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:10:36 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:10:36.611+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:10:36.954 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:10:36 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:10:36.611+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:10:36.955 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:10:36 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:10:36.612+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:10:37.314 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Dec 16 22:10:36 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-mon-smithi045[80759]: 2023-12-16T22:10:36.952+0000 7f45a1f8f700 -1 mon.smithi045@0(probing) e2 get_health_metrics reporting 30916 slow ops, oldest is monmgrreport(gid 14214, 0 checks, 1 progress events) 2023-12-16T22:10:37.315 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:10:37 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:10:37.032+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:10:37.315 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:10:37 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:10:37.032+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:10:37.315 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:10:37 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:10:37.032+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:10:37.315 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:10:37 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:10:37.033+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:10:38.065 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:10:37 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:10:37.809+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:10:38.065 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:10:37 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:10:37.809+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:10:38.065 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:10:37 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:10:37.809+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:10:38.065 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:10:37 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:10:37.809+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:10:38.065 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:10:37 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:10:37.615+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:10:38.066 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:10:37 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:10:37.615+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:10:38.066 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:10:37 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:10:37.615+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:10:38.066 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:10:37 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:10:37.616+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:10:38.565 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:10:38 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:10:38.076+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:10:38.565 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:10:38 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:10:38.076+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:10:38.565 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:10:38 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:10:38.076+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:10:38.565 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:10:38 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:10:38.077+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:10:39.049 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:10:38 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:10:38.603+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:10:39.050 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:10:38 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:10:38.603+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:10:39.050 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:10:38 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:10:38.603+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:10:39.050 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:10:38 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:10:38.605+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:10:39.050 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:10:38 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:10:38.791+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:10:39.050 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:10:38 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:10:38.791+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:10:39.050 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:10:38 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:10:38.791+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:10:39.051 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:10:38 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:10:38.793+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:10:39.314 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:10:39 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:10:39.048+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:10:39.314 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:10:39 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:10:39.048+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:10:39.315 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:10:39 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:10:39.048+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:10:39.315 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:10:39 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:10:39.049+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:10:39.814 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:10:39 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:10:39.776+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:10:39.815 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:10:39 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:10:39.776+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:10:39.815 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:10:39 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:10:39.776+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:10:39.815 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:10:39 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:10:39.777+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:10:39.815 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:10:39 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:10:39.562+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:10:39.816 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:10:39 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:10:39.562+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:10:39.816 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:10:39 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:10:39.562+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:10:39.816 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:10:39 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:10:39.563+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:10:40.314 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:10:40 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:10:40.049+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:10:40.315 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:10:40 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:10:40.049+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:10:40.315 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:10:40 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:10:40.049+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:10:40.315 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:10:40 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:10:40.049+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:10:40.815 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:10:40 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:10:40.533+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:10:40.815 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:10:40 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:10:40.533+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:10:40.815 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:10:40 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:10:40.533+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:10:40.815 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:10:40 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:10:40.534+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:10:40.815 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:10:40 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:10:40.764+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:10:40.816 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:10:40 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:10:40.764+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:10:40.816 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:10:40 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:10:40.764+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:10:40.816 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:10:40 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:10:40.765+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:10:41.314 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:10:41 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:10:41.007+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:10:41.314 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:10:41 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:10:41.007+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:10:41.315 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:10:41 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:10:41.007+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:16.461573+0000 (oldest deadline 2023-12-16T22:00:36.461573+0000) 2023-12-16T22:10:41.315 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:10:41 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:10:41.009+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:10:41.814 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:10:41 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:10:41.522+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:10:41.814 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:10:41 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:10:41.522+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:10:41.815 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:10:41 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:10:41.522+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:10:41.815 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:10:41 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:10:41.523+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:10:41.815 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:10:41 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:10:41.763+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:10:41.815 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:10:41 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:10:41.763+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:10:41.815 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:10:41 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:10:41.763+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:10.264793+0000 (oldest deadline 2023-12-16T22:00:30.264793+0000) 2023-12-16T22:10:41.815 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:10:41 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:10:41.764+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:10:42.314 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Dec 16 22:10:41 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-mon-smithi045[80759]: 2023-12-16T22:10:41.956+0000 7f45a1f8f700 -1 mon.smithi045@0(probing) e2 get_health_metrics reporting 31008 slow ops, oldest is monmgrreport(gid 14214, 0 checks, 1 progress events) 2023-12-16T22:10:42.315 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:10:42 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:10:42.007+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:10:42.814 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:10:42 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:10:42.748+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:10:42.815 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:10:42 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:10:42.525+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:10:42.815 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:10:42 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:10:42.525+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:10:42.815 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:10:42 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:10:42.525+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:10:42.815 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:10:42 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:10:42.526+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:10:43.314 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:10:43 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:10:43.046+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:10:43.814 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:10:43 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:10:43.731+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:10:43.815 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:10:43 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:10:43.559+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:10:43.815 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:10:43 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:10:43.559+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:10:43.815 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:10:43 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:10:43.559+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:10:43.815 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:10:43 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:10:43.560+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:10:44.314 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:10:44 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:10:44.021+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:10:45.065 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:10:44 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:10:44.721+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:10:45.065 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:10:44 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:10:44.598+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:10:45.065 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:10:44 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:10:44.598+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:10:45.066 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:10:44 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:10:44.598+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:10:45.066 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:10:44 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:10:44.599+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:10:45.564 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:10:45 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:10:45.071+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:10:46.064 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:10:45 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:10:45.753+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:10:46.065 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:10:45 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:10:45.621+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:10:46.065 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:10:45 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:10:45.621+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:10:46.065 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:10:45 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:10:45.621+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:10:46.065 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:10:45 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:10:45.621+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:10:46.564 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:10:46 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:10:46.094+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:10:46.962 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:10:46 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:10:46.776+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:10:46.962 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:10:46 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:10:46.595+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:10:46.962 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:10:46 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:10:46.595+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:10:46.962 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:10:46 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:10:46.595+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:10:46.963 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:10:46 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:10:46.596+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:10:47.314 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:10:47 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:10:47.084+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:10:47.315 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Dec 16 22:10:46 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-mon-smithi045[80759]: 2023-12-16T22:10:46.960+0000 7f45a1f8f700 -1 mon.smithi045@0(probing) e2 get_health_metrics reporting 31097 slow ops, oldest is monmgrreport(gid 14214, 0 checks, 1 progress events) 2023-12-16T22:10:48.046 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:10:47 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:10:47.801+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:10:48.046 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:10:47 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:10:47.615+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:10:48.046 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:10:47 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:10:47.615+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:10:48.047 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:10:47 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:10:47.615+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:10:48.047 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:10:47 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:10:47.616+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:10:48.314 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:10:48 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:10:48.045+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:10:49.054 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:10:48 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:10:48.799+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:10:49.054 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:10:48 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:10:48.632+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:10:49.055 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:10:48 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:10:48.632+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:10:49.055 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:10:48 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:10:48.632+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:10:49.055 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:10:48 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:10:48.633+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:10:49.314 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:10:49 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:10:49.053+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:10:50.016 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:10:49 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:10:49.834+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:10:50.017 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:10:49 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:10:49.677+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:10:50.017 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:10:49 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:10:49.677+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:10:50.017 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:10:49 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:10:49.677+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:10:50.017 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:10:49 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:10:49.677+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:10:50.314 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:10:50 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:10:50.015+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:10:51.057 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:10:50 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:10:50.870+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:10:51.057 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:10:50 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:10:50.726+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:10:51.058 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:10:50 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:10:50.726+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:10:51.058 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:10:50 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:10:50.726+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:10:51.058 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:10:50 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:10:50.727+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:10:51.314 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:10:51 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:10:51.056+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:10:52.010 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:10:51 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:10:51.829+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:10:52.010 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Dec 16 22:10:51 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-mon-smithi045[80759]: 2023-12-16T22:10:51.964+0000 7f45a1f8f700 -1 mon.smithi045@0(probing) e2 get_health_metrics reporting 31190 slow ops, oldest is monmgrreport(gid 14214, 0 checks, 1 progress events) 2023-12-16T22:10:52.010 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:10:51 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:10:51.742+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:10:52.010 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:10:51 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:10:51.742+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:10:52.011 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:10:51 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:10:51.742+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:10:52.011 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:10:51 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:10:51.743+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:10:52.314 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:10:52 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:10:52.009+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:10:53.014 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:10:52 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:10:52.789+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:10:53.015 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:10:52 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:10:52.733+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:10:53.015 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:10:52 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:10:52.733+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:10:53.015 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:10:52 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:10:52.733+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:10:53.015 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:10:52 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:10:52.735+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:10:53.314 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:10:53 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:10:53.013+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:10:54.020 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:10:53 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:10:53.750+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:10:54.020 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:10:53 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:10:53.726+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:10:54.020 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:10:53 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:10:53.726+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:10:54.021 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:10:53 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:10:53.726+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:10:54.021 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:10:53 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:10:53.727+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:10:54.314 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:10:54 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:10:54.019+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:10:54.973 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:10:54 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:10:54.745+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:10:54.974 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:10:54 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:10:54.689+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:10:54.974 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:10:54 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:10:54.689+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:10:54.974 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:10:54 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:10:54.689+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:10:54.974 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:10:54 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:10:54.690+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:10:55.314 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:10:54 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:10:54.972+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:10:55.954 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:10:55 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:10:55.701+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:10:55.954 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:10:55 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:10:55.683+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:10:55.954 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:10:55 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:10:55.683+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:10:55.954 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:10:55 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:10:55.683+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:10:55.955 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:10:55 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:10:55.683+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:10:56.314 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:10:55 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:10:55.953+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:10:56.970 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:10:56 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:10:56.929+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:10:56.970 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:10:56 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:10:56.687+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:10:56.970 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:10:56 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:10:56.702+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:10:56.970 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:10:56 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:10:56.702+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:10:56.971 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:10:56 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:10:56.702+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:10:56.971 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:10:56 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:10:56.702+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:10:57.314 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Dec 16 22:10:56 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-mon-smithi045[80759]: 2023-12-16T22:10:56.969+0000 7f45a1f8f700 -1 mon.smithi045@0(probing) e2 get_health_metrics reporting 31279 slow ops, oldest is monmgrreport(gid 14214, 0 checks, 1 progress events) 2023-12-16T22:10:57.933 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:10:57 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:10:57.734+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:10:57.933 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:10:57 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:10:57.656+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:10:57.934 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:10:57 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:10:57.656+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:10:57.934 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:10:57 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:10:57.656+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:10:57.934 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:10:57 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:10:57.657+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:10:58.314 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:10:57 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:10:57.932+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:10:59.064 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:10:58 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:10:58.940+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:10:59.065 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:10:58 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:10:58.698+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:10:59.065 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:10:58 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:10:58.695+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:10:59.065 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:10:58 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:10:58.695+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:10:59.065 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:10:58 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:10:58.695+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:10:59.065 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:10:58 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:10:58.695+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:11:00.064 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:10:59 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:10:59.907+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:11:00.065 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:10:59 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:10:59.744+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:11:00.065 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:10:59 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:10:59.721+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:11:00.065 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:10:59 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:10:59.721+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:11:00.065 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:10:59 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:10:59.721+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:11:00.065 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:10:59 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:10:59.723+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:11:01.064 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:11:00 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:11:00.878+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:11:01.065 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:11:00 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:11:00.746+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:11:01.065 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:11:00 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:11:00.729+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:11:01.065 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:11:00 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:11:00.729+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:11:01.065 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:11:00 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:11:00.729+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:11:01.065 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:11:00 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:11:00.730+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:11:01.985 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:11:01 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:11:01.837+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:11:01.986 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:11:01 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:11:01.725+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:11:01.986 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:11:01 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:11:01.727+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:11:01.986 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:11:01 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:11:01.727+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:11:01.986 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:11:01 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:11:01.727+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:11:01.986 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:11:01 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:11:01.728+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:11:01.987 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Dec 16 22:11:01 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-mon-smithi045[80759]: 2023-12-16T22:11:01.972+0000 7f45a1f8f700 -1 mon.smithi045@0(probing) e2 get_health_metrics reporting 31371 slow ops, oldest is monmgrreport(gid 14214, 0 checks, 1 progress events) 2023-12-16T22:11:03.055 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:11:02 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:11:02.689+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:11:03.055 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:11:02 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:11:02.732+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:11:03.056 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:11:02 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:11:02.732+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:11:03.056 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:11:02 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:11:02.732+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:11:03.056 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:11:02 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:11:02.733+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:11:03.056 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:11:02 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:11:02.845+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:11:04.064 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:11:03 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:11:03.780+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:11:04.064 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:11:03 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:11:03.780+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:11:04.065 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:11:03 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:11:03.780+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:11:04.065 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:11:03 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:11:03.781+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:11:04.066 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:11:03 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:11:03.834+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:10:43.026858+0000 (oldest deadline 2023-12-16T22:11:03.026858+0000) 2023-12-16T22:11:04.066 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:11:03 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:11:03.834+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:10:43.026858+0000 (oldest deadline 2023-12-16T22:11:03.026858+0000) 2023-12-16T22:11:04.066 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:11:03 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:11:03.834+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:10:43.026858+0000 (oldest deadline 2023-12-16T22:11:03.026858+0000) 2023-12-16T22:11:04.066 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:11:03 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:11:03.835+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:11:04.067 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:11:03 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:11:03.660+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:10:42.731761+0000 (oldest deadline 2023-12-16T22:11:02.731761+0000) 2023-12-16T22:11:04.067 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:11:03 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:11:03.660+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:10:42.731761+0000 (oldest deadline 2023-12-16T22:11:02.731761+0000) 2023-12-16T22:11:04.067 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:11:03 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:11:03.660+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:10:42.731761+0000 (oldest deadline 2023-12-16T22:11:02.731761+0000) 2023-12-16T22:11:04.067 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:11:03 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:11:03.660+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:11:05.064 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:11:04 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:11:04.828+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:10:43.026858+0000 (oldest deadline 2023-12-16T22:11:03.026858+0000) 2023-12-16T22:11:05.065 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:11:04 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:11:04.828+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:10:43.026858+0000 (oldest deadline 2023-12-16T22:11:03.026858+0000) 2023-12-16T22:11:05.065 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:11:04 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:11:04.828+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:10:43.026858+0000 (oldest deadline 2023-12-16T22:11:03.026858+0000) 2023-12-16T22:11:05.065 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:11:04 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:11:04.829+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:11:05.065 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:11:04 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:11:04.710+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:10:42.731761+0000 (oldest deadline 2023-12-16T22:11:02.731761+0000) 2023-12-16T22:11:05.065 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:11:04 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:11:04.710+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:10:42.731761+0000 (oldest deadline 2023-12-16T22:11:02.731761+0000) 2023-12-16T22:11:05.066 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:11:04 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:11:04.710+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:10:42.731761+0000 (oldest deadline 2023-12-16T22:11:02.731761+0000) 2023-12-16T22:11:05.066 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:11:04 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:11:04.711+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:11:05.066 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:11:04 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:11:04.746+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:11:05.066 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:11:04 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:11:04.746+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:11:05.066 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:11:04 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:11:04.746+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:00:28.078142+0000 (oldest deadline 2023-12-16T22:00:48.078142+0000) 2023-12-16T22:11:05.066 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:11:04 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:11:04.747+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:11:06.064 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:11:05 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:11:05.749+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:11:06.065 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:11:05 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:11:05.754+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:10:42.731761+0000 (oldest deadline 2023-12-16T22:11:02.731761+0000) 2023-12-16T22:11:06.065 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:11:05 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:11:05.754+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:10:42.731761+0000 (oldest deadline 2023-12-16T22:11:02.731761+0000) 2023-12-16T22:11:06.065 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:11:05 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:11:05.754+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:10:42.731761+0000 (oldest deadline 2023-12-16T22:11:02.731761+0000) 2023-12-16T22:11:06.065 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:11:05 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:11:05.755+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:11:06.066 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:11:05 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:11:05.842+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:10:43.026858+0000 (oldest deadline 2023-12-16T22:11:03.026858+0000) 2023-12-16T22:11:06.066 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:11:05 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:11:05.842+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:10:43.026858+0000 (oldest deadline 2023-12-16T22:11:03.026858+0000) 2023-12-16T22:11:06.066 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:11:05 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:11:05.842+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:10:43.026858+0000 (oldest deadline 2023-12-16T22:11:03.026858+0000) 2023-12-16T22:11:06.066 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:11:05 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:11:05.842+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:11:06.979 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:11:06 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:11:06.709+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:11:06.980 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:11:06 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:11:06.881+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:10:43.026858+0000 (oldest deadline 2023-12-16T22:11:03.026858+0000) 2023-12-16T22:11:06.980 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:11:06 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:11:06.881+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:10:43.026858+0000 (oldest deadline 2023-12-16T22:11:03.026858+0000) 2023-12-16T22:11:06.980 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:11:06 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:11:06.881+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:10:43.026858+0000 (oldest deadline 2023-12-16T22:11:03.026858+0000) 2023-12-16T22:11:06.980 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:11:06 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:11:06.882+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:11:06.980 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:11:06 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:11:06.717+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:10:42.731761+0000 (oldest deadline 2023-12-16T22:11:02.731761+0000) 2023-12-16T22:11:06.981 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:11:06 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:11:06.717+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:10:42.731761+0000 (oldest deadline 2023-12-16T22:11:02.731761+0000) 2023-12-16T22:11:06.981 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:11:06 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:11:06.717+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:10:42.731761+0000 (oldest deadline 2023-12-16T22:11:02.731761+0000) 2023-12-16T22:11:06.981 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:11:06 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:11:06.718+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:11:07.315 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Dec 16 22:11:06 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-mon-smithi045[80759]: 2023-12-16T22:11:06.977+0000 7f45a1f8f700 -1 mon.smithi045@0(probing) e2 get_health_metrics reporting 31460 slow ops, oldest is monmgrreport(gid 14214, 0 checks, 1 progress events) 2023-12-16T22:11:08.064 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:11:07 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:11:07.680+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:11:08.065 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:11:07 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:11:07.908+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:10:43.026858+0000 (oldest deadline 2023-12-16T22:11:03.026858+0000) 2023-12-16T22:11:08.065 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:11:07 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:11:07.908+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:10:43.026858+0000 (oldest deadline 2023-12-16T22:11:03.026858+0000) 2023-12-16T22:11:08.065 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:11:07 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:11:07.908+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:10:43.026858+0000 (oldest deadline 2023-12-16T22:11:03.026858+0000) 2023-12-16T22:11:08.065 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:11:07 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:11:07.908+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:11:08.065 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:11:07 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:11:07.756+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:10:42.731761+0000 (oldest deadline 2023-12-16T22:11:02.731761+0000) 2023-12-16T22:11:08.066 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:11:07 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:11:07.756+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:10:42.731761+0000 (oldest deadline 2023-12-16T22:11:02.731761+0000) 2023-12-16T22:11:08.066 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:11:07 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:11:07.756+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:10:42.731761+0000 (oldest deadline 2023-12-16T22:11:02.731761+0000) 2023-12-16T22:11:08.066 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:11:07 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:11:07.757+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:11:09.064 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:11:08 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:11:08.711+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:11:09.065 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:11:08 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:11:08.762+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:10:42.731761+0000 (oldest deadline 2023-12-16T22:11:02.731761+0000) 2023-12-16T22:11:09.065 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:11:08 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:11:08.762+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:10:42.731761+0000 (oldest deadline 2023-12-16T22:11:02.731761+0000) 2023-12-16T22:11:09.065 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:11:08 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:11:08.762+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:10:42.731761+0000 (oldest deadline 2023-12-16T22:11:02.731761+0000) 2023-12-16T22:11:09.065 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:11:08 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:11:08.763+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:11:09.066 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:11:08 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:11:08.955+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:10:43.026858+0000 (oldest deadline 2023-12-16T22:11:03.026858+0000) 2023-12-16T22:11:09.066 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:11:08 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:11:08.955+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:10:43.026858+0000 (oldest deadline 2023-12-16T22:11:03.026858+0000) 2023-12-16T22:11:09.066 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:11:08 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:11:08.955+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:10:43.026858+0000 (oldest deadline 2023-12-16T22:11:03.026858+0000) 2023-12-16T22:11:09.066 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:11:08 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:11:08.956+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:11:09.953 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:11:09 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:11:09.669+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:11:09.954 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:11:09 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:11:09.749+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:10:42.731761+0000 (oldest deadline 2023-12-16T22:11:02.731761+0000) 2023-12-16T22:11:09.954 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:11:09 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:11:09.749+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:10:42.731761+0000 (oldest deadline 2023-12-16T22:11:02.731761+0000) 2023-12-16T22:11:09.954 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:11:09 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:11:09.749+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:10:42.731761+0000 (oldest deadline 2023-12-16T22:11:02.731761+0000) 2023-12-16T22:11:09.954 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:11:09 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:11:09.749+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:11:10.214 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:11:09 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:11:09.952+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:10:43.026858+0000 (oldest deadline 2023-12-16T22:11:03.026858+0000) 2023-12-16T22:11:10.215 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:11:09 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:11:09.952+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:10:43.026858+0000 (oldest deadline 2023-12-16T22:11:03.026858+0000) 2023-12-16T22:11:10.215 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:11:09 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:11:09.952+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:10:43.026858+0000 (oldest deadline 2023-12-16T22:11:03.026858+0000) 2023-12-16T22:11:10.215 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:11:09 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:11:09.953+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:11:11.064 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:11:10 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:11:10.670+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:11:11.065 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:11:10 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:11:10.919+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:10:43.026858+0000 (oldest deadline 2023-12-16T22:11:03.026858+0000) 2023-12-16T22:11:11.065 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:11:10 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:11:10.919+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:10:43.026858+0000 (oldest deadline 2023-12-16T22:11:03.026858+0000) 2023-12-16T22:11:11.065 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:11:10 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:11:10.919+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:10:43.026858+0000 (oldest deadline 2023-12-16T22:11:03.026858+0000) 2023-12-16T22:11:11.065 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:11:10 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:11:10.920+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:11:11.065 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:11:10 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:11:10.713+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:10:42.731761+0000 (oldest deadline 2023-12-16T22:11:02.731761+0000) 2023-12-16T22:11:11.066 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:11:10 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:11:10.713+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:10:42.731761+0000 (oldest deadline 2023-12-16T22:11:02.731761+0000) 2023-12-16T22:11:11.066 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:11:10 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:11:10.713+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:10:42.731761+0000 (oldest deadline 2023-12-16T22:11:02.731761+0000) 2023-12-16T22:11:11.066 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:11:10 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:11:10.713+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:11:11.970 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:11:11 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:11:11.660+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:11:11.970 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:11:11 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:11:11.730+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:10:42.731761+0000 (oldest deadline 2023-12-16T22:11:02.731761+0000) 2023-12-16T22:11:11.970 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:11:11 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:11:11.730+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:10:42.731761+0000 (oldest deadline 2023-12-16T22:11:02.731761+0000) 2023-12-16T22:11:11.971 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:11:11 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:11:11.730+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:10:42.731761+0000 (oldest deadline 2023-12-16T22:11:02.731761+0000) 2023-12-16T22:11:11.971 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:11:11 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:11:11.731+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:11:12.271 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Dec 16 22:11:11 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-mon-smithi045[80759]: 2023-12-16T22:11:11.980+0000 7f45a1f8f700 -1 mon.smithi045@0(probing) e2 get_health_metrics reporting 31553 slow ops, oldest is monmgrreport(gid 14214, 0 checks, 1 progress events) 2023-12-16T22:11:12.271 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:11:11 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:11:11.969+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:10:43.026858+0000 (oldest deadline 2023-12-16T22:11:03.026858+0000) 2023-12-16T22:11:12.271 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:11:11 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:11:11.969+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:10:43.026858+0000 (oldest deadline 2023-12-16T22:11:03.026858+0000) 2023-12-16T22:11:12.272 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:11:11 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:11:11.969+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:10:43.026858+0000 (oldest deadline 2023-12-16T22:11:03.026858+0000) 2023-12-16T22:11:12.272 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:11:11 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:11:11.970+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:11:12.978 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:11:12 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:11:12.633+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:11:12.979 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:11:12 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:11:12.763+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:10:42.731761+0000 (oldest deadline 2023-12-16T22:11:02.731761+0000) 2023-12-16T22:11:12.979 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:11:12 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:11:12.763+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:10:42.731761+0000 (oldest deadline 2023-12-16T22:11:02.731761+0000) 2023-12-16T22:11:12.979 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:11:12 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:11:12.763+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:10:42.731761+0000 (oldest deadline 2023-12-16T22:11:02.731761+0000) 2023-12-16T22:11:12.979 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:11:12 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:11:12.763+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:11:13.314 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:11:12 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:11:12.977+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:10:43.026858+0000 (oldest deadline 2023-12-16T22:11:03.026858+0000) 2023-12-16T22:11:13.314 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:11:12 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:11:12.977+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:10:43.026858+0000 (oldest deadline 2023-12-16T22:11:03.026858+0000) 2023-12-16T22:11:13.314 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:11:12 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:11:12.977+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:10:43.026858+0000 (oldest deadline 2023-12-16T22:11:03.026858+0000) 2023-12-16T22:11:13.315 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:11:12 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:11:12.978+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:11:13.961 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:11:13 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:11:13.684+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:11:13.961 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:11:13 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:11:13.729+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:10:42.731761+0000 (oldest deadline 2023-12-16T22:11:02.731761+0000) 2023-12-16T22:11:13.962 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:11:13 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:11:13.729+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:10:42.731761+0000 (oldest deadline 2023-12-16T22:11:02.731761+0000) 2023-12-16T22:11:13.962 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:11:13 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:11:13.729+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:10:42.731761+0000 (oldest deadline 2023-12-16T22:11:02.731761+0000) 2023-12-16T22:11:13.962 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:11:13 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:11:13.730+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:11:14.314 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:11:13 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:11:13.960+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:10:43.026858+0000 (oldest deadline 2023-12-16T22:11:03.026858+0000) 2023-12-16T22:11:14.314 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:11:13 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:11:13.960+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:10:43.026858+0000 (oldest deadline 2023-12-16T22:11:03.026858+0000) 2023-12-16T22:11:14.314 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:11:13 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:11:13.960+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:10:43.026858+0000 (oldest deadline 2023-12-16T22:11:03.026858+0000) 2023-12-16T22:11:14.315 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:11:13 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:11:13.961+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:11:15.064 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:11:14 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:11:14.720+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:11:15.065 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:11:14 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:11:14.732+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:10:42.731761+0000 (oldest deadline 2023-12-16T22:11:02.731761+0000) 2023-12-16T22:11:15.065 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:11:14 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:11:14.732+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:10:42.731761+0000 (oldest deadline 2023-12-16T22:11:02.731761+0000) 2023-12-16T22:11:15.065 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:11:14 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:11:14.732+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:10:42.731761+0000 (oldest deadline 2023-12-16T22:11:02.731761+0000) 2023-12-16T22:11:15.065 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:11:14 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:11:14.733+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:11:15.065 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:11:14 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:11:14.966+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:10:43.026858+0000 (oldest deadline 2023-12-16T22:11:03.026858+0000) 2023-12-16T22:11:15.065 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:11:14 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:11:14.966+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:10:43.026858+0000 (oldest deadline 2023-12-16T22:11:03.026858+0000) 2023-12-16T22:11:15.066 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:11:14 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:11:14.966+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:10:43.026858+0000 (oldest deadline 2023-12-16T22:11:03.026858+0000) 2023-12-16T22:11:15.066 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:11:14 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:11:14.967+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:11:16.064 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:11:15 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:11:15.712+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:11:16.065 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:11:15 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:11:15.943+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:10:43.026858+0000 (oldest deadline 2023-12-16T22:11:03.026858+0000) 2023-12-16T22:11:16.065 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:11:15 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:11:15.943+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:10:43.026858+0000 (oldest deadline 2023-12-16T22:11:03.026858+0000) 2023-12-16T22:11:16.065 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:11:15 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:11:15.943+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:10:43.026858+0000 (oldest deadline 2023-12-16T22:11:03.026858+0000) 2023-12-16T22:11:16.065 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:11:15 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:11:15.944+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:11:16.065 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:11:15 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:11:15.714+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:10:42.731761+0000 (oldest deadline 2023-12-16T22:11:02.731761+0000) 2023-12-16T22:11:16.065 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:11:15 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:11:15.714+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:10:42.731761+0000 (oldest deadline 2023-12-16T22:11:02.731761+0000) 2023-12-16T22:11:16.066 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:11:15 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:11:15.714+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:10:42.731761+0000 (oldest deadline 2023-12-16T22:11:02.731761+0000) 2023-12-16T22:11:16.066 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:11:15 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:11:15.715+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:11:17.004 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Dec 16 22:11:16 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-mon-smithi045[80759]: 2023-12-16T22:11:16.984+0000 7f45a1f8f700 -1 mon.smithi045@0(probing) e2 get_health_metrics reporting 31642 slow ops, oldest is monmgrreport(gid 14214, 0 checks, 1 progress events) 2023-12-16T22:11:17.005 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:11:16 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:11:16.759+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:11:17.005 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:11:16 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:11:16.751+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:10:42.731761+0000 (oldest deadline 2023-12-16T22:11:02.731761+0000) 2023-12-16T22:11:17.005 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:11:16 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:11:16.751+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:10:42.731761+0000 (oldest deadline 2023-12-16T22:11:02.731761+0000) 2023-12-16T22:11:17.005 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:11:16 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:11:16.751+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:10:42.731761+0000 (oldest deadline 2023-12-16T22:11:02.731761+0000) 2023-12-16T22:11:17.005 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:11:16 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:11:16.752+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:11:17.006 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:11:16 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:11:16.987+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:10:43.026858+0000 (oldest deadline 2023-12-16T22:11:03.026858+0000) 2023-12-16T22:11:17.006 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:11:16 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:11:16.987+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:10:43.026858+0000 (oldest deadline 2023-12-16T22:11:03.026858+0000) 2023-12-16T22:11:17.006 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:11:16 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:11:16.987+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:10:43.026858+0000 (oldest deadline 2023-12-16T22:11:03.026858+0000) 2023-12-16T22:11:17.006 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:11:16 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:11:16.988+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:11:18.018 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:11:17 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:11:17.727+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:11:18.019 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:11:17 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:11:17.721+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:10:42.731761+0000 (oldest deadline 2023-12-16T22:11:02.731761+0000) 2023-12-16T22:11:18.019 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:11:17 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:11:17.721+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:10:42.731761+0000 (oldest deadline 2023-12-16T22:11:02.731761+0000) 2023-12-16T22:11:18.019 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:11:17 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:11:17.721+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:10:42.731761+0000 (oldest deadline 2023-12-16T22:11:02.731761+0000) 2023-12-16T22:11:18.019 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:11:17 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:11:17.722+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:11:18.314 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:11:18 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:11:18.018+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:10:43.026858+0000 (oldest deadline 2023-12-16T22:11:03.026858+0000) 2023-12-16T22:11:18.314 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:11:18 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:11:18.018+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:10:43.026858+0000 (oldest deadline 2023-12-16T22:11:03.026858+0000) 2023-12-16T22:11:18.315 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:11:18 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:11:18.018+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:10:43.026858+0000 (oldest deadline 2023-12-16T22:11:03.026858+0000) 2023-12-16T22:11:18.315 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:11:18 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:11:18.019+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:11:19.056 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:11:18 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:11:18.747+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:11:19.057 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:11:18 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:11:18.711+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:10:42.731761+0000 (oldest deadline 2023-12-16T22:11:02.731761+0000) 2023-12-16T22:11:19.057 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:11:18 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:11:18.711+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:10:42.731761+0000 (oldest deadline 2023-12-16T22:11:02.731761+0000) 2023-12-16T22:11:19.057 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:11:18 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:11:18.711+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:10:42.731761+0000 (oldest deadline 2023-12-16T22:11:02.731761+0000) 2023-12-16T22:11:19.057 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:11:18 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:11:18.712+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:11:19.314 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:11:19 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:11:19.055+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:10:43.026858+0000 (oldest deadline 2023-12-16T22:11:03.026858+0000) 2023-12-16T22:11:19.314 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:11:19 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:11:19.055+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:10:43.026858+0000 (oldest deadline 2023-12-16T22:11:03.026858+0000) 2023-12-16T22:11:19.314 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:11:19 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:11:19.055+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:10:43.026858+0000 (oldest deadline 2023-12-16T22:11:03.026858+0000) 2023-12-16T22:11:19.315 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:11:19 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:11:19.057+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:11:20.062 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:11:19 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:11:19.704+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:11:20.062 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:11:19 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:11:19.737+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:10:42.731761+0000 (oldest deadline 2023-12-16T22:11:02.731761+0000) 2023-12-16T22:11:20.062 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:11:19 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:11:19.737+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:10:42.731761+0000 (oldest deadline 2023-12-16T22:11:02.731761+0000) 2023-12-16T22:11:20.062 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:11:19 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:11:19.737+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:10:42.731761+0000 (oldest deadline 2023-12-16T22:11:02.731761+0000) 2023-12-16T22:11:20.062 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:11:19 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:11:19.737+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:11:20.314 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:11:20 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:11:20.061+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:10:43.026858+0000 (oldest deadline 2023-12-16T22:11:03.026858+0000) 2023-12-16T22:11:20.314 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:11:20 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:11:20.061+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:10:43.026858+0000 (oldest deadline 2023-12-16T22:11:03.026858+0000) 2023-12-16T22:11:20.314 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:11:20 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:11:20.061+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:10:43.026858+0000 (oldest deadline 2023-12-16T22:11:03.026858+0000) 2023-12-16T22:11:20.315 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:11:20 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:11:20.062+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:11:21.064 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:11:20 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:11:20.739+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:11:21.065 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:11:20 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:11:20.735+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:10:42.731761+0000 (oldest deadline 2023-12-16T22:11:02.731761+0000) 2023-12-16T22:11:21.065 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:11:20 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:11:20.735+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:10:42.731761+0000 (oldest deadline 2023-12-16T22:11:02.731761+0000) 2023-12-16T22:11:21.065 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:11:20 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:11:20.735+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:10:42.731761+0000 (oldest deadline 2023-12-16T22:11:02.731761+0000) 2023-12-16T22:11:21.065 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:11:20 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:11:20.736+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:11:21.487 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:11:21 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:11:21.071+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:10:43.026858+0000 (oldest deadline 2023-12-16T22:11:03.026858+0000) 2023-12-16T22:11:21.487 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:11:21 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:11:21.071+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:10:43.026858+0000 (oldest deadline 2023-12-16T22:11:03.026858+0000) 2023-12-16T22:11:21.487 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:11:21 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:11:21.071+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:10:43.026858+0000 (oldest deadline 2023-12-16T22:11:03.026858+0000) 2023-12-16T22:11:21.487 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:11:21 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:11:21.072+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:11:21.990 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:11:21 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:11:21.747+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:11:21.990 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:11:21 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:11:21.737+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:10:42.731761+0000 (oldest deadline 2023-12-16T22:11:02.731761+0000) 2023-12-16T22:11:21.990 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:11:21 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:11:21.737+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:10:42.731761+0000 (oldest deadline 2023-12-16T22:11:02.731761+0000) 2023-12-16T22:11:21.990 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:11:21 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:11:21.737+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:10:42.731761+0000 (oldest deadline 2023-12-16T22:11:02.731761+0000) 2023-12-16T22:11:21.991 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:11:21 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:11:21.738+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:11:22.314 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Dec 16 22:11:21 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-mon-smithi045[80759]: 2023-12-16T22:11:21.989+0000 7f45a1f8f700 -1 mon.smithi045@0(probing) e2 get_health_metrics reporting 31734 slow ops, oldest is monmgrreport(gid 14214, 0 checks, 1 progress events) 2023-12-16T22:11:22.314 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:11:22 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:11:22.043+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:10:43.026858+0000 (oldest deadline 2023-12-16T22:11:03.026858+0000) 2023-12-16T22:11:22.315 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:11:22 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:11:22.043+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:10:43.026858+0000 (oldest deadline 2023-12-16T22:11:03.026858+0000) 2023-12-16T22:11:22.315 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:11:22 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:11:22.043+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:10:43.026858+0000 (oldest deadline 2023-12-16T22:11:03.026858+0000) 2023-12-16T22:11:22.315 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:11:22 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:11:22.044+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:11:23.021 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:11:22 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:11:22.793+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:11:23.021 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:11:22 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:11:22.707+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:10:42.731761+0000 (oldest deadline 2023-12-16T22:11:02.731761+0000) 2023-12-16T22:11:23.022 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:11:22 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:11:22.707+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:10:42.731761+0000 (oldest deadline 2023-12-16T22:11:02.731761+0000) 2023-12-16T22:11:23.022 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:11:22 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:11:22.707+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:10:42.731761+0000 (oldest deadline 2023-12-16T22:11:02.731761+0000) 2023-12-16T22:11:23.022 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:11:22 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:11:22.707+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:11:23.314 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:11:23 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:11:23.020+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:10:43.026858+0000 (oldest deadline 2023-12-16T22:11:03.026858+0000) 2023-12-16T22:11:23.314 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:11:23 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:11:23.020+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:10:43.026858+0000 (oldest deadline 2023-12-16T22:11:03.026858+0000) 2023-12-16T22:11:23.314 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:11:23 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:11:23.020+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:10:43.026858+0000 (oldest deadline 2023-12-16T22:11:03.026858+0000) 2023-12-16T22:11:23.314 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:11:23 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:11:23.021+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:11:24.052 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:11:23 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:11:23.783+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:11:24.052 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:11:23 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:11:23.700+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:10:42.731761+0000 (oldest deadline 2023-12-16T22:11:02.731761+0000) 2023-12-16T22:11:24.052 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:11:23 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:11:23.700+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:10:42.731761+0000 (oldest deadline 2023-12-16T22:11:02.731761+0000) 2023-12-16T22:11:24.053 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:11:23 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:11:23.700+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:10:42.731761+0000 (oldest deadline 2023-12-16T22:11:02.731761+0000) 2023-12-16T22:11:24.053 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:11:23 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:11:23.701+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:11:24.314 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:11:24 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:11:24.051+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:10:43.026858+0000 (oldest deadline 2023-12-16T22:11:03.026858+0000) 2023-12-16T22:11:24.314 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:11:24 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:11:24.051+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:10:43.026858+0000 (oldest deadline 2023-12-16T22:11:03.026858+0000) 2023-12-16T22:11:24.314 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:11:24 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:11:24.051+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:10:43.026858+0000 (oldest deadline 2023-12-16T22:11:03.026858+0000) 2023-12-16T22:11:24.315 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:11:24 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:11:24.052+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:11:24.814 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:11:24 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:11:24.786+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:11:24.815 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:11:24 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:11:24.670+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:10:42.731761+0000 (oldest deadline 2023-12-16T22:11:02.731761+0000) 2023-12-16T22:11:24.815 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:11:24 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:11:24.670+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:10:42.731761+0000 (oldest deadline 2023-12-16T22:11:02.731761+0000) 2023-12-16T22:11:24.815 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:11:24 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:11:24.670+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:10:42.731761+0000 (oldest deadline 2023-12-16T22:11:02.731761+0000) 2023-12-16T22:11:24.816 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:11:24 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:11:24.670+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:11:25.564 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:11:25 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:11:25.076+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:10:43.026858+0000 (oldest deadline 2023-12-16T22:11:03.026858+0000) 2023-12-16T22:11:25.565 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:11:25 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:11:25.076+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:10:43.026858+0000 (oldest deadline 2023-12-16T22:11:03.026858+0000) 2023-12-16T22:11:25.565 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:11:25 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:11:25.076+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:10:43.026858+0000 (oldest deadline 2023-12-16T22:11:03.026858+0000) 2023-12-16T22:11:25.565 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:11:25 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:11:25.077+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:11:26.040 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:11:25 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:11:25.824+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:11:26.040 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:11:25 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:11:25.622+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:10:42.731761+0000 (oldest deadline 2023-12-16T22:11:02.731761+0000) 2023-12-16T22:11:26.040 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:11:25 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:11:25.622+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:10:42.731761+0000 (oldest deadline 2023-12-16T22:11:02.731761+0000) 2023-12-16T22:11:26.041 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:11:25 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:11:25.622+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:10:42.731761+0000 (oldest deadline 2023-12-16T22:11:02.731761+0000) 2023-12-16T22:11:26.041 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:11:25 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:11:25.624+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:11:26.314 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:11:26 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:11:26.039+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:10:43.026858+0000 (oldest deadline 2023-12-16T22:11:03.026858+0000) 2023-12-16T22:11:26.315 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:11:26 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:11:26.039+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:10:43.026858+0000 (oldest deadline 2023-12-16T22:11:03.026858+0000) 2023-12-16T22:11:26.315 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:11:26 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:11:26.039+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:10:43.026858+0000 (oldest deadline 2023-12-16T22:11:03.026858+0000) 2023-12-16T22:11:26.315 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:11:26 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:11:26.040+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:11:26.994 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:11:26 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:11:26.795+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:11:26.994 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:11:26 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:11:26.584+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:10:42.731761+0000 (oldest deadline 2023-12-16T22:11:02.731761+0000) 2023-12-16T22:11:26.994 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:11:26 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:11:26.584+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:10:42.731761+0000 (oldest deadline 2023-12-16T22:11:02.731761+0000) 2023-12-16T22:11:26.994 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:11:26 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:11:26.584+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:10:42.731761+0000 (oldest deadline 2023-12-16T22:11:02.731761+0000) 2023-12-16T22:11:26.995 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:11:26 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:11:26.584+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:11:27.314 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Dec 16 22:11:26 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-mon-smithi045[80759]: 2023-12-16T22:11:26.992+0000 7f45a1f8f700 -1 mon.smithi045@0(probing) e2 get_health_metrics reporting 31824 slow ops, oldest is monmgrreport(gid 14214, 0 checks, 1 progress events) 2023-12-16T22:11:27.315 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:11:27 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:11:27.008+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:10:43.026858+0000 (oldest deadline 2023-12-16T22:11:03.026858+0000) 2023-12-16T22:11:27.315 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:11:27 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:11:27.008+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:10:43.026858+0000 (oldest deadline 2023-12-16T22:11:03.026858+0000) 2023-12-16T22:11:27.315 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:11:27 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:11:27.008+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:10:43.026858+0000 (oldest deadline 2023-12-16T22:11:03.026858+0000) 2023-12-16T22:11:27.315 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:11:27 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:11:27.009+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:11:27.814 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:11:27 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:11:27.535+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:10:42.731761+0000 (oldest deadline 2023-12-16T22:11:02.731761+0000) 2023-12-16T22:11:27.815 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:11:27 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:11:27.535+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:10:42.731761+0000 (oldest deadline 2023-12-16T22:11:02.731761+0000) 2023-12-16T22:11:27.815 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:11:27 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:11:27.535+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:10:42.731761+0000 (oldest deadline 2023-12-16T22:11:02.731761+0000) 2023-12-16T22:11:27.815 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:11:27 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:11:27.536+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:11:28.314 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:11:27 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:11:27.827+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:11:28.315 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:11:28 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:11:28.048+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:10:43.026858+0000 (oldest deadline 2023-12-16T22:11:03.026858+0000) 2023-12-16T22:11:28.315 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:11:28 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:11:28.048+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:10:43.026858+0000 (oldest deadline 2023-12-16T22:11:03.026858+0000) 2023-12-16T22:11:28.316 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:11:28 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:11:28.048+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:10:43.026858+0000 (oldest deadline 2023-12-16T22:11:03.026858+0000) 2023-12-16T22:11:28.316 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:11:28 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:11:28.048+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:11:29.010 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:11:28 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:11:28.810+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:11:07.849373+0000 (oldest deadline 2023-12-16T22:11:27.849373+0000) 2023-12-16T22:11:29.010 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:11:28 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:11:28.810+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:11:07.849373+0000 (oldest deadline 2023-12-16T22:11:27.849373+0000) 2023-12-16T22:11:29.010 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:11:28 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:11:28.810+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:11:07.849373+0000 (oldest deadline 2023-12-16T22:11:27.849373+0000) 2023-12-16T22:11:29.011 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:11:28 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:11:28.810+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:11:29.011 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:11:28 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:11:28.576+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:10:42.731761+0000 (oldest deadline 2023-12-16T22:11:02.731761+0000) 2023-12-16T22:11:29.011 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:11:28 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:11:28.576+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:10:42.731761+0000 (oldest deadline 2023-12-16T22:11:02.731761+0000) 2023-12-16T22:11:29.011 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:11:28 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:11:28.576+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:10:42.731761+0000 (oldest deadline 2023-12-16T22:11:02.731761+0000) 2023-12-16T22:11:29.011 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:11:28 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:11:28.577+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:11:29.314 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:11:29 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:11:29.008+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:10:43.026858+0000 (oldest deadline 2023-12-16T22:11:03.026858+0000) 2023-12-16T22:11:29.314 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:11:29 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:11:29.008+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:10:43.026858+0000 (oldest deadline 2023-12-16T22:11:03.026858+0000) 2023-12-16T22:11:29.315 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:11:29 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:11:29.008+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:10:43.026858+0000 (oldest deadline 2023-12-16T22:11:03.026858+0000) 2023-12-16T22:11:29.315 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:11:29 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:11:29.009+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:11:29.814 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:11:29 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:11:29.537+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:10:42.731761+0000 (oldest deadline 2023-12-16T22:11:02.731761+0000) 2023-12-16T22:11:29.814 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:11:29 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:11:29.537+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:10:42.731761+0000 (oldest deadline 2023-12-16T22:11:02.731761+0000) 2023-12-16T22:11:29.815 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:11:29 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:11:29.537+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:10:42.731761+0000 (oldest deadline 2023-12-16T22:11:02.731761+0000) 2023-12-16T22:11:29.815 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:11:29 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:11:29.538+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:11:30.216 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:11:29 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:11:29.968+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:10:43.026858+0000 (oldest deadline 2023-12-16T22:11:03.026858+0000) 2023-12-16T22:11:30.216 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:11:29 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:11:29.968+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:10:43.026858+0000 (oldest deadline 2023-12-16T22:11:03.026858+0000) 2023-12-16T22:11:30.216 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:11:29 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:11:29.968+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:10:43.026858+0000 (oldest deadline 2023-12-16T22:11:03.026858+0000) 2023-12-16T22:11:30.217 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:11:29 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:11:29.969+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:11:30.217 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:11:29 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:11:29.819+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:11:07.849373+0000 (oldest deadline 2023-12-16T22:11:27.849373+0000) 2023-12-16T22:11:30.217 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:11:29 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:11:29.819+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:11:07.849373+0000 (oldest deadline 2023-12-16T22:11:27.849373+0000) 2023-12-16T22:11:30.217 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:11:29 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:11:29.819+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:11:07.849373+0000 (oldest deadline 2023-12-16T22:11:27.849373+0000) 2023-12-16T22:11:30.217 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:11:29 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:11:29.820+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:11:30.941 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:11:30 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:11:30.776+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:11:07.849373+0000 (oldest deadline 2023-12-16T22:11:27.849373+0000) 2023-12-16T22:11:30.941 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:11:30 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:11:30.776+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:11:07.849373+0000 (oldest deadline 2023-12-16T22:11:27.849373+0000) 2023-12-16T22:11:30.942 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:11:30 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:11:30.776+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:11:07.849373+0000 (oldest deadline 2023-12-16T22:11:27.849373+0000) 2023-12-16T22:11:30.942 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:11:30 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:11:30.776+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:11:30.942 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:11:30 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:11:30.566+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:10:42.731761+0000 (oldest deadline 2023-12-16T22:11:02.731761+0000) 2023-12-16T22:11:30.942 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:11:30 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:11:30.566+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:10:42.731761+0000 (oldest deadline 2023-12-16T22:11:02.731761+0000) 2023-12-16T22:11:30.942 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:11:30 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:11:30.566+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:10:42.731761+0000 (oldest deadline 2023-12-16T22:11:02.731761+0000) 2023-12-16T22:11:30.942 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:11:30 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:11:30.566+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:11:31.314 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:11:30 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:11:30.939+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:10:43.026858+0000 (oldest deadline 2023-12-16T22:11:03.026858+0000) 2023-12-16T22:11:31.315 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:11:30 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:11:30.939+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:10:43.026858+0000 (oldest deadline 2023-12-16T22:11:03.026858+0000) 2023-12-16T22:11:31.315 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:11:30 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:11:30.939+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:10:43.026858+0000 (oldest deadline 2023-12-16T22:11:03.026858+0000) 2023-12-16T22:11:31.315 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:11:30 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:11:30.940+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:11:31.814 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:11:31 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:11:31.767+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:11:07.849373+0000 (oldest deadline 2023-12-16T22:11:27.849373+0000) 2023-12-16T22:11:31.815 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:11:31 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:11:31.767+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:11:07.849373+0000 (oldest deadline 2023-12-16T22:11:27.849373+0000) 2023-12-16T22:11:31.815 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:11:31 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:11:31.767+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:11:07.849373+0000 (oldest deadline 2023-12-16T22:11:27.849373+0000) 2023-12-16T22:11:31.815 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:11:31 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:11:31.767+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:11:31.815 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:11:31 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:11:31.541+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:10:42.731761+0000 (oldest deadline 2023-12-16T22:11:02.731761+0000) 2023-12-16T22:11:31.815 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:11:31 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:11:31.541+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:10:42.731761+0000 (oldest deadline 2023-12-16T22:11:02.731761+0000) 2023-12-16T22:11:31.816 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:11:31 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:11:31.541+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:10:42.731761+0000 (oldest deadline 2023-12-16T22:11:02.731761+0000) 2023-12-16T22:11:31.816 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:11:31 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:11:31.541+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:11:32.314 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Dec 16 22:11:31 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-mon-smithi045[80759]: 2023-12-16T22:11:31.997+0000 7f45a1f8f700 -1 mon.smithi045@0(probing) e2 get_health_metrics reporting 31916 slow ops, oldest is monmgrreport(gid 14214, 0 checks, 1 progress events) 2023-12-16T22:11:32.315 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:11:31 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:11:31.909+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:10:43.026858+0000 (oldest deadline 2023-12-16T22:11:03.026858+0000) 2023-12-16T22:11:32.315 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:11:31 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:11:31.909+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:10:43.026858+0000 (oldest deadline 2023-12-16T22:11:03.026858+0000) 2023-12-16T22:11:32.315 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:11:31 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:11:31.909+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:10:43.026858+0000 (oldest deadline 2023-12-16T22:11:03.026858+0000) 2023-12-16T22:11:32.315 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:11:31 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:11:31.910+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:11:32.814 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:11:32 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:11:32.554+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:10:42.731761+0000 (oldest deadline 2023-12-16T22:11:02.731761+0000) 2023-12-16T22:11:32.814 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:11:32 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:11:32.554+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:10:42.731761+0000 (oldest deadline 2023-12-16T22:11:02.731761+0000) 2023-12-16T22:11:32.815 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:11:32 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:11:32.554+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:10:42.731761+0000 (oldest deadline 2023-12-16T22:11:02.731761+0000) 2023-12-16T22:11:32.815 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:11:32 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:11:32.555+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:11:32.815 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:11:32 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:11:32.731+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:11:07.849373+0000 (oldest deadline 2023-12-16T22:11:27.849373+0000) 2023-12-16T22:11:32.815 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:11:32 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:11:32.731+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:11:07.849373+0000 (oldest deadline 2023-12-16T22:11:27.849373+0000) 2023-12-16T22:11:32.815 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:11:32 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:11:32.731+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:11:07.849373+0000 (oldest deadline 2023-12-16T22:11:27.849373+0000) 2023-12-16T22:11:32.815 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:11:32 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:11:32.731+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:11:33.314 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:11:32 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:11:32.880+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:10:43.026858+0000 (oldest deadline 2023-12-16T22:11:03.026858+0000) 2023-12-16T22:11:33.315 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:11:32 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:11:32.880+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:10:43.026858+0000 (oldest deadline 2023-12-16T22:11:03.026858+0000) 2023-12-16T22:11:33.315 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:11:32 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:11:32.880+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:10:43.026858+0000 (oldest deadline 2023-12-16T22:11:03.026858+0000) 2023-12-16T22:11:33.315 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:11:32 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:11:32.881+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:11:33.775 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:11:33 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:11:33.514+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:10:42.731761+0000 (oldest deadline 2023-12-16T22:11:02.731761+0000) 2023-12-16T22:11:33.776 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:11:33 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:11:33.514+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:10:42.731761+0000 (oldest deadline 2023-12-16T22:11:02.731761+0000) 2023-12-16T22:11:33.776 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:11:33 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:11:33.514+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:10:42.731761+0000 (oldest deadline 2023-12-16T22:11:02.731761+0000) 2023-12-16T22:11:33.776 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:11:33 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:11:33.515+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:11:33.776 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:11:33 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:11:33.697+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:11:07.849373+0000 (oldest deadline 2023-12-16T22:11:27.849373+0000) 2023-12-16T22:11:33.776 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:11:33 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:11:33.697+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:11:07.849373+0000 (oldest deadline 2023-12-16T22:11:27.849373+0000) 2023-12-16T22:11:33.776 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:11:33 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:11:33.697+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:11:07.849373+0000 (oldest deadline 2023-12-16T22:11:27.849373+0000) 2023-12-16T22:11:33.777 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:11:33 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:11:33.699+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:11:34.064 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:11:33 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:11:33.876+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:10:43.026858+0000 (oldest deadline 2023-12-16T22:11:03.026858+0000) 2023-12-16T22:11:34.065 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:11:33 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:11:33.876+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:10:43.026858+0000 (oldest deadline 2023-12-16T22:11:03.026858+0000) 2023-12-16T22:11:34.065 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:11:33 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:11:33.876+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:10:43.026858+0000 (oldest deadline 2023-12-16T22:11:03.026858+0000) 2023-12-16T22:11:34.065 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:11:33 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:11:33.877+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:11:34.814 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:11:34 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:11:34.479+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:10:42.731761+0000 (oldest deadline 2023-12-16T22:11:02.731761+0000) 2023-12-16T22:11:34.815 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:11:34 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:11:34.479+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:10:42.731761+0000 (oldest deadline 2023-12-16T22:11:02.731761+0000) 2023-12-16T22:11:34.815 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:11:34 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:11:34.479+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:10:42.731761+0000 (oldest deadline 2023-12-16T22:11:02.731761+0000) 2023-12-16T22:11:34.815 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:11:34 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:11:34.479+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:11:34.815 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:11:34 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:11:34.695+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:11:07.849373+0000 (oldest deadline 2023-12-16T22:11:27.849373+0000) 2023-12-16T22:11:34.815 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:11:34 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:11:34.695+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:11:07.849373+0000 (oldest deadline 2023-12-16T22:11:27.849373+0000) 2023-12-16T22:11:34.815 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:11:34 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:11:34.695+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:11:07.849373+0000 (oldest deadline 2023-12-16T22:11:27.849373+0000) 2023-12-16T22:11:34.816 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:11:34 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:11:34.697+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:11:35.314 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:11:34 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:11:34.844+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:10:43.026858+0000 (oldest deadline 2023-12-16T22:11:03.026858+0000) 2023-12-16T22:11:35.314 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:11:34 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:11:34.844+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:10:43.026858+0000 (oldest deadline 2023-12-16T22:11:03.026858+0000) 2023-12-16T22:11:35.315 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:11:34 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:11:34.844+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:10:43.026858+0000 (oldest deadline 2023-12-16T22:11:03.026858+0000) 2023-12-16T22:11:35.315 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:11:34 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:11:34.845+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:11:35.814 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:11:35 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:11:35.441+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:10:42.731761+0000 (oldest deadline 2023-12-16T22:11:02.731761+0000) 2023-12-16T22:11:35.815 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:11:35 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:11:35.441+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:10:42.731761+0000 (oldest deadline 2023-12-16T22:11:02.731761+0000) 2023-12-16T22:11:35.815 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:11:35 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:11:35.441+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:10:42.731761+0000 (oldest deadline 2023-12-16T22:11:02.731761+0000) 2023-12-16T22:11:35.815 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:11:35 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:11:35.442+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:11:35.815 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:11:35 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:11:35.669+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:11:07.849373+0000 (oldest deadline 2023-12-16T22:11:27.849373+0000) 2023-12-16T22:11:35.816 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:11:35 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:11:35.669+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:11:07.849373+0000 (oldest deadline 2023-12-16T22:11:27.849373+0000) 2023-12-16T22:11:35.816 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:11:35 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:11:35.669+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:11:07.849373+0000 (oldest deadline 2023-12-16T22:11:27.849373+0000) 2023-12-16T22:11:35.816 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:11:35 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:11:35.670+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:11:36.314 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:11:35 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:11:35.856+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:10:43.026858+0000 (oldest deadline 2023-12-16T22:11:03.026858+0000) 2023-12-16T22:11:36.314 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:11:35 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:11:35.856+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:10:43.026858+0000 (oldest deadline 2023-12-16T22:11:03.026858+0000) 2023-12-16T22:11:36.315 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:11:35 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:11:35.856+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:10:43.026858+0000 (oldest deadline 2023-12-16T22:11:03.026858+0000) 2023-12-16T22:11:36.315 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:11:35 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:11:35.857+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:11:36.814 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:11:36 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:11:36.655+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:11:07.849373+0000 (oldest deadline 2023-12-16T22:11:27.849373+0000) 2023-12-16T22:11:36.815 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:11:36 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:11:36.655+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:11:07.849373+0000 (oldest deadline 2023-12-16T22:11:27.849373+0000) 2023-12-16T22:11:36.815 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:11:36 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:11:36.655+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:11:07.849373+0000 (oldest deadline 2023-12-16T22:11:27.849373+0000) 2023-12-16T22:11:36.815 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:11:36 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:11:36.657+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:11:36.815 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:11:36 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:11:36.448+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:10:42.731761+0000 (oldest deadline 2023-12-16T22:11:02.731761+0000) 2023-12-16T22:11:36.815 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:11:36 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:11:36.448+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:10:42.731761+0000 (oldest deadline 2023-12-16T22:11:02.731761+0000) 2023-12-16T22:11:36.815 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:11:36 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:11:36.448+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:10:42.731761+0000 (oldest deadline 2023-12-16T22:11:02.731761+0000) 2023-12-16T22:11:36.816 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:11:36 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:11:36.449+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:11:37.128 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Dec 16 22:11:37 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-mon-smithi045[80759]: 2023-12-16T22:11:37.001+0000 7f45a1f8f700 -1 mon.smithi045@0(probing) e2 get_health_metrics reporting 32006 slow ops, oldest is monmgrreport(gid 14214, 0 checks, 1 progress events) 2023-12-16T22:11:37.128 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:11:36 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:11:36.825+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:10:43.026858+0000 (oldest deadline 2023-12-16T22:11:03.026858+0000) 2023-12-16T22:11:37.128 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:11:36 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:11:36.825+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:10:43.026858+0000 (oldest deadline 2023-12-16T22:11:03.026858+0000) 2023-12-16T22:11:37.128 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:11:36 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:11:36.825+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:10:43.026858+0000 (oldest deadline 2023-12-16T22:11:03.026858+0000) 2023-12-16T22:11:37.128 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:11:36 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:11:36.826+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:11:37.815 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:11:37 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:11:37.698+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:11:07.849373+0000 (oldest deadline 2023-12-16T22:11:27.849373+0000) 2023-12-16T22:11:37.815 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:11:37 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:11:37.698+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:11:07.849373+0000 (oldest deadline 2023-12-16T22:11:27.849373+0000) 2023-12-16T22:11:37.815 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:11:37 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:11:37.698+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:11:07.849373+0000 (oldest deadline 2023-12-16T22:11:27.849373+0000) 2023-12-16T22:11:37.815 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:11:37 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:11:37.699+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:11:37.815 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:11:37 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:11:37.454+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:10:42.731761+0000 (oldest deadline 2023-12-16T22:11:02.731761+0000) 2023-12-16T22:11:37.816 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:11:37 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:11:37.454+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:10:42.731761+0000 (oldest deadline 2023-12-16T22:11:02.731761+0000) 2023-12-16T22:11:37.816 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:11:37 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:11:37.454+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:10:42.731761+0000 (oldest deadline 2023-12-16T22:11:02.731761+0000) 2023-12-16T22:11:37.816 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:11:37 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:11:37.455+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:11:38.314 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:11:37 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:11:37.866+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:10:43.026858+0000 (oldest deadline 2023-12-16T22:11:03.026858+0000) 2023-12-16T22:11:38.315 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:11:37 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:11:37.866+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:10:43.026858+0000 (oldest deadline 2023-12-16T22:11:03.026858+0000) 2023-12-16T22:11:38.315 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:11:37 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:11:37.866+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:10:43.026858+0000 (oldest deadline 2023-12-16T22:11:03.026858+0000) 2023-12-16T22:11:38.315 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:11:37 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:11:37.867+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:11:38.814 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:11:38 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:11:38.505+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:10:42.731761+0000 (oldest deadline 2023-12-16T22:11:02.731761+0000) 2023-12-16T22:11:38.815 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:11:38 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:11:38.505+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:10:42.731761+0000 (oldest deadline 2023-12-16T22:11:02.731761+0000) 2023-12-16T22:11:38.815 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:11:38 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:11:38.505+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:10:42.731761+0000 (oldest deadline 2023-12-16T22:11:02.731761+0000) 2023-12-16T22:11:38.815 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:11:38 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:11:38.505+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:11:38.815 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:11:38 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:11:38.713+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:11:07.849373+0000 (oldest deadline 2023-12-16T22:11:27.849373+0000) 2023-12-16T22:11:38.815 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:11:38 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:11:38.713+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:11:07.849373+0000 (oldest deadline 2023-12-16T22:11:27.849373+0000) 2023-12-16T22:11:38.815 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:11:38 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:11:38.713+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:11:07.849373+0000 (oldest deadline 2023-12-16T22:11:27.849373+0000) 2023-12-16T22:11:38.815 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:11:38 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:11:38.715+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:11:39.314 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:11:38 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:11:38.870+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:10:43.026858+0000 (oldest deadline 2023-12-16T22:11:03.026858+0000) 2023-12-16T22:11:39.315 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:11:38 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:11:38.870+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:10:43.026858+0000 (oldest deadline 2023-12-16T22:11:03.026858+0000) 2023-12-16T22:11:39.315 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:11:38 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:11:38.870+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:10:43.026858+0000 (oldest deadline 2023-12-16T22:11:03.026858+0000) 2023-12-16T22:11:39.315 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:11:38 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:11:38.871+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:11:39.814 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:11:39 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:11:39.464+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:10:42.731761+0000 (oldest deadline 2023-12-16T22:11:02.731761+0000) 2023-12-16T22:11:39.815 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:11:39 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:11:39.464+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:10:42.731761+0000 (oldest deadline 2023-12-16T22:11:02.731761+0000) 2023-12-16T22:11:39.815 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:11:39 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:11:39.464+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:10:42.731761+0000 (oldest deadline 2023-12-16T22:11:02.731761+0000) 2023-12-16T22:11:39.815 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:11:39 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:11:39.465+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:11:39.815 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:11:39 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:11:39.671+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:11:07.849373+0000 (oldest deadline 2023-12-16T22:11:27.849373+0000) 2023-12-16T22:11:39.815 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:11:39 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:11:39.671+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:11:07.849373+0000 (oldest deadline 2023-12-16T22:11:27.849373+0000) 2023-12-16T22:11:39.816 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:11:39 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:11:39.671+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:11:07.849373+0000 (oldest deadline 2023-12-16T22:11:27.849373+0000) 2023-12-16T22:11:39.816 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:11:39 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:11:39.672+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:11:40.217 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:11:39 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:11:39.856+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:10:43.026858+0000 (oldest deadline 2023-12-16T22:11:03.026858+0000) 2023-12-16T22:11:40.217 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:11:39 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:11:39.856+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:10:43.026858+0000 (oldest deadline 2023-12-16T22:11:03.026858+0000) 2023-12-16T22:11:40.217 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:11:39 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:11:39.856+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:10:43.026858+0000 (oldest deadline 2023-12-16T22:11:03.026858+0000) 2023-12-16T22:11:40.221 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:11:39 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:11:39.857+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:11:40.564 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:11:40 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:11:40.422+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:10:42.731761+0000 (oldest deadline 2023-12-16T22:11:02.731761+0000) 2023-12-16T22:11:40.565 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:11:40 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:11:40.422+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:10:42.731761+0000 (oldest deadline 2023-12-16T22:11:02.731761+0000) 2023-12-16T22:11:40.565 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:11:40 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:11:40.422+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:10:42.731761+0000 (oldest deadline 2023-12-16T22:11:02.731761+0000) 2023-12-16T22:11:40.565 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:11:40 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:11:40.423+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:11:41.064 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:11:40 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:11:40.895+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:10:43.026858+0000 (oldest deadline 2023-12-16T22:11:03.026858+0000) 2023-12-16T22:11:41.064 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:11:40 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:11:40.895+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:10:43.026858+0000 (oldest deadline 2023-12-16T22:11:03.026858+0000) 2023-12-16T22:11:41.065 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:11:40 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:11:40.895+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:10:43.026858+0000 (oldest deadline 2023-12-16T22:11:03.026858+0000) 2023-12-16T22:11:41.065 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:11:40 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:11:40.896+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:11:41.065 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:11:40 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:11:40.695+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:11:07.849373+0000 (oldest deadline 2023-12-16T22:11:27.849373+0000) 2023-12-16T22:11:41.065 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:11:40 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:11:40.695+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:11:07.849373+0000 (oldest deadline 2023-12-16T22:11:27.849373+0000) 2023-12-16T22:11:41.065 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:11:40 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:11:40.695+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:11:07.849373+0000 (oldest deadline 2023-12-16T22:11:27.849373+0000) 2023-12-16T22:11:41.065 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:11:40 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:11:40.697+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:11:41.746 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:11:41 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:11:41.468+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:10:42.731761+0000 (oldest deadline 2023-12-16T22:11:02.731761+0000) 2023-12-16T22:11:41.746 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:11:41 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:11:41.468+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:10:42.731761+0000 (oldest deadline 2023-12-16T22:11:02.731761+0000) 2023-12-16T22:11:41.747 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:11:41 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:11:41.468+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:10:42.731761+0000 (oldest deadline 2023-12-16T22:11:02.731761+0000) 2023-12-16T22:11:41.747 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:11:41 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:11:41.469+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:11:41.747 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:11:41 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:11:41.680+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:11:07.849373+0000 (oldest deadline 2023-12-16T22:11:27.849373+0000) 2023-12-16T22:11:41.747 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:11:41 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:11:41.680+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:11:07.849373+0000 (oldest deadline 2023-12-16T22:11:27.849373+0000) 2023-12-16T22:11:41.747 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:11:41 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:11:41.680+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:11:07.849373+0000 (oldest deadline 2023-12-16T22:11:27.849373+0000) 2023-12-16T22:11:41.748 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:11:41 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:11:41.681+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:11:42.006 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:11:41 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:11:41.903+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:10:43.026858+0000 (oldest deadline 2023-12-16T22:11:03.026858+0000) 2023-12-16T22:11:42.006 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:11:41 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:11:41.903+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:10:43.026858+0000 (oldest deadline 2023-12-16T22:11:03.026858+0000) 2023-12-16T22:11:42.006 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:11:41 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:11:41.903+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:10:43.026858+0000 (oldest deadline 2023-12-16T22:11:03.026858+0000) 2023-12-16T22:11:42.006 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:11:41 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:11:41.903+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:11:42.314 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Dec 16 22:11:42 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-mon-smithi045[80759]: 2023-12-16T22:11:42.005+0000 7f45a1f8f700 -1 mon.smithi045@0(probing) e2 get_health_metrics reporting 32097 slow ops, oldest is monmgrreport(gid 14214, 0 checks, 1 progress events) 2023-12-16T22:11:42.815 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:11:42 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:11:42.502+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:10:42.731761+0000 (oldest deadline 2023-12-16T22:11:02.731761+0000) 2023-12-16T22:11:42.815 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:11:42 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:11:42.502+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:10:42.731761+0000 (oldest deadline 2023-12-16T22:11:02.731761+0000) 2023-12-16T22:11:42.815 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:11:42 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:11:42.502+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:10:42.731761+0000 (oldest deadline 2023-12-16T22:11:02.731761+0000) 2023-12-16T22:11:42.815 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:11:42 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:11:42.503+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:11:42.816 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:11:42 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:11:42.669+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:11:07.849373+0000 (oldest deadline 2023-12-16T22:11:27.849373+0000) 2023-12-16T22:11:42.816 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:11:42 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:11:42.669+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:11:07.849373+0000 (oldest deadline 2023-12-16T22:11:27.849373+0000) 2023-12-16T22:11:42.816 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:11:42 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:11:42.669+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:11:07.849373+0000 (oldest deadline 2023-12-16T22:11:27.849373+0000) 2023-12-16T22:11:42.816 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:11:42 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:11:42.670+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:11:43.314 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:11:42 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:11:42.902+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:10:43.026858+0000 (oldest deadline 2023-12-16T22:11:03.026858+0000) 2023-12-16T22:11:43.314 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:11:42 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:11:42.902+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:10:43.026858+0000 (oldest deadline 2023-12-16T22:11:03.026858+0000) 2023-12-16T22:11:43.315 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:11:42 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:11:42.902+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:10:43.026858+0000 (oldest deadline 2023-12-16T22:11:03.026858+0000) 2023-12-16T22:11:43.315 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:11:42 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:11:42.903+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:11:43.814 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:11:43 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:11:43.482+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:10:42.731761+0000 (oldest deadline 2023-12-16T22:11:02.731761+0000) 2023-12-16T22:11:43.815 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:11:43 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:11:43.482+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:10:42.731761+0000 (oldest deadline 2023-12-16T22:11:02.731761+0000) 2023-12-16T22:11:43.815 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:11:43 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:11:43.482+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:10:42.731761+0000 (oldest deadline 2023-12-16T22:11:02.731761+0000) 2023-12-16T22:11:43.815 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:11:43 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:11:43.483+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:11:43.815 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:11:43 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:11:43.688+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:11:07.849373+0000 (oldest deadline 2023-12-16T22:11:27.849373+0000) 2023-12-16T22:11:43.815 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:11:43 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:11:43.688+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:11:07.849373+0000 (oldest deadline 2023-12-16T22:11:27.849373+0000) 2023-12-16T22:11:43.816 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:11:43 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:11:43.688+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:11:07.849373+0000 (oldest deadline 2023-12-16T22:11:27.849373+0000) 2023-12-16T22:11:43.816 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:11:43 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:11:43.689+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:11:44.314 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:11:43 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:11:43.896+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:10:43.026858+0000 (oldest deadline 2023-12-16T22:11:03.026858+0000) 2023-12-16T22:11:44.314 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:11:43 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:11:43.896+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:10:43.026858+0000 (oldest deadline 2023-12-16T22:11:03.026858+0000) 2023-12-16T22:11:44.315 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:11:43 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:11:43.896+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:10:43.026858+0000 (oldest deadline 2023-12-16T22:11:03.026858+0000) 2023-12-16T22:11:44.315 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:11:43 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:11:43.897+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:11:44.814 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:11:44 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:11:44.446+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:10:42.731761+0000 (oldest deadline 2023-12-16T22:11:02.731761+0000) 2023-12-16T22:11:44.815 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:11:44 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:11:44.446+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:10:42.731761+0000 (oldest deadline 2023-12-16T22:11:02.731761+0000) 2023-12-16T22:11:44.815 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:11:44 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:11:44.446+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:10:42.731761+0000 (oldest deadline 2023-12-16T22:11:02.731761+0000) 2023-12-16T22:11:44.815 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:11:44 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:11:44.447+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:11:44.815 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:11:44 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:11:44.644+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:11:07.849373+0000 (oldest deadline 2023-12-16T22:11:27.849373+0000) 2023-12-16T22:11:44.815 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:11:44 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:11:44.644+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:11:07.849373+0000 (oldest deadline 2023-12-16T22:11:27.849373+0000) 2023-12-16T22:11:44.815 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:11:44 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:11:44.644+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:11:07.849373+0000 (oldest deadline 2023-12-16T22:11:27.849373+0000) 2023-12-16T22:11:44.816 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:11:44 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:11:44.645+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:11:45.314 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:11:44 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:11:44.849+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:10:43.026858+0000 (oldest deadline 2023-12-16T22:11:03.026858+0000) 2023-12-16T22:11:45.314 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:11:44 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:11:44.849+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:10:43.026858+0000 (oldest deadline 2023-12-16T22:11:03.026858+0000) 2023-12-16T22:11:45.315 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:11:44 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:11:44.849+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:10:43.026858+0000 (oldest deadline 2023-12-16T22:11:03.026858+0000) 2023-12-16T22:11:45.315 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:11:44 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:11:44.850+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:11:45.802 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:11:45 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:11:45.645+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:11:07.849373+0000 (oldest deadline 2023-12-16T22:11:27.849373+0000) 2023-12-16T22:11:45.802 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:11:45 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:11:45.645+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:11:07.849373+0000 (oldest deadline 2023-12-16T22:11:27.849373+0000) 2023-12-16T22:11:45.802 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:11:45 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:11:45.645+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:11:07.849373+0000 (oldest deadline 2023-12-16T22:11:27.849373+0000) 2023-12-16T22:11:45.802 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:11:45 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:11:45.646+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:11:45.802 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:11:45 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:11:45.440+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:10:42.731761+0000 (oldest deadline 2023-12-16T22:11:02.731761+0000) 2023-12-16T22:11:45.802 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:11:45 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:11:45.440+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:10:42.731761+0000 (oldest deadline 2023-12-16T22:11:02.731761+0000) 2023-12-16T22:11:45.803 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:11:45 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:11:45.440+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:10:42.731761+0000 (oldest deadline 2023-12-16T22:11:02.731761+0000) 2023-12-16T22:11:45.803 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:11:45 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:11:45.440+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:11:46.063 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:11:45 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:11:45.800+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:10:43.026858+0000 (oldest deadline 2023-12-16T22:11:03.026858+0000) 2023-12-16T22:11:46.063 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:11:45 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:11:45.800+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:10:43.026858+0000 (oldest deadline 2023-12-16T22:11:03.026858+0000) 2023-12-16T22:11:46.063 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:11:45 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:11:45.800+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:10:43.026858+0000 (oldest deadline 2023-12-16T22:11:03.026858+0000) 2023-12-16T22:11:46.063 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:11:45 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:11:45.802+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:11:46.677 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:11:46 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:11:46.415+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:10:42.731761+0000 (oldest deadline 2023-12-16T22:11:02.731761+0000) 2023-12-16T22:11:46.677 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:11:46 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:11:46.415+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:10:42.731761+0000 (oldest deadline 2023-12-16T22:11:02.731761+0000) 2023-12-16T22:11:46.677 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:11:46 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:11:46.415+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:10:42.731761+0000 (oldest deadline 2023-12-16T22:11:02.731761+0000) 2023-12-16T22:11:46.677 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:11:46 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:11:46.416+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:11:47.011 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:11:46 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:11:46.676+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:11:07.849373+0000 (oldest deadline 2023-12-16T22:11:27.849373+0000) 2023-12-16T22:11:47.012 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:11:46 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:11:46.676+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:11:07.849373+0000 (oldest deadline 2023-12-16T22:11:27.849373+0000) 2023-12-16T22:11:47.012 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:11:46 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:11:46.676+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:11:07.849373+0000 (oldest deadline 2023-12-16T22:11:27.849373+0000) 2023-12-16T22:11:47.012 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:11:46 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:11:46.676+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:11:47.012 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:11:46 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:11:46.758+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:10:43.026858+0000 (oldest deadline 2023-12-16T22:11:03.026858+0000) 2023-12-16T22:11:47.012 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:11:46 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:11:46.758+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:10:43.026858+0000 (oldest deadline 2023-12-16T22:11:03.026858+0000) 2023-12-16T22:11:47.013 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:11:46 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:11:46.758+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:10:43.026858+0000 (oldest deadline 2023-12-16T22:11:03.026858+0000) 2023-12-16T22:11:47.013 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:11:46 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:11:46.759+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:11:47.314 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Dec 16 22:11:47 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-mon-smithi045[80759]: 2023-12-16T22:11:47.009+0000 7f45a1f8f700 -1 mon.smithi045@0(probing) e2 get_health_metrics reporting 32187 slow ops, oldest is monmgrreport(gid 14214, 0 checks, 1 progress events) 2023-12-16T22:11:47.761 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:11:47 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:11:47.696+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:11:07.849373+0000 (oldest deadline 2023-12-16T22:11:27.849373+0000) 2023-12-16T22:11:47.761 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:11:47 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:11:47.696+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:11:07.849373+0000 (oldest deadline 2023-12-16T22:11:27.849373+0000) 2023-12-16T22:11:47.761 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:11:47 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:11:47.696+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:11:07.849373+0000 (oldest deadline 2023-12-16T22:11:27.849373+0000) 2023-12-16T22:11:47.761 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:11:47 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:11:47.696+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:11:47.762 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:11:47 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:11:47.463+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:10:42.731761+0000 (oldest deadline 2023-12-16T22:11:02.731761+0000) 2023-12-16T22:11:47.762 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:11:47 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:11:47.463+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:10:42.731761+0000 (oldest deadline 2023-12-16T22:11:02.731761+0000) 2023-12-16T22:11:47.762 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:11:47 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:11:47.463+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:10:42.731761+0000 (oldest deadline 2023-12-16T22:11:02.731761+0000) 2023-12-16T22:11:47.762 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:11:47 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:11:47.464+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:11:48.064 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:11:47 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:11:47.760+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:10:43.026858+0000 (oldest deadline 2023-12-16T22:11:03.026858+0000) 2023-12-16T22:11:48.064 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:11:47 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:11:47.760+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:10:43.026858+0000 (oldest deadline 2023-12-16T22:11:03.026858+0000) 2023-12-16T22:11:48.064 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:11:47 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:11:47.760+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:10:43.026858+0000 (oldest deadline 2023-12-16T22:11:03.026858+0000) 2023-12-16T22:11:48.065 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:11:47 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:11:47.761+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:11:48.673 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:11:48 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:11:48.422+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:10:42.731761+0000 (oldest deadline 2023-12-16T22:11:02.731761+0000) 2023-12-16T22:11:48.673 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:11:48 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:11:48.422+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:10:42.731761+0000 (oldest deadline 2023-12-16T22:11:02.731761+0000) 2023-12-16T22:11:48.673 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:11:48 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:11:48.422+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:10:42.731761+0000 (oldest deadline 2023-12-16T22:11:02.731761+0000) 2023-12-16T22:11:48.673 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:11:48 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:11:48.422+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:11:49.064 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:11:48 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:11:48.777+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:10:43.026858+0000 (oldest deadline 2023-12-16T22:11:03.026858+0000) 2023-12-16T22:11:49.065 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:11:48 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:11:48.777+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:10:43.026858+0000 (oldest deadline 2023-12-16T22:11:03.026858+0000) 2023-12-16T22:11:49.065 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:11:48 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:11:48.777+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:10:43.026858+0000 (oldest deadline 2023-12-16T22:11:03.026858+0000) 2023-12-16T22:11:49.065 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:11:48 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:11:48.778+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:11:49.065 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:11:48 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:11:48.672+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:11:07.849373+0000 (oldest deadline 2023-12-16T22:11:27.849373+0000) 2023-12-16T22:11:49.065 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:11:48 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:11:48.672+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:11:07.849373+0000 (oldest deadline 2023-12-16T22:11:27.849373+0000) 2023-12-16T22:11:49.066 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:11:48 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:11:48.672+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:11:07.849373+0000 (oldest deadline 2023-12-16T22:11:27.849373+0000) 2023-12-16T22:11:49.066 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:11:48 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:11:48.672+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:11:49.796 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:11:49 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:11:49.383+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:10:42.731761+0000 (oldest deadline 2023-12-16T22:11:02.731761+0000) 2023-12-16T22:11:49.796 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:11:49 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:11:49.383+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:10:42.731761+0000 (oldest deadline 2023-12-16T22:11:02.731761+0000) 2023-12-16T22:11:49.796 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:11:49 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:11:49.383+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:10:42.731761+0000 (oldest deadline 2023-12-16T22:11:02.731761+0000) 2023-12-16T22:11:49.797 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:11:49 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:11:49.384+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:11:49.797 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:11:49 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:11:49.623+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:11:07.849373+0000 (oldest deadline 2023-12-16T22:11:27.849373+0000) 2023-12-16T22:11:49.797 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:11:49 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:11:49.623+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:11:07.849373+0000 (oldest deadline 2023-12-16T22:11:27.849373+0000) 2023-12-16T22:11:49.797 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:11:49 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:11:49.624+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:11:07.849373+0000 (oldest deadline 2023-12-16T22:11:27.849373+0000) 2023-12-16T22:11:49.797 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:11:49 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:11:49.625+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:11:50.064 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:11:49 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:11:49.795+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:10:43.026858+0000 (oldest deadline 2023-12-16T22:11:03.026858+0000) 2023-12-16T22:11:50.064 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:11:49 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:11:49.795+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:10:43.026858+0000 (oldest deadline 2023-12-16T22:11:03.026858+0000) 2023-12-16T22:11:50.065 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:11:49 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:11:49.795+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:10:43.026858+0000 (oldest deadline 2023-12-16T22:11:03.026858+0000) 2023-12-16T22:11:50.065 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:11:49 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:11:49.796+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:11:50.564 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:11:50 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:11:50.432+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:10:42.731761+0000 (oldest deadline 2023-12-16T22:11:02.731761+0000) 2023-12-16T22:11:50.564 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:11:50 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:11:50.432+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:10:42.731761+0000 (oldest deadline 2023-12-16T22:11:02.731761+0000) 2023-12-16T22:11:50.565 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:11:50 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:11:50.432+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:10:42.731761+0000 (oldest deadline 2023-12-16T22:11:02.731761+0000) 2023-12-16T22:11:50.565 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:11:50 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:11:50.433+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:11:51.064 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:11:50 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:11:50.788+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:10:43.026858+0000 (oldest deadline 2023-12-16T22:11:03.026858+0000) 2023-12-16T22:11:51.064 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:11:50 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:11:50.788+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:10:43.026858+0000 (oldest deadline 2023-12-16T22:11:03.026858+0000) 2023-12-16T22:11:51.065 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:11:50 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:11:50.788+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:10:43.026858+0000 (oldest deadline 2023-12-16T22:11:03.026858+0000) 2023-12-16T22:11:51.065 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:11:50 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:11:50.788+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:11:51.065 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:11:50 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:11:50.664+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:11:07.849373+0000 (oldest deadline 2023-12-16T22:11:27.849373+0000) 2023-12-16T22:11:51.065 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:11:50 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:11:50.664+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:11:07.849373+0000 (oldest deadline 2023-12-16T22:11:27.849373+0000) 2023-12-16T22:11:51.065 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:11:50 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:11:50.664+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:11:07.849373+0000 (oldest deadline 2023-12-16T22:11:27.849373+0000) 2023-12-16T22:11:51.065 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:11:50 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:11:50.665+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:11:51.748 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:11:51 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:11:51.426+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:10:42.731761+0000 (oldest deadline 2023-12-16T22:11:02.731761+0000) 2023-12-16T22:11:51.749 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:11:51 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:11:51.426+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:10:42.731761+0000 (oldest deadline 2023-12-16T22:11:02.731761+0000) 2023-12-16T22:11:51.749 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:11:51 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:11:51.426+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:10:42.731761+0000 (oldest deadline 2023-12-16T22:11:02.731761+0000) 2023-12-16T22:11:51.749 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:11:51 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:11:51.427+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:11:51.749 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:11:51 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:11:51.652+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:11:07.849373+0000 (oldest deadline 2023-12-16T22:11:27.849373+0000) 2023-12-16T22:11:51.749 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:11:51 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:11:51.652+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:11:07.849373+0000 (oldest deadline 2023-12-16T22:11:27.849373+0000) 2023-12-16T22:11:51.750 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:11:51 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:11:51.652+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:11:07.849373+0000 (oldest deadline 2023-12-16T22:11:27.849373+0000) 2023-12-16T22:11:51.750 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:11:51 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:11:51.653+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:11:52.015 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:11:51 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:11:51.748+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:10:43.026858+0000 (oldest deadline 2023-12-16T22:11:03.026858+0000) 2023-12-16T22:11:52.015 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:11:51 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:11:51.748+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:10:43.026858+0000 (oldest deadline 2023-12-16T22:11:03.026858+0000) 2023-12-16T22:11:52.015 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:11:51 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:11:51.748+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:10:43.026858+0000 (oldest deadline 2023-12-16T22:11:03.026858+0000) 2023-12-16T22:11:52.015 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:11:51 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:11:51.749+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:11:52.314 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Dec 16 22:11:52 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-mon-smithi045[80759]: 2023-12-16T22:11:52.014+0000 7f45a1f8f700 -1 mon.smithi045@0(probing) e2 get_health_metrics reporting 32280 slow ops, oldest is monmgrreport(gid 14214, 0 checks, 1 progress events) 2023-12-16T22:11:52.780 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:11:52 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:11:52.418+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:10:42.731761+0000 (oldest deadline 2023-12-16T22:11:02.731761+0000) 2023-12-16T22:11:52.781 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:11:52 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:11:52.418+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:10:42.731761+0000 (oldest deadline 2023-12-16T22:11:02.731761+0000) 2023-12-16T22:11:52.781 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:11:52 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:11:52.418+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:10:42.731761+0000 (oldest deadline 2023-12-16T22:11:02.731761+0000) 2023-12-16T22:11:52.781 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:11:52 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:11:52.418+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:11:52.781 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:11:52 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:11:52.651+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:11:07.849373+0000 (oldest deadline 2023-12-16T22:11:27.849373+0000) 2023-12-16T22:11:52.781 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:11:52 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:11:52.651+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:11:07.849373+0000 (oldest deadline 2023-12-16T22:11:27.849373+0000) 2023-12-16T22:11:52.781 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:11:52 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:11:52.651+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:11:07.849373+0000 (oldest deadline 2023-12-16T22:11:27.849373+0000) 2023-12-16T22:11:52.782 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:11:52 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:11:52.652+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:11:53.064 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:11:52 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:11:52.779+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:10:43.026858+0000 (oldest deadline 2023-12-16T22:11:03.026858+0000) 2023-12-16T22:11:53.064 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:11:52 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:11:52.779+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:10:43.026858+0000 (oldest deadline 2023-12-16T22:11:03.026858+0000) 2023-12-16T22:11:53.065 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:11:52 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:11:52.779+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:10:43.026858+0000 (oldest deadline 2023-12-16T22:11:03.026858+0000) 2023-12-16T22:11:53.065 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:11:52 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:11:52.780+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:11:53.776 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:11:53 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:11:53.632+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:11:07.849373+0000 (oldest deadline 2023-12-16T22:11:27.849373+0000) 2023-12-16T22:11:53.776 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:11:53 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:11:53.632+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:11:07.849373+0000 (oldest deadline 2023-12-16T22:11:27.849373+0000) 2023-12-16T22:11:53.776 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:11:53 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:11:53.632+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:11:07.849373+0000 (oldest deadline 2023-12-16T22:11:27.849373+0000) 2023-12-16T22:11:53.776 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:11:53 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:11:53.633+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:11:53.777 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:11:53 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:11:53.388+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:10:42.731761+0000 (oldest deadline 2023-12-16T22:11:02.731761+0000) 2023-12-16T22:11:53.777 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:11:53 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:11:53.388+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:10:42.731761+0000 (oldest deadline 2023-12-16T22:11:02.731761+0000) 2023-12-16T22:11:53.777 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:11:53 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:11:53.388+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:10:42.731761+0000 (oldest deadline 2023-12-16T22:11:02.731761+0000) 2023-12-16T22:11:53.777 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:11:53 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:11:53.389+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:11:54.064 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:11:53 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:11:53.775+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:10:43.026858+0000 (oldest deadline 2023-12-16T22:11:03.026858+0000) 2023-12-16T22:11:54.065 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:11:53 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:11:53.775+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:10:43.026858+0000 (oldest deadline 2023-12-16T22:11:03.026858+0000) 2023-12-16T22:11:54.065 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:11:53 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:11:53.775+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:10:43.026858+0000 (oldest deadline 2023-12-16T22:11:03.026858+0000) 2023-12-16T22:11:54.065 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:11:53 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:11:53.776+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:11:54.756 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:11:54 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:11:54.597+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:11:07.849373+0000 (oldest deadline 2023-12-16T22:11:27.849373+0000) 2023-12-16T22:11:54.756 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:11:54 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:11:54.597+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:11:07.849373+0000 (oldest deadline 2023-12-16T22:11:27.849373+0000) 2023-12-16T22:11:54.756 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:11:54 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:11:54.597+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:11:07.849373+0000 (oldest deadline 2023-12-16T22:11:27.849373+0000) 2023-12-16T22:11:54.756 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:11:54 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:11:54.598+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:11:54.757 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:11:54 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:11:54.353+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:10:42.731761+0000 (oldest deadline 2023-12-16T22:11:02.731761+0000) 2023-12-16T22:11:54.757 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:11:54 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:11:54.353+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:10:42.731761+0000 (oldest deadline 2023-12-16T22:11:02.731761+0000) 2023-12-16T22:11:54.757 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:11:54 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:11:54.353+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:10:42.731761+0000 (oldest deadline 2023-12-16T22:11:02.731761+0000) 2023-12-16T22:11:54.757 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:11:54 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:11:54.354+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:11:55.064 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:11:54 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:11:54.755+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:10:43.026858+0000 (oldest deadline 2023-12-16T22:11:03.026858+0000) 2023-12-16T22:11:55.064 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:11:54 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:11:54.755+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:10:43.026858+0000 (oldest deadline 2023-12-16T22:11:03.026858+0000) 2023-12-16T22:11:55.064 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:11:54 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:11:54.755+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:10:43.026858+0000 (oldest deadline 2023-12-16T22:11:03.026858+0000) 2023-12-16T22:11:55.065 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:11:54 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:11:54.755+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:11:55.561 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:11:55 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:11:55.306+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:10:42.731761+0000 (oldest deadline 2023-12-16T22:11:02.731761+0000) 2023-12-16T22:11:55.561 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:11:55 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:11:55.306+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:10:42.731761+0000 (oldest deadline 2023-12-16T22:11:02.731761+0000) 2023-12-16T22:11:55.561 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:11:55 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:11:55.306+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:10:42.731761+0000 (oldest deadline 2023-12-16T22:11:02.731761+0000) 2023-12-16T22:11:55.561 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:11:55 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:11:55.307+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:11:55.814 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:11:55 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:11:55.737+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:10:43.026858+0000 (oldest deadline 2023-12-16T22:11:03.026858+0000) 2023-12-16T22:11:55.814 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:11:55 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:11:55.737+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:10:43.026858+0000 (oldest deadline 2023-12-16T22:11:03.026858+0000) 2023-12-16T22:11:55.815 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:11:55 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:11:55.737+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:10:43.026858+0000 (oldest deadline 2023-12-16T22:11:03.026858+0000) 2023-12-16T22:11:55.815 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:11:55 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:11:55.738+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:11:55.815 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:11:55 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:11:55.560+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:11:07.849373+0000 (oldest deadline 2023-12-16T22:11:27.849373+0000) 2023-12-16T22:11:55.815 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:11:55 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:11:55.560+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:11:07.849373+0000 (oldest deadline 2023-12-16T22:11:27.849373+0000) 2023-12-16T22:11:55.815 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:11:55 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:11:55.560+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:11:07.849373+0000 (oldest deadline 2023-12-16T22:11:27.849373+0000) 2023-12-16T22:11:55.815 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:11:55 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:11:55.561+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:11:56.564 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:11:56 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:11:56.281+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:10:42.731761+0000 (oldest deadline 2023-12-16T22:11:02.731761+0000) 2023-12-16T22:11:56.564 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:11:56 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:11:56.281+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:10:42.731761+0000 (oldest deadline 2023-12-16T22:11:02.731761+0000) 2023-12-16T22:11:56.564 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:11:56 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:11:56.281+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:10:42.731761+0000 (oldest deadline 2023-12-16T22:11:02.731761+0000) 2023-12-16T22:11:56.565 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:11:56 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:11:56.282+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:11:57.020 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:11:56 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:11:56.733+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:10:43.026858+0000 (oldest deadline 2023-12-16T22:11:03.026858+0000) 2023-12-16T22:11:57.020 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:11:56 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:11:56.733+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:10:43.026858+0000 (oldest deadline 2023-12-16T22:11:03.026858+0000) 2023-12-16T22:11:57.020 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:11:56 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:11:56.733+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:10:43.026858+0000 (oldest deadline 2023-12-16T22:11:03.026858+0000) 2023-12-16T22:11:57.020 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:11:56 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:11:56.734+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:11:57.020 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:11:56 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:11:56.602+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:11:07.849373+0000 (oldest deadline 2023-12-16T22:11:27.849373+0000) 2023-12-16T22:11:57.020 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:11:56 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:11:56.602+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:11:07.849373+0000 (oldest deadline 2023-12-16T22:11:27.849373+0000) 2023-12-16T22:11:57.021 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:11:56 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:11:56.602+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:11:07.849373+0000 (oldest deadline 2023-12-16T22:11:27.849373+0000) 2023-12-16T22:11:57.021 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:11:56 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:11:56.603+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:11:57.272 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Dec 16 22:11:57 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-mon-smithi045[80759]: 2023-12-16T22:11:57.018+0000 7f45a1f8f700 -1 mon.smithi045@0(probing) e2 get_health_metrics reporting 32366 slow ops, oldest is monmgrreport(gid 14214, 0 checks, 1 progress events) 2023-12-16T22:11:57.564 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:11:57 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:11:57.271+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:10:42.731761+0000 (oldest deadline 2023-12-16T22:11:02.731761+0000) 2023-12-16T22:11:57.564 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:11:57 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:11:57.271+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:10:42.731761+0000 (oldest deadline 2023-12-16T22:11:02.731761+0000) 2023-12-16T22:11:57.565 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:11:57 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:11:57.271+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:10:42.731761+0000 (oldest deadline 2023-12-16T22:11:02.731761+0000) 2023-12-16T22:11:57.565 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:11:57 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:11:57.272+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:11:58.064 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:11:57 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:11:57.621+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:11:07.849373+0000 (oldest deadline 2023-12-16T22:11:27.849373+0000) 2023-12-16T22:11:58.065 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:11:57 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:11:57.621+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:11:07.849373+0000 (oldest deadline 2023-12-16T22:11:27.849373+0000) 2023-12-16T22:11:58.065 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:11:57 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:11:57.621+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:11:07.849373+0000 (oldest deadline 2023-12-16T22:11:27.849373+0000) 2023-12-16T22:11:58.065 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:11:57 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:11:57.622+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:11:58.065 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:11:57 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:11:57.719+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:10:43.026858+0000 (oldest deadline 2023-12-16T22:11:03.026858+0000) 2023-12-16T22:11:58.065 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:11:57 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:11:57.719+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:10:43.026858+0000 (oldest deadline 2023-12-16T22:11:03.026858+0000) 2023-12-16T22:11:58.065 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:11:57 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:11:57.719+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:10:43.026858+0000 (oldest deadline 2023-12-16T22:11:03.026858+0000) 2023-12-16T22:11:58.065 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:11:57 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:11:57.720+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:11:58.564 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:11:58 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:11:58.299+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:10:42.731761+0000 (oldest deadline 2023-12-16T22:11:02.731761+0000) 2023-12-16T22:11:58.564 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:11:58 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:11:58.299+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:10:42.731761+0000 (oldest deadline 2023-12-16T22:11:02.731761+0000) 2023-12-16T22:11:58.564 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:11:58 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:11:58.299+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:10:42.731761+0000 (oldest deadline 2023-12-16T22:11:02.731761+0000) 2023-12-16T22:11:58.565 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:11:58 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:11:58.300+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:11:59.064 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:11:58 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:11:58.752+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:10:43.026858+0000 (oldest deadline 2023-12-16T22:11:03.026858+0000) 2023-12-16T22:11:59.064 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:11:58 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:11:58.752+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:10:43.026858+0000 (oldest deadline 2023-12-16T22:11:03.026858+0000) 2023-12-16T22:11:59.065 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:11:58 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:11:58.752+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:10:43.026858+0000 (oldest deadline 2023-12-16T22:11:03.026858+0000) 2023-12-16T22:11:59.065 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:11:58 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:11:58.752+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:11:59.065 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:11:58 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:11:58.639+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:11:07.849373+0000 (oldest deadline 2023-12-16T22:11:27.849373+0000) 2023-12-16T22:11:59.065 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:11:58 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:11:58.639+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:11:07.849373+0000 (oldest deadline 2023-12-16T22:11:27.849373+0000) 2023-12-16T22:11:59.065 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:11:58 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:11:58.639+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:11:07.849373+0000 (oldest deadline 2023-12-16T22:11:27.849373+0000) 2023-12-16T22:11:59.065 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:11:58 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:11:58.640+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:11:59.564 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:11:59 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:11:59.264+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:10:42.731761+0000 (oldest deadline 2023-12-16T22:11:02.731761+0000) 2023-12-16T22:11:59.564 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:11:59 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:11:59.264+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:10:42.731761+0000 (oldest deadline 2023-12-16T22:11:02.731761+0000) 2023-12-16T22:11:59.565 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:11:59 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:11:59.264+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:10:42.731761+0000 (oldest deadline 2023-12-16T22:11:02.731761+0000) 2023-12-16T22:11:59.565 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:11:59 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:11:59.266+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:12:00.065 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:11:59 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:11:59.746+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:10:43.026858+0000 (oldest deadline 2023-12-16T22:11:03.026858+0000) 2023-12-16T22:12:00.066 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:11:59 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:11:59.746+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:10:43.026858+0000 (oldest deadline 2023-12-16T22:11:03.026858+0000) 2023-12-16T22:12:00.066 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:11:59 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:11:59.746+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:10:43.026858+0000 (oldest deadline 2023-12-16T22:11:03.026858+0000) 2023-12-16T22:12:00.066 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:11:59 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:11:59.746+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:12:00.066 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:11:59 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:11:59.656+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:11:07.849373+0000 (oldest deadline 2023-12-16T22:11:27.849373+0000) 2023-12-16T22:12:00.066 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:11:59 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:11:59.656+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:11:07.849373+0000 (oldest deadline 2023-12-16T22:11:27.849373+0000) 2023-12-16T22:12:00.067 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:11:59 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:11:59.656+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:11:07.849373+0000 (oldest deadline 2023-12-16T22:11:27.849373+0000) 2023-12-16T22:12:00.067 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:11:59 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:11:59.657+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:12:00.564 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:12:00 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:12:00.272+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:10:42.731761+0000 (oldest deadline 2023-12-16T22:11:02.731761+0000) 2023-12-16T22:12:00.565 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:12:00 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:12:00.272+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:10:42.731761+0000 (oldest deadline 2023-12-16T22:11:02.731761+0000) 2023-12-16T22:12:00.565 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:12:00 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:12:00.272+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:10:42.731761+0000 (oldest deadline 2023-12-16T22:11:02.731761+0000) 2023-12-16T22:12:00.565 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:12:00 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:12:00.272+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:12:01.064 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:12:00 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:12:00.780+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:10:43.026858+0000 (oldest deadline 2023-12-16T22:11:03.026858+0000) 2023-12-16T22:12:01.064 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:12:00 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:12:00.780+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:10:43.026858+0000 (oldest deadline 2023-12-16T22:11:03.026858+0000) 2023-12-16T22:12:01.065 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:12:00 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:12:00.780+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:10:43.026858+0000 (oldest deadline 2023-12-16T22:11:03.026858+0000) 2023-12-16T22:12:01.065 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:12:00 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:12:00.781+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:12:01.065 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:12:00 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:12:00.674+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:11:07.849373+0000 (oldest deadline 2023-12-16T22:11:27.849373+0000) 2023-12-16T22:12:01.065 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:12:00 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:12:00.674+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:11:07.849373+0000 (oldest deadline 2023-12-16T22:11:27.849373+0000) 2023-12-16T22:12:01.065 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:12:00 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:12:00.674+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:11:07.849373+0000 (oldest deadline 2023-12-16T22:11:27.849373+0000) 2023-12-16T22:12:01.065 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:12:00 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:12:00.675+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:12:01.675 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:12:01 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:12:01.319+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:10:42.731761+0000 (oldest deadline 2023-12-16T22:11:02.731761+0000) 2023-12-16T22:12:01.675 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:12:01 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:12:01.319+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:10:42.731761+0000 (oldest deadline 2023-12-16T22:11:02.731761+0000) 2023-12-16T22:12:01.675 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:12:01 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:12:01.319+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:10:42.731761+0000 (oldest deadline 2023-12-16T22:11:02.731761+0000) 2023-12-16T22:12:01.675 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:12:01 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:12:01.320+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:12:02.024 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:12:01 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:12:01.674+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:11:07.849373+0000 (oldest deadline 2023-12-16T22:11:27.849373+0000) 2023-12-16T22:12:02.025 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:12:01 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:12:01.674+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:11:07.849373+0000 (oldest deadline 2023-12-16T22:11:27.849373+0000) 2023-12-16T22:12:02.025 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:12:01 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:12:01.674+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:11:07.849373+0000 (oldest deadline 2023-12-16T22:11:27.849373+0000) 2023-12-16T22:12:02.025 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:12:01 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:12:01.674+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:12:02.025 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:12:01 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:12:01.813+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:10:43.026858+0000 (oldest deadline 2023-12-16T22:11:03.026858+0000) 2023-12-16T22:12:02.026 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:12:01 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:12:01.813+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:10:43.026858+0000 (oldest deadline 2023-12-16T22:11:03.026858+0000) 2023-12-16T22:12:02.026 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:12:01 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:12:01.813+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:10:43.026858+0000 (oldest deadline 2023-12-16T22:11:03.026858+0000) 2023-12-16T22:12:02.026 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:12:01 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:12:01.814+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:12:02.314 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Dec 16 22:12:02 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-mon-smithi045[80759]: 2023-12-16T22:12:02.023+0000 7f45a1f8f700 -1 mon.smithi045@0(probing) e2 get_health_metrics reporting 32456 slow ops, oldest is monmgrreport(gid 14214, 0 checks, 1 progress events) 2023-12-16T22:12:02.684 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:12:02 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:12:02.338+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:10:42.731761+0000 (oldest deadline 2023-12-16T22:11:02.731761+0000) 2023-12-16T22:12:02.684 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:12:02 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:12:02.338+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:10:42.731761+0000 (oldest deadline 2023-12-16T22:11:02.731761+0000) 2023-12-16T22:12:02.684 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:12:02 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:12:02.338+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:10:42.731761+0000 (oldest deadline 2023-12-16T22:11:02.731761+0000) 2023-12-16T22:12:02.684 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:12:02 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:12:02.339+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:12:03.064 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:12:02 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:12:02.824+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:10:43.026858+0000 (oldest deadline 2023-12-16T22:11:03.026858+0000) 2023-12-16T22:12:03.065 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:12:02 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:12:02.824+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:10:43.026858+0000 (oldest deadline 2023-12-16T22:11:03.026858+0000) 2023-12-16T22:12:03.065 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:12:02 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:12:02.824+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:10:43.026858+0000 (oldest deadline 2023-12-16T22:11:03.026858+0000) 2023-12-16T22:12:03.065 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:12:02 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:12:02.825+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:12:03.065 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:12:02 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:12:02.682+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:11:07.849373+0000 (oldest deadline 2023-12-16T22:11:27.849373+0000) 2023-12-16T22:12:03.065 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:12:02 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:12:02.682+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:11:07.849373+0000 (oldest deadline 2023-12-16T22:11:27.849373+0000) 2023-12-16T22:12:03.065 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:12:02 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:12:02.682+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:11:07.849373+0000 (oldest deadline 2023-12-16T22:11:27.849373+0000) 2023-12-16T22:12:03.065 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:12:02 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:12:02.683+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:12:03.640 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:12:03 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:12:03.387+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:10:42.731761+0000 (oldest deadline 2023-12-16T22:11:02.731761+0000) 2023-12-16T22:12:03.640 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:12:03 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:12:03.387+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:10:42.731761+0000 (oldest deadline 2023-12-16T22:11:02.731761+0000) 2023-12-16T22:12:03.641 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:12:03 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:12:03.387+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:10:42.731761+0000 (oldest deadline 2023-12-16T22:11:02.731761+0000) 2023-12-16T22:12:03.641 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:12:03 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:12:03.388+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:12:04.064 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:12:03 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:12:03.810+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:10:43.026858+0000 (oldest deadline 2023-12-16T22:11:03.026858+0000) 2023-12-16T22:12:04.065 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:12:03 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:12:03.810+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:10:43.026858+0000 (oldest deadline 2023-12-16T22:11:03.026858+0000) 2023-12-16T22:12:04.065 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:12:03 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:12:03.810+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:10:43.026858+0000 (oldest deadline 2023-12-16T22:11:03.026858+0000) 2023-12-16T22:12:04.065 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:12:03 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:12:03.810+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:12:04.065 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:12:03 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:12:03.717+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:11:07.849373+0000 (oldest deadline 2023-12-16T22:11:27.849373+0000) 2023-12-16T22:12:04.065 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:12:03 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:12:03.717+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:11:07.849373+0000 (oldest deadline 2023-12-16T22:11:27.849373+0000) 2023-12-16T22:12:04.065 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:12:03 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:12:03.717+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:11:07.849373+0000 (oldest deadline 2023-12-16T22:11:27.849373+0000) 2023-12-16T22:12:04.066 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:12:03 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:12:03.718+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:12:04.737 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:12:04 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:12:04.409+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:10:42.731761+0000 (oldest deadline 2023-12-16T22:11:02.731761+0000) 2023-12-16T22:12:04.738 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:12:04 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:12:04.409+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:10:42.731761+0000 (oldest deadline 2023-12-16T22:11:02.731761+0000) 2023-12-16T22:12:04.738 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:12:04 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:12:04.409+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:10:42.731761+0000 (oldest deadline 2023-12-16T22:11:02.731761+0000) 2023-12-16T22:12:04.738 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:12:04 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:12:04.410+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:12:05.064 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:12:04 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:12:04.736+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:11:07.849373+0000 (oldest deadline 2023-12-16T22:11:27.849373+0000) 2023-12-16T22:12:05.064 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:12:04 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:12:04.736+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:11:07.849373+0000 (oldest deadline 2023-12-16T22:11:27.849373+0000) 2023-12-16T22:12:05.064 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:12:04 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:12:04.736+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:11:07.849373+0000 (oldest deadline 2023-12-16T22:11:27.849373+0000) 2023-12-16T22:12:05.065 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:12:04 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:12:04.737+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:12:05.065 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:12:04 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:12:04.774+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:10:43.026858+0000 (oldest deadline 2023-12-16T22:11:03.026858+0000) 2023-12-16T22:12:05.065 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:12:04 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:12:04.774+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:10:43.026858+0000 (oldest deadline 2023-12-16T22:11:03.026858+0000) 2023-12-16T22:12:05.065 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:12:04 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:12:04.774+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:10:43.026858+0000 (oldest deadline 2023-12-16T22:11:03.026858+0000) 2023-12-16T22:12:05.065 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:12:04 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:12:04.775+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:12:05.736 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:12:05 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:12:05.407+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:10:42.731761+0000 (oldest deadline 2023-12-16T22:11:02.731761+0000) 2023-12-16T22:12:05.737 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:12:05 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:12:05.407+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:10:42.731761+0000 (oldest deadline 2023-12-16T22:11:02.731761+0000) 2023-12-16T22:12:05.737 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:12:05 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:12:05.407+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:10:42.731761+0000 (oldest deadline 2023-12-16T22:11:02.731761+0000) 2023-12-16T22:12:05.737 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:12:05 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:12:05.408+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:12:06.065 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:12:05 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:12:05.751+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:11:07.849373+0000 (oldest deadline 2023-12-16T22:11:27.849373+0000) 2023-12-16T22:12:06.066 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:12:05 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:12:05.751+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:11:07.849373+0000 (oldest deadline 2023-12-16T22:11:27.849373+0000) 2023-12-16T22:12:06.066 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:12:05 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:12:05.751+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:11:07.849373+0000 (oldest deadline 2023-12-16T22:11:27.849373+0000) 2023-12-16T22:12:06.066 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:12:05 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:12:05.752+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:12:06.066 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:12:05 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:12:05.735+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:10:43.026858+0000 (oldest deadline 2023-12-16T22:11:03.026858+0000) 2023-12-16T22:12:06.066 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:12:05 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:12:05.735+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:10:43.026858+0000 (oldest deadline 2023-12-16T22:11:03.026858+0000) 2023-12-16T22:12:06.066 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:12:05 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:12:05.735+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:10:43.026858+0000 (oldest deadline 2023-12-16T22:11:03.026858+0000) 2023-12-16T22:12:06.067 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:12:05 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:12:05.736+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:12:06.702 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:12:06 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:12:06.383+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:10:42.731761+0000 (oldest deadline 2023-12-16T22:11:02.731761+0000) 2023-12-16T22:12:06.702 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:12:06 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:12:06.383+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:10:42.731761+0000 (oldest deadline 2023-12-16T22:11:02.731761+0000) 2023-12-16T22:12:06.702 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:12:06 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:12:06.383+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:10:42.731761+0000 (oldest deadline 2023-12-16T22:11:02.731761+0000) 2023-12-16T22:12:06.703 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:12:06 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:12:06.384+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:12:07.028 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:12:06 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:12:06.794+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:11:07.849373+0000 (oldest deadline 2023-12-16T22:11:27.849373+0000) 2023-12-16T22:12:07.028 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:12:06 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:12:06.794+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:11:07.849373+0000 (oldest deadline 2023-12-16T22:11:27.849373+0000) 2023-12-16T22:12:07.028 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:12:06 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:12:06.794+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:11:07.849373+0000 (oldest deadline 2023-12-16T22:11:27.849373+0000) 2023-12-16T22:12:07.029 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:12:06 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:12:06.794+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:12:07.029 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:12:06 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:12:06.701+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:10:43.026858+0000 (oldest deadline 2023-12-16T22:11:03.026858+0000) 2023-12-16T22:12:07.029 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:12:06 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:12:06.701+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:10:43.026858+0000 (oldest deadline 2023-12-16T22:11:03.026858+0000) 2023-12-16T22:12:07.029 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:12:06 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:12:06.701+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:10:43.026858+0000 (oldest deadline 2023-12-16T22:11:03.026858+0000) 2023-12-16T22:12:07.029 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:12:06 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:12:06.702+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:12:07.314 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Dec 16 22:12:07 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-mon-smithi045[80759]: 2023-12-16T22:12:07.027+0000 7f45a1f8f700 -1 mon.smithi045@0(probing) e2 get_health_metrics reporting 32547 slow ops, oldest is monmgrreport(gid 14214, 0 checks, 1 progress events) 2023-12-16T22:12:07.694 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:12:07 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:12:07.375+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:10:42.731761+0000 (oldest deadline 2023-12-16T22:11:02.731761+0000) 2023-12-16T22:12:07.694 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:12:07 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:12:07.375+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:10:42.731761+0000 (oldest deadline 2023-12-16T22:11:02.731761+0000) 2023-12-16T22:12:07.694 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:12:07 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:12:07.375+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:10:42.731761+0000 (oldest deadline 2023-12-16T22:11:02.731761+0000) 2023-12-16T22:12:07.694 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:12:07 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:12:07.376+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:12:08.064 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:12:07 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:12:07.826+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:11:07.849373+0000 (oldest deadline 2023-12-16T22:11:27.849373+0000) 2023-12-16T22:12:08.065 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:12:07 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:12:07.826+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:11:07.849373+0000 (oldest deadline 2023-12-16T22:11:27.849373+0000) 2023-12-16T22:12:08.065 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:12:07 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:12:07.826+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:11:07.849373+0000 (oldest deadline 2023-12-16T22:11:27.849373+0000) 2023-12-16T22:12:08.065 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:12:07 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:12:07.826+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:12:08.065 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:12:07 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:12:07.693+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:10:43.026858+0000 (oldest deadline 2023-12-16T22:11:03.026858+0000) 2023-12-16T22:12:08.065 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:12:07 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:12:07.693+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:10:43.026858+0000 (oldest deadline 2023-12-16T22:11:03.026858+0000) 2023-12-16T22:12:08.066 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:12:07 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:12:07.693+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:10:43.026858+0000 (oldest deadline 2023-12-16T22:11:03.026858+0000) 2023-12-16T22:12:08.066 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:12:07 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:12:07.694+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:12:08.718 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:12:08 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:12:08.376+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:10:42.731761+0000 (oldest deadline 2023-12-16T22:11:02.731761+0000) 2023-12-16T22:12:08.718 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:12:08 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:12:08.376+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:10:42.731761+0000 (oldest deadline 2023-12-16T22:11:02.731761+0000) 2023-12-16T22:12:08.718 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:12:08 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:12:08.376+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:10:42.731761+0000 (oldest deadline 2023-12-16T22:11:02.731761+0000) 2023-12-16T22:12:08.718 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:12:08 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:12:08.376+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:12:09.064 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:12:08 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:12:08.782+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:11:07.849373+0000 (oldest deadline 2023-12-16T22:11:27.849373+0000) 2023-12-16T22:12:09.065 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:12:08 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:12:08.782+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:11:07.849373+0000 (oldest deadline 2023-12-16T22:11:27.849373+0000) 2023-12-16T22:12:09.065 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:12:08 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:12:08.782+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:11:07.849373+0000 (oldest deadline 2023-12-16T22:11:27.849373+0000) 2023-12-16T22:12:09.065 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:12:08 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:12:08.782+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:12:09.065 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:12:08 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:12:08.717+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:10:43.026858+0000 (oldest deadline 2023-12-16T22:11:03.026858+0000) 2023-12-16T22:12:09.065 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:12:08 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:12:08.717+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:10:43.026858+0000 (oldest deadline 2023-12-16T22:11:03.026858+0000) 2023-12-16T22:12:09.066 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:12:08 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:12:08.717+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:10:43.026858+0000 (oldest deadline 2023-12-16T22:11:03.026858+0000) 2023-12-16T22:12:09.066 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:12:08 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:12:08.718+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:12:09.729 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:12:09 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:12:09.398+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:10:42.731761+0000 (oldest deadline 2023-12-16T22:11:02.731761+0000) 2023-12-16T22:12:09.729 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:12:09 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:12:09.398+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:10:42.731761+0000 (oldest deadline 2023-12-16T22:11:02.731761+0000) 2023-12-16T22:12:09.730 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:12:09 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:12:09.398+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:10:42.731761+0000 (oldest deadline 2023-12-16T22:11:02.731761+0000) 2023-12-16T22:12:09.730 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:12:09 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:12:09.399+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:12:10.064 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:12:09 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:12:09.794+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:11:07.849373+0000 (oldest deadline 2023-12-16T22:11:27.849373+0000) 2023-12-16T22:12:10.064 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:12:09 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:12:09.794+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:11:07.849373+0000 (oldest deadline 2023-12-16T22:11:27.849373+0000) 2023-12-16T22:12:10.064 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:12:09 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:12:09.794+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:11:07.849373+0000 (oldest deadline 2023-12-16T22:11:27.849373+0000) 2023-12-16T22:12:10.065 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:12:09 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:12:09.795+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:12:10.065 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:12:09 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:12:09.727+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:10:43.026858+0000 (oldest deadline 2023-12-16T22:11:03.026858+0000) 2023-12-16T22:12:10.065 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:12:09 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:12:09.728+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:10:43.026858+0000 (oldest deadline 2023-12-16T22:11:03.026858+0000) 2023-12-16T22:12:10.066 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:12:09 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:12:09.728+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:10:43.026858+0000 (oldest deadline 2023-12-16T22:11:03.026858+0000) 2023-12-16T22:12:10.066 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:12:09 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:12:09.729+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:12:10.564 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:12:10 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:12:10.428+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:10:42.731761+0000 (oldest deadline 2023-12-16T22:11:02.731761+0000) 2023-12-16T22:12:10.565 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:12:10 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:12:10.428+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:10:42.731761+0000 (oldest deadline 2023-12-16T22:11:02.731761+0000) 2023-12-16T22:12:10.565 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:12:10 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:12:10.428+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:10:42.731761+0000 (oldest deadline 2023-12-16T22:11:02.731761+0000) 2023-12-16T22:12:10.565 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:12:10 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:12:10.430+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:12:11.064 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:12:10 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:12:10.680+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:10:43.026858+0000 (oldest deadline 2023-12-16T22:11:03.026858+0000) 2023-12-16T22:12:11.065 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:12:10 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:12:10.680+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:10:43.026858+0000 (oldest deadline 2023-12-16T22:11:03.026858+0000) 2023-12-16T22:12:11.065 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:12:10 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:12:10.680+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:10:43.026858+0000 (oldest deadline 2023-12-16T22:11:03.026858+0000) 2023-12-16T22:12:11.065 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:12:10 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:12:10.681+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:12:11.065 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:12:10 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:12:10.779+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:11:07.849373+0000 (oldest deadline 2023-12-16T22:11:27.849373+0000) 2023-12-16T22:12:11.065 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:12:10 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:12:10.779+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:11:07.849373+0000 (oldest deadline 2023-12-16T22:11:27.849373+0000) 2023-12-16T22:12:11.065 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:12:10 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:12:10.779+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:11:07.849373+0000 (oldest deadline 2023-12-16T22:11:27.849373+0000) 2023-12-16T22:12:11.065 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:12:10 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:12:10.780+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:12:11.706 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:12:11 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:12:11.423+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:10:42.731761+0000 (oldest deadline 2023-12-16T22:11:02.731761+0000) 2023-12-16T22:12:11.706 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:12:11 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:12:11.423+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:10:42.731761+0000 (oldest deadline 2023-12-16T22:11:02.731761+0000) 2023-12-16T22:12:11.706 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:12:11 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:12:11.423+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:10:42.731761+0000 (oldest deadline 2023-12-16T22:11:02.731761+0000) 2023-12-16T22:12:11.707 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:12:11 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:12:11.424+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:12:12.033 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:12:11 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:12:11.812+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:11:07.849373+0000 (oldest deadline 2023-12-16T22:11:27.849373+0000) 2023-12-16T22:12:12.033 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:12:11 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:12:11.812+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:11:07.849373+0000 (oldest deadline 2023-12-16T22:11:27.849373+0000) 2023-12-16T22:12:12.034 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:12:11 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:12:11.812+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:11:07.849373+0000 (oldest deadline 2023-12-16T22:11:27.849373+0000) 2023-12-16T22:12:12.034 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:12:11 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:12:11.813+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:12:12.034 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:12:11 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:12:11.705+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:10:43.026858+0000 (oldest deadline 2023-12-16T22:11:03.026858+0000) 2023-12-16T22:12:12.034 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:12:11 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:12:11.705+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:10:43.026858+0000 (oldest deadline 2023-12-16T22:11:03.026858+0000) 2023-12-16T22:12:12.034 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:12:11 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:12:11.705+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:10:43.026858+0000 (oldest deadline 2023-12-16T22:11:03.026858+0000) 2023-12-16T22:12:12.034 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:12:11 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:12:11.706+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:12:12.314 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Dec 16 22:12:12 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-mon-smithi045[80759]: 2023-12-16T22:12:12.031+0000 7f45a1f8f700 -1 mon.smithi045@0(probing) e2 get_health_metrics reporting 32638 slow ops, oldest is monmgrreport(gid 14214, 0 checks, 1 progress events) 2023-12-16T22:12:12.734 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:12:12 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:12:12.389+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:10:42.731761+0000 (oldest deadline 2023-12-16T22:11:02.731761+0000) 2023-12-16T22:12:12.734 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:12:12 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:12:12.389+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:10:42.731761+0000 (oldest deadline 2023-12-16T22:11:02.731761+0000) 2023-12-16T22:12:12.734 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:12:12 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:12:12.389+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:10:42.731761+0000 (oldest deadline 2023-12-16T22:11:02.731761+0000) 2023-12-16T22:12:12.734 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:12:12 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:12:12.390+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:12:13.064 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:12:12 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:12:12.829+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:11:07.849373+0000 (oldest deadline 2023-12-16T22:11:27.849373+0000) 2023-12-16T22:12:13.064 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:12:12 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:12:12.829+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:11:07.849373+0000 (oldest deadline 2023-12-16T22:11:27.849373+0000) 2023-12-16T22:12:13.065 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:12:12 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:12:12.829+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:11:07.849373+0000 (oldest deadline 2023-12-16T22:11:27.849373+0000) 2023-12-16T22:12:13.065 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:12:12 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:12:12.830+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:12:13.065 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:12:12 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:12:12.733+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:10:43.026858+0000 (oldest deadline 2023-12-16T22:11:03.026858+0000) 2023-12-16T22:12:13.065 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:12:12 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:12:12.733+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:10:43.026858+0000 (oldest deadline 2023-12-16T22:11:03.026858+0000) 2023-12-16T22:12:13.065 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:12:12 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:12:12.733+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:10:43.026858+0000 (oldest deadline 2023-12-16T22:11:03.026858+0000) 2023-12-16T22:12:13.065 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:12:12 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:12:12.734+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:12:13.641 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:12:13 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:12:13.378+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:10:42.731761+0000 (oldest deadline 2023-12-16T22:11:02.731761+0000) 2023-12-16T22:12:13.641 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:12:13 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:12:13.378+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:10:42.731761+0000 (oldest deadline 2023-12-16T22:11:02.731761+0000) 2023-12-16T22:12:13.641 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:12:13 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:12:13.378+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:10:42.731761+0000 (oldest deadline 2023-12-16T22:11:02.731761+0000) 2023-12-16T22:12:13.642 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:12:13 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:12:13.378+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:12:14.064 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:12:13 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:12:13.686+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:10:43.026858+0000 (oldest deadline 2023-12-16T22:11:03.026858+0000) 2023-12-16T22:12:14.064 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:12:13 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:12:13.686+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:10:43.026858+0000 (oldest deadline 2023-12-16T22:11:03.026858+0000) 2023-12-16T22:12:14.065 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:12:13 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:12:13.686+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:10:43.026858+0000 (oldest deadline 2023-12-16T22:11:03.026858+0000) 2023-12-16T22:12:14.065 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:12:13 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:12:13.687+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:12:14.065 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:12:13 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:12:13.783+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:11:07.849373+0000 (oldest deadline 2023-12-16T22:11:27.849373+0000) 2023-12-16T22:12:14.065 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:12:13 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:12:13.783+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:11:07.849373+0000 (oldest deadline 2023-12-16T22:11:27.849373+0000) 2023-12-16T22:12:14.065 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:12:13 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:12:13.783+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:11:07.849373+0000 (oldest deadline 2023-12-16T22:11:27.849373+0000) 2023-12-16T22:12:14.065 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:12:13 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:12:13.784+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:12:14.735 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:12:14 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:12:14.369+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:10:42.731761+0000 (oldest deadline 2023-12-16T22:11:02.731761+0000) 2023-12-16T22:12:14.735 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:12:14 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:12:14.369+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:10:42.731761+0000 (oldest deadline 2023-12-16T22:11:02.731761+0000) 2023-12-16T22:12:14.735 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:12:14 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:12:14.369+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:10:42.731761+0000 (oldest deadline 2023-12-16T22:11:02.731761+0000) 2023-12-16T22:12:14.735 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:12:14 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:12:14.370+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:12:15.064 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:12:14 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:12:14.734+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:10:43.026858+0000 (oldest deadline 2023-12-16T22:11:03.026858+0000) 2023-12-16T22:12:15.064 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:12:14 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:12:14.734+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:10:43.026858+0000 (oldest deadline 2023-12-16T22:11:03.026858+0000) 2023-12-16T22:12:15.065 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:12:14 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:12:14.734+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:10:43.026858+0000 (oldest deadline 2023-12-16T22:11:03.026858+0000) 2023-12-16T22:12:15.065 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:12:14 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:12:14.735+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:12:15.065 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:12:14 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:12:14.750+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:11:07.849373+0000 (oldest deadline 2023-12-16T22:11:27.849373+0000) 2023-12-16T22:12:15.065 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:12:14 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:12:14.750+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:11:07.849373+0000 (oldest deadline 2023-12-16T22:11:27.849373+0000) 2023-12-16T22:12:15.065 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:12:14 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:12:14.750+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:11:07.849373+0000 (oldest deadline 2023-12-16T22:11:27.849373+0000) 2023-12-16T22:12:15.065 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:12:14 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:12:14.751+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:12:15.734 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:12:15 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:12:15.404+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:10:42.731761+0000 (oldest deadline 2023-12-16T22:11:02.731761+0000) 2023-12-16T22:12:15.735 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:12:15 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:12:15.404+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:10:42.731761+0000 (oldest deadline 2023-12-16T22:11:02.731761+0000) 2023-12-16T22:12:15.735 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:12:15 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:12:15.404+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:10:42.731761+0000 (oldest deadline 2023-12-16T22:11:02.731761+0000) 2023-12-16T22:12:15.735 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:12:15 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:12:15.406+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:12:16.064 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:12:15 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:12:15.758+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:10:43.026858+0000 (oldest deadline 2023-12-16T22:11:03.026858+0000) 2023-12-16T22:12:16.064 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:12:15 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:12:15.758+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:10:43.026858+0000 (oldest deadline 2023-12-16T22:11:03.026858+0000) 2023-12-16T22:12:16.065 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:12:15 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:12:15.758+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:10:43.026858+0000 (oldest deadline 2023-12-16T22:11:03.026858+0000) 2023-12-16T22:12:16.065 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:12:15 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:12:15.760+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:12:16.065 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:12:15 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:12:15.733+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:11:07.849373+0000 (oldest deadline 2023-12-16T22:11:27.849373+0000) 2023-12-16T22:12:16.065 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:12:15 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:12:15.733+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:11:07.849373+0000 (oldest deadline 2023-12-16T22:11:27.849373+0000) 2023-12-16T22:12:16.065 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:12:15 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:12:15.733+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:11:07.849373+0000 (oldest deadline 2023-12-16T22:11:27.849373+0000) 2023-12-16T22:12:16.065 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:12:15 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:12:15.734+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:12:16.706 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:12:16 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:12:16.381+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:10:42.731761+0000 (oldest deadline 2023-12-16T22:11:02.731761+0000) 2023-12-16T22:12:16.706 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:12:16 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:12:16.381+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:10:42.731761+0000 (oldest deadline 2023-12-16T22:11:02.731761+0000) 2023-12-16T22:12:16.707 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:12:16 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:12:16.381+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:10:42.731761+0000 (oldest deadline 2023-12-16T22:11:02.731761+0000) 2023-12-16T22:12:16.707 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:12:16 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:12:16.382+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:12:17.036 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:12:16 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:12:16.711+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:10:43.026858+0000 (oldest deadline 2023-12-16T22:11:03.026858+0000) 2023-12-16T22:12:17.037 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:12:16 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:12:16.711+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:10:43.026858+0000 (oldest deadline 2023-12-16T22:11:03.026858+0000) 2023-12-16T22:12:17.037 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:12:16 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:12:16.711+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:10:43.026858+0000 (oldest deadline 2023-12-16T22:11:03.026858+0000) 2023-12-16T22:12:17.037 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:12:16 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:12:16.712+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:12:17.037 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:12:16 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:12:16.705+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:11:07.849373+0000 (oldest deadline 2023-12-16T22:11:27.849373+0000) 2023-12-16T22:12:17.037 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:12:16 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:12:16.705+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:11:07.849373+0000 (oldest deadline 2023-12-16T22:11:27.849373+0000) 2023-12-16T22:12:17.038 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:12:16 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:12:16.705+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:11:07.849373+0000 (oldest deadline 2023-12-16T22:11:27.849373+0000) 2023-12-16T22:12:17.038 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:12:16 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:12:16.706+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:12:17.314 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Dec 16 22:12:17 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-mon-smithi045[80759]: 2023-12-16T22:12:17.035+0000 7f45a1f8f700 -1 mon.smithi045@0(probing) e2 get_health_metrics reporting 32729 slow ops, oldest is monmgrreport(gid 14214, 0 checks, 1 progress events) 2023-12-16T22:12:17.678 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:12:17 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:12:17.393+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:10:42.731761+0000 (oldest deadline 2023-12-16T22:11:02.731761+0000) 2023-12-16T22:12:17.679 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:12:17 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:12:17.393+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:10:42.731761+0000 (oldest deadline 2023-12-16T22:11:02.731761+0000) 2023-12-16T22:12:17.679 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:12:17 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:12:17.393+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:10:42.731761+0000 (oldest deadline 2023-12-16T22:11:02.731761+0000) 2023-12-16T22:12:17.679 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:12:17 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:12:17.395+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:12:18.064 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:12:17 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:12:17.749+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:10:43.026858+0000 (oldest deadline 2023-12-16T22:11:03.026858+0000) 2023-12-16T22:12:18.065 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:12:17 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:12:17.749+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:10:43.026858+0000 (oldest deadline 2023-12-16T22:11:03.026858+0000) 2023-12-16T22:12:18.065 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:12:17 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:12:17.749+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:10:43.026858+0000 (oldest deadline 2023-12-16T22:11:03.026858+0000) 2023-12-16T22:12:18.065 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:12:17 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:12:17.750+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:12:18.065 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:12:17 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:12:17.677+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:11:07.849373+0000 (oldest deadline 2023-12-16T22:11:27.849373+0000) 2023-12-16T22:12:18.065 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:12:17 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:12:17.677+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:11:07.849373+0000 (oldest deadline 2023-12-16T22:11:27.849373+0000) 2023-12-16T22:12:18.065 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:12:17 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:12:17.677+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:11:07.849373+0000 (oldest deadline 2023-12-16T22:11:27.849373+0000) 2023-12-16T22:12:18.065 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:12:17 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:12:17.679+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:12:18.705 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:12:18 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:12:18.385+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:10:42.731761+0000 (oldest deadline 2023-12-16T22:11:02.731761+0000) 2023-12-16T22:12:18.705 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:12:18 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:12:18.385+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:10:42.731761+0000 (oldest deadline 2023-12-16T22:11:02.731761+0000) 2023-12-16T22:12:18.705 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:12:18 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:12:18.385+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:10:42.731761+0000 (oldest deadline 2023-12-16T22:11:02.731761+0000) 2023-12-16T22:12:18.705 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:12:18 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:12:18.386+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:12:19.064 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:12:18 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:12:18.742+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:10:43.026858+0000 (oldest deadline 2023-12-16T22:11:03.026858+0000) 2023-12-16T22:12:19.064 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:12:18 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:12:18.742+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:10:43.026858+0000 (oldest deadline 2023-12-16T22:11:03.026858+0000) 2023-12-16T22:12:19.065 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:12:18 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:12:18.742+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:10:43.026858+0000 (oldest deadline 2023-12-16T22:11:03.026858+0000) 2023-12-16T22:12:19.065 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:12:18 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:12:18.743+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:12:19.065 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:12:18 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:12:18.704+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:11:07.849373+0000 (oldest deadline 2023-12-16T22:11:27.849373+0000) 2023-12-16T22:12:19.065 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:12:18 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:12:18.704+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:11:07.849373+0000 (oldest deadline 2023-12-16T22:11:27.849373+0000) 2023-12-16T22:12:19.065 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:12:18 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:12:18.704+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:11:07.849373+0000 (oldest deadline 2023-12-16T22:11:27.849373+0000) 2023-12-16T22:12:19.065 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:12:18 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:12:18.704+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:12:19.683 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:12:19 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:12:19.397+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:10:42.731761+0000 (oldest deadline 2023-12-16T22:11:02.731761+0000) 2023-12-16T22:12:19.683 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:12:19 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:12:19.397+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:10:42.731761+0000 (oldest deadline 2023-12-16T22:11:02.731761+0000) 2023-12-16T22:12:19.683 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:12:19 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:12:19.397+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:10:42.731761+0000 (oldest deadline 2023-12-16T22:11:02.731761+0000) 2023-12-16T22:12:19.683 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:12:19 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:12:19.398+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:12:20.064 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:12:19 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:12:19.763+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:10:43.026858+0000 (oldest deadline 2023-12-16T22:11:03.026858+0000) 2023-12-16T22:12:20.064 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:12:19 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:12:19.763+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:10:43.026858+0000 (oldest deadline 2023-12-16T22:11:03.026858+0000) 2023-12-16T22:12:20.064 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:12:19 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:12:19.763+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:10:43.026858+0000 (oldest deadline 2023-12-16T22:11:03.026858+0000) 2023-12-16T22:12:20.065 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:12:19 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:12:19.764+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:12:20.065 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:12:19 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:12:19.682+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:11:07.849373+0000 (oldest deadline 2023-12-16T22:11:27.849373+0000) 2023-12-16T22:12:20.065 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:12:19 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:12:19.682+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:11:07.849373+0000 (oldest deadline 2023-12-16T22:11:27.849373+0000) 2023-12-16T22:12:20.065 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:12:19 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:12:19.682+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:11:07.849373+0000 (oldest deadline 2023-12-16T22:11:27.849373+0000) 2023-12-16T22:12:20.065 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:12:19 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:12:19.682+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:12:20.564 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:12:20 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:12:20.418+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:10:42.731761+0000 (oldest deadline 2023-12-16T22:11:02.731761+0000) 2023-12-16T22:12:20.564 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:12:20 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:12:20.418+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:10:42.731761+0000 (oldest deadline 2023-12-16T22:11:02.731761+0000) 2023-12-16T22:12:20.565 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:12:20 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:12:20.418+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:10:42.731761+0000 (oldest deadline 2023-12-16T22:11:02.731761+0000) 2023-12-16T22:12:20.565 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:12:20 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:12:20.418+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:12:21.064 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:12:20 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:12:20.812+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:10:43.026858+0000 (oldest deadline 2023-12-16T22:11:03.026858+0000) 2023-12-16T22:12:21.064 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:12:20 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:12:20.812+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:10:43.026858+0000 (oldest deadline 2023-12-16T22:11:03.026858+0000) 2023-12-16T22:12:21.065 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:12:20 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:12:20.812+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:10:43.026858+0000 (oldest deadline 2023-12-16T22:11:03.026858+0000) 2023-12-16T22:12:21.065 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:12:20 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:12:20.813+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:12:21.065 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:12:20 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:12:20.727+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:11:07.849373+0000 (oldest deadline 2023-12-16T22:11:27.849373+0000) 2023-12-16T22:12:21.065 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:12:20 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:12:20.727+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:11:07.849373+0000 (oldest deadline 2023-12-16T22:11:27.849373+0000) 2023-12-16T22:12:21.065 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:12:20 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:12:20.727+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:11:07.849373+0000 (oldest deadline 2023-12-16T22:11:27.849373+0000) 2023-12-16T22:12:21.066 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:12:20 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:12:20.728+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:12:21.740 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:12:21 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:12:21.390+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:10:42.731761+0000 (oldest deadline 2023-12-16T22:11:02.731761+0000) 2023-12-16T22:12:21.740 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:12:21 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:12:21.390+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:10:42.731761+0000 (oldest deadline 2023-12-16T22:11:02.731761+0000) 2023-12-16T22:12:21.740 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:12:21 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:12:21.390+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:10:42.731761+0000 (oldest deadline 2023-12-16T22:11:02.731761+0000) 2023-12-16T22:12:21.740 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:12:21 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:12:21.391+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:12:22.042 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:12:21 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:12:21.765+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:10:43.026858+0000 (oldest deadline 2023-12-16T22:11:03.026858+0000) 2023-12-16T22:12:22.042 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:12:21 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:12:21.765+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:10:43.026858+0000 (oldest deadline 2023-12-16T22:11:03.026858+0000) 2023-12-16T22:12:22.042 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:12:21 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:12:21.765+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:10:43.026858+0000 (oldest deadline 2023-12-16T22:11:03.026858+0000) 2023-12-16T22:12:22.042 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:12:21 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:12:21.766+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:12:22.043 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:12:21 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:12:21.739+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:11:07.849373+0000 (oldest deadline 2023-12-16T22:11:27.849373+0000) 2023-12-16T22:12:22.043 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:12:21 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:12:21.739+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:11:07.849373+0000 (oldest deadline 2023-12-16T22:11:27.849373+0000) 2023-12-16T22:12:22.043 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:12:21 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:12:21.739+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:11:07.849373+0000 (oldest deadline 2023-12-16T22:11:27.849373+0000) 2023-12-16T22:12:22.043 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:12:21 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:12:21.740+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:12:22.314 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Dec 16 22:12:22 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-mon-smithi045[80759]: 2023-12-16T22:12:22.040+0000 7f45a1f8f700 -1 mon.smithi045@0(probing) e2 get_health_metrics reporting 32819 slow ops, oldest is monmgrreport(gid 14214, 0 checks, 1 progress events) 2023-12-16T22:12:22.723 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:12:22 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:12:22.437+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:10:42.731761+0000 (oldest deadline 2023-12-16T22:11:02.731761+0000) 2023-12-16T22:12:22.723 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:12:22 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:12:22.437+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:10:42.731761+0000 (oldest deadline 2023-12-16T22:11:02.731761+0000) 2023-12-16T22:12:22.723 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:12:22 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:12:22.437+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:10:42.731761+0000 (oldest deadline 2023-12-16T22:11:02.731761+0000) 2023-12-16T22:12:22.723 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:12:22 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:12:22.438+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:12:23.065 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:12:22 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:12:22.801+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:10:43.026858+0000 (oldest deadline 2023-12-16T22:11:03.026858+0000) 2023-12-16T22:12:23.065 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:12:22 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:12:22.801+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:10:43.026858+0000 (oldest deadline 2023-12-16T22:11:03.026858+0000) 2023-12-16T22:12:23.065 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:12:22 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:12:22.801+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:10:43.026858+0000 (oldest deadline 2023-12-16T22:11:03.026858+0000) 2023-12-16T22:12:23.065 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:12:22 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:12:22.802+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:12:23.065 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:12:22 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:12:22.722+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:11:07.849373+0000 (oldest deadline 2023-12-16T22:11:27.849373+0000) 2023-12-16T22:12:23.066 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:12:22 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:12:22.722+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:11:07.849373+0000 (oldest deadline 2023-12-16T22:11:27.849373+0000) 2023-12-16T22:12:23.066 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:12:22 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:12:22.722+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:11:07.849373+0000 (oldest deadline 2023-12-16T22:11:27.849373+0000) 2023-12-16T22:12:23.066 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:12:22 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:12:22.723+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:12:23.772 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:12:23 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:12:23.484+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:10:42.731761+0000 (oldest deadline 2023-12-16T22:11:02.731761+0000) 2023-12-16T22:12:23.772 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:12:23 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:12:23.484+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:10:42.731761+0000 (oldest deadline 2023-12-16T22:11:02.731761+0000) 2023-12-16T22:12:23.772 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:12:23 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:12:23.484+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:10:42.731761+0000 (oldest deadline 2023-12-16T22:11:02.731761+0000) 2023-12-16T22:12:23.772 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:12:23 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:12:23.485+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:12:24.064 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:12:23 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:12:23.771+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:11:07.849373+0000 (oldest deadline 2023-12-16T22:11:27.849373+0000) 2023-12-16T22:12:24.064 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:12:23 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:12:23.771+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:11:07.849373+0000 (oldest deadline 2023-12-16T22:11:27.849373+0000) 2023-12-16T22:12:24.065 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:12:23 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:12:23.771+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:11:07.849373+0000 (oldest deadline 2023-12-16T22:11:27.849373+0000) 2023-12-16T22:12:24.065 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:12:23 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:12:23.772+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:12:24.065 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:12:23 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:12:23.851+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:10:43.026858+0000 (oldest deadline 2023-12-16T22:11:03.026858+0000) 2023-12-16T22:12:24.065 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:12:23 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:12:23.851+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:10:43.026858+0000 (oldest deadline 2023-12-16T22:11:03.026858+0000) 2023-12-16T22:12:24.065 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:12:23 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:12:23.851+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:10:43.026858+0000 (oldest deadline 2023-12-16T22:11:03.026858+0000) 2023-12-16T22:12:24.065 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:12:23 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:12:23.852+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:12:24.785 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:12:24 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:12:24.518+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:10:42.731761+0000 (oldest deadline 2023-12-16T22:11:02.731761+0000) 2023-12-16T22:12:24.786 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:12:24 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:12:24.518+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:10:42.731761+0000 (oldest deadline 2023-12-16T22:11:02.731761+0000) 2023-12-16T22:12:24.786 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:12:24 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:12:24.518+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:10:42.731761+0000 (oldest deadline 2023-12-16T22:11:02.731761+0000) 2023-12-16T22:12:24.786 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:12:24 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:12:24.519+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:12:25.064 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:12:24 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:12:24.784+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:11:07.849373+0000 (oldest deadline 2023-12-16T22:11:27.849373+0000) 2023-12-16T22:12:25.064 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:12:24 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:12:24.784+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:11:07.849373+0000 (oldest deadline 2023-12-16T22:11:27.849373+0000) 2023-12-16T22:12:25.065 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:12:24 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:12:24.784+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:11:07.849373+0000 (oldest deadline 2023-12-16T22:11:27.849373+0000) 2023-12-16T22:12:25.065 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:12:24 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:12:24.785+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:12:25.066 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:12:24 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:12:24.810+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:10:43.026858+0000 (oldest deadline 2023-12-16T22:11:03.026858+0000) 2023-12-16T22:12:25.066 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:12:24 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:12:24.810+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:10:43.026858+0000 (oldest deadline 2023-12-16T22:11:03.026858+0000) 2023-12-16T22:12:25.066 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:12:24 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:12:24.810+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:10:43.026858+0000 (oldest deadline 2023-12-16T22:11:03.026858+0000) 2023-12-16T22:12:25.066 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:12:24 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:12:24.811+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:12:25.801 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:12:25 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:12:25.536+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:10:42.731761+0000 (oldest deadline 2023-12-16T22:11:02.731761+0000) 2023-12-16T22:12:25.802 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:12:25 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:12:25.536+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:10:42.731761+0000 (oldest deadline 2023-12-16T22:11:02.731761+0000) 2023-12-16T22:12:25.802 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:12:25 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:12:25.536+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:10:42.731761+0000 (oldest deadline 2023-12-16T22:11:02.731761+0000) 2023-12-16T22:12:25.802 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:12:25 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:12:25.537+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:12:26.064 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:12:25 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:12:25.800+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:11:07.849373+0000 (oldest deadline 2023-12-16T22:11:27.849373+0000) 2023-12-16T22:12:26.065 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:12:25 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:12:25.800+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:11:07.849373+0000 (oldest deadline 2023-12-16T22:11:27.849373+0000) 2023-12-16T22:12:26.065 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:12:25 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:12:25.800+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:11:07.849373+0000 (oldest deadline 2023-12-16T22:11:27.849373+0000) 2023-12-16T22:12:26.065 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:12:25 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:12:25.801+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:12:26.065 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:12:25 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:12:25.808+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:10:43.026858+0000 (oldest deadline 2023-12-16T22:11:03.026858+0000) 2023-12-16T22:12:26.065 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:12:25 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:12:25.808+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:10:43.026858+0000 (oldest deadline 2023-12-16T22:11:03.026858+0000) 2023-12-16T22:12:26.065 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:12:25 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:12:25.808+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:10:43.026858+0000 (oldest deadline 2023-12-16T22:11:03.026858+0000) 2023-12-16T22:12:26.066 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:12:25 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:12:25.810+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:12:26.814 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:12:26 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:12:26.561+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:10:42.731761+0000 (oldest deadline 2023-12-16T22:11:02.731761+0000) 2023-12-16T22:12:26.814 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:12:26 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:12:26.561+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:10:42.731761+0000 (oldest deadline 2023-12-16T22:11:02.731761+0000) 2023-12-16T22:12:26.814 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:12:26 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:12:26.561+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:10:42.731761+0000 (oldest deadline 2023-12-16T22:11:02.731761+0000) 2023-12-16T22:12:26.815 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:12:26 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:12:26.561+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:12:27.170 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Dec 16 22:12:27 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-mon-smithi045[80759]: 2023-12-16T22:12:27.044+0000 7f45a1f8f700 -1 mon.smithi045@0(probing) e2 get_health_metrics reporting 32910 slow ops, oldest is monmgrreport(gid 14214, 0 checks, 1 progress events) 2023-12-16T22:12:27.170 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:12:26 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:12:26.825+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:11:07.849373+0000 (oldest deadline 2023-12-16T22:11:27.849373+0000) 2023-12-16T22:12:27.170 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:12:26 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:12:26.825+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:11:07.849373+0000 (oldest deadline 2023-12-16T22:11:27.849373+0000) 2023-12-16T22:12:27.171 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:12:26 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:12:26.825+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:11:07.849373+0000 (oldest deadline 2023-12-16T22:11:27.849373+0000) 2023-12-16T22:12:27.171 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:12:26 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:12:26.826+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:12:27.171 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:12:26 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:12:26.855+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:10:43.026858+0000 (oldest deadline 2023-12-16T22:11:03.026858+0000) 2023-12-16T22:12:27.171 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:12:26 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:12:26.855+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:10:43.026858+0000 (oldest deadline 2023-12-16T22:11:03.026858+0000) 2023-12-16T22:12:27.171 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:12:26 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:12:26.855+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:10:43.026858+0000 (oldest deadline 2023-12-16T22:11:03.026858+0000) 2023-12-16T22:12:27.171 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:12:26 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:12:26.857+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:12:27.814 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:12:27 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:12:27.521+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:10:42.731761+0000 (oldest deadline 2023-12-16T22:11:02.731761+0000) 2023-12-16T22:12:27.814 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:12:27 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:12:27.521+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:10:42.731761+0000 (oldest deadline 2023-12-16T22:11:02.731761+0000) 2023-12-16T22:12:27.815 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:12:27 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:12:27.521+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:10:42.731761+0000 (oldest deadline 2023-12-16T22:11:02.731761+0000) 2023-12-16T22:12:27.815 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:12:27 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:12:27.521+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:12:28.314 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:12:27 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:12:27.855+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:11:07.849373+0000 (oldest deadline 2023-12-16T22:11:27.849373+0000) 2023-12-16T22:12:28.315 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:12:27 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:12:27.855+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:11:07.849373+0000 (oldest deadline 2023-12-16T22:11:27.849373+0000) 2023-12-16T22:12:28.315 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:12:27 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:12:27.855+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:11:07.849373+0000 (oldest deadline 2023-12-16T22:11:27.849373+0000) 2023-12-16T22:12:28.315 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:12:27 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:12:27.856+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:12:28.315 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:12:27 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:12:27.880+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:10:43.026858+0000 (oldest deadline 2023-12-16T22:11:03.026858+0000) 2023-12-16T22:12:28.315 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:12:27 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:12:27.880+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:10:43.026858+0000 (oldest deadline 2023-12-16T22:11:03.026858+0000) 2023-12-16T22:12:28.316 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:12:27 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:12:27.880+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:10:43.026858+0000 (oldest deadline 2023-12-16T22:11:03.026858+0000) 2023-12-16T22:12:28.316 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:12:27 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:12:27.881+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:12:28.817 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:12:28 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:12:28.492+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:10:42.731761+0000 (oldest deadline 2023-12-16T22:11:02.731761+0000) 2023-12-16T22:12:28.817 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:12:28 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:12:28.492+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:10:42.731761+0000 (oldest deadline 2023-12-16T22:11:02.731761+0000) 2023-12-16T22:12:28.817 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:12:28 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:12:28.492+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:10:42.731761+0000 (oldest deadline 2023-12-16T22:11:02.731761+0000) 2023-12-16T22:12:28.817 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:12:28 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:12:28.493+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:12:29.314 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:12:28 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:12:28.867+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:11:07.849373+0000 (oldest deadline 2023-12-16T22:11:27.849373+0000) 2023-12-16T22:12:29.314 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:12:28 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:12:28.867+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:11:07.849373+0000 (oldest deadline 2023-12-16T22:11:27.849373+0000) 2023-12-16T22:12:29.315 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:12:28 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:12:28.867+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:11:07.849373+0000 (oldest deadline 2023-12-16T22:11:27.849373+0000) 2023-12-16T22:12:29.315 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:12:28 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:12:28.869+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:12:29.315 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:12:28 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:12:28.871+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:10:43.026858+0000 (oldest deadline 2023-12-16T22:11:03.026858+0000) 2023-12-16T22:12:29.315 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:12:28 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:12:28.871+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:10:43.026858+0000 (oldest deadline 2023-12-16T22:11:03.026858+0000) 2023-12-16T22:12:29.316 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:12:28 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:12:28.871+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:10:43.026858+0000 (oldest deadline 2023-12-16T22:11:03.026858+0000) 2023-12-16T22:12:29.316 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:12:28 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:12:28.871+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:12:29.814 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:12:29 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:12:29.516+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:10:42.731761+0000 (oldest deadline 2023-12-16T22:11:02.731761+0000) 2023-12-16T22:12:29.814 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:12:29 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:12:29.516+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:10:42.731761+0000 (oldest deadline 2023-12-16T22:11:02.731761+0000) 2023-12-16T22:12:29.814 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:12:29 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:12:29.516+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:10:42.731761+0000 (oldest deadline 2023-12-16T22:11:02.731761+0000) 2023-12-16T22:12:29.815 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:12:29 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:12:29.517+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:12:30.220 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:12:29 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:12:29.852+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:11:07.849373+0000 (oldest deadline 2023-12-16T22:11:27.849373+0000) 2023-12-16T22:12:30.221 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:12:29 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:12:29.852+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:11:07.849373+0000 (oldest deadline 2023-12-16T22:11:27.849373+0000) 2023-12-16T22:12:30.221 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:12:29 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:12:29.852+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:11:07.849373+0000 (oldest deadline 2023-12-16T22:11:27.849373+0000) 2023-12-16T22:12:30.221 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:12:29 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:12:29.853+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:12:30.221 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:12:29 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:12:29.858+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:10:43.026858+0000 (oldest deadline 2023-12-16T22:11:03.026858+0000) 2023-12-16T22:12:30.221 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:12:29 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:12:29.858+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:10:43.026858+0000 (oldest deadline 2023-12-16T22:11:03.026858+0000) 2023-12-16T22:12:30.222 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:12:29 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:12:29.858+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:10:43.026858+0000 (oldest deadline 2023-12-16T22:11:03.026858+0000) 2023-12-16T22:12:30.222 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:12:29 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:12:29.858+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:12:30.807 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:12:30 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:12:30.537+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:10:42.731761+0000 (oldest deadline 2023-12-16T22:11:02.731761+0000) 2023-12-16T22:12:30.808 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:12:30 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:12:30.537+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:10:42.731761+0000 (oldest deadline 2023-12-16T22:11:02.731761+0000) 2023-12-16T22:12:30.808 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:12:30 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:12:30.537+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:10:42.731761+0000 (oldest deadline 2023-12-16T22:11:02.731761+0000) 2023-12-16T22:12:30.808 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:12:30 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:12:30.538+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:12:31.064 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:12:30 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:12:30.837+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:10:43.026858+0000 (oldest deadline 2023-12-16T22:11:03.026858+0000) 2023-12-16T22:12:31.065 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:12:30 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:12:30.837+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:10:43.026858+0000 (oldest deadline 2023-12-16T22:11:03.026858+0000) 2023-12-16T22:12:31.065 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:12:30 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:12:30.837+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:10:43.026858+0000 (oldest deadline 2023-12-16T22:11:03.026858+0000) 2023-12-16T22:12:31.065 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:12:30 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:12:30.838+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:12:31.065 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:12:30 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:12:30.806+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:11:07.849373+0000 (oldest deadline 2023-12-16T22:11:27.849373+0000) 2023-12-16T22:12:31.065 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:12:30 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:12:30.806+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:11:07.849373+0000 (oldest deadline 2023-12-16T22:11:27.849373+0000) 2023-12-16T22:12:31.066 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:12:30 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:12:30.806+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:11:07.849373+0000 (oldest deadline 2023-12-16T22:11:27.849373+0000) 2023-12-16T22:12:31.066 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:12:30 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:12:30.807+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:12:31.784 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:12:31 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:12:31.516+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:10:42.731761+0000 (oldest deadline 2023-12-16T22:11:02.731761+0000) 2023-12-16T22:12:31.784 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:12:31 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:12:31.516+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:10:42.731761+0000 (oldest deadline 2023-12-16T22:11:02.731761+0000) 2023-12-16T22:12:31.784 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:12:31 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:12:31.516+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:10:42.731761+0000 (oldest deadline 2023-12-16T22:11:02.731761+0000) 2023-12-16T22:12:31.784 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:12:31 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:12:31.518+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:12:32.049 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:12:31 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:12:31.794+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:10:43.026858+0000 (oldest deadline 2023-12-16T22:11:03.026858+0000) 2023-12-16T22:12:32.050 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:12:31 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:12:31.794+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:10:43.026858+0000 (oldest deadline 2023-12-16T22:11:03.026858+0000) 2023-12-16T22:12:32.050 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:12:31 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:12:31.794+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:10:43.026858+0000 (oldest deadline 2023-12-16T22:11:03.026858+0000) 2023-12-16T22:12:32.050 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:12:31 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:12:31.794+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:12:32.050 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:12:31 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:12:31.782+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:11:07.849373+0000 (oldest deadline 2023-12-16T22:11:27.849373+0000) 2023-12-16T22:12:32.050 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:12:31 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:12:31.782+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:11:07.849373+0000 (oldest deadline 2023-12-16T22:11:27.849373+0000) 2023-12-16T22:12:32.050 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:12:31 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:12:31.782+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:11:07.849373+0000 (oldest deadline 2023-12-16T22:11:27.849373+0000) 2023-12-16T22:12:32.050 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:12:31 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:12:31.783+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:12:32.314 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Dec 16 22:12:32 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-mon-smithi045[80759]: 2023-12-16T22:12:32.047+0000 7f45a1f8f700 -1 mon.smithi045@0(probing) e2 get_health_metrics reporting 33001 slow ops, oldest is monmgrreport(gid 14214, 0 checks, 1 progress events) 2023-12-16T22:12:32.778 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:12:32 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:12:32.487+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:10:42.731761+0000 (oldest deadline 2023-12-16T22:11:02.731761+0000) 2023-12-16T22:12:32.779 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:12:32 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:12:32.487+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:10:42.731761+0000 (oldest deadline 2023-12-16T22:11:02.731761+0000) 2023-12-16T22:12:32.779 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:12:32 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:12:32.487+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:10:42.731761+0000 (oldest deadline 2023-12-16T22:11:02.731761+0000) 2023-12-16T22:12:32.779 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:12:32 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:12:32.488+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:12:33.064 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:12:32 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:12:32.776+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:10:43.026858+0000 (oldest deadline 2023-12-16T22:11:03.026858+0000) 2023-12-16T22:12:33.064 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:12:32 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:12:32.776+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:10:43.026858+0000 (oldest deadline 2023-12-16T22:11:03.026858+0000) 2023-12-16T22:12:33.065 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:12:32 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:12:32.776+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:10:43.026858+0000 (oldest deadline 2023-12-16T22:11:03.026858+0000) 2023-12-16T22:12:33.065 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:12:32 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:12:32.777+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:12:33.065 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:12:32 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:12:32.827+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:11:07.849373+0000 (oldest deadline 2023-12-16T22:11:27.849373+0000) 2023-12-16T22:12:33.065 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:12:32 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:12:32.827+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:11:07.849373+0000 (oldest deadline 2023-12-16T22:11:27.849373+0000) 2023-12-16T22:12:33.065 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:12:32 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:12:32.827+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:11:07.849373+0000 (oldest deadline 2023-12-16T22:11:27.849373+0000) 2023-12-16T22:12:33.065 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:12:32 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:12:32.828+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:12:33.798 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:12:33 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:12:33.485+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:10:42.731761+0000 (oldest deadline 2023-12-16T22:11:02.731761+0000) 2023-12-16T22:12:33.798 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:12:33 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:12:33.486+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:10:42.731761+0000 (oldest deadline 2023-12-16T22:11:02.731761+0000) 2023-12-16T22:12:33.798 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:12:33 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:12:33.486+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:10:42.731761+0000 (oldest deadline 2023-12-16T22:11:02.731761+0000) 2023-12-16T22:12:33.799 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:12:33 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:12:33.487+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:12:34.064 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:12:33 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:12:33.796+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:10:43.026858+0000 (oldest deadline 2023-12-16T22:11:03.026858+0000) 2023-12-16T22:12:34.064 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:12:33 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:12:33.796+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:10:43.026858+0000 (oldest deadline 2023-12-16T22:11:03.026858+0000) 2023-12-16T22:12:34.065 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:12:33 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:12:33.796+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:10:43.026858+0000 (oldest deadline 2023-12-16T22:11:03.026858+0000) 2023-12-16T22:12:34.065 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:12:33 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:12:33.797+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:12:34.065 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:12:33 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:12:33.832+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:11:07.849373+0000 (oldest deadline 2023-12-16T22:11:27.849373+0000) 2023-12-16T22:12:34.065 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:12:33 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:12:33.832+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:11:07.849373+0000 (oldest deadline 2023-12-16T22:11:27.849373+0000) 2023-12-16T22:12:34.065 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:12:33 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:12:33.832+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:11:07.849373+0000 (oldest deadline 2023-12-16T22:11:27.849373+0000) 2023-12-16T22:12:34.065 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:12:33 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:12:33.833+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:12:34.807 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:12:34 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:12:34.459+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:10:42.731761+0000 (oldest deadline 2023-12-16T22:11:02.731761+0000) 2023-12-16T22:12:34.807 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:12:34 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:12:34.459+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:10:42.731761+0000 (oldest deadline 2023-12-16T22:11:02.731761+0000) 2023-12-16T22:12:34.808 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:12:34 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:12:34.459+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:10:42.731761+0000 (oldest deadline 2023-12-16T22:11:02.731761+0000) 2023-12-16T22:12:34.808 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:12:34 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:12:34.459+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:12:35.064 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:12:34 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:12:34.806+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:10:43.026858+0000 (oldest deadline 2023-12-16T22:11:03.026858+0000) 2023-12-16T22:12:35.064 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:12:34 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:12:34.806+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:10:43.026858+0000 (oldest deadline 2023-12-16T22:11:03.026858+0000) 2023-12-16T22:12:35.065 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:12:34 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:12:34.806+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:10:43.026858+0000 (oldest deadline 2023-12-16T22:11:03.026858+0000) 2023-12-16T22:12:35.065 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:12:34 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:12:34.807+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:12:35.065 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:12:34 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:12:34.823+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:11:07.849373+0000 (oldest deadline 2023-12-16T22:11:27.849373+0000) 2023-12-16T22:12:35.065 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:12:34 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:12:34.823+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:11:07.849373+0000 (oldest deadline 2023-12-16T22:11:27.849373+0000) 2023-12-16T22:12:35.065 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:12:34 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:12:34.823+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:11:07.849373+0000 (oldest deadline 2023-12-16T22:11:27.849373+0000) 2023-12-16T22:12:35.065 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:12:34 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:12:34.825+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:12:35.801 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:12:35 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:12:35.476+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:10:42.731761+0000 (oldest deadline 2023-12-16T22:11:02.731761+0000) 2023-12-16T22:12:35.801 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:12:35 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:12:35.476+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:10:42.731761+0000 (oldest deadline 2023-12-16T22:11:02.731761+0000) 2023-12-16T22:12:35.801 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:12:35 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:12:35.476+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:10:42.731761+0000 (oldest deadline 2023-12-16T22:11:02.731761+0000) 2023-12-16T22:12:35.802 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:12:35 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:12:35.476+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:12:36.064 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:12:35 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:12:35.845+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:10:43.026858+0000 (oldest deadline 2023-12-16T22:11:03.026858+0000) 2023-12-16T22:12:36.065 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:12:35 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:12:35.845+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:10:43.026858+0000 (oldest deadline 2023-12-16T22:11:03.026858+0000) 2023-12-16T22:12:36.065 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:12:35 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:12:35.845+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:10:43.026858+0000 (oldest deadline 2023-12-16T22:11:03.026858+0000) 2023-12-16T22:12:36.065 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:12:35 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:12:35.845+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:12:36.065 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:12:35 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:12:35.800+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:11:07.849373+0000 (oldest deadline 2023-12-16T22:11:27.849373+0000) 2023-12-16T22:12:36.065 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:12:35 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:12:35.800+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:11:07.849373+0000 (oldest deadline 2023-12-16T22:11:27.849373+0000) 2023-12-16T22:12:36.065 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:12:35 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:12:35.800+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:11:07.849373+0000 (oldest deadline 2023-12-16T22:11:27.849373+0000) 2023-12-16T22:12:36.065 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:12:35 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:12:35.800+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:12:36.814 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:12:36 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:12:36.468+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:10:42.731761+0000 (oldest deadline 2023-12-16T22:11:02.731761+0000) 2023-12-16T22:12:36.815 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:12:36 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:12:36.468+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:10:42.731761+0000 (oldest deadline 2023-12-16T22:11:02.731761+0000) 2023-12-16T22:12:36.815 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:12:36 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:12:36.468+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:10:42.731761+0000 (oldest deadline 2023-12-16T22:11:02.731761+0000) 2023-12-16T22:12:36.815 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:12:36 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:12:36.469+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:12:37.065 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Dec 16 22:12:37 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-mon-smithi045[80759]: 2023-12-16T22:12:37.053+0000 7f45a1f8f700 -1 mon.smithi045@0(probing) e2 get_health_metrics reporting 33092 slow ops, oldest is monmgrreport(gid 14214, 0 checks, 1 progress events) 2023-12-16T22:12:37.065 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:12:36 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:12:36.864+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:10:43.026858+0000 (oldest deadline 2023-12-16T22:11:03.026858+0000) 2023-12-16T22:12:37.066 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:12:36 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:12:36.864+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:10:43.026858+0000 (oldest deadline 2023-12-16T22:11:03.026858+0000) 2023-12-16T22:12:37.066 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:12:36 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:12:36.864+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:10:43.026858+0000 (oldest deadline 2023-12-16T22:11:03.026858+0000) 2023-12-16T22:12:37.066 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:12:36 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:12:36.864+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:12:37.066 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:12:36 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:12:36.814+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:11:07.849373+0000 (oldest deadline 2023-12-16T22:11:27.849373+0000) 2023-12-16T22:12:37.066 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:12:36 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:12:36.814+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:11:07.849373+0000 (oldest deadline 2023-12-16T22:11:27.849373+0000) 2023-12-16T22:12:37.066 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:12:36 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:12:36.814+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:11:07.849373+0000 (oldest deadline 2023-12-16T22:11:27.849373+0000) 2023-12-16T22:12:37.066 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:12:36 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:12:36.815+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:12:37.772 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:12:37 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:12:37.458+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:10:42.731761+0000 (oldest deadline 2023-12-16T22:11:02.731761+0000) 2023-12-16T22:12:37.772 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:12:37 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:12:37.458+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:10:42.731761+0000 (oldest deadline 2023-12-16T22:11:02.731761+0000) 2023-12-16T22:12:37.772 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:12:37 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:12:37.458+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:10:42.731761+0000 (oldest deadline 2023-12-16T22:11:02.731761+0000) 2023-12-16T22:12:37.772 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:12:37 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:12:37.459+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:12:38.064 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:12:37 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:12:37.891+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:10:43.026858+0000 (oldest deadline 2023-12-16T22:11:03.026858+0000) 2023-12-16T22:12:38.064 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:12:37 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:12:37.891+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:10:43.026858+0000 (oldest deadline 2023-12-16T22:11:03.026858+0000) 2023-12-16T22:12:38.065 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:12:37 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:12:37.891+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:10:43.026858+0000 (oldest deadline 2023-12-16T22:11:03.026858+0000) 2023-12-16T22:12:38.065 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:12:37 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:12:37.892+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:12:38.065 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:12:37 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:12:37.771+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:11:07.849373+0000 (oldest deadline 2023-12-16T22:11:27.849373+0000) 2023-12-16T22:12:38.065 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:12:37 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:12:37.771+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:11:07.849373+0000 (oldest deadline 2023-12-16T22:11:27.849373+0000) 2023-12-16T22:12:38.065 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:12:37 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:12:37.771+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:11:07.849373+0000 (oldest deadline 2023-12-16T22:11:27.849373+0000) 2023-12-16T22:12:38.066 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:12:37 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:12:37.771+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:12:38.720 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:12:38 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:12:38.467+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:10:42.731761+0000 (oldest deadline 2023-12-16T22:11:02.731761+0000) 2023-12-16T22:12:38.720 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:12:38 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:12:38.467+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:10:42.731761+0000 (oldest deadline 2023-12-16T22:11:02.731761+0000) 2023-12-16T22:12:38.720 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:12:38 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:12:38.467+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:10:42.731761+0000 (oldest deadline 2023-12-16T22:11:02.731761+0000) 2023-12-16T22:12:38.720 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:12:38 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:12:38.468+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:12:39.064 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:12:38 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:12:38.869+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:10:43.026858+0000 (oldest deadline 2023-12-16T22:11:03.026858+0000) 2023-12-16T22:12:39.065 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:12:38 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:12:38.869+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:10:43.026858+0000 (oldest deadline 2023-12-16T22:11:03.026858+0000) 2023-12-16T22:12:39.065 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:12:38 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:12:38.869+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:10:43.026858+0000 (oldest deadline 2023-12-16T22:11:03.026858+0000) 2023-12-16T22:12:39.065 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:12:38 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:12:38.869+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:12:39.065 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:12:38 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:12:38.822+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:11:07.849373+0000 (oldest deadline 2023-12-16T22:11:27.849373+0000) 2023-12-16T22:12:39.065 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:12:38 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:12:38.822+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:11:07.849373+0000 (oldest deadline 2023-12-16T22:11:27.849373+0000) 2023-12-16T22:12:39.065 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:12:38 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:12:38.822+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:11:07.849373+0000 (oldest deadline 2023-12-16T22:11:27.849373+0000) 2023-12-16T22:12:39.066 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:12:38 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:12:38.823+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:12:39.792 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:12:39 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:12:39.423+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:10:42.731761+0000 (oldest deadline 2023-12-16T22:11:02.731761+0000) 2023-12-16T22:12:39.793 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:12:39 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:12:39.423+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:10:42.731761+0000 (oldest deadline 2023-12-16T22:11:02.731761+0000) 2023-12-16T22:12:39.793 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:12:39 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:12:39.423+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:10:42.731761+0000 (oldest deadline 2023-12-16T22:11:02.731761+0000) 2023-12-16T22:12:39.793 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:12:39 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:12:39.424+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:12:40.064 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:12:39 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:12:39.791+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:11:07.849373+0000 (oldest deadline 2023-12-16T22:11:27.849373+0000) 2023-12-16T22:12:40.064 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:12:39 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:12:39.792+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:11:07.849373+0000 (oldest deadline 2023-12-16T22:11:27.849373+0000) 2023-12-16T22:12:40.065 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:12:39 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:12:39.792+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:11:07.849373+0000 (oldest deadline 2023-12-16T22:11:27.849373+0000) 2023-12-16T22:12:40.065 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:12:39 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:12:39.792+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:12:40.065 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:12:39 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:12:39.823+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:10:43.026858+0000 (oldest deadline 2023-12-16T22:11:03.026858+0000) 2023-12-16T22:12:40.065 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:12:39 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:12:39.823+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:10:43.026858+0000 (oldest deadline 2023-12-16T22:11:03.026858+0000) 2023-12-16T22:12:40.065 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:12:39 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:12:39.823+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:10:43.026858+0000 (oldest deadline 2023-12-16T22:11:03.026858+0000) 2023-12-16T22:12:40.065 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:12:39 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:12:39.825+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:12:40.564 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:12:40 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:12:40.384+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:10:42.731761+0000 (oldest deadline 2023-12-16T22:11:02.731761+0000) 2023-12-16T22:12:40.564 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:12:40 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:12:40.384+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:10:42.731761+0000 (oldest deadline 2023-12-16T22:11:02.731761+0000) 2023-12-16T22:12:40.565 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:12:40 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:12:40.384+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:10:42.731761+0000 (oldest deadline 2023-12-16T22:11:02.731761+0000) 2023-12-16T22:12:40.565 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:12:40 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:12:40.384+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:12:41.064 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:12:40 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:12:40.795+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:11:07.849373+0000 (oldest deadline 2023-12-16T22:11:27.849373+0000) 2023-12-16T22:12:41.065 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:12:40 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:12:40.795+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:11:07.849373+0000 (oldest deadline 2023-12-16T22:11:27.849373+0000) 2023-12-16T22:12:41.065 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:12:40 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:12:40.795+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:11:07.849373+0000 (oldest deadline 2023-12-16T22:11:27.849373+0000) 2023-12-16T22:12:41.065 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:12:40 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:12:40.796+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:12:41.065 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:12:40 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:12:40.804+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:10:43.026858+0000 (oldest deadline 2023-12-16T22:11:03.026858+0000) 2023-12-16T22:12:41.065 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:12:40 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:12:40.804+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:10:43.026858+0000 (oldest deadline 2023-12-16T22:11:03.026858+0000) 2023-12-16T22:12:41.065 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:12:40 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:12:40.804+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:10:43.026858+0000 (oldest deadline 2023-12-16T22:11:03.026858+0000) 2023-12-16T22:12:41.066 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:12:40 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:12:40.805+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:12:41.751 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:12:41 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:12:41.395+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:10:42.731761+0000 (oldest deadline 2023-12-16T22:11:02.731761+0000) 2023-12-16T22:12:41.751 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:12:41 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:12:41.395+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:10:42.731761+0000 (oldest deadline 2023-12-16T22:11:02.731761+0000) 2023-12-16T22:12:41.751 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:12:41 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:12:41.395+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:10:42.731761+0000 (oldest deadline 2023-12-16T22:11:02.731761+0000) 2023-12-16T22:12:41.752 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:12:41 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:12:41.395+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:12:42.058 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:12:41 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:12:41.778+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:11:07.849373+0000 (oldest deadline 2023-12-16T22:11:27.849373+0000) 2023-12-16T22:12:42.058 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:12:41 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:12:41.778+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:11:07.849373+0000 (oldest deadline 2023-12-16T22:11:27.849373+0000) 2023-12-16T22:12:42.058 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:12:41 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:12:41.778+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:11:07.849373+0000 (oldest deadline 2023-12-16T22:11:27.849373+0000) 2023-12-16T22:12:42.058 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:12:41 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:12:41.779+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:12:42.058 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:12:41 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:12:41.814+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:10:43.026858+0000 (oldest deadline 2023-12-16T22:11:03.026858+0000) 2023-12-16T22:12:42.059 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:12:41 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:12:41.814+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:10:43.026858+0000 (oldest deadline 2023-12-16T22:11:03.026858+0000) 2023-12-16T22:12:42.059 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:12:41 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:12:41.814+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:10:43.026858+0000 (oldest deadline 2023-12-16T22:11:03.026858+0000) 2023-12-16T22:12:42.059 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:12:41 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:12:41.815+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:12:42.314 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Dec 16 22:12:42 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-mon-smithi045[80759]: 2023-12-16T22:12:42.057+0000 7f45a1f8f700 -1 mon.smithi045@0(probing) e2 get_health_metrics reporting 33179 slow ops, oldest is monmgrreport(gid 14214, 0 checks, 1 progress events) 2023-12-16T22:12:42.773 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:12:42 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:12:42.437+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:10:42.731761+0000 (oldest deadline 2023-12-16T22:11:02.731761+0000) 2023-12-16T22:12:42.774 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:12:42 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:12:42.437+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:10:42.731761+0000 (oldest deadline 2023-12-16T22:11:02.731761+0000) 2023-12-16T22:12:42.774 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:12:42 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:12:42.437+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:10:42.731761+0000 (oldest deadline 2023-12-16T22:11:02.731761+0000) 2023-12-16T22:12:42.774 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:12:42 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:12:42.439+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:12:43.064 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:12:42 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:12:42.840+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:10:43.026858+0000 (oldest deadline 2023-12-16T22:11:03.026858+0000) 2023-12-16T22:12:43.065 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:12:42 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:12:42.840+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:10:43.026858+0000 (oldest deadline 2023-12-16T22:11:03.026858+0000) 2023-12-16T22:12:43.065 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:12:42 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:12:42.840+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:10:43.026858+0000 (oldest deadline 2023-12-16T22:11:03.026858+0000) 2023-12-16T22:12:43.065 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:12:42 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:12:42.840+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:12:43.065 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:12:42 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:12:42.772+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:11:07.849373+0000 (oldest deadline 2023-12-16T22:11:27.849373+0000) 2023-12-16T22:12:43.065 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:12:42 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:12:42.772+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:11:07.849373+0000 (oldest deadline 2023-12-16T22:11:27.849373+0000) 2023-12-16T22:12:43.066 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:12:42 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:12:42.772+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:11:07.849373+0000 (oldest deadline 2023-12-16T22:11:27.849373+0000) 2023-12-16T22:12:43.066 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:12:42 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:12:42.773+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:12:43.752 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:12:43 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:12:43.454+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:10:42.731761+0000 (oldest deadline 2023-12-16T22:11:02.731761+0000) 2023-12-16T22:12:43.752 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:12:43 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:12:43.454+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:10:42.731761+0000 (oldest deadline 2023-12-16T22:11:02.731761+0000) 2023-12-16T22:12:43.752 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:12:43 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:12:43.454+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:10:42.731761+0000 (oldest deadline 2023-12-16T22:11:02.731761+0000) 2023-12-16T22:12:43.752 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:12:43 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:12:43.456+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:12:44.064 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:12:43 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:12:43.751+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:11:07.849373+0000 (oldest deadline 2023-12-16T22:11:27.849373+0000) 2023-12-16T22:12:44.064 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:12:43 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:12:43.751+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:11:07.849373+0000 (oldest deadline 2023-12-16T22:11:27.849373+0000) 2023-12-16T22:12:44.065 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:12:43 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:12:43.751+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:11:07.849373+0000 (oldest deadline 2023-12-16T22:11:27.849373+0000) 2023-12-16T22:12:44.065 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:12:43 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:12:43.752+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:12:44.065 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:12:43 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:12:43.798+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:10:43.026858+0000 (oldest deadline 2023-12-16T22:11:03.026858+0000) 2023-12-16T22:12:44.065 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:12:43 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:12:43.798+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:10:43.026858+0000 (oldest deadline 2023-12-16T22:11:03.026858+0000) 2023-12-16T22:12:44.065 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:12:43 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:12:43.798+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:10:43.026858+0000 (oldest deadline 2023-12-16T22:11:03.026858+0000) 2023-12-16T22:12:44.065 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:12:43 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:12:43.799+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:12:44.771 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:12:44 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:12:44.428+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:10:42.731761+0000 (oldest deadline 2023-12-16T22:11:02.731761+0000) 2023-12-16T22:12:44.771 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:12:44 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:12:44.428+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:10:42.731761+0000 (oldest deadline 2023-12-16T22:11:02.731761+0000) 2023-12-16T22:12:44.771 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:12:44 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:12:44.428+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:10:42.731761+0000 (oldest deadline 2023-12-16T22:11:02.731761+0000) 2023-12-16T22:12:44.771 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:12:44 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:12:44.429+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:12:45.064 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:12:44 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:12:44.806+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:10:43.026858+0000 (oldest deadline 2023-12-16T22:11:03.026858+0000) 2023-12-16T22:12:45.065 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:12:44 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:12:44.806+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:10:43.026858+0000 (oldest deadline 2023-12-16T22:11:03.026858+0000) 2023-12-16T22:12:45.065 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:12:44 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:12:44.806+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:10:43.026858+0000 (oldest deadline 2023-12-16T22:11:03.026858+0000) 2023-12-16T22:12:45.065 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:12:44 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:12:44.808+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:12:45.065 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:12:44 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:12:44.770+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:11:07.849373+0000 (oldest deadline 2023-12-16T22:11:27.849373+0000) 2023-12-16T22:12:45.065 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:12:44 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:12:44.770+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:11:07.849373+0000 (oldest deadline 2023-12-16T22:11:27.849373+0000) 2023-12-16T22:12:45.066 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:12:44 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:12:44.770+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:11:07.849373+0000 (oldest deadline 2023-12-16T22:11:27.849373+0000) 2023-12-16T22:12:45.066 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:12:44 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:12:44.771+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:12:45.750 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:12:45 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:12:45.420+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:10:42.731761+0000 (oldest deadline 2023-12-16T22:11:02.731761+0000) 2023-12-16T22:12:45.750 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:12:45 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:12:45.420+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:10:42.731761+0000 (oldest deadline 2023-12-16T22:11:02.731761+0000) 2023-12-16T22:12:45.750 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:12:45 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:12:45.420+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:10:42.731761+0000 (oldest deadline 2023-12-16T22:11:02.731761+0000) 2023-12-16T22:12:45.751 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:12:45 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:12:45.421+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:12:46.064 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:12:45 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:12:45.749+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:11:07.849373+0000 (oldest deadline 2023-12-16T22:11:27.849373+0000) 2023-12-16T22:12:46.065 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:12:45 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:12:45.749+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:11:07.849373+0000 (oldest deadline 2023-12-16T22:11:27.849373+0000) 2023-12-16T22:12:46.065 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:12:45 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:12:45.749+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:11:07.849373+0000 (oldest deadline 2023-12-16T22:11:27.849373+0000) 2023-12-16T22:12:46.065 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:12:45 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:12:45.750+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:12:46.065 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:12:45 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:12:45.790+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:10:43.026858+0000 (oldest deadline 2023-12-16T22:11:03.026858+0000) 2023-12-16T22:12:46.065 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:12:45 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:12:45.790+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:10:43.026858+0000 (oldest deadline 2023-12-16T22:11:03.026858+0000) 2023-12-16T22:12:46.066 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:12:45 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:12:45.790+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:10:43.026858+0000 (oldest deadline 2023-12-16T22:11:03.026858+0000) 2023-12-16T22:12:46.066 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:12:45 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:12:45.791+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:12:46.763 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:12:46 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:12:46.419+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:10:42.731761+0000 (oldest deadline 2023-12-16T22:11:02.731761+0000) 2023-12-16T22:12:46.764 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:12:46 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:12:46.419+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:10:42.731761+0000 (oldest deadline 2023-12-16T22:11:02.731761+0000) 2023-12-16T22:12:46.764 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:12:46 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:12:46.419+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:10:42.731761+0000 (oldest deadline 2023-12-16T22:11:02.731761+0000) 2023-12-16T22:12:46.764 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:12:46 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:12:46.420+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:12:47.062 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:12:46 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:12:46.762+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:11:07.849373+0000 (oldest deadline 2023-12-16T22:11:27.849373+0000) 2023-12-16T22:12:47.062 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:12:46 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:12:46.762+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:11:07.849373+0000 (oldest deadline 2023-12-16T22:11:27.849373+0000) 2023-12-16T22:12:47.063 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:12:46 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:12:46.762+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:11:07.849373+0000 (oldest deadline 2023-12-16T22:11:27.849373+0000) 2023-12-16T22:12:47.063 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:12:46 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:12:46.763+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:12:47.063 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:12:46 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:12:46.834+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:10:43.026858+0000 (oldest deadline 2023-12-16T22:11:03.026858+0000) 2023-12-16T22:12:47.063 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:12:46 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:12:46.834+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:10:43.026858+0000 (oldest deadline 2023-12-16T22:11:03.026858+0000) 2023-12-16T22:12:47.063 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:12:46 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:12:46.834+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:10:43.026858+0000 (oldest deadline 2023-12-16T22:11:03.026858+0000) 2023-12-16T22:12:47.063 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:12:46 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:12:46.835+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:12:47.314 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Dec 16 22:12:47 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-mon-smithi045[80759]: 2023-12-16T22:12:47.061+0000 7f45a1f8f700 -1 mon.smithi045@0(probing) e2 get_health_metrics reporting 33263 slow ops, oldest is monmgrreport(gid 14214, 0 checks, 1 progress events) 2023-12-16T22:12:47.791 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:12:47 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:12:47.407+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:10:42.731761+0000 (oldest deadline 2023-12-16T22:11:02.731761+0000) 2023-12-16T22:12:47.791 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:12:47 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:12:47.407+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:10:42.731761+0000 (oldest deadline 2023-12-16T22:11:02.731761+0000) 2023-12-16T22:12:47.792 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:12:47 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:12:47.407+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:10:42.731761+0000 (oldest deadline 2023-12-16T22:11:02.731761+0000) 2023-12-16T22:12:47.792 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:12:47 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:12:47.409+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:12:48.064 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:12:47 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:12:47.790+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:10:43.026858+0000 (oldest deadline 2023-12-16T22:11:03.026858+0000) 2023-12-16T22:12:48.065 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:12:47 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:12:47.790+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:10:43.026858+0000 (oldest deadline 2023-12-16T22:11:03.026858+0000) 2023-12-16T22:12:48.065 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:12:47 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:12:47.790+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:10:43.026858+0000 (oldest deadline 2023-12-16T22:11:03.026858+0000) 2023-12-16T22:12:48.065 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:12:47 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:12:47.791+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:12:48.065 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:12:47 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:12:47.808+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:11:07.849373+0000 (oldest deadline 2023-12-16T22:11:27.849373+0000) 2023-12-16T22:12:48.065 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:12:47 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:12:47.808+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:11:07.849373+0000 (oldest deadline 2023-12-16T22:11:27.849373+0000) 2023-12-16T22:12:48.065 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:12:47 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:12:47.808+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:11:07.849373+0000 (oldest deadline 2023-12-16T22:11:27.849373+0000) 2023-12-16T22:12:48.066 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:12:47 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:12:47.809+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:12:48.721 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:12:48 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:12:48.438+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:10:42.731761+0000 (oldest deadline 2023-12-16T22:11:02.731761+0000) 2023-12-16T22:12:48.721 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:12:48 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:12:48.438+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:10:42.731761+0000 (oldest deadline 2023-12-16T22:11:02.731761+0000) 2023-12-16T22:12:48.721 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:12:48 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:12:48.438+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:10:42.731761+0000 (oldest deadline 2023-12-16T22:11:02.731761+0000) 2023-12-16T22:12:48.721 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:12:48 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:12:48.439+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:12:49.064 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:12:48 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:12:48.821+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:10:43.026858+0000 (oldest deadline 2023-12-16T22:11:03.026858+0000) 2023-12-16T22:12:49.064 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:12:48 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:12:48.821+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:10:43.026858+0000 (oldest deadline 2023-12-16T22:11:03.026858+0000) 2023-12-16T22:12:49.065 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:12:48 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:12:48.821+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:10:43.026858+0000 (oldest deadline 2023-12-16T22:11:03.026858+0000) 2023-12-16T22:12:49.065 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:12:48 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:12:48.822+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:12:49.065 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:12:48 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:12:48.856+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:11:07.849373+0000 (oldest deadline 2023-12-16T22:11:27.849373+0000) 2023-12-16T22:12:49.065 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:12:48 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:12:48.856+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:11:07.849373+0000 (oldest deadline 2023-12-16T22:11:27.849373+0000) 2023-12-16T22:12:49.065 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:12:48 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:12:48.856+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:11:07.849373+0000 (oldest deadline 2023-12-16T22:11:27.849373+0000) 2023-12-16T22:12:49.065 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:12:48 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:12:48.857+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:12:49.814 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:12:49 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:12:49.432+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:10:42.731761+0000 (oldest deadline 2023-12-16T22:11:02.731761+0000) 2023-12-16T22:12:49.814 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:12:49 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:12:49.432+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:10:42.731761+0000 (oldest deadline 2023-12-16T22:11:02.731761+0000) 2023-12-16T22:12:49.815 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:12:49 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:12:49.432+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:10:42.731761+0000 (oldest deadline 2023-12-16T22:11:02.731761+0000) 2023-12-16T22:12:49.815 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:12:49 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:12:49.433+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:12:50.224 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:12:49 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:12:49.871+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:10:43.026858+0000 (oldest deadline 2023-12-16T22:11:03.026858+0000) 2023-12-16T22:12:50.225 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:12:49 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:12:49.871+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:10:43.026858+0000 (oldest deadline 2023-12-16T22:11:03.026858+0000) 2023-12-16T22:12:50.226 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:12:49 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:12:49.871+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:10:43.026858+0000 (oldest deadline 2023-12-16T22:11:03.026858+0000) 2023-12-16T22:12:50.226 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:12:49 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:12:49.872+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:12:50.226 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:12:49 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:12:49.869+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:11:07.849373+0000 (oldest deadline 2023-12-16T22:11:27.849373+0000) 2023-12-16T22:12:50.226 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:12:49 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:12:49.869+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:11:07.849373+0000 (oldest deadline 2023-12-16T22:11:27.849373+0000) 2023-12-16T22:12:50.226 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:12:49 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:12:49.869+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:11:07.849373+0000 (oldest deadline 2023-12-16T22:11:27.849373+0000) 2023-12-16T22:12:50.226 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:12:49 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:12:49.870+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:12:50.814 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:12:50 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:12:50.476+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:10:42.731761+0000 (oldest deadline 2023-12-16T22:11:02.731761+0000) 2023-12-16T22:12:50.814 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:12:50 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:12:50.476+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:10:42.731761+0000 (oldest deadline 2023-12-16T22:11:02.731761+0000) 2023-12-16T22:12:50.815 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:12:50 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:12:50.476+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:10:42.731761+0000 (oldest deadline 2023-12-16T22:11:02.731761+0000) 2023-12-16T22:12:50.815 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:12:50 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:12:50.477+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:12:51.315 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:12:50 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:12:50.840+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:10:43.026858+0000 (oldest deadline 2023-12-16T22:11:03.026858+0000) 2023-12-16T22:12:51.315 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:12:50 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:12:50.840+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:10:43.026858+0000 (oldest deadline 2023-12-16T22:11:03.026858+0000) 2023-12-16T22:12:51.315 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:12:50 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:12:50.840+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:10:43.026858+0000 (oldest deadline 2023-12-16T22:11:03.026858+0000) 2023-12-16T22:12:51.315 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:12:50 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:12:50.841+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:12:51.316 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:12:50 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:12:50.904+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:11:07.849373+0000 (oldest deadline 2023-12-16T22:11:27.849373+0000) 2023-12-16T22:12:51.316 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:12:50 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:12:50.904+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:11:07.849373+0000 (oldest deadline 2023-12-16T22:11:27.849373+0000) 2023-12-16T22:12:51.316 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:12:50 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:12:50.904+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:11:07.849373+0000 (oldest deadline 2023-12-16T22:11:27.849373+0000) 2023-12-16T22:12:51.317 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:12:50 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:12:50.905+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:12:51.752 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:12:51 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:12:51.486+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:10:42.731761+0000 (oldest deadline 2023-12-16T22:11:02.731761+0000) 2023-12-16T22:12:51.752 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:12:51 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:12:51.486+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:10:42.731761+0000 (oldest deadline 2023-12-16T22:11:02.731761+0000) 2023-12-16T22:12:51.753 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:12:51 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:12:51.486+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:10:42.731761+0000 (oldest deadline 2023-12-16T22:11:02.731761+0000) 2023-12-16T22:12:51.753 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:12:51 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:12:51.487+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:12:52.064 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:12:51 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:12:51.918+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:11:07.849373+0000 (oldest deadline 2023-12-16T22:11:27.849373+0000) 2023-12-16T22:12:52.065 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:12:51 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:12:51.918+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:11:07.849373+0000 (oldest deadline 2023-12-16T22:11:27.849373+0000) 2023-12-16T22:12:52.065 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:12:51 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:12:51.918+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:11:07.849373+0000 (oldest deadline 2023-12-16T22:11:27.849373+0000) 2023-12-16T22:12:52.065 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:12:51 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:12:51.919+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:12:52.065 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:12:51 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:12:51.888+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:10:43.026858+0000 (oldest deadline 2023-12-16T22:11:03.026858+0000) 2023-12-16T22:12:52.065 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:12:51 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:12:51.888+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:10:43.026858+0000 (oldest deadline 2023-12-16T22:11:03.026858+0000) 2023-12-16T22:12:52.066 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:12:51 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:12:51.888+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:10:43.026858+0000 (oldest deadline 2023-12-16T22:11:03.026858+0000) 2023-12-16T22:12:52.066 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:12:51 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:12:51.889+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:12:52.442 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Dec 16 22:12:52 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-mon-smithi045[80759]: 2023-12-16T22:12:52.066+0000 7f45a1f8f700 -1 mon.smithi045@0(probing) e2 get_health_metrics reporting 33348 slow ops, oldest is monmgrreport(gid 14214, 0 checks, 1 progress events) 2023-12-16T22:12:52.814 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:12:52 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:12:52.441+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:10:42.731761+0000 (oldest deadline 2023-12-16T22:11:02.731761+0000) 2023-12-16T22:12:52.814 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:12:52 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:12:52.441+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:10:42.731761+0000 (oldest deadline 2023-12-16T22:11:02.731761+0000) 2023-12-16T22:12:52.815 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:12:52 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:12:52.441+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:10:42.731761+0000 (oldest deadline 2023-12-16T22:11:02.731761+0000) 2023-12-16T22:12:52.815 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:12:52 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:12:52.442+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:12:53.314 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:12:52 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:12:52.863+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:10:43.026858+0000 (oldest deadline 2023-12-16T22:11:03.026858+0000) 2023-12-16T22:12:53.315 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:12:52 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:12:52.863+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:10:43.026858+0000 (oldest deadline 2023-12-16T22:11:03.026858+0000) 2023-12-16T22:12:53.315 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:12:52 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:12:52.863+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:10:43.026858+0000 (oldest deadline 2023-12-16T22:11:03.026858+0000) 2023-12-16T22:12:53.315 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:12:52 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:12:52.864+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:12:53.315 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:12:52 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:12:52.910+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:11:07.849373+0000 (oldest deadline 2023-12-16T22:11:27.849373+0000) 2023-12-16T22:12:53.315 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:12:52 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:12:52.910+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:11:07.849373+0000 (oldest deadline 2023-12-16T22:11:27.849373+0000) 2023-12-16T22:12:53.315 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:12:52 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:12:52.910+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:11:07.849373+0000 (oldest deadline 2023-12-16T22:11:27.849373+0000) 2023-12-16T22:12:53.316 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:12:52 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:12:52.911+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:12:53.814 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:12:53 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:12:53.469+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:10:42.731761+0000 (oldest deadline 2023-12-16T22:11:02.731761+0000) 2023-12-16T22:12:53.814 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:12:53 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:12:53.469+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:10:42.731761+0000 (oldest deadline 2023-12-16T22:11:02.731761+0000) 2023-12-16T22:12:53.814 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:12:53 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:12:53.469+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:10:42.731761+0000 (oldest deadline 2023-12-16T22:11:02.731761+0000) 2023-12-16T22:12:53.815 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:12:53 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:12:53.470+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:12:54.314 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:12:53 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:12:53.912+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:11:07.849373+0000 (oldest deadline 2023-12-16T22:11:27.849373+0000) 2023-12-16T22:12:54.315 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:12:53 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:12:53.912+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:11:07.849373+0000 (oldest deadline 2023-12-16T22:11:27.849373+0000) 2023-12-16T22:12:54.315 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:12:53 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:12:53.912+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:11:07.849373+0000 (oldest deadline 2023-12-16T22:11:27.849373+0000) 2023-12-16T22:12:54.315 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:12:53 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:12:53.912+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:12:54.315 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:12:53 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:12:53.855+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:10:43.026858+0000 (oldest deadline 2023-12-16T22:11:03.026858+0000) 2023-12-16T22:12:54.315 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:12:53 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:12:53.855+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:10:43.026858+0000 (oldest deadline 2023-12-16T22:11:03.026858+0000) 2023-12-16T22:12:54.315 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:12:53 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:12:53.855+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:10:43.026858+0000 (oldest deadline 2023-12-16T22:11:03.026858+0000) 2023-12-16T22:12:54.315 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:12:53 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:12:53.857+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:12:54.814 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:12:54 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:12:54.444+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:10:42.731761+0000 (oldest deadline 2023-12-16T22:11:02.731761+0000) 2023-12-16T22:12:54.814 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:12:54 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:12:54.444+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:10:42.731761+0000 (oldest deadline 2023-12-16T22:11:02.731761+0000) 2023-12-16T22:12:54.815 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:12:54 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:12:54.444+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:10:42.731761+0000 (oldest deadline 2023-12-16T22:11:02.731761+0000) 2023-12-16T22:12:54.815 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:12:54 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:12:54.445+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:12:55.314 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:12:54 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:12:54.939+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:11:07.849373+0000 (oldest deadline 2023-12-16T22:11:27.849373+0000) 2023-12-16T22:12:55.315 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:12:54 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:12:54.939+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:11:07.849373+0000 (oldest deadline 2023-12-16T22:11:27.849373+0000) 2023-12-16T22:12:55.315 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:12:54 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:12:54.939+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:11:07.849373+0000 (oldest deadline 2023-12-16T22:11:27.849373+0000) 2023-12-16T22:12:55.315 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:12:54 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:12:54.940+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:12:55.315 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:12:54 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:12:54.844+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:10:43.026858+0000 (oldest deadline 2023-12-16T22:11:03.026858+0000) 2023-12-16T22:12:55.315 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:12:54 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:12:54.844+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:10:43.026858+0000 (oldest deadline 2023-12-16T22:11:03.026858+0000) 2023-12-16T22:12:55.316 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:12:54 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:12:54.844+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:10:43.026858+0000 (oldest deadline 2023-12-16T22:11:03.026858+0000) 2023-12-16T22:12:55.316 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:12:54 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:12:54.845+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:12:55.815 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:12:55 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:12:55.494+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:10:42.731761+0000 (oldest deadline 2023-12-16T22:11:02.731761+0000) 2023-12-16T22:12:55.815 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:12:55 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:12:55.494+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:10:42.731761+0000 (oldest deadline 2023-12-16T22:11:02.731761+0000) 2023-12-16T22:12:55.815 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:12:55 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:12:55.494+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:10:42.731761+0000 (oldest deadline 2023-12-16T22:11:02.731761+0000) 2023-12-16T22:12:55.815 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:12:55 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:12:55.495+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:12:56.314 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:12:55 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:12:55.971+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:11:07.849373+0000 (oldest deadline 2023-12-16T22:11:27.849373+0000) 2023-12-16T22:12:56.314 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:12:55 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:12:55.971+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:11:07.849373+0000 (oldest deadline 2023-12-16T22:11:27.849373+0000) 2023-12-16T22:12:56.315 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:12:55 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:12:55.971+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:11:07.849373+0000 (oldest deadline 2023-12-16T22:11:27.849373+0000) 2023-12-16T22:12:56.315 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:12:55 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:12:55.972+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:12:56.315 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:12:55 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:12:55.862+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:10:43.026858+0000 (oldest deadline 2023-12-16T22:11:03.026858+0000) 2023-12-16T22:12:56.315 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:12:55 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:12:55.862+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:10:43.026858+0000 (oldest deadline 2023-12-16T22:11:03.026858+0000) 2023-12-16T22:12:56.315 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:12:55 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:12:55.862+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:10:43.026858+0000 (oldest deadline 2023-12-16T22:11:03.026858+0000) 2023-12-16T22:12:56.315 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:12:55 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:12:55.863+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:12:56.814 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:12:56 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:12:56.476+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:10:42.731761+0000 (oldest deadline 2023-12-16T22:11:02.731761+0000) 2023-12-16T22:12:56.814 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:12:56 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:12:56.476+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:10:42.731761+0000 (oldest deadline 2023-12-16T22:11:02.731761+0000) 2023-12-16T22:12:56.814 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:12:56 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:12:56.476+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:10:42.731761+0000 (oldest deadline 2023-12-16T22:11:02.731761+0000) 2023-12-16T22:12:56.815 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:12:56 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:12:56.477+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:12:57.194 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Dec 16 22:12:57 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-mon-smithi045[80759]: 2023-12-16T22:12:57.070+0000 7f45a1f8f700 -1 mon.smithi045@0(probing) e2 get_health_metrics reporting 33433 slow ops, oldest is monmgrreport(gid 14214, 0 checks, 1 progress events) 2023-12-16T22:12:57.194 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:12:56 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:12:56.913+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:10:43.026858+0000 (oldest deadline 2023-12-16T22:11:03.026858+0000) 2023-12-16T22:12:57.194 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:12:56 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:12:56.913+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:10:43.026858+0000 (oldest deadline 2023-12-16T22:11:03.026858+0000) 2023-12-16T22:12:57.195 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:12:56 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:12:56.913+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:10:43.026858+0000 (oldest deadline 2023-12-16T22:11:03.026858+0000) 2023-12-16T22:12:57.195 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:12:56 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:12:56.914+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:12:57.195 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:12:56 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:12:56.927+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:11:07.849373+0000 (oldest deadline 2023-12-16T22:11:27.849373+0000) 2023-12-16T22:12:57.195 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:12:56 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:12:56.927+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:11:07.849373+0000 (oldest deadline 2023-12-16T22:11:27.849373+0000) 2023-12-16T22:12:57.195 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:12:56 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:12:56.927+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:11:07.849373+0000 (oldest deadline 2023-12-16T22:11:27.849373+0000) 2023-12-16T22:12:57.195 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:12:56 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:12:56.928+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:12:57.814 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:12:57 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:12:57.448+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:10:42.731761+0000 (oldest deadline 2023-12-16T22:11:02.731761+0000) 2023-12-16T22:12:57.814 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:12:57 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:12:57.448+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:10:42.731761+0000 (oldest deadline 2023-12-16T22:11:02.731761+0000) 2023-12-16T22:12:57.815 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:12:57 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:12:57.448+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:10:42.731761+0000 (oldest deadline 2023-12-16T22:11:02.731761+0000) 2023-12-16T22:12:57.815 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:12:57 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:12:57.450+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:12:58.314 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:12:57 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:12:57.864+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:10:43.026858+0000 (oldest deadline 2023-12-16T22:11:03.026858+0000) 2023-12-16T22:12:58.315 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:12:57 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:12:57.864+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:10:43.026858+0000 (oldest deadline 2023-12-16T22:11:03.026858+0000) 2023-12-16T22:12:58.315 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:12:57 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:12:57.864+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:10:43.026858+0000 (oldest deadline 2023-12-16T22:11:03.026858+0000) 2023-12-16T22:12:58.315 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:12:57 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:12:57.865+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:12:58.315 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:12:57 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:12:57.958+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:11:07.849373+0000 (oldest deadline 2023-12-16T22:11:27.849373+0000) 2023-12-16T22:12:58.315 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:12:57 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:12:57.958+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:11:07.849373+0000 (oldest deadline 2023-12-16T22:11:27.849373+0000) 2023-12-16T22:12:58.315 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:12:57 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:12:57.958+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:11:07.849373+0000 (oldest deadline 2023-12-16T22:11:27.849373+0000) 2023-12-16T22:12:58.315 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:12:57 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:12:57.958+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:12:58.720 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:12:58 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:12:58.447+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:10:42.731761+0000 (oldest deadline 2023-12-16T22:11:02.731761+0000) 2023-12-16T22:12:58.720 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:12:58 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:12:58.447+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:10:42.731761+0000 (oldest deadline 2023-12-16T22:11:02.731761+0000) 2023-12-16T22:12:58.720 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:12:58 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:12:58.447+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:10:42.731761+0000 (oldest deadline 2023-12-16T22:11:02.731761+0000) 2023-12-16T22:12:58.720 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:12:58 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:12:58.448+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:12:59.064 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:12:58 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:12:58.961+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:11:07.849373+0000 (oldest deadline 2023-12-16T22:11:27.849373+0000) 2023-12-16T22:12:59.064 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:12:58 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:12:58.961+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:11:07.849373+0000 (oldest deadline 2023-12-16T22:11:27.849373+0000) 2023-12-16T22:12:59.065 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:12:58 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:12:58.961+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:11:07.849373+0000 (oldest deadline 2023-12-16T22:11:27.849373+0000) 2023-12-16T22:12:59.065 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:12:58 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:12:58.962+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:12:59.065 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:12:58 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:12:58.838+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:10:43.026858+0000 (oldest deadline 2023-12-16T22:11:03.026858+0000) 2023-12-16T22:12:59.065 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:12:58 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:12:58.838+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:10:43.026858+0000 (oldest deadline 2023-12-16T22:11:03.026858+0000) 2023-12-16T22:12:59.065 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:12:58 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:12:58.838+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:10:43.026858+0000 (oldest deadline 2023-12-16T22:11:03.026858+0000) 2023-12-16T22:12:59.065 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:12:58 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:12:58.838+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:12:59.814 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:12:59 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:12:59.416+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:10:42.731761+0000 (oldest deadline 2023-12-16T22:11:02.731761+0000) 2023-12-16T22:12:59.814 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:12:59 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:12:59.416+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:10:42.731761+0000 (oldest deadline 2023-12-16T22:11:02.731761+0000) 2023-12-16T22:12:59.814 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:12:59 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:12:59.416+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:10:42.731761+0000 (oldest deadline 2023-12-16T22:11:02.731761+0000) 2023-12-16T22:12:59.815 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:12:59 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:12:59.417+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:13:00.223 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:12:59 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:12:59.871+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:10:43.026858+0000 (oldest deadline 2023-12-16T22:11:03.026858+0000) 2023-12-16T22:13:00.224 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:12:59 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:12:59.871+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:10:43.026858+0000 (oldest deadline 2023-12-16T22:11:03.026858+0000) 2023-12-16T22:13:00.224 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:12:59 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:12:59.871+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:10:43.026858+0000 (oldest deadline 2023-12-16T22:11:03.026858+0000) 2023-12-16T22:13:00.224 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:12:59 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:12:59.872+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:13:00.224 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:12:59 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:12:59.933+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:11:07.849373+0000 (oldest deadline 2023-12-16T22:11:27.849373+0000) 2023-12-16T22:13:00.224 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:12:59 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:12:59.933+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:11:07.849373+0000 (oldest deadline 2023-12-16T22:11:27.849373+0000) 2023-12-16T22:13:00.225 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:12:59 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:12:59.933+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:11:07.849373+0000 (oldest deadline 2023-12-16T22:11:27.849373+0000) 2023-12-16T22:13:00.225 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:12:59 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:12:59.934+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:13:00.564 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:13:00 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:13:00.387+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:10:42.731761+0000 (oldest deadline 2023-12-16T22:11:02.731761+0000) 2023-12-16T22:13:00.564 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:13:00 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:13:00.387+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:10:42.731761+0000 (oldest deadline 2023-12-16T22:11:02.731761+0000) 2023-12-16T22:13:00.565 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:13:00 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:13:00.387+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:10:42.731761+0000 (oldest deadline 2023-12-16T22:11:02.731761+0000) 2023-12-16T22:13:00.565 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:13:00 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:13:00.388+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:13:01.314 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:13:00 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:13:00.911+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:11:07.849373+0000 (oldest deadline 2023-12-16T22:11:27.849373+0000) 2023-12-16T22:13:01.314 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:13:00 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:13:00.911+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:11:07.849373+0000 (oldest deadline 2023-12-16T22:11:27.849373+0000) 2023-12-16T22:13:01.315 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:13:00 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:13:00.911+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:11:07.849373+0000 (oldest deadline 2023-12-16T22:11:27.849373+0000) 2023-12-16T22:13:01.315 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:13:00 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:13:00.912+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:13:01.315 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:13:00 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:13:00.912+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:10:43.026858+0000 (oldest deadline 2023-12-16T22:11:03.026858+0000) 2023-12-16T22:13:01.315 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:13:00 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:13:00.912+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:10:43.026858+0000 (oldest deadline 2023-12-16T22:11:03.026858+0000) 2023-12-16T22:13:01.315 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:13:00 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:13:00.912+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:10:43.026858+0000 (oldest deadline 2023-12-16T22:11:03.026858+0000) 2023-12-16T22:13:01.315 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:13:00 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:13:00.913+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:13:01.753 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:13:01 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:13:01.383+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:10:42.731761+0000 (oldest deadline 2023-12-16T22:11:02.731761+0000) 2023-12-16T22:13:01.753 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:13:01 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:13:01.383+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:10:42.731761+0000 (oldest deadline 2023-12-16T22:11:02.731761+0000) 2023-12-16T22:13:01.753 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:13:01 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:13:01.383+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:10:42.731761+0000 (oldest deadline 2023-12-16T22:11:02.731761+0000) 2023-12-16T22:13:01.753 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:13:01 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:13:01.384+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:13:02.064 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:13:01 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:13:01.939+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:10:43.026858+0000 (oldest deadline 2023-12-16T22:11:03.026858+0000) 2023-12-16T22:13:02.064 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:13:01 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:13:01.939+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:10:43.026858+0000 (oldest deadline 2023-12-16T22:11:03.026858+0000) 2023-12-16T22:13:02.065 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:13:01 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:13:01.939+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:10:43.026858+0000 (oldest deadline 2023-12-16T22:11:03.026858+0000) 2023-12-16T22:13:02.065 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:13:01 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:13:01.940+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:13:02.065 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:13:01 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:13:01.944+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:11:07.849373+0000 (oldest deadline 2023-12-16T22:11:27.849373+0000) 2023-12-16T22:13:02.065 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:13:01 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:13:01.944+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:11:07.849373+0000 (oldest deadline 2023-12-16T22:11:27.849373+0000) 2023-12-16T22:13:02.065 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:13:01 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:13:01.944+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:11:07.849373+0000 (oldest deadline 2023-12-16T22:11:27.849373+0000) 2023-12-16T22:13:02.065 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:13:01 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:13:01.945+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:13:02.393 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Dec 16 22:13:02 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-mon-smithi045[80759]: 2023-12-16T22:13:02.075+0000 7f45a1f8f700 -1 mon.smithi045@0(probing) e2 get_health_metrics reporting 33517 slow ops, oldest is monmgrreport(gid 14214, 0 checks, 1 progress events) 2023-12-16T22:13:02.814 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:13:02 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:13:02.392+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:10:42.731761+0000 (oldest deadline 2023-12-16T22:11:02.731761+0000) 2023-12-16T22:13:02.814 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:13:02 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:13:02.392+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:10:42.731761+0000 (oldest deadline 2023-12-16T22:11:02.731761+0000) 2023-12-16T22:13:02.815 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:13:02 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:13:02.392+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:10:42.731761+0000 (oldest deadline 2023-12-16T22:11:02.731761+0000) 2023-12-16T22:13:02.815 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:13:02 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:13:02.394+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:13:03.314 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:13:02 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:13:02.982+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:10:43.026858+0000 (oldest deadline 2023-12-16T22:11:03.026858+0000) 2023-12-16T22:13:03.314 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:13:02 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:13:02.982+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:10:43.026858+0000 (oldest deadline 2023-12-16T22:11:03.026858+0000) 2023-12-16T22:13:03.315 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:13:02 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:13:02.982+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:10:43.026858+0000 (oldest deadline 2023-12-16T22:11:03.026858+0000) 2023-12-16T22:13:03.315 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:13:02 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:13:02.984+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:13:03.315 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:13:02 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:13:02.963+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:11:07.849373+0000 (oldest deadline 2023-12-16T22:11:27.849373+0000) 2023-12-16T22:13:03.315 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:13:02 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:13:02.963+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:11:07.849373+0000 (oldest deadline 2023-12-16T22:11:27.849373+0000) 2023-12-16T22:13:03.316 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:13:02 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:13:02.963+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:11:07.849373+0000 (oldest deadline 2023-12-16T22:11:27.849373+0000) 2023-12-16T22:13:03.316 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:13:02 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:13:02.964+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:13:03.814 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:13:03 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:13:03.416+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:10:42.731761+0000 (oldest deadline 2023-12-16T22:11:02.731761+0000) 2023-12-16T22:13:03.815 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:13:03 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:13:03.416+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:10:42.731761+0000 (oldest deadline 2023-12-16T22:11:02.731761+0000) 2023-12-16T22:13:03.815 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:13:03 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:13:03.416+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:10:42.731761+0000 (oldest deadline 2023-12-16T22:11:02.731761+0000) 2023-12-16T22:13:03.815 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:13:03 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:13:03.416+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:13:04.314 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:13:03 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:13:03.944+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:11:07.849373+0000 (oldest deadline 2023-12-16T22:11:27.849373+0000) 2023-12-16T22:13:04.314 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:13:03 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:13:03.944+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:11:07.849373+0000 (oldest deadline 2023-12-16T22:11:27.849373+0000) 2023-12-16T22:13:04.315 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:13:03 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:13:03.944+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:11:07.849373+0000 (oldest deadline 2023-12-16T22:11:27.849373+0000) 2023-12-16T22:13:04.315 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:13:03 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:13:03.945+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:13:04.315 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:13:03 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:13:03.977+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:10:43.026858+0000 (oldest deadline 2023-12-16T22:11:03.026858+0000) 2023-12-16T22:13:04.315 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:13:03 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:13:03.977+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:10:43.026858+0000 (oldest deadline 2023-12-16T22:11:03.026858+0000) 2023-12-16T22:13:04.315 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:13:03 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:13:03.977+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:10:43.026858+0000 (oldest deadline 2023-12-16T22:11:03.026858+0000) 2023-12-16T22:13:04.316 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:13:03 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:13:03.979+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:13:04.814 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:13:04 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:13:04.411+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:10:42.731761+0000 (oldest deadline 2023-12-16T22:11:02.731761+0000) 2023-12-16T22:13:04.815 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:13:04 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:13:04.411+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:10:42.731761+0000 (oldest deadline 2023-12-16T22:11:02.731761+0000) 2023-12-16T22:13:04.815 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:13:04 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:13:04.411+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:10:42.731761+0000 (oldest deadline 2023-12-16T22:11:02.731761+0000) 2023-12-16T22:13:04.815 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:13:04 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:13:04.412+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:13:05.314 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:13:04 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:13:04.926+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:11:07.849373+0000 (oldest deadline 2023-12-16T22:11:27.849373+0000) 2023-12-16T22:13:05.314 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:13:04 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:13:04.926+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:11:07.849373+0000 (oldest deadline 2023-12-16T22:11:27.849373+0000) 2023-12-16T22:13:05.315 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:13:04 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:13:04.926+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:11:07.849373+0000 (oldest deadline 2023-12-16T22:11:27.849373+0000) 2023-12-16T22:13:05.315 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:13:04 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:13:04.927+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:13:05.315 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:13:04 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:13:04.972+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:10:43.026858+0000 (oldest deadline 2023-12-16T22:11:03.026858+0000) 2023-12-16T22:13:05.315 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:13:04 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:13:04.972+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:10:43.026858+0000 (oldest deadline 2023-12-16T22:11:03.026858+0000) 2023-12-16T22:13:05.315 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:13:04 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:13:04.972+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:10:43.026858+0000 (oldest deadline 2023-12-16T22:11:03.026858+0000) 2023-12-16T22:13:05.315 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:13:04 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:13:04.973+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:13:05.814 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:13:05 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:13:05.420+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:10:42.731761+0000 (oldest deadline 2023-12-16T22:11:02.731761+0000) 2023-12-16T22:13:05.814 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:13:05 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:13:05.420+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:10:42.731761+0000 (oldest deadline 2023-12-16T22:11:02.731761+0000) 2023-12-16T22:13:05.815 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:13:05 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:13:05.420+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:10:42.731761+0000 (oldest deadline 2023-12-16T22:11:02.731761+0000) 2023-12-16T22:13:05.815 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:13:05 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:13:05.421+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:13:06.314 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:13:05 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:13:05.937+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:10:43.026858+0000 (oldest deadline 2023-12-16T22:11:03.026858+0000) 2023-12-16T22:13:06.315 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:13:05 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:13:05.937+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:10:43.026858+0000 (oldest deadline 2023-12-16T22:11:03.026858+0000) 2023-12-16T22:13:06.315 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:13:05 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:13:05.937+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:10:43.026858+0000 (oldest deadline 2023-12-16T22:11:03.026858+0000) 2023-12-16T22:13:06.315 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:13:05 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:13:05.938+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:13:06.315 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:13:05 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:13:05.935+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:11:07.849373+0000 (oldest deadline 2023-12-16T22:11:27.849373+0000) 2023-12-16T22:13:06.315 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:13:05 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:13:05.935+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:11:07.849373+0000 (oldest deadline 2023-12-16T22:11:27.849373+0000) 2023-12-16T22:13:06.315 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:13:05 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:13:05.935+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:11:07.849373+0000 (oldest deadline 2023-12-16T22:11:27.849373+0000) 2023-12-16T22:13:06.315 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:13:05 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:13:05.936+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:13:06.814 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:13:06 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:13:06.461+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:10:42.731761+0000 (oldest deadline 2023-12-16T22:11:02.731761+0000) 2023-12-16T22:13:06.814 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:13:06 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:13:06.461+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:10:42.731761+0000 (oldest deadline 2023-12-16T22:11:02.731761+0000) 2023-12-16T22:13:06.815 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:13:06 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:13:06.461+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:10:42.731761+0000 (oldest deadline 2023-12-16T22:11:02.731761+0000) 2023-12-16T22:13:06.815 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:13:06 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:13:06.463+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:13:07.205 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Dec 16 22:13:07 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-mon-smithi045[80759]: 2023-12-16T22:13:07.079+0000 7f45a1f8f700 -1 mon.smithi045@0(probing) e2 get_health_metrics reporting 33605 slow ops, oldest is monmgrreport(gid 14214, 0 checks, 1 progress events) 2023-12-16T22:13:07.206 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:13:06 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:13:06.903+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:11:07.849373+0000 (oldest deadline 2023-12-16T22:11:27.849373+0000) 2023-12-16T22:13:07.206 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:13:06 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:13:06.903+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:11:07.849373+0000 (oldest deadline 2023-12-16T22:11:27.849373+0000) 2023-12-16T22:13:07.206 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:13:06 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:13:06.903+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:11:07.849373+0000 (oldest deadline 2023-12-16T22:11:27.849373+0000) 2023-12-16T22:13:07.206 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:13:06 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:13:06.904+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:13:07.206 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:13:06 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:13:06.907+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:10:43.026858+0000 (oldest deadline 2023-12-16T22:11:03.026858+0000) 2023-12-16T22:13:07.207 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:13:06 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:13:06.907+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:10:43.026858+0000 (oldest deadline 2023-12-16T22:11:03.026858+0000) 2023-12-16T22:13:07.207 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:13:06 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:13:06.907+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:10:43.026858+0000 (oldest deadline 2023-12-16T22:11:03.026858+0000) 2023-12-16T22:13:07.207 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:13:06 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:13:06.908+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:13:07.814 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:13:07 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:13:07.496+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:10:42.731761+0000 (oldest deadline 2023-12-16T22:11:02.731761+0000) 2023-12-16T22:13:07.814 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:13:07 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:13:07.496+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:10:42.731761+0000 (oldest deadline 2023-12-16T22:11:02.731761+0000) 2023-12-16T22:13:07.815 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:13:07 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:13:07.496+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:10:42.731761+0000 (oldest deadline 2023-12-16T22:11:02.731761+0000) 2023-12-16T22:13:07.815 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:13:07 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:13:07.498+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:13:08.314 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:13:07 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:13:07.864+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:10:43.026858+0000 (oldest deadline 2023-12-16T22:11:03.026858+0000) 2023-12-16T22:13:08.314 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:13:07 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:13:07.864+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:10:43.026858+0000 (oldest deadline 2023-12-16T22:11:03.026858+0000) 2023-12-16T22:13:08.315 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:13:07 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:13:07.864+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:10:43.026858+0000 (oldest deadline 2023-12-16T22:11:03.026858+0000) 2023-12-16T22:13:08.315 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:13:07 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:13:07.865+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:13:08.315 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:13:07 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:13:07.942+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:11:07.849373+0000 (oldest deadline 2023-12-16T22:11:27.849373+0000) 2023-12-16T22:13:08.315 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:13:07 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:13:07.942+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:11:07.849373+0000 (oldest deadline 2023-12-16T22:11:27.849373+0000) 2023-12-16T22:13:08.315 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:13:07 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:13:07.942+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:11:07.849373+0000 (oldest deadline 2023-12-16T22:11:27.849373+0000) 2023-12-16T22:13:08.315 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:13:07 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:13:07.943+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:13:08.814 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:13:08 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:13:08.508+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:10:42.731761+0000 (oldest deadline 2023-12-16T22:11:02.731761+0000) 2023-12-16T22:13:08.814 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:13:08 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:13:08.508+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:10:42.731761+0000 (oldest deadline 2023-12-16T22:11:02.731761+0000) 2023-12-16T22:13:08.815 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:13:08 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:13:08.508+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:10:42.731761+0000 (oldest deadline 2023-12-16T22:11:02.731761+0000) 2023-12-16T22:13:08.815 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:13:08 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:13:08.509+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:13:09.314 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:13:08 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:13:08.855+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:10:43.026858+0000 (oldest deadline 2023-12-16T22:11:03.026858+0000) 2023-12-16T22:13:09.315 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:13:08 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:13:08.855+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:10:43.026858+0000 (oldest deadline 2023-12-16T22:11:03.026858+0000) 2023-12-16T22:13:09.315 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:13:08 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:13:08.855+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:10:43.026858+0000 (oldest deadline 2023-12-16T22:11:03.026858+0000) 2023-12-16T22:13:09.315 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:13:08 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:13:08.857+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:13:09.315 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:13:08 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:13:08.929+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:11:07.849373+0000 (oldest deadline 2023-12-16T22:11:27.849373+0000) 2023-12-16T22:13:09.315 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:13:08 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:13:08.929+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:11:07.849373+0000 (oldest deadline 2023-12-16T22:11:27.849373+0000) 2023-12-16T22:13:09.316 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:13:08 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:13:08.929+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:11:07.849373+0000 (oldest deadline 2023-12-16T22:11:27.849373+0000) 2023-12-16T22:13:09.316 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:13:08 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:13:08.930+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:13:09.814 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:13:09 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:13:09.472+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:10:42.731761+0000 (oldest deadline 2023-12-16T22:11:02.731761+0000) 2023-12-16T22:13:09.814 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:13:09 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:13:09.472+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:10:42.731761+0000 (oldest deadline 2023-12-16T22:11:02.731761+0000) 2023-12-16T22:13:09.814 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:13:09 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:13:09.472+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:10:42.731761+0000 (oldest deadline 2023-12-16T22:11:02.731761+0000) 2023-12-16T22:13:09.815 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:13:09 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:13:09.473+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:13:10.224 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:13:09 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:13:09.917+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:11:07.849373+0000 (oldest deadline 2023-12-16T22:11:27.849373+0000) 2023-12-16T22:13:10.224 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:13:09 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:13:09.917+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:11:07.849373+0000 (oldest deadline 2023-12-16T22:11:27.849373+0000) 2023-12-16T22:13:10.224 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:13:09 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:13:09.917+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:11:07.849373+0000 (oldest deadline 2023-12-16T22:11:27.849373+0000) 2023-12-16T22:13:10.224 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:13:09 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:13:09.919+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:13:10.224 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:13:09 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:13:09.836+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:10:43.026858+0000 (oldest deadline 2023-12-16T22:11:03.026858+0000) 2023-12-16T22:13:10.225 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:13:09 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:13:09.836+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:10:43.026858+0000 (oldest deadline 2023-12-16T22:11:03.026858+0000) 2023-12-16T22:13:10.225 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:13:09 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:13:09.836+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:10:43.026858+0000 (oldest deadline 2023-12-16T22:11:03.026858+0000) 2023-12-16T22:13:10.225 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:13:09 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:13:09.837+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:13:10.564 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:13:10 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:13:10.461+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:10:42.731761+0000 (oldest deadline 2023-12-16T22:11:02.731761+0000) 2023-12-16T22:13:10.565 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:13:10 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:13:10.461+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:10:42.731761+0000 (oldest deadline 2023-12-16T22:11:02.731761+0000) 2023-12-16T22:13:10.565 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:13:10 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:13:10.461+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:10:42.731761+0000 (oldest deadline 2023-12-16T22:11:02.731761+0000) 2023-12-16T22:13:10.565 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:13:10 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:13:10.463+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:13:11.314 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:13:10 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:13:10.919+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:11:07.849373+0000 (oldest deadline 2023-12-16T22:11:27.849373+0000) 2023-12-16T22:13:11.314 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:13:10 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:13:10.919+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:11:07.849373+0000 (oldest deadline 2023-12-16T22:11:27.849373+0000) 2023-12-16T22:13:11.315 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:13:10 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:13:10.919+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:11:07.849373+0000 (oldest deadline 2023-12-16T22:11:27.849373+0000) 2023-12-16T22:13:11.315 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:13:10 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:13:10.920+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:13:11.315 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:13:10 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:13:10.869+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:10:43.026858+0000 (oldest deadline 2023-12-16T22:11:03.026858+0000) 2023-12-16T22:13:11.315 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:13:10 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:13:10.869+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:10:43.026858+0000 (oldest deadline 2023-12-16T22:11:03.026858+0000) 2023-12-16T22:13:11.315 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:13:10 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:13:10.869+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:10:43.026858+0000 (oldest deadline 2023-12-16T22:11:03.026858+0000) 2023-12-16T22:13:11.315 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:13:10 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:13:10.869+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:13:11.753 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:13:11 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:13:11.429+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:10:42.731761+0000 (oldest deadline 2023-12-16T22:11:02.731761+0000) 2023-12-16T22:13:11.754 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:13:11 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:13:11.429+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:10:42.731761+0000 (oldest deadline 2023-12-16T22:11:02.731761+0000) 2023-12-16T22:13:11.754 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:13:11 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:13:11.429+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:10:42.731761+0000 (oldest deadline 2023-12-16T22:11:02.731761+0000) 2023-12-16T22:13:11.754 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:13:11 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:13:11.430+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:13:12.064 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:13:11 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:13:11.887+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:11:07.849373+0000 (oldest deadline 2023-12-16T22:11:27.849373+0000) 2023-12-16T22:13:12.064 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:13:11 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:13:11.887+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:11:07.849373+0000 (oldest deadline 2023-12-16T22:11:27.849373+0000) 2023-12-16T22:13:12.065 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:13:11 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:13:11.887+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:11:07.849373+0000 (oldest deadline 2023-12-16T22:11:27.849373+0000) 2023-12-16T22:13:12.065 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:13:11 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:13:11.888+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:13:12.065 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:13:11 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:13:11.846+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:10:43.026858+0000 (oldest deadline 2023-12-16T22:11:03.026858+0000) 2023-12-16T22:13:12.065 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:13:11 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:13:11.846+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:10:43.026858+0000 (oldest deadline 2023-12-16T22:11:03.026858+0000) 2023-12-16T22:13:12.065 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:13:11 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:13:11.846+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:10:43.026858+0000 (oldest deadline 2023-12-16T22:11:03.026858+0000) 2023-12-16T22:13:12.065 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:13:11 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:13:11.847+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:13:12.454 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Dec 16 22:13:12 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-mon-smithi045[80759]: 2023-12-16T22:13:12.083+0000 7f45a1f8f700 -1 mon.smithi045@0(probing) e2 get_health_metrics reporting 33693 slow ops, oldest is monmgrreport(gid 14214, 0 checks, 1 progress events) 2023-12-16T22:13:12.813 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:13:12 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:13:12.454+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:10:42.731761+0000 (oldest deadline 2023-12-16T22:11:02.731761+0000) 2023-12-16T22:13:12.814 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:13:12 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:13:12.454+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:10:42.731761+0000 (oldest deadline 2023-12-16T22:11:02.731761+0000) 2023-12-16T22:13:12.814 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:13:12 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:13:12.454+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:10:42.731761+0000 (oldest deadline 2023-12-16T22:11:02.731761+0000) 2023-12-16T22:13:12.814 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:13:12 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:13:12.455+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:13:13.064 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:13:12 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:13:12.910+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:11:07.849373+0000 (oldest deadline 2023-12-16T22:11:27.849373+0000) 2023-12-16T22:13:13.064 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:13:12 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:13:12.910+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:11:07.849373+0000 (oldest deadline 2023-12-16T22:11:27.849373+0000) 2023-12-16T22:13:13.065 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:13:12 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:13:12.910+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:11:07.849373+0000 (oldest deadline 2023-12-16T22:11:27.849373+0000) 2023-12-16T22:13:13.065 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:13:12 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:13:12.911+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:13:13.065 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:13:12 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:13:12.811+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:10:43.026858+0000 (oldest deadline 2023-12-16T22:11:03.026858+0000) 2023-12-16T22:13:13.065 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:13:12 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:13:12.811+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:10:43.026858+0000 (oldest deadline 2023-12-16T22:11:03.026858+0000) 2023-12-16T22:13:13.065 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:13:12 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:13:12.811+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:10:43.026858+0000 (oldest deadline 2023-12-16T22:11:03.026858+0000) 2023-12-16T22:13:13.065 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:13:12 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:13:12.813+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:13:13.789 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:13:13 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:13:13.467+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:10:42.731761+0000 (oldest deadline 2023-12-16T22:11:02.731761+0000) 2023-12-16T22:13:13.790 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:13:13 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:13:13.467+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:10:42.731761+0000 (oldest deadline 2023-12-16T22:11:02.731761+0000) 2023-12-16T22:13:13.790 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:13:13 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:13:13.467+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:10:42.731761+0000 (oldest deadline 2023-12-16T22:11:02.731761+0000) 2023-12-16T22:13:13.790 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:13:13 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:13:13.467+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:13:14.064 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:13:13 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:13:13.908+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:11:07.849373+0000 (oldest deadline 2023-12-16T22:11:27.849373+0000) 2023-12-16T22:13:14.064 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:13:13 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:13:13.908+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:11:07.849373+0000 (oldest deadline 2023-12-16T22:11:27.849373+0000) 2023-12-16T22:13:14.065 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:13:13 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:13:13.908+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:11:07.849373+0000 (oldest deadline 2023-12-16T22:11:27.849373+0000) 2023-12-16T22:13:14.065 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:13:13 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:13:13.909+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:13:14.065 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:13:13 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:13:13.788+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:10:43.026858+0000 (oldest deadline 2023-12-16T22:11:03.026858+0000) 2023-12-16T22:13:14.065 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:13:13 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:13:13.788+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:10:43.026858+0000 (oldest deadline 2023-12-16T22:11:03.026858+0000) 2023-12-16T22:13:14.065 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:13:13 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:13:13.788+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:10:43.026858+0000 (oldest deadline 2023-12-16T22:11:03.026858+0000) 2023-12-16T22:13:14.065 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:13:13 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:13:13.789+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:13:14.747 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:13:14 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:13:14.473+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:10:42.731761+0000 (oldest deadline 2023-12-16T22:11:02.731761+0000) 2023-12-16T22:13:14.747 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:13:14 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:13:14.473+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:10:42.731761+0000 (oldest deadline 2023-12-16T22:11:02.731761+0000) 2023-12-16T22:13:14.747 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:13:14 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:13:14.473+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:10:42.731761+0000 (oldest deadline 2023-12-16T22:11:02.731761+0000) 2023-12-16T22:13:14.747 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:13:14 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:13:14.473+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:13:15.064 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:13:14 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:13:14.873+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:11:07.849373+0000 (oldest deadline 2023-12-16T22:11:27.849373+0000) 2023-12-16T22:13:15.065 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:13:14 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:13:14.873+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:11:07.849373+0000 (oldest deadline 2023-12-16T22:11:27.849373+0000) 2023-12-16T22:13:15.065 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:13:14 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:13:14.873+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:11:07.849373+0000 (oldest deadline 2023-12-16T22:11:27.849373+0000) 2023-12-16T22:13:15.065 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:13:14 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:13:14.873+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:13:15.065 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:13:14 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:13:14.746+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:10:43.026858+0000 (oldest deadline 2023-12-16T22:11:03.026858+0000) 2023-12-16T22:13:15.065 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:13:14 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:13:14.746+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:10:43.026858+0000 (oldest deadline 2023-12-16T22:11:03.026858+0000) 2023-12-16T22:13:15.065 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:13:14 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:13:14.746+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:10:43.026858+0000 (oldest deadline 2023-12-16T22:11:03.026858+0000) 2023-12-16T22:13:15.066 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:13:14 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:13:14.747+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:13:15.788 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:13:15 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:13:15.481+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:10:42.731761+0000 (oldest deadline 2023-12-16T22:11:02.731761+0000) 2023-12-16T22:13:15.789 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:13:15 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:13:15.481+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:10:42.731761+0000 (oldest deadline 2023-12-16T22:11:02.731761+0000) 2023-12-16T22:13:15.789 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:13:15 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:13:15.481+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:10:42.731761+0000 (oldest deadline 2023-12-16T22:11:02.731761+0000) 2023-12-16T22:13:15.789 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:13:15 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:13:15.482+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:13:16.064 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:13:15 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:13:15.787+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:10:43.026858+0000 (oldest deadline 2023-12-16T22:11:03.026858+0000) 2023-12-16T22:13:16.065 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:13:15 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:13:15.787+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:10:43.026858+0000 (oldest deadline 2023-12-16T22:11:03.026858+0000) 2023-12-16T22:13:16.065 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:13:15 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:13:15.787+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:10:43.026858+0000 (oldest deadline 2023-12-16T22:11:03.026858+0000) 2023-12-16T22:13:16.065 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:13:15 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:13:15.788+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:13:16.065 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:13:15 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:13:15.903+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:11:07.849373+0000 (oldest deadline 2023-12-16T22:11:27.849373+0000) 2023-12-16T22:13:16.065 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:13:15 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:13:15.903+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:11:07.849373+0000 (oldest deadline 2023-12-16T22:11:27.849373+0000) 2023-12-16T22:13:16.065 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:13:15 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:13:15.903+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:11:07.849373+0000 (oldest deadline 2023-12-16T22:11:27.849373+0000) 2023-12-16T22:13:16.066 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:13:15 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:13:15.904+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:13:16.782 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:13:16 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:13:16.511+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:10:42.731761+0000 (oldest deadline 2023-12-16T22:11:02.731761+0000) 2023-12-16T22:13:16.783 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:13:16 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:13:16.511+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:10:42.731761+0000 (oldest deadline 2023-12-16T22:11:02.731761+0000) 2023-12-16T22:13:16.783 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:13:16 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:13:16.511+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:10:42.731761+0000 (oldest deadline 2023-12-16T22:11:02.731761+0000) 2023-12-16T22:13:16.783 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:13:16 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:13:16.512+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:13:17.064 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:13:16 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:13:16.782+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:10:43.026858+0000 (oldest deadline 2023-12-16T22:11:03.026858+0000) 2023-12-16T22:13:17.065 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:13:16 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:13:16.782+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:10:43.026858+0000 (oldest deadline 2023-12-16T22:11:03.026858+0000) 2023-12-16T22:13:17.065 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:13:16 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:13:16.782+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:10:43.026858+0000 (oldest deadline 2023-12-16T22:11:03.026858+0000) 2023-12-16T22:13:17.065 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:13:16 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:13:16.783+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:13:17.065 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:13:16 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:13:16.941+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:11:07.849373+0000 (oldest deadline 2023-12-16T22:11:27.849373+0000) 2023-12-16T22:13:17.065 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:13:16 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:13:16.941+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:11:07.849373+0000 (oldest deadline 2023-12-16T22:11:27.849373+0000) 2023-12-16T22:13:17.065 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:13:16 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:13:16.941+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:11:07.849373+0000 (oldest deadline 2023-12-16T22:11:27.849373+0000) 2023-12-16T22:13:17.066 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:13:16 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:13:16.942+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:13:17.558 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Dec 16 22:13:17 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-mon-smithi045[80759]: 2023-12-16T22:13:17.088+0000 7f45a1f8f700 -1 mon.smithi045@0(probing) e2 get_health_metrics reporting 33781 slow ops, oldest is monmgrreport(gid 14214, 0 checks, 1 progress events) 2023-12-16T22:13:17.814 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:13:17 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:13:17.789+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:10:43.026858+0000 (oldest deadline 2023-12-16T22:11:03.026858+0000) 2023-12-16T22:13:17.814 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:13:17 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:13:17.789+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:10:43.026858+0000 (oldest deadline 2023-12-16T22:11:03.026858+0000) 2023-12-16T22:13:17.815 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:13:17 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:13:17.789+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:10:43.026858+0000 (oldest deadline 2023-12-16T22:11:03.026858+0000) 2023-12-16T22:13:17.815 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:13:17 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:13:17.790+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:13:17.815 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:13:17 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:13:17.557+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:10:42.731761+0000 (oldest deadline 2023-12-16T22:11:02.731761+0000) 2023-12-16T22:13:17.815 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:13:17 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:13:17.557+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:10:42.731761+0000 (oldest deadline 2023-12-16T22:11:02.731761+0000) 2023-12-16T22:13:17.815 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:13:17 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:13:17.557+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:10:42.731761+0000 (oldest deadline 2023-12-16T22:11:02.731761+0000) 2023-12-16T22:13:17.815 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:13:17 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:13:17.558+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:13:18.314 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:13:17 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:13:17.980+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:11:07.849373+0000 (oldest deadline 2023-12-16T22:11:27.849373+0000) 2023-12-16T22:13:18.314 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:13:17 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:13:17.980+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:11:07.849373+0000 (oldest deadline 2023-12-16T22:11:27.849373+0000) 2023-12-16T22:13:18.314 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:13:17 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:13:17.980+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:11:07.849373+0000 (oldest deadline 2023-12-16T22:11:27.849373+0000) 2023-12-16T22:13:18.315 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:13:17 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:13:17.981+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:13:18.946 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:13:18 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:13:18.813+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:10:43.026858+0000 (oldest deadline 2023-12-16T22:11:03.026858+0000) 2023-12-16T22:13:18.946 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:13:18 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:13:18.813+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:10:43.026858+0000 (oldest deadline 2023-12-16T22:11:03.026858+0000) 2023-12-16T22:13:18.947 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:13:18 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:13:18.813+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:10:43.026858+0000 (oldest deadline 2023-12-16T22:11:03.026858+0000) 2023-12-16T22:13:18.947 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:13:18 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:13:18.814+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:13:18.947 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:13:18 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:13:18.571+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:10:42.731761+0000 (oldest deadline 2023-12-16T22:11:02.731761+0000) 2023-12-16T22:13:18.947 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:13:18 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:13:18.571+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:10:42.731761+0000 (oldest deadline 2023-12-16T22:11:02.731761+0000) 2023-12-16T22:13:18.947 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:13:18 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:13:18.571+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:10:42.731761+0000 (oldest deadline 2023-12-16T22:11:02.731761+0000) 2023-12-16T22:13:18.947 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:13:18 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:13:18.572+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:13:19.314 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:13:18 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:13:18.945+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:11:07.849373+0000 (oldest deadline 2023-12-16T22:11:27.849373+0000) 2023-12-16T22:13:19.314 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:13:18 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:13:18.945+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:11:07.849373+0000 (oldest deadline 2023-12-16T22:11:27.849373+0000) 2023-12-16T22:13:19.314 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:13:18 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:13:18.945+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:11:07.849373+0000 (oldest deadline 2023-12-16T22:11:27.849373+0000) 2023-12-16T22:13:19.315 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:13:18 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:13:18.946+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:13:19.948 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:13:19 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:13:19.814+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:10:43.026858+0000 (oldest deadline 2023-12-16T22:11:03.026858+0000) 2023-12-16T22:13:19.949 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:13:19 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:13:19.814+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:10:43.026858+0000 (oldest deadline 2023-12-16T22:11:03.026858+0000) 2023-12-16T22:13:19.949 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:13:19 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:13:19.814+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:10:43.026858+0000 (oldest deadline 2023-12-16T22:11:03.026858+0000) 2023-12-16T22:13:19.949 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:13:19 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:13:19.815+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:13:19.949 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:13:19 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:13:19.607+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:10:42.731761+0000 (oldest deadline 2023-12-16T22:11:02.731761+0000) 2023-12-16T22:13:19.949 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:13:19 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:13:19.607+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:10:42.731761+0000 (oldest deadline 2023-12-16T22:11:02.731761+0000) 2023-12-16T22:13:19.949 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:13:19 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:13:19.607+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:10:42.731761+0000 (oldest deadline 2023-12-16T22:11:02.731761+0000) 2023-12-16T22:13:19.950 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:13:19 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:13:19.607+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:13:20.225 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:13:19 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:13:19.947+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:11:07.849373+0000 (oldest deadline 2023-12-16T22:11:27.849373+0000) 2023-12-16T22:13:20.225 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:13:19 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:13:19.947+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:11:07.849373+0000 (oldest deadline 2023-12-16T22:11:27.849373+0000) 2023-12-16T22:13:20.225 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:13:19 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:13:19.947+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:11:07.849373+0000 (oldest deadline 2023-12-16T22:11:27.849373+0000) 2023-12-16T22:13:20.225 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:13:19 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:13:19.948+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:13:20.836 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:13:20 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:13:20.577+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:10:42.731761+0000 (oldest deadline 2023-12-16T22:11:02.731761+0000) 2023-12-16T22:13:20.929 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:13:20 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:13:20.577+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:10:42.731761+0000 (oldest deadline 2023-12-16T22:11:02.731761+0000) 2023-12-16T22:13:20.929 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:13:20 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:13:20.577+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:10:42.731761+0000 (oldest deadline 2023-12-16T22:11:02.731761+0000) 2023-12-16T22:13:20.930 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:13:20 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:13:20.577+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:13:21.295 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:13:20 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:13:20.834+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:10:43.026858+0000 (oldest deadline 2023-12-16T22:11:03.026858+0000) 2023-12-16T22:13:21.295 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:13:20 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:13:20.834+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:10:43.026858+0000 (oldest deadline 2023-12-16T22:11:03.026858+0000) 2023-12-16T22:13:21.295 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:13:20 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:13:20.834+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:10:43.026858+0000 (oldest deadline 2023-12-16T22:11:03.026858+0000) 2023-12-16T22:13:21.295 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:13:20 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:13:20.835+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:13:21.296 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:13:20 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:13:20.923+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:11:07.849373+0000 (oldest deadline 2023-12-16T22:11:27.849373+0000) 2023-12-16T22:13:21.296 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:13:20 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:13:20.923+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:11:07.849373+0000 (oldest deadline 2023-12-16T22:11:27.849373+0000) 2023-12-16T22:13:21.296 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:13:20 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:13:20.923+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:11:07.849373+0000 (oldest deadline 2023-12-16T22:11:27.849373+0000) 2023-12-16T22:13:21.296 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:13:20 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:13:20.924+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:13:21.814 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:13:21 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:13:21.554+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:10:42.731761+0000 (oldest deadline 2023-12-16T22:11:02.731761+0000) 2023-12-16T22:13:21.814 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:13:21 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:13:21.554+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:10:42.731761+0000 (oldest deadline 2023-12-16T22:11:02.731761+0000) 2023-12-16T22:13:21.814 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:13:21 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:13:21.555+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:10:42.731761+0000 (oldest deadline 2023-12-16T22:11:02.731761+0000) 2023-12-16T22:13:21.815 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:13:21 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:13:21.555+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:13:22.116 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Dec 16 22:13:22 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-mon-smithi045[80759]: 2023-12-16T22:13:22.092+0000 7f45a1f8f700 -1 mon.smithi045@0(probing) e2 get_health_metrics reporting 33868 slow ops, oldest is monmgrreport(gid 14214, 0 checks, 1 progress events) 2023-12-16T22:13:22.116 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:13:21 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:13:21.862+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:10:43.026858+0000 (oldest deadline 2023-12-16T22:11:03.026858+0000) 2023-12-16T22:13:22.117 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:13:21 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:13:21.862+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:10:43.026858+0000 (oldest deadline 2023-12-16T22:11:03.026858+0000) 2023-12-16T22:13:22.117 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:13:21 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:13:21.862+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:10:43.026858+0000 (oldest deadline 2023-12-16T22:11:03.026858+0000) 2023-12-16T22:13:22.117 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:13:21 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:13:21.863+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:13:22.117 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:13:21 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:13:21.899+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:11:07.849373+0000 (oldest deadline 2023-12-16T22:11:27.849373+0000) 2023-12-16T22:13:22.117 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:13:21 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:13:21.899+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:11:07.849373+0000 (oldest deadline 2023-12-16T22:11:27.849373+0000) 2023-12-16T22:13:22.117 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:13:21 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:13:21.899+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:11:07.849373+0000 (oldest deadline 2023-12-16T22:11:27.849373+0000) 2023-12-16T22:13:22.117 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:13:21 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:13:21.900+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:13:22.518 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:13:22 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:13:22.517+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:10:42.731761+0000 (oldest deadline 2023-12-16T22:11:02.731761+0000) 2023-12-16T22:13:22.519 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:13:22 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:13:22.517+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:10:42.731761+0000 (oldest deadline 2023-12-16T22:11:02.731761+0000) 2023-12-16T22:13:22.519 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:13:22 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:13:22.517+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:10:42.731761+0000 (oldest deadline 2023-12-16T22:11:02.731761+0000) 2023-12-16T22:13:22.814 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:13:22 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:13:22.519+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:13:23.314 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:13:22 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:13:22.895+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:11:07.849373+0000 (oldest deadline 2023-12-16T22:11:27.849373+0000) 2023-12-16T22:13:23.315 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:13:22 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:13:22.895+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:11:07.849373+0000 (oldest deadline 2023-12-16T22:11:27.849373+0000) 2023-12-16T22:13:23.315 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:13:22 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:13:22.895+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:11:07.849373+0000 (oldest deadline 2023-12-16T22:11:27.849373+0000) 2023-12-16T22:13:23.315 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:13:22 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:13:22.895+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:13:23.317 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:13:22 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:13:22.838+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:10:43.026858+0000 (oldest deadline 2023-12-16T22:11:03.026858+0000) 2023-12-16T22:13:23.317 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:13:22 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:13:22.838+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:10:43.026858+0000 (oldest deadline 2023-12-16T22:11:03.026858+0000) 2023-12-16T22:13:23.317 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:13:22 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:13:22.838+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:10:43.026858+0000 (oldest deadline 2023-12-16T22:11:03.026858+0000) 2023-12-16T22:13:23.318 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:13:22 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:13:22.838+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:13:23.814 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:13:23 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:13:23.510+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:10:42.731761+0000 (oldest deadline 2023-12-16T22:11:02.731761+0000) 2023-12-16T22:13:23.814 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:13:23 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:13:23.510+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:10:42.731761+0000 (oldest deadline 2023-12-16T22:11:02.731761+0000) 2023-12-16T22:13:23.815 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:13:23 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:13:23.510+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:10:42.731761+0000 (oldest deadline 2023-12-16T22:11:02.731761+0000) 2023-12-16T22:13:23.815 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:13:23 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:13:23.511+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:13:24.314 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:13:23 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:13:23.861+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:10:43.026858+0000 (oldest deadline 2023-12-16T22:11:03.026858+0000) 2023-12-16T22:13:24.315 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:13:23 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:13:23.861+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:10:43.026858+0000 (oldest deadline 2023-12-16T22:11:03.026858+0000) 2023-12-16T22:13:24.315 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:13:23 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:13:23.861+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:10:43.026858+0000 (oldest deadline 2023-12-16T22:11:03.026858+0000) 2023-12-16T22:13:24.315 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:13:23 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:13:23.863+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:13:24.315 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:13:23 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:13:23.894+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:11:07.849373+0000 (oldest deadline 2023-12-16T22:11:27.849373+0000) 2023-12-16T22:13:24.315 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:13:23 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:13:23.894+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:11:07.849373+0000 (oldest deadline 2023-12-16T22:11:27.849373+0000) 2023-12-16T22:13:24.315 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:13:23 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:13:23.894+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:11:07.849373+0000 (oldest deadline 2023-12-16T22:11:27.849373+0000) 2023-12-16T22:13:24.315 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:13:23 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:13:23.895+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:13:24.814 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:13:24 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:13:24.531+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:10:42.731761+0000 (oldest deadline 2023-12-16T22:11:02.731761+0000) 2023-12-16T22:13:24.814 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:13:24 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:13:24.531+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:10:42.731761+0000 (oldest deadline 2023-12-16T22:11:02.731761+0000) 2023-12-16T22:13:24.815 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:13:24 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:13:24.531+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:10:42.731761+0000 (oldest deadline 2023-12-16T22:11:02.731761+0000) 2023-12-16T22:13:24.815 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:13:24 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:13:24.532+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:13:25.314 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:13:24 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:13:24.862+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:10:43.026858+0000 (oldest deadline 2023-12-16T22:11:03.026858+0000) 2023-12-16T22:13:25.315 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:13:24 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:13:24.862+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:10:43.026858+0000 (oldest deadline 2023-12-16T22:11:03.026858+0000) 2023-12-16T22:13:25.315 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:13:24 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:13:24.862+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:10:43.026858+0000 (oldest deadline 2023-12-16T22:11:03.026858+0000) 2023-12-16T22:13:25.315 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:13:24 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:13:24.862+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:13:25.315 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:13:24 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:13:24.888+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:11:07.849373+0000 (oldest deadline 2023-12-16T22:11:27.849373+0000) 2023-12-16T22:13:25.315 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:13:24 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:13:24.888+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:11:07.849373+0000 (oldest deadline 2023-12-16T22:11:27.849373+0000) 2023-12-16T22:13:25.315 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:13:24 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:13:24.888+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:11:07.849373+0000 (oldest deadline 2023-12-16T22:11:27.849373+0000) 2023-12-16T22:13:25.315 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:13:24 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:13:24.889+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:13:25.814 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:13:25 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:13:25.528+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:10:42.731761+0000 (oldest deadline 2023-12-16T22:11:02.731761+0000) 2023-12-16T22:13:25.815 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:13:25 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:13:25.528+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:10:42.731761+0000 (oldest deadline 2023-12-16T22:11:02.731761+0000) 2023-12-16T22:13:25.815 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:13:25 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:13:25.528+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:10:42.731761+0000 (oldest deadline 2023-12-16T22:11:02.731761+0000) 2023-12-16T22:13:25.815 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:13:25 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:13:25.529+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:13:26.314 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:13:25 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:13:25.878+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:11:07.849373+0000 (oldest deadline 2023-12-16T22:11:27.849373+0000) 2023-12-16T22:13:26.314 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:13:25 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:13:25.878+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:11:07.849373+0000 (oldest deadline 2023-12-16T22:11:27.849373+0000) 2023-12-16T22:13:26.314 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:13:25 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:13:25.878+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:11:07.849373+0000 (oldest deadline 2023-12-16T22:11:27.849373+0000) 2023-12-16T22:13:26.315 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:13:25 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:13:25.879+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:13:26.315 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:13:25 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:13:25.838+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:10:43.026858+0000 (oldest deadline 2023-12-16T22:11:03.026858+0000) 2023-12-16T22:13:26.315 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:13:25 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:13:25.838+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:10:43.026858+0000 (oldest deadline 2023-12-16T22:11:03.026858+0000) 2023-12-16T22:13:26.315 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:13:25 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:13:25.838+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:10:43.026858+0000 (oldest deadline 2023-12-16T22:11:03.026858+0000) 2023-12-16T22:13:26.315 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:13:25 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:13:25.838+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:13:26.814 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:13:26 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:13:26.520+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:10:42.731761+0000 (oldest deadline 2023-12-16T22:11:02.731761+0000) 2023-12-16T22:13:26.814 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:13:26 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:13:26.520+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:10:42.731761+0000 (oldest deadline 2023-12-16T22:11:02.731761+0000) 2023-12-16T22:13:26.814 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:13:26 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:13:26.520+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:10:42.731761+0000 (oldest deadline 2023-12-16T22:11:02.731761+0000) 2023-12-16T22:13:26.815 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:13:26 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:13:26.520+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:13:27.097 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:13:26 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:13:26.900+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:11:07.849373+0000 (oldest deadline 2023-12-16T22:11:27.849373+0000) 2023-12-16T22:13:27.097 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:13:26 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:13:26.900+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:11:07.849373+0000 (oldest deadline 2023-12-16T22:11:27.849373+0000) 2023-12-16T22:13:27.098 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:13:26 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:13:26.900+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:11:07.849373+0000 (oldest deadline 2023-12-16T22:11:27.849373+0000) 2023-12-16T22:13:27.098 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:13:26 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:13:26.901+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:13:27.098 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:13:26 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:13:26.822+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:10:43.026858+0000 (oldest deadline 2023-12-16T22:11:03.026858+0000) 2023-12-16T22:13:27.098 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:13:26 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:13:26.822+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:10:43.026858+0000 (oldest deadline 2023-12-16T22:11:03.026858+0000) 2023-12-16T22:13:27.098 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:13:26 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:13:26.822+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:10:43.026858+0000 (oldest deadline 2023-12-16T22:11:03.026858+0000) 2023-12-16T22:13:27.098 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:13:26 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:13:26.823+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:13:27.518 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Dec 16 22:13:27 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-mon-smithi045[80759]: 2023-12-16T22:13:27.096+0000 7f45a1f8f700 -1 mon.smithi045@0(probing) e2 get_health_metrics reporting 33956 slow ops, oldest is monmgrreport(gid 14214, 0 checks, 1 progress events) 2023-12-16T22:13:27.814 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:13:27 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:13:27.517+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:10:42.731761+0000 (oldest deadline 2023-12-16T22:11:02.731761+0000) 2023-12-16T22:13:27.814 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:13:27 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:13:27.517+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:10:42.731761+0000 (oldest deadline 2023-12-16T22:11:02.731761+0000) 2023-12-16T22:13:27.815 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:13:27 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:13:27.517+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:10:42.731761+0000 (oldest deadline 2023-12-16T22:11:02.731761+0000) 2023-12-16T22:13:27.815 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:13:27 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:13:27.518+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:13:28.314 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:13:27 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:13:27.841+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:10:43.026858+0000 (oldest deadline 2023-12-16T22:11:03.026858+0000) 2023-12-16T22:13:28.314 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:13:27 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:13:27.841+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:10:43.026858+0000 (oldest deadline 2023-12-16T22:11:03.026858+0000) 2023-12-16T22:13:28.314 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:13:27 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:13:27.841+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:10:43.026858+0000 (oldest deadline 2023-12-16T22:11:03.026858+0000) 2023-12-16T22:13:28.315 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:13:27 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:13:27.842+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:13:28.315 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:13:27 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:13:27.907+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:11:07.849373+0000 (oldest deadline 2023-12-16T22:11:27.849373+0000) 2023-12-16T22:13:28.315 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:13:27 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:13:27.907+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:11:07.849373+0000 (oldest deadline 2023-12-16T22:11:27.849373+0000) 2023-12-16T22:13:28.315 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:13:27 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:13:27.907+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:11:07.849373+0000 (oldest deadline 2023-12-16T22:11:27.849373+0000) 2023-12-16T22:13:28.315 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:13:27 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:13:27.908+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:13:28.793 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:13:28 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:13:28.504+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:10:42.731761+0000 (oldest deadline 2023-12-16T22:11:02.731761+0000) 2023-12-16T22:13:28.794 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:13:28 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:13:28.504+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:10:42.731761+0000 (oldest deadline 2023-12-16T22:11:02.731761+0000) 2023-12-16T22:13:28.794 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:13:28 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:13:28.504+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:10:42.731761+0000 (oldest deadline 2023-12-16T22:11:02.731761+0000) 2023-12-16T22:13:28.794 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:13:28 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:13:28.505+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:13:29.064 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:13:28 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:13:28.793+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:10:43.026858+0000 (oldest deadline 2023-12-16T22:11:03.026858+0000) 2023-12-16T22:13:29.064 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:13:28 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:13:28.793+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:10:43.026858+0000 (oldest deadline 2023-12-16T22:11:03.026858+0000) 2023-12-16T22:13:29.064 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:13:28 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:13:28.793+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:10:43.026858+0000 (oldest deadline 2023-12-16T22:11:03.026858+0000) 2023-12-16T22:13:29.065 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:13:28 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:13:28.794+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:13:29.065 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:13:28 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:13:28.904+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:11:07.849373+0000 (oldest deadline 2023-12-16T22:11:27.849373+0000) 2023-12-16T22:13:29.065 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:13:28 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:13:28.904+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:11:07.849373+0000 (oldest deadline 2023-12-16T22:11:27.849373+0000) 2023-12-16T22:13:29.065 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:13:28 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:13:28.904+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:11:07.849373+0000 (oldest deadline 2023-12-16T22:11:27.849373+0000) 2023-12-16T22:13:29.065 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:13:28 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:13:28.905+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:13:29.814 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:13:29 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:13:29.526+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:10:42.731761+0000 (oldest deadline 2023-12-16T22:11:02.731761+0000) 2023-12-16T22:13:29.814 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:13:29 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:13:29.526+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:10:42.731761+0000 (oldest deadline 2023-12-16T22:11:02.731761+0000) 2023-12-16T22:13:29.814 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:13:29 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:13:29.526+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:10:42.731761+0000 (oldest deadline 2023-12-16T22:11:02.731761+0000) 2023-12-16T22:13:29.815 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:13:29 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:13:29.528+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:13:30.225 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:13:29 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:13:29.835+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:10:43.026858+0000 (oldest deadline 2023-12-16T22:11:03.026858+0000) 2023-12-16T22:13:30.225 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:13:29 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:13:29.835+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:10:43.026858+0000 (oldest deadline 2023-12-16T22:11:03.026858+0000) 2023-12-16T22:13:30.226 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:13:29 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:13:29.835+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:10:43.026858+0000 (oldest deadline 2023-12-16T22:11:03.026858+0000) 2023-12-16T22:13:30.226 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:13:29 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:13:29.836+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:13:30.226 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:13:29 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:13:29.932+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:11:07.849373+0000 (oldest deadline 2023-12-16T22:11:27.849373+0000) 2023-12-16T22:13:30.226 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:13:29 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:13:29.932+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:11:07.849373+0000 (oldest deadline 2023-12-16T22:11:27.849373+0000) 2023-12-16T22:13:30.226 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:13:29 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:13:29.932+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:11:07.849373+0000 (oldest deadline 2023-12-16T22:11:27.849373+0000) 2023-12-16T22:13:30.226 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:13:29 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:13:29.932+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:13:30.814 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:13:30 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:13:30.543+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:10:42.731761+0000 (oldest deadline 2023-12-16T22:11:02.731761+0000) 2023-12-16T22:13:30.814 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:13:30 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:13:30.543+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:10:42.731761+0000 (oldest deadline 2023-12-16T22:11:02.731761+0000) 2023-12-16T22:13:30.815 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:13:30 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:13:30.543+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:10:42.731761+0000 (oldest deadline 2023-12-16T22:11:02.731761+0000) 2023-12-16T22:13:30.815 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:13:30 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:13:30.544+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:13:31.314 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:13:30 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:13:30.968+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:11:07.849373+0000 (oldest deadline 2023-12-16T22:11:27.849373+0000) 2023-12-16T22:13:31.314 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:13:30 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:13:30.968+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:11:07.849373+0000 (oldest deadline 2023-12-16T22:11:27.849373+0000) 2023-12-16T22:13:31.315 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:13:30 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:13:30.968+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:11:07.849373+0000 (oldest deadline 2023-12-16T22:11:27.849373+0000) 2023-12-16T22:13:31.315 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:13:30 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:13:30.968+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:13:31.315 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:13:30 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:13:30.858+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:10:43.026858+0000 (oldest deadline 2023-12-16T22:11:03.026858+0000) 2023-12-16T22:13:31.315 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:13:30 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:13:30.858+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:10:43.026858+0000 (oldest deadline 2023-12-16T22:11:03.026858+0000) 2023-12-16T22:13:31.315 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:13:30 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:13:30.858+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:10:43.026858+0000 (oldest deadline 2023-12-16T22:11:03.026858+0000) 2023-12-16T22:13:31.315 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:13:30 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:13:30.858+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:13:31.814 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:13:31 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:13:31.520+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:10:42.731761+0000 (oldest deadline 2023-12-16T22:11:02.731761+0000) 2023-12-16T22:13:31.814 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:13:31 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:13:31.520+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:10:42.731761+0000 (oldest deadline 2023-12-16T22:11:02.731761+0000) 2023-12-16T22:13:31.814 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:13:31 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:13:31.520+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:10:42.731761+0000 (oldest deadline 2023-12-16T22:11:02.731761+0000) 2023-12-16T22:13:31.815 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:13:31 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:13:31.521+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:13:32.102 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:13:31 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:13:31.998+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:11:07.849373+0000 (oldest deadline 2023-12-16T22:11:27.849373+0000) 2023-12-16T22:13:32.102 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:13:31 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:13:31.998+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:11:07.849373+0000 (oldest deadline 2023-12-16T22:11:27.849373+0000) 2023-12-16T22:13:32.102 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:13:31 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:13:31.998+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:11:07.849373+0000 (oldest deadline 2023-12-16T22:11:27.849373+0000) 2023-12-16T22:13:32.102 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:13:32 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:13:31.999+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:13:32.102 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:13:31 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:13:31.821+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:10:43.026858+0000 (oldest deadline 2023-12-16T22:11:03.026858+0000) 2023-12-16T22:13:32.103 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:13:31 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:13:31.821+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:10:43.026858+0000 (oldest deadline 2023-12-16T22:11:03.026858+0000) 2023-12-16T22:13:32.103 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:13:31 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:13:31.821+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:10:43.026858+0000 (oldest deadline 2023-12-16T22:11:03.026858+0000) 2023-12-16T22:13:32.103 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:13:31 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:13:31.822+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:13:32.505 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Dec 16 22:13:32 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-mon-smithi045[80759]: 2023-12-16T22:13:32.101+0000 7f45a1f8f700 -1 mon.smithi045@0(probing) e2 get_health_metrics reporting 34047 slow ops, oldest is monmgrreport(gid 14214, 0 checks, 1 progress events) 2023-12-16T22:13:32.816 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:13:32 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:13:32.504+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:10:42.731761+0000 (oldest deadline 2023-12-16T22:11:02.731761+0000) 2023-12-16T22:13:32.817 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:13:32 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:13:32.504+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:10:42.731761+0000 (oldest deadline 2023-12-16T22:11:02.731761+0000) 2023-12-16T22:13:32.817 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:13:32 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:13:32.504+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:10:42.731761+0000 (oldest deadline 2023-12-16T22:11:02.731761+0000) 2023-12-16T22:13:32.817 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:13:32 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:13:32.505+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:13:33.314 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:13:32 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:13:32.839+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:10:43.026858+0000 (oldest deadline 2023-12-16T22:11:03.026858+0000) 2023-12-16T22:13:33.314 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:13:32 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:13:32.839+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:10:43.026858+0000 (oldest deadline 2023-12-16T22:11:03.026858+0000) 2023-12-16T22:13:33.315 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:13:32 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:13:32.839+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:10:43.026858+0000 (oldest deadline 2023-12-16T22:11:03.026858+0000) 2023-12-16T22:13:33.315 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:13:32 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:13:32.840+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:13:33.315 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:13:33 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:13:33.011+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:11:07.849373+0000 (oldest deadline 2023-12-16T22:11:27.849373+0000) 2023-12-16T22:13:33.315 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:13:33 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:13:33.011+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:11:07.849373+0000 (oldest deadline 2023-12-16T22:11:27.849373+0000) 2023-12-16T22:13:33.315 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:13:33 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:13:33.011+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:11:07.849373+0000 (oldest deadline 2023-12-16T22:11:27.849373+0000) 2023-12-16T22:13:33.315 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:13:33 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:13:33.012+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:13:33.814 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:13:33 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:13:33.549+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:10:42.731761+0000 (oldest deadline 2023-12-16T22:11:02.731761+0000) 2023-12-16T22:13:33.814 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:13:33 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:13:33.549+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:10:42.731761+0000 (oldest deadline 2023-12-16T22:11:02.731761+0000) 2023-12-16T22:13:33.814 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:13:33 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:13:33.549+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:10:42.731761+0000 (oldest deadline 2023-12-16T22:11:02.731761+0000) 2023-12-16T22:13:33.815 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:13:33 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:13:33.550+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:13:34.314 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:13:33 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:13:33.874+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:10:43.026858+0000 (oldest deadline 2023-12-16T22:11:03.026858+0000) 2023-12-16T22:13:34.315 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:13:33 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:13:33.874+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:10:43.026858+0000 (oldest deadline 2023-12-16T22:11:03.026858+0000) 2023-12-16T22:13:34.315 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:13:33 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:13:33.874+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:10:43.026858+0000 (oldest deadline 2023-12-16T22:11:03.026858+0000) 2023-12-16T22:13:34.315 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:13:33 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:13:33.875+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:13:34.315 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:13:34 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:13:34.045+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:11:07.849373+0000 (oldest deadline 2023-12-16T22:11:27.849373+0000) 2023-12-16T22:13:34.315 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:13:34 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:13:34.045+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:11:07.849373+0000 (oldest deadline 2023-12-16T22:11:27.849373+0000) 2023-12-16T22:13:34.315 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:13:34 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:13:34.045+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:11:07.849373+0000 (oldest deadline 2023-12-16T22:11:27.849373+0000) 2023-12-16T22:13:34.316 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:13:34 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:13:34.046+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:13:34.814 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:13:34 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:13:34.563+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:10:42.731761+0000 (oldest deadline 2023-12-16T22:11:02.731761+0000) 2023-12-16T22:13:34.814 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:13:34 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:13:34.563+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:10:42.731761+0000 (oldest deadline 2023-12-16T22:11:02.731761+0000) 2023-12-16T22:13:34.815 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:13:34 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:13:34.563+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:10:42.731761+0000 (oldest deadline 2023-12-16T22:11:02.731761+0000) 2023-12-16T22:13:34.815 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:13:34 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:13:34.564+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:13:35.314 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:13:34 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:13:34.843+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:10:43.026858+0000 (oldest deadline 2023-12-16T22:11:03.026858+0000) 2023-12-16T22:13:35.315 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:13:34 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:13:34.843+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:10:43.026858+0000 (oldest deadline 2023-12-16T22:11:03.026858+0000) 2023-12-16T22:13:35.315 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:13:34 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:13:34.843+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:10:43.026858+0000 (oldest deadline 2023-12-16T22:11:03.026858+0000) 2023-12-16T22:13:35.315 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:13:34 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:13:34.844+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:13:35.315 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:13:35 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:13:35.033+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:11:07.849373+0000 (oldest deadline 2023-12-16T22:11:27.849373+0000) 2023-12-16T22:13:35.315 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:13:35 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:13:35.033+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:11:07.849373+0000 (oldest deadline 2023-12-16T22:11:27.849373+0000) 2023-12-16T22:13:35.315 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:13:35 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:13:35.033+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:11:07.849373+0000 (oldest deadline 2023-12-16T22:11:27.849373+0000) 2023-12-16T22:13:35.315 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:13:35 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:13:35.034+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:13:36.009 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:13:35 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:13:35.574+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:10:42.731761+0000 (oldest deadline 2023-12-16T22:11:02.731761+0000) 2023-12-16T22:13:36.009 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:13:35 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:13:35.574+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:10:42.731761+0000 (oldest deadline 2023-12-16T22:11:02.731761+0000) 2023-12-16T22:13:36.009 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:13:35 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:13:35.574+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:10:42.731761+0000 (oldest deadline 2023-12-16T22:11:02.731761+0000) 2023-12-16T22:13:36.009 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:13:35 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:13:35.575+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:13:36.010 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:13:35 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:13:35.816+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:10:43.026858+0000 (oldest deadline 2023-12-16T22:11:03.026858+0000) 2023-12-16T22:13:36.010 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:13:35 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:13:35.816+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:10:43.026858+0000 (oldest deadline 2023-12-16T22:11:03.026858+0000) 2023-12-16T22:13:36.010 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:13:35 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:13:35.816+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:10:43.026858+0000 (oldest deadline 2023-12-16T22:11:03.026858+0000) 2023-12-16T22:13:36.010 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:13:35 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:13:35.818+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:13:36.314 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:13:36 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:13:36.008+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:11:07.849373+0000 (oldest deadline 2023-12-16T22:11:27.849373+0000) 2023-12-16T22:13:36.314 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:13:36 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:13:36.008+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:11:07.849373+0000 (oldest deadline 2023-12-16T22:11:27.849373+0000) 2023-12-16T22:13:36.315 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:13:36 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:13:36.008+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:11:07.849373+0000 (oldest deadline 2023-12-16T22:11:27.849373+0000) 2023-12-16T22:13:36.315 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:13:36 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:13:36.009+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:13:37.033 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:13:36 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:13:36.776+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:10:43.026858+0000 (oldest deadline 2023-12-16T22:11:03.026858+0000) 2023-12-16T22:13:37.033 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:13:36 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:13:36.776+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:10:43.026858+0000 (oldest deadline 2023-12-16T22:11:03.026858+0000) 2023-12-16T22:13:37.034 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:13:36 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:13:36.776+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:10:43.026858+0000 (oldest deadline 2023-12-16T22:11:03.026858+0000) 2023-12-16T22:13:37.034 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:13:36 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:13:36.777+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:13:37.034 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:13:36 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:13:36.590+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:10:42.731761+0000 (oldest deadline 2023-12-16T22:11:02.731761+0000) 2023-12-16T22:13:37.034 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:13:36 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:13:36.590+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:10:42.731761+0000 (oldest deadline 2023-12-16T22:11:02.731761+0000) 2023-12-16T22:13:37.034 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:13:36 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:13:36.590+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:10:42.731761+0000 (oldest deadline 2023-12-16T22:11:02.731761+0000) 2023-12-16T22:13:37.034 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:13:36 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:13:36.591+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:13:37.314 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Dec 16 22:13:37 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-mon-smithi045[80759]: 2023-12-16T22:13:37.104+0000 7f45a1f8f700 -1 mon.smithi045@0(probing) e2 get_health_metrics reporting 34138 slow ops, oldest is monmgrreport(gid 14214, 0 checks, 1 progress events) 2023-12-16T22:13:37.314 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:13:37 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:13:37.031+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:11:07.849373+0000 (oldest deadline 2023-12-16T22:11:27.849373+0000) 2023-12-16T22:13:37.315 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:13:37 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:13:37.031+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:11:07.849373+0000 (oldest deadline 2023-12-16T22:11:27.849373+0000) 2023-12-16T22:13:37.315 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:13:37 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:13:37.031+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:11:07.849373+0000 (oldest deadline 2023-12-16T22:11:27.849373+0000) 2023-12-16T22:13:37.315 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:13:37 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:13:37.032+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:13:37.987 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:13:37 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:13:37.817+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:10:43.026858+0000 (oldest deadline 2023-12-16T22:11:03.026858+0000) 2023-12-16T22:13:37.987 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:13:37 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:13:37.817+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:10:43.026858+0000 (oldest deadline 2023-12-16T22:11:03.026858+0000) 2023-12-16T22:13:37.987 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:13:37 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:13:37.817+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:10:43.026858+0000 (oldest deadline 2023-12-16T22:11:03.026858+0000) 2023-12-16T22:13:37.987 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:13:37 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:13:37.818+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:13:37.988 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:13:37 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:13:37.610+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:10:42.731761+0000 (oldest deadline 2023-12-16T22:11:02.731761+0000) 2023-12-16T22:13:37.988 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:13:37 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:13:37.610+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:10:42.731761+0000 (oldest deadline 2023-12-16T22:11:02.731761+0000) 2023-12-16T22:13:37.988 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:13:37 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:13:37.610+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:10:42.731761+0000 (oldest deadline 2023-12-16T22:11:02.731761+0000) 2023-12-16T22:13:37.988 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:13:37 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:13:37.611+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:13:38.314 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:13:37 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:13:37.985+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:11:07.849373+0000 (oldest deadline 2023-12-16T22:11:27.849373+0000) 2023-12-16T22:13:38.314 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:13:37 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:13:37.985+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:11:07.849373+0000 (oldest deadline 2023-12-16T22:11:27.849373+0000) 2023-12-16T22:13:38.314 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:13:37 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:13:37.985+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:11:07.849373+0000 (oldest deadline 2023-12-16T22:11:27.849373+0000) 2023-12-16T22:13:38.315 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:13:37 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:13:37.986+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:13:39.020 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:13:38 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:13:38.787+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:10:43.026858+0000 (oldest deadline 2023-12-16T22:11:03.026858+0000) 2023-12-16T22:13:39.020 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:13:38 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:13:38.787+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:10:43.026858+0000 (oldest deadline 2023-12-16T22:11:03.026858+0000) 2023-12-16T22:13:39.020 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:13:38 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:13:38.787+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:10:43.026858+0000 (oldest deadline 2023-12-16T22:11:03.026858+0000) 2023-12-16T22:13:39.020 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:13:38 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:13:38.788+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:13:39.020 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:13:38 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:13:38.655+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:10:42.731761+0000 (oldest deadline 2023-12-16T22:11:02.731761+0000) 2023-12-16T22:13:39.021 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:13:38 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:13:38.655+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:10:42.731761+0000 (oldest deadline 2023-12-16T22:11:02.731761+0000) 2023-12-16T22:13:39.021 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:13:38 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:13:38.655+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:10:42.731761+0000 (oldest deadline 2023-12-16T22:11:02.731761+0000) 2023-12-16T22:13:39.021 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:13:38 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:13:38.656+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:13:39.314 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:13:39 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:13:39.018+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:11:07.849373+0000 (oldest deadline 2023-12-16T22:11:27.849373+0000) 2023-12-16T22:13:39.314 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:13:39 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:13:39.018+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:11:07.849373+0000 (oldest deadline 2023-12-16T22:11:27.849373+0000) 2023-12-16T22:13:39.314 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:13:39 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:13:39.018+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:11:07.849373+0000 (oldest deadline 2023-12-16T22:11:27.849373+0000) 2023-12-16T22:13:39.315 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:13:39 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:13:39.019+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:13:40.023 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:13:39 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:13:39.818+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:10:43.026858+0000 (oldest deadline 2023-12-16T22:11:03.026858+0000) 2023-12-16T22:13:40.023 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:13:39 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:13:39.818+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:10:43.026858+0000 (oldest deadline 2023-12-16T22:11:03.026858+0000) 2023-12-16T22:13:40.023 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:13:39 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:13:39.818+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:10:43.026858+0000 (oldest deadline 2023-12-16T22:11:03.026858+0000) 2023-12-16T22:13:40.023 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:13:39 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:13:39.819+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:13:40.023 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:13:39 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:13:39.619+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:10:42.731761+0000 (oldest deadline 2023-12-16T22:11:02.731761+0000) 2023-12-16T22:13:40.024 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:13:39 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:13:39.619+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:10:42.731761+0000 (oldest deadline 2023-12-16T22:11:02.731761+0000) 2023-12-16T22:13:40.024 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:13:39 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:13:39.619+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:10:42.731761+0000 (oldest deadline 2023-12-16T22:11:02.731761+0000) 2023-12-16T22:13:40.024 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:13:39 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:13:39.619+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:13:40.314 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:13:40 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:13:40.019+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:11:07.849373+0000 (oldest deadline 2023-12-16T22:11:27.849373+0000) 2023-12-16T22:13:40.314 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:13:40 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:13:40.019+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:11:07.849373+0000 (oldest deadline 2023-12-16T22:11:27.849373+0000) 2023-12-16T22:13:40.315 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:13:40 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:13:40.019+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:11:07.849373+0000 (oldest deadline 2023-12-16T22:11:27.849373+0000) 2023-12-16T22:13:40.315 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:13:40 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:13:40.021+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:13:40.986 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:13:40 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:13:40.840+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:10:43.026858+0000 (oldest deadline 2023-12-16T22:11:03.026858+0000) 2023-12-16T22:13:40.987 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:13:40 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:13:40.840+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:10:43.026858+0000 (oldest deadline 2023-12-16T22:11:03.026858+0000) 2023-12-16T22:13:40.987 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:13:40 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:13:40.840+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:10:43.026858+0000 (oldest deadline 2023-12-16T22:11:03.026858+0000) 2023-12-16T22:13:40.987 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:13:40 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:13:40.841+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:13:40.988 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:13:40 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:13:40.636+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:10:42.731761+0000 (oldest deadline 2023-12-16T22:11:02.731761+0000) 2023-12-16T22:13:40.988 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:13:40 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:13:40.636+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:10:42.731761+0000 (oldest deadline 2023-12-16T22:11:02.731761+0000) 2023-12-16T22:13:40.988 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:13:40 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:13:40.636+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:10:42.731761+0000 (oldest deadline 2023-12-16T22:11:02.731761+0000) 2023-12-16T22:13:40.989 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:13:40 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:13:40.637+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:13:41.314 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:13:40 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:13:40.984+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:11:07.849373+0000 (oldest deadline 2023-12-16T22:11:27.849373+0000) 2023-12-16T22:13:41.314 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:13:40 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:13:40.984+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:11:07.849373+0000 (oldest deadline 2023-12-16T22:11:27.849373+0000) 2023-12-16T22:13:41.315 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:13:40 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:13:40.984+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:11:07.849373+0000 (oldest deadline 2023-12-16T22:11:27.849373+0000) 2023-12-16T22:13:41.315 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:13:40 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:13:40.985+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:13:41.892 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:13:41 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:13:41.621+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:10:42.731761+0000 (oldest deadline 2023-12-16T22:11:02.731761+0000) 2023-12-16T22:13:41.892 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:13:41 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:13:41.621+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:10:42.731761+0000 (oldest deadline 2023-12-16T22:11:02.731761+0000) 2023-12-16T22:13:41.892 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:13:41 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:13:41.621+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:10:42.731761+0000 (oldest deadline 2023-12-16T22:11:02.731761+0000) 2023-12-16T22:13:41.892 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:13:41 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:13:41.621+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:13:42.234 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Dec 16 22:13:42 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-mon-smithi045[80759]: 2023-12-16T22:13:42.109+0000 7f45a1f8f700 -1 mon.smithi045@0(probing) e2 get_health_metrics reporting 34228 slow ops, oldest is monmgrreport(gid 14214, 0 checks, 1 progress events) 2023-12-16T22:13:42.235 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:13:42 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:13:42.022+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:11:07.849373+0000 (oldest deadline 2023-12-16T22:11:27.849373+0000) 2023-12-16T22:13:42.235 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:13:42 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:13:42.022+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:11:07.849373+0000 (oldest deadline 2023-12-16T22:11:27.849373+0000) 2023-12-16T22:13:42.235 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:13:42 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:13:42.022+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:11:07.849373+0000 (oldest deadline 2023-12-16T22:11:27.849373+0000) 2023-12-16T22:13:42.235 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:13:42 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:13:42.023+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:13:42.235 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:13:41 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:13:41.891+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:10:43.026858+0000 (oldest deadline 2023-12-16T22:11:03.026858+0000) 2023-12-16T22:13:42.236 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:13:41 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:13:41.891+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:10:43.026858+0000 (oldest deadline 2023-12-16T22:11:03.026858+0000) 2023-12-16T22:13:42.236 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:13:41 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:13:41.891+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:10:43.026858+0000 (oldest deadline 2023-12-16T22:11:03.026858+0000) 2023-12-16T22:13:42.236 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:13:41 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:13:41.892+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:13:42.938 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:13:42 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:13:42.652+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:10:42.731761+0000 (oldest deadline 2023-12-16T22:11:02.731761+0000) 2023-12-16T22:13:42.938 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:13:42 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:13:42.652+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:10:42.731761+0000 (oldest deadline 2023-12-16T22:11:02.731761+0000) 2023-12-16T22:13:42.938 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:13:42 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:13:42.652+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:10:42.731761+0000 (oldest deadline 2023-12-16T22:11:02.731761+0000) 2023-12-16T22:13:42.938 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:13:42 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:13:42.653+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:13:43.314 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:13:42 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:13:42.937+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:10:43.026858+0000 (oldest deadline 2023-12-16T22:11:03.026858+0000) 2023-12-16T22:13:43.314 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:13:42 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:13:42.937+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:10:43.026858+0000 (oldest deadline 2023-12-16T22:11:03.026858+0000) 2023-12-16T22:13:43.315 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:13:42 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:13:42.937+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:10:43.026858+0000 (oldest deadline 2023-12-16T22:11:03.026858+0000) 2023-12-16T22:13:43.315 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:13:42 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:13:42.938+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:13:43.315 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:13:42 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:13:42.994+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:11:07.849373+0000 (oldest deadline 2023-12-16T22:11:27.849373+0000) 2023-12-16T22:13:43.315 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:13:42 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:13:42.994+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:11:07.849373+0000 (oldest deadline 2023-12-16T22:11:27.849373+0000) 2023-12-16T22:13:43.315 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:13:42 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:13:42.994+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:11:07.849373+0000 (oldest deadline 2023-12-16T22:11:27.849373+0000) 2023-12-16T22:13:43.315 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:13:42 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:13:42.995+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:13:43.913 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:13:43 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:13:43.616+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:10:42.731761+0000 (oldest deadline 2023-12-16T22:11:02.731761+0000) 2023-12-16T22:13:43.913 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:13:43 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:13:43.616+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:10:42.731761+0000 (oldest deadline 2023-12-16T22:11:02.731761+0000) 2023-12-16T22:13:43.913 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:13:43 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:13:43.616+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:10:42.731761+0000 (oldest deadline 2023-12-16T22:11:02.731761+0000) 2023-12-16T22:13:43.913 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:13:43 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:13:43.617+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:13:44.314 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:13:44 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:13:44.023+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:11:07.849373+0000 (oldest deadline 2023-12-16T22:11:27.849373+0000) 2023-12-16T22:13:44.314 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:13:44 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:13:44.023+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:11:07.849373+0000 (oldest deadline 2023-12-16T22:11:27.849373+0000) 2023-12-16T22:13:44.315 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:13:44 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:13:44.023+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:11:07.849373+0000 (oldest deadline 2023-12-16T22:11:27.849373+0000) 2023-12-16T22:13:44.315 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:13:44 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:13:44.024+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:13:44.315 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:13:43 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:13:43.912+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:10:43.026858+0000 (oldest deadline 2023-12-16T22:11:03.026858+0000) 2023-12-16T22:13:44.315 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:13:43 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:13:43.912+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:10:43.026858+0000 (oldest deadline 2023-12-16T22:11:03.026858+0000) 2023-12-16T22:13:44.315 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:13:43 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:13:43.912+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:10:43.026858+0000 (oldest deadline 2023-12-16T22:11:03.026858+0000) 2023-12-16T22:13:44.315 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:13:43 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:13:43.913+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:13:44.881 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:13:44 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:13:44.583+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:10:42.731761+0000 (oldest deadline 2023-12-16T22:11:02.731761+0000) 2023-12-16T22:13:44.881 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:13:44 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:13:44.583+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:10:42.731761+0000 (oldest deadline 2023-12-16T22:11:02.731761+0000) 2023-12-16T22:13:44.881 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:13:44 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:13:44.583+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:10:42.731761+0000 (oldest deadline 2023-12-16T22:11:02.731761+0000) 2023-12-16T22:13:44.881 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:13:44 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:13:44.585+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:13:45.314 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:13:44 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:13:44.991+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:11:07.849373+0000 (oldest deadline 2023-12-16T22:11:27.849373+0000) 2023-12-16T22:13:45.314 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:13:44 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:13:44.991+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:11:07.849373+0000 (oldest deadline 2023-12-16T22:11:27.849373+0000) 2023-12-16T22:13:45.315 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:13:44 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:13:44.991+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:11:07.849373+0000 (oldest deadline 2023-12-16T22:11:27.849373+0000) 2023-12-16T22:13:45.315 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:13:44 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:13:44.992+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:13:45.315 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:13:44 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:13:44.877+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:10:43.026858+0000 (oldest deadline 2023-12-16T22:11:03.026858+0000) 2023-12-16T22:13:45.315 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:13:44 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:13:44.877+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:10:43.026858+0000 (oldest deadline 2023-12-16T22:11:03.026858+0000) 2023-12-16T22:13:45.315 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:13:44 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:13:44.877+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:10:43.026858+0000 (oldest deadline 2023-12-16T22:11:03.026858+0000) 2023-12-16T22:13:45.316 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:13:44 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:13:44.878+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:13:45.871 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:13:45 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:13:45.568+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:10:42.731761+0000 (oldest deadline 2023-12-16T22:11:02.731761+0000) 2023-12-16T22:13:45.871 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:13:45 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:13:45.568+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:10:42.731761+0000 (oldest deadline 2023-12-16T22:11:02.731761+0000) 2023-12-16T22:13:45.871 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:13:45 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:13:45.568+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:10:42.731761+0000 (oldest deadline 2023-12-16T22:11:02.731761+0000) 2023-12-16T22:13:45.871 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:13:45 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:13:45.568+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:13:46.316 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:13:45 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:13:45.947+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:11:07.849373+0000 (oldest deadline 2023-12-16T22:11:27.849373+0000) 2023-12-16T22:13:46.317 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:13:45 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:13:45.947+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:11:07.849373+0000 (oldest deadline 2023-12-16T22:11:27.849373+0000) 2023-12-16T22:13:46.317 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:13:45 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:13:45.947+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:11:07.849373+0000 (oldest deadline 2023-12-16T22:11:27.849373+0000) 2023-12-16T22:13:46.317 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:13:45 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:13:45.948+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:13:46.318 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:13:45 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:13:45.883+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:10:43.026858+0000 (oldest deadline 2023-12-16T22:11:03.026858+0000) 2023-12-16T22:13:46.318 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:13:45 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:13:45.883+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:10:43.026858+0000 (oldest deadline 2023-12-16T22:11:03.026858+0000) 2023-12-16T22:13:46.318 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:13:45 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:13:45.883+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:10:43.026858+0000 (oldest deadline 2023-12-16T22:11:03.026858+0000) 2023-12-16T22:13:46.318 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:13:45 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:13:45.884+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:13:46.892 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:13:46 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:13:46.586+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:10:42.731761+0000 (oldest deadline 2023-12-16T22:11:02.731761+0000) 2023-12-16T22:13:46.892 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:13:46 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:13:46.586+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:10:42.731761+0000 (oldest deadline 2023-12-16T22:11:02.731761+0000) 2023-12-16T22:13:46.892 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:13:46 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:13:46.586+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:10:42.731761+0000 (oldest deadline 2023-12-16T22:11:02.731761+0000) 2023-12-16T22:13:46.892 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:13:46 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:13:46.588+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:13:47.238 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Dec 16 22:13:47 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-mon-smithi045[80759]: 2023-12-16T22:13:47.113+0000 7f45a1f8f700 -1 mon.smithi045@0(probing) e2 get_health_metrics reporting 34319 slow ops, oldest is monmgrreport(gid 14214, 0 checks, 1 progress events) 2023-12-16T22:13:47.239 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:13:46 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:13:46.996+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:11:07.849373+0000 (oldest deadline 2023-12-16T22:11:27.849373+0000) 2023-12-16T22:13:47.239 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:13:46 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:13:46.996+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:11:07.849373+0000 (oldest deadline 2023-12-16T22:11:27.849373+0000) 2023-12-16T22:13:47.239 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:13:46 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:13:46.996+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:11:07.849373+0000 (oldest deadline 2023-12-16T22:11:27.849373+0000) 2023-12-16T22:13:47.239 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:13:46 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:13:46.997+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:13:47.240 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:13:46 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:13:46.891+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:10:43.026858+0000 (oldest deadline 2023-12-16T22:11:03.026858+0000) 2023-12-16T22:13:47.240 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:13:46 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:13:46.891+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:10:43.026858+0000 (oldest deadline 2023-12-16T22:11:03.026858+0000) 2023-12-16T22:13:47.240 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:13:46 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:13:46.891+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:10:43.026858+0000 (oldest deadline 2023-12-16T22:11:03.026858+0000) 2023-12-16T22:13:47.240 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:13:46 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:13:46.892+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:13:47.814 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:13:47 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:13:47.543+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:10:42.731761+0000 (oldest deadline 2023-12-16T22:11:02.731761+0000) 2023-12-16T22:13:47.814 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:13:47 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:13:47.543+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:10:42.731761+0000 (oldest deadline 2023-12-16T22:11:02.731761+0000) 2023-12-16T22:13:47.814 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:13:47 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:13:47.543+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:10:42.731761+0000 (oldest deadline 2023-12-16T22:11:02.731761+0000) 2023-12-16T22:13:47.815 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:13:47 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:13:47.543+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:13:48.314 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:13:47 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:13:47.867+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:10:43.026858+0000 (oldest deadline 2023-12-16T22:11:03.026858+0000) 2023-12-16T22:13:48.314 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:13:47 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:13:47.867+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:10:43.026858+0000 (oldest deadline 2023-12-16T22:11:03.026858+0000) 2023-12-16T22:13:48.314 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:13:47 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:13:47.867+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:10:43.026858+0000 (oldest deadline 2023-12-16T22:11:03.026858+0000) 2023-12-16T22:13:48.315 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:13:47 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:13:47.869+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:13:48.315 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:13:48 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:13:48.016+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:11:07.849373+0000 (oldest deadline 2023-12-16T22:11:27.849373+0000) 2023-12-16T22:13:48.315 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:13:48 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:13:48.016+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:11:07.849373+0000 (oldest deadline 2023-12-16T22:11:27.849373+0000) 2023-12-16T22:13:48.315 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:13:48 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:13:48.016+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:11:07.849373+0000 (oldest deadline 2023-12-16T22:11:27.849373+0000) 2023-12-16T22:13:48.315 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:13:48 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:13:48.017+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:13:48.814 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:13:48 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:13:48.522+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:10:42.731761+0000 (oldest deadline 2023-12-16T22:11:02.731761+0000) 2023-12-16T22:13:48.815 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:13:48 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:13:48.522+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:10:42.731761+0000 (oldest deadline 2023-12-16T22:11:02.731761+0000) 2023-12-16T22:13:48.815 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:13:48 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:13:48.522+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:10:42.731761+0000 (oldest deadline 2023-12-16T22:11:02.731761+0000) 2023-12-16T22:13:48.815 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:13:48 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:13:48.523+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:13:49.314 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:13:48 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:13:48.872+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:10:43.026858+0000 (oldest deadline 2023-12-16T22:11:03.026858+0000) 2023-12-16T22:13:49.315 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:13:48 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:13:48.872+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:10:43.026858+0000 (oldest deadline 2023-12-16T22:11:03.026858+0000) 2023-12-16T22:13:49.315 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:13:48 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:13:48.872+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:10:43.026858+0000 (oldest deadline 2023-12-16T22:11:03.026858+0000) 2023-12-16T22:13:49.315 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:13:48 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:13:48.873+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:13:49.315 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:13:49 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:13:49.047+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:11:07.849373+0000 (oldest deadline 2023-12-16T22:11:27.849373+0000) 2023-12-16T22:13:49.315 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:13:49 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:13:49.047+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:11:07.849373+0000 (oldest deadline 2023-12-16T22:11:27.849373+0000) 2023-12-16T22:13:49.316 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:13:49 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:13:49.047+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:11:07.849373+0000 (oldest deadline 2023-12-16T22:11:27.849373+0000) 2023-12-16T22:13:49.316 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:13:49 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:13:49.048+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:13:49.847 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:13:49 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:13:49.569+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:10:42.731761+0000 (oldest deadline 2023-12-16T22:11:02.731761+0000) 2023-12-16T22:13:49.847 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:13:49 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:13:49.569+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:10:42.731761+0000 (oldest deadline 2023-12-16T22:11:02.731761+0000) 2023-12-16T22:13:49.847 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:13:49 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:13:49.569+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:10:42.731761+0000 (oldest deadline 2023-12-16T22:11:02.731761+0000) 2023-12-16T22:13:49.847 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:13:49 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:13:49.569+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:13:50.227 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:13:49 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:13:49.846+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:10:43.026858+0000 (oldest deadline 2023-12-16T22:11:03.026858+0000) 2023-12-16T22:13:50.227 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:13:49 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:13:49.846+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:10:43.026858+0000 (oldest deadline 2023-12-16T22:11:03.026858+0000) 2023-12-16T22:13:50.227 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:13:49 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:13:49.846+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:10:43.026858+0000 (oldest deadline 2023-12-16T22:11:03.026858+0000) 2023-12-16T22:13:50.227 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:13:49 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:13:49.846+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:13:50.227 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:13:50 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:13:50.025+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:11:07.849373+0000 (oldest deadline 2023-12-16T22:11:27.849373+0000) 2023-12-16T22:13:50.228 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:13:50 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:13:50.025+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:11:07.849373+0000 (oldest deadline 2023-12-16T22:11:27.849373+0000) 2023-12-16T22:13:50.228 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:13:50 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:13:50.025+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:11:07.849373+0000 (oldest deadline 2023-12-16T22:11:27.849373+0000) 2023-12-16T22:13:50.228 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:13:50 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:13:50.026+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:13:50.888 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:13:50 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:13:50.595+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:10:42.731761+0000 (oldest deadline 2023-12-16T22:11:02.731761+0000) 2023-12-16T22:13:50.888 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:13:50 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:13:50.595+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:10:42.731761+0000 (oldest deadline 2023-12-16T22:11:02.731761+0000) 2023-12-16T22:13:50.888 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:13:50 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:13:50.595+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:10:42.731761+0000 (oldest deadline 2023-12-16T22:11:02.731761+0000) 2023-12-16T22:13:50.888 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:13:50 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:13:50.596+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:13:51.314 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:13:51 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:13:51.069+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:11:07.849373+0000 (oldest deadline 2023-12-16T22:11:27.849373+0000) 2023-12-16T22:13:51.314 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:13:51 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:13:51.069+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:11:07.849373+0000 (oldest deadline 2023-12-16T22:11:27.849373+0000) 2023-12-16T22:13:51.314 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:13:51 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:13:51.069+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:11:07.849373+0000 (oldest deadline 2023-12-16T22:11:27.849373+0000) 2023-12-16T22:13:51.315 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:13:51 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:13:51.070+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:13:51.315 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:13:50 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:13:50.887+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:10:43.026858+0000 (oldest deadline 2023-12-16T22:11:03.026858+0000) 2023-12-16T22:13:51.315 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:13:50 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:13:50.887+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:10:43.026858+0000 (oldest deadline 2023-12-16T22:11:03.026858+0000) 2023-12-16T22:13:51.315 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:13:50 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:13:50.887+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:10:43.026858+0000 (oldest deadline 2023-12-16T22:11:03.026858+0000) 2023-12-16T22:13:51.315 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:13:50 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:13:50.888+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:13:51.909 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:13:51 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:13:51.568+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:10:42.731761+0000 (oldest deadline 2023-12-16T22:11:02.731761+0000) 2023-12-16T22:13:51.909 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:13:51 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:13:51.568+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:10:42.731761+0000 (oldest deadline 2023-12-16T22:11:02.731761+0000) 2023-12-16T22:13:51.909 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:13:51 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:13:51.568+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:10:42.731761+0000 (oldest deadline 2023-12-16T22:11:02.731761+0000) 2023-12-16T22:13:51.910 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:13:51 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:13:51.569+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:13:52.245 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Dec 16 22:13:52 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-mon-smithi045[80759]: 2023-12-16T22:13:52.118+0000 7f45a1f8f700 -1 mon.smithi045@0(probing) e2 get_health_metrics reporting 34411 slow ops, oldest is monmgrreport(gid 14214, 0 checks, 1 progress events) 2023-12-16T22:13:52.245 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:13:52 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:13:52.109+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:11:07.849373+0000 (oldest deadline 2023-12-16T22:11:27.849373+0000) 2023-12-16T22:13:52.246 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:13:52 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:13:52.109+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:11:07.849373+0000 (oldest deadline 2023-12-16T22:11:27.849373+0000) 2023-12-16T22:13:52.246 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:13:52 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:13:52.109+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:11:07.849373+0000 (oldest deadline 2023-12-16T22:11:27.849373+0000) 2023-12-16T22:13:52.246 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:13:52 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:13:52.109+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:13:52.246 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:13:51 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:13:51.908+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:10:43.026858+0000 (oldest deadline 2023-12-16T22:11:03.026858+0000) 2023-12-16T22:13:52.246 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:13:51 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:13:51.908+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:10:43.026858+0000 (oldest deadline 2023-12-16T22:11:03.026858+0000) 2023-12-16T22:13:52.246 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:13:51 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:13:51.908+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:10:43.026858+0000 (oldest deadline 2023-12-16T22:11:03.026858+0000) 2023-12-16T22:13:52.246 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:13:51 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:13:51.909+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:13:52.957 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:13:52 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:13:52.578+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:10:42.731761+0000 (oldest deadline 2023-12-16T22:11:02.731761+0000) 2023-12-16T22:13:52.957 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:13:52 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:13:52.578+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:10:42.731761+0000 (oldest deadline 2023-12-16T22:11:02.731761+0000) 2023-12-16T22:13:52.957 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:13:52 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:13:52.578+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:10:42.731761+0000 (oldest deadline 2023-12-16T22:11:02.731761+0000) 2023-12-16T22:13:52.957 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:13:52 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:13:52.578+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:13:53.314 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:13:53 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:13:53.121+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:11:07.849373+0000 (oldest deadline 2023-12-16T22:11:27.849373+0000) 2023-12-16T22:13:53.315 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:13:53 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:13:53.121+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:11:07.849373+0000 (oldest deadline 2023-12-16T22:11:27.849373+0000) 2023-12-16T22:13:53.315 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:13:53 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:13:53.121+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:11:07.849373+0000 (oldest deadline 2023-12-16T22:11:27.849373+0000) 2023-12-16T22:13:53.315 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:13:53 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:13:53.123+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:13:53.315 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:13:52 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:13:52.956+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:10:43.026858+0000 (oldest deadline 2023-12-16T22:11:03.026858+0000) 2023-12-16T22:13:53.315 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:13:52 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:13:52.956+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:10:43.026858+0000 (oldest deadline 2023-12-16T22:11:03.026858+0000) 2023-12-16T22:13:53.315 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:13:52 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:13:52.956+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:10:43.026858+0000 (oldest deadline 2023-12-16T22:11:03.026858+0000) 2023-12-16T22:13:53.316 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:13:52 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:13:52.956+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:13:53.937 ERROR:paramiko.transport:Socket exception: No route to host (113) 2023-12-16T22:13:53.942 DEBUG:teuthology.orchestra.run:got remote process result: None 2023-12-16T22:13:53.942 INFO:tasks.workunit:Stopping ['suites/fsstress.sh'] on client.1... 2023-12-16T22:13:53.943 ERROR:teuthology.run_tasks:Manager failed: kclient Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_5f3731ee8689ef3b0ef7d571e4cc2cb968c78d73/teuthology/run_tasks.py", line 154, in run_tasks suppress = manager.__exit__(*exc_info) File "/usr/lib/python3.8/contextlib.py", line 120, in __exit__ next(self.gen) File "/home/teuthworker/src/github.com_ceph_ceph-c_c7586e95519b7be76f3b6e3a4b333888baf46bc9/qa/tasks/kclient.py", line 139, in task forced = umount_all() File "/home/teuthworker/src/github.com_ceph_ceph-c_c7586e95519b7be76f3b6e3a4b333888baf46bc9/qa/tasks/kclient.py", line 119, in umount_all if mount.is_mounted(): File "/home/teuthworker/src/github.com_ceph_ceph-c_c7586e95519b7be76f3b6e3a4b333888baf46bc9/qa/tasks/cephfs/mount.py", line 220, in is_mounted file = self.client_remote.read_file('/proc/self/mounts',stdout=StringIO()) File "/home/teuthworker/src/git.ceph.com_teuthology_5f3731ee8689ef3b0ef7d571e4cc2cb968c78d73/teuthology/orchestra/remote.py", line 254, in read_file proc = self.run(args=args, stdout=stdout, stderr=StringIO(), File "/home/teuthworker/src/git.ceph.com_teuthology_5f3731ee8689ef3b0ef7d571e4cc2cb968c78d73/teuthology/orchestra/remote.py", line 523, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_5f3731ee8689ef3b0ef7d571e4cc2cb968c78d73/teuthology/orchestra/run.py", line 450, in run r.execute() File "/home/teuthworker/src/git.ceph.com_teuthology_5f3731ee8689ef3b0ef7d571e4cc2cb968c78d73/teuthology/orchestra/run.py", line 100, in execute self.client.exec_command(self.command) File "/home/teuthworker/src/git.ceph.com_teuthology_5f3731ee8689ef3b0ef7d571e4cc2cb968c78d73/virtualenv/lib/python3.8/site-packages/paramiko/client.py", line 525, in exec_command chan = self._transport.open_session(timeout=timeout) File "/home/teuthworker/src/git.ceph.com_teuthology_5f3731ee8689ef3b0ef7d571e4cc2cb968c78d73/virtualenv/lib/python3.8/site-packages/paramiko/transport.py", line 920, in open_session return self.open_channel( File "/home/teuthworker/src/git.ceph.com_teuthology_5f3731ee8689ef3b0ef7d571e4cc2cb968c78d73/virtualenv/lib/python3.8/site-packages/paramiko/transport.py", line 1051, in open_channel raise e File "/home/teuthworker/src/git.ceph.com_teuthology_5f3731ee8689ef3b0ef7d571e4cc2cb968c78d73/virtualenv/lib/python3.8/site-packages/paramiko/transport.py", line 2110, in run ptype, m = self.packetizer.read_message() File "/home/teuthworker/src/git.ceph.com_teuthology_5f3731ee8689ef3b0ef7d571e4cc2cb968c78d73/virtualenv/lib/python3.8/site-packages/paramiko/packet.py", line 459, in read_message header = self.read_all(self.__block_size_in, check_rekey=True) File "/home/teuthworker/src/git.ceph.com_teuthology_5f3731ee8689ef3b0ef7d571e4cc2cb968c78d73/virtualenv/lib/python3.8/site-packages/paramiko/packet.py", line 301, in read_all x = self.__socket.recv(n) File "/home/teuthworker/src/git.ceph.com_teuthology_5f3731ee8689ef3b0ef7d571e4cc2cb968c78d73/virtualenv/lib/python3.8/site-packages/gevent/_socketcommon.py", line 657, in recv return self._sock.recv(*args) OSError: [Errno 113] No route to host 2023-12-16T22:13:53.956 DEBUG:teuthology.run_tasks:Unwinding manager cephadm 2023-12-16T22:13:53.971 INFO:tasks.cephadm:Teardown begin 2023-12-16T22:13:53.971 ERROR:teuthology.contextutil:Saw exception from nested tasks Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_5f3731ee8689ef3b0ef7d571e4cc2cb968c78d73/teuthology/contextutil.py", line 32, in nested yield vars File "/home/teuthworker/src/github.com_ceph_ceph-c_c7586e95519b7be76f3b6e3a4b333888baf46bc9/qa/tasks/cephadm.py", line 1794, in task yield File "/home/teuthworker/src/git.ceph.com_teuthology_5f3731ee8689ef3b0ef7d571e4cc2cb968c78d73/teuthology/run_tasks.py", line 154, in run_tasks suppress = manager.__exit__(*exc_info) File "/usr/lib/python3.8/contextlib.py", line 120, in __exit__ next(self.gen) File "/home/teuthworker/src/github.com_ceph_ceph-c_c7586e95519b7be76f3b6e3a4b333888baf46bc9/qa/tasks/kclient.py", line 139, in task forced = umount_all() File "/home/teuthworker/src/github.com_ceph_ceph-c_c7586e95519b7be76f3b6e3a4b333888baf46bc9/qa/tasks/kclient.py", line 119, in umount_all if mount.is_mounted(): File "/home/teuthworker/src/github.com_ceph_ceph-c_c7586e95519b7be76f3b6e3a4b333888baf46bc9/qa/tasks/cephfs/mount.py", line 220, in is_mounted file = self.client_remote.read_file('/proc/self/mounts',stdout=StringIO()) File "/home/teuthworker/src/git.ceph.com_teuthology_5f3731ee8689ef3b0ef7d571e4cc2cb968c78d73/teuthology/orchestra/remote.py", line 254, in read_file proc = self.run(args=args, stdout=stdout, stderr=StringIO(), File "/home/teuthworker/src/git.ceph.com_teuthology_5f3731ee8689ef3b0ef7d571e4cc2cb968c78d73/teuthology/orchestra/remote.py", line 523, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_5f3731ee8689ef3b0ef7d571e4cc2cb968c78d73/teuthology/orchestra/run.py", line 450, in run r.execute() File "/home/teuthworker/src/git.ceph.com_teuthology_5f3731ee8689ef3b0ef7d571e4cc2cb968c78d73/teuthology/orchestra/run.py", line 100, in execute self.client.exec_command(self.command) File "/home/teuthworker/src/git.ceph.com_teuthology_5f3731ee8689ef3b0ef7d571e4cc2cb968c78d73/virtualenv/lib/python3.8/site-packages/paramiko/client.py", line 525, in exec_command chan = self._transport.open_session(timeout=timeout) File "/home/teuthworker/src/git.ceph.com_teuthology_5f3731ee8689ef3b0ef7d571e4cc2cb968c78d73/virtualenv/lib/python3.8/site-packages/paramiko/transport.py", line 920, in open_session return self.open_channel( File "/home/teuthworker/src/git.ceph.com_teuthology_5f3731ee8689ef3b0ef7d571e4cc2cb968c78d73/virtualenv/lib/python3.8/site-packages/paramiko/transport.py", line 1051, in open_channel raise e File "/home/teuthworker/src/git.ceph.com_teuthology_5f3731ee8689ef3b0ef7d571e4cc2cb968c78d73/virtualenv/lib/python3.8/site-packages/paramiko/transport.py", line 2110, in run ptype, m = self.packetizer.read_message() File "/home/teuthworker/src/git.ceph.com_teuthology_5f3731ee8689ef3b0ef7d571e4cc2cb968c78d73/virtualenv/lib/python3.8/site-packages/paramiko/packet.py", line 459, in read_message header = self.read_all(self.__block_size_in, check_rekey=True) File "/home/teuthworker/src/git.ceph.com_teuthology_5f3731ee8689ef3b0ef7d571e4cc2cb968c78d73/virtualenv/lib/python3.8/site-packages/paramiko/packet.py", line 301, in read_all x = self.__socket.recv(n) File "/home/teuthworker/src/git.ceph.com_teuthology_5f3731ee8689ef3b0ef7d571e4cc2cb968c78d73/virtualenv/lib/python3.8/site-packages/gevent/_socketcommon.py", line 657, in recv return self._sock.recv(*args) OSError: [Errno 113] No route to host 2023-12-16T22:13:53.976 DEBUG:teuthology.orchestra.run.smithi045:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2023-12-16T22:13:53.988 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:13:53 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:13:53.575+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:10:42.731761+0000 (oldest deadline 2023-12-16T22:11:02.731761+0000) 2023-12-16T22:13:53.988 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:13:53 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:13:53.575+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:10:42.731761+0000 (oldest deadline 2023-12-16T22:11:02.731761+0000) 2023-12-16T22:13:53.988 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:13:53 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:13:53.575+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:10:42.731761+0000 (oldest deadline 2023-12-16T22:11:02.731761+0000) 2023-12-16T22:13:53.988 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:13:53 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:13:53.576+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:13:54.023 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi169.front.sepia.ceph.com' 2023-12-16T22:13:54.024 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi169.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-12-16T22:13:54.314 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:13:53 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:13:53.939+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:10:43.026858+0000 (oldest deadline 2023-12-16T22:11:03.026858+0000) 2023-12-16T22:13:54.315 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:13:53 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:13:53.939+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:10:43.026858+0000 (oldest deadline 2023-12-16T22:11:03.026858+0000) 2023-12-16T22:13:54.315 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:13:53 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:13:53.939+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:10:43.026858+0000 (oldest deadline 2023-12-16T22:11:03.026858+0000) 2023-12-16T22:13:54.315 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:13:53 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:13:53.940+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:13:54.315 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:13:54 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:13:54.084+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:11:07.849373+0000 (oldest deadline 2023-12-16T22:11:27.849373+0000) 2023-12-16T22:13:54.315 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:13:54 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:13:54.084+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:11:07.849373+0000 (oldest deadline 2023-12-16T22:11:27.849373+0000) 2023-12-16T22:13:54.315 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:13:54 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:13:54.084+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:11:07.849373+0000 (oldest deadline 2023-12-16T22:11:27.849373+0000) 2023-12-16T22:13:54.316 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:13:54 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:13:54.085+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:13:54.932 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:13:54 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:13:54.566+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:10:42.731761+0000 (oldest deadline 2023-12-16T22:11:02.731761+0000) 2023-12-16T22:13:54.932 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:13:54 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:13:54.566+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:10:42.731761+0000 (oldest deadline 2023-12-16T22:11:02.731761+0000) 2023-12-16T22:13:54.932 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:13:54 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:13:54.566+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:10:42.731761+0000 (oldest deadline 2023-12-16T22:11:02.731761+0000) 2023-12-16T22:13:54.932 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:13:54 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:13:54.567+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:13:55.314 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:13:55 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:13:55.129+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:11:07.849373+0000 (oldest deadline 2023-12-16T22:11:27.849373+0000) 2023-12-16T22:13:55.314 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:13:55 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:13:55.129+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:11:07.849373+0000 (oldest deadline 2023-12-16T22:11:27.849373+0000) 2023-12-16T22:13:55.315 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:13:55 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:13:55.129+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:11:07.849373+0000 (oldest deadline 2023-12-16T22:11:27.849373+0000) 2023-12-16T22:13:55.315 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:13:55 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:13:55.130+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:13:55.315 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:13:54 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:13:54.931+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:10:43.026858+0000 (oldest deadline 2023-12-16T22:11:03.026858+0000) 2023-12-16T22:13:55.315 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:13:54 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:13:54.931+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:10:43.026858+0000 (oldest deadline 2023-12-16T22:11:03.026858+0000) 2023-12-16T22:13:55.315 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:13:54 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:13:54.931+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:10:43.026858+0000 (oldest deadline 2023-12-16T22:11:03.026858+0000) 2023-12-16T22:13:55.315 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:13:54 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:13:54.932+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:13:55.814 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:13:55 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:13:55.539+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:10:42.731761+0000 (oldest deadline 2023-12-16T22:11:02.731761+0000) 2023-12-16T22:13:55.814 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:13:55 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:13:55.539+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:10:42.731761+0000 (oldest deadline 2023-12-16T22:11:02.731761+0000) 2023-12-16T22:13:55.814 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:13:55 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:13:55.539+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:10:42.731761+0000 (oldest deadline 2023-12-16T22:11:02.731761+0000) 2023-12-16T22:13:55.815 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:13:55 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:13:55.540+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:13:56.314 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:13:55 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:13:55.937+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:10:43.026858+0000 (oldest deadline 2023-12-16T22:11:03.026858+0000) 2023-12-16T22:13:56.314 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:13:55 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:13:55.937+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:10:43.026858+0000 (oldest deadline 2023-12-16T22:11:03.026858+0000) 2023-12-16T22:13:56.315 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:13:55 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:13:55.937+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:10:43.026858+0000 (oldest deadline 2023-12-16T22:11:03.026858+0000) 2023-12-16T22:13:56.315 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:13:55 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:13:55.938+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:13:56.315 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:13:56 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:13:56.121+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:11:07.849373+0000 (oldest deadline 2023-12-16T22:11:27.849373+0000) 2023-12-16T22:13:56.315 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:13:56 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:13:56.121+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:11:07.849373+0000 (oldest deadline 2023-12-16T22:11:27.849373+0000) 2023-12-16T22:13:56.315 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:13:56 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:13:56.121+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:11:07.849373+0000 (oldest deadline 2023-12-16T22:11:27.849373+0000) 2023-12-16T22:13:56.315 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:13:56 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:13:56.122+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:13:56.814 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:13:56 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:13:56.552+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:10:42.731761+0000 (oldest deadline 2023-12-16T22:11:02.731761+0000) 2023-12-16T22:13:56.815 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:13:56 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:13:56.552+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:10:42.731761+0000 (oldest deadline 2023-12-16T22:11:02.731761+0000) 2023-12-16T22:13:56.815 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:13:56 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:13:56.552+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:10:42.731761+0000 (oldest deadline 2023-12-16T22:11:02.731761+0000) 2023-12-16T22:13:56.815 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:13:56 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:13:56.553+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:13:57.105 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.169 2023-12-16T22:13:57.147 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Dec 16 22:13:57 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-mon-smithi045[80759]: 2023-12-16T22:13:57.122+0000 7f45a1f8f700 -1 mon.smithi045@0(probing) e2 get_health_metrics reporting 34500 slow ops, oldest is monmgrreport(gid 14214, 0 checks, 1 progress events) 2023-12-16T22:13:57.147 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:13:56 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:13:56.895+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:10:43.026858+0000 (oldest deadline 2023-12-16T22:11:03.026858+0000) 2023-12-16T22:13:57.148 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:13:56 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:13:56.895+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:10:43.026858+0000 (oldest deadline 2023-12-16T22:11:03.026858+0000) 2023-12-16T22:13:57.148 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:13:56 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:13:56.895+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:10:43.026858+0000 (oldest deadline 2023-12-16T22:11:03.026858+0000) 2023-12-16T22:13:57.148 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:13:56 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:13:56.896+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:13:57.541 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:13:57 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:13:57.154+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:11:07.849373+0000 (oldest deadline 2023-12-16T22:11:27.849373+0000) 2023-12-16T22:13:57.541 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:13:57 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:13:57.154+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:11:07.849373+0000 (oldest deadline 2023-12-16T22:11:27.849373+0000) 2023-12-16T22:13:57.541 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:13:57 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:13:57.154+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:11:07.849373+0000 (oldest deadline 2023-12-16T22:11:27.849373+0000) 2023-12-16T22:13:57.541 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:13:57 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:13:57.155+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:13:57.814 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:13:57 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:13:57.540+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:10:42.731761+0000 (oldest deadline 2023-12-16T22:11:02.731761+0000) 2023-12-16T22:13:57.815 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:13:57 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:13:57.540+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:10:42.731761+0000 (oldest deadline 2023-12-16T22:11:02.731761+0000) 2023-12-16T22:13:57.815 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:13:57 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:13:57.540+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:10:42.731761+0000 (oldest deadline 2023-12-16T22:11:02.731761+0000) 2023-12-16T22:13:57.815 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:13:57 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:13:57.541+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:13:58.118 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:13:57 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:13:57.858+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:10:43.026858+0000 (oldest deadline 2023-12-16T22:11:03.026858+0000) 2023-12-16T22:13:58.118 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:13:57 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:13:57.858+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:10:43.026858+0000 (oldest deadline 2023-12-16T22:11:03.026858+0000) 2023-12-16T22:13:58.118 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:13:57 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:13:57.858+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:10:43.026858+0000 (oldest deadline 2023-12-16T22:11:03.026858+0000) 2023-12-16T22:13:58.118 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:13:57 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:13:57.859+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:13:58.564 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:13:58 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:13:58.117+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:11:07.849373+0000 (oldest deadline 2023-12-16T22:11:27.849373+0000) 2023-12-16T22:13:58.564 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:13:58 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:13:58.117+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:11:07.849373+0000 (oldest deadline 2023-12-16T22:11:27.849373+0000) 2023-12-16T22:13:58.564 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:13:58 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:13:58.117+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:11:07.849373+0000 (oldest deadline 2023-12-16T22:11:27.849373+0000) 2023-12-16T22:13:58.565 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:13:58 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:13:58.118+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:13:59.064 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:13:58 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:13:58.814+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:10:43.026858+0000 (oldest deadline 2023-12-16T22:11:03.026858+0000) 2023-12-16T22:13:59.065 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:13:58 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:13:58.814+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:10:43.026858+0000 (oldest deadline 2023-12-16T22:11:03.026858+0000) 2023-12-16T22:13:59.065 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:13:58 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:13:58.814+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:10:43.026858+0000 (oldest deadline 2023-12-16T22:11:03.026858+0000) 2023-12-16T22:13:59.065 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:13:58 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:13:58.814+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:13:59.065 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:13:58 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:13:58.572+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:10:42.731761+0000 (oldest deadline 2023-12-16T22:11:02.731761+0000) 2023-12-16T22:13:59.065 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:13:58 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:13:58.572+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:10:42.731761+0000 (oldest deadline 2023-12-16T22:11:02.731761+0000) 2023-12-16T22:13:59.065 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:13:58 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:13:58.572+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:10:42.731761+0000 (oldest deadline 2023-12-16T22:11:02.731761+0000) 2023-12-16T22:13:59.066 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:13:58 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:13:58.573+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:13:59.564 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:13:59 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:13:59.129+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:11:07.849373+0000 (oldest deadline 2023-12-16T22:11:27.849373+0000) 2023-12-16T22:13:59.564 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:13:59 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:13:59.129+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:11:07.849373+0000 (oldest deadline 2023-12-16T22:11:27.849373+0000) 2023-12-16T22:13:59.564 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:13:59 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:13:59.129+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:11:07.849373+0000 (oldest deadline 2023-12-16T22:11:27.849373+0000) 2023-12-16T22:13:59.564 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:13:59 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:13:59.130+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:13:59.856 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:13:59 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:13:59.591+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:10:42.731761+0000 (oldest deadline 2023-12-16T22:11:02.731761+0000) 2023-12-16T22:13:59.857 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:13:59 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:13:59.591+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:10:42.731761+0000 (oldest deadline 2023-12-16T22:11:02.731761+0000) 2023-12-16T22:13:59.857 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:13:59 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:13:59.591+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:10:42.731761+0000 (oldest deadline 2023-12-16T22:11:02.731761+0000) 2023-12-16T22:13:59.857 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:13:59 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:13:59.592+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:14:00.228 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:13:59 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:13:59.856+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:10:43.026858+0000 (oldest deadline 2023-12-16T22:11:03.026858+0000) 2023-12-16T22:14:00.228 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:13:59 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:13:59.856+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:10:43.026858+0000 (oldest deadline 2023-12-16T22:11:03.026858+0000) 2023-12-16T22:14:00.228 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:13:59 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:13:59.856+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:10:43.026858+0000 (oldest deadline 2023-12-16T22:11:03.026858+0000) 2023-12-16T22:14:00.228 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:13:59 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:13:59.856+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:14:00.229 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:14:00 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:14:00.091+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:11:07.849373+0000 (oldest deadline 2023-12-16T22:11:27.849373+0000) 2023-12-16T22:14:00.229 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:14:00 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:14:00.091+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:11:07.849373+0000 (oldest deadline 2023-12-16T22:11:27.849373+0000) 2023-12-16T22:14:00.229 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:14:00 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:14:00.091+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:11:07.849373+0000 (oldest deadline 2023-12-16T22:11:27.849373+0000) 2023-12-16T22:14:00.229 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:14:00 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:14:00.092+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:14:00.887 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:14:00 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:14:00.574+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:10:42.731761+0000 (oldest deadline 2023-12-16T22:11:02.731761+0000) 2023-12-16T22:14:00.888 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:14:00 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:14:00.574+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:10:42.731761+0000 (oldest deadline 2023-12-16T22:11:02.731761+0000) 2023-12-16T22:14:00.888 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:14:00 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:14:00.574+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:10:42.731761+0000 (oldest deadline 2023-12-16T22:11:02.731761+0000) 2023-12-16T22:14:00.888 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:14:00 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:14:00.575+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:14:01.231 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:14:01 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:14:01.076+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:11:07.849373+0000 (oldest deadline 2023-12-16T22:11:27.849373+0000) 2023-12-16T22:14:01.231 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:14:01 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:14:01.076+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:11:07.849373+0000 (oldest deadline 2023-12-16T22:11:27.849373+0000) 2023-12-16T22:14:01.231 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:14:01 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:14:01.076+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:11:07.849373+0000 (oldest deadline 2023-12-16T22:11:27.849373+0000) 2023-12-16T22:14:01.231 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:14:01 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:14:01.076+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:14:01.231 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:14:00 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:14:00.886+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:10:43.026858+0000 (oldest deadline 2023-12-16T22:11:03.026858+0000) 2023-12-16T22:14:01.232 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:14:00 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:14:00.886+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:10:43.026858+0000 (oldest deadline 2023-12-16T22:11:03.026858+0000) 2023-12-16T22:14:01.232 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:14:00 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:14:00.886+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:10:43.026858+0000 (oldest deadline 2023-12-16T22:11:03.026858+0000) 2023-12-16T22:14:01.232 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:14:00 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:14:00.887+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:14:01.814 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:14:01 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:14:01.545+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:10:42.731761+0000 (oldest deadline 2023-12-16T22:11:02.731761+0000) 2023-12-16T22:14:01.814 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:14:01 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:14:01.545+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:10:42.731761+0000 (oldest deadline 2023-12-16T22:11:02.731761+0000) 2023-12-16T22:14:01.814 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:14:01 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:14:01.545+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:10:42.731761+0000 (oldest deadline 2023-12-16T22:11:02.731761+0000) 2023-12-16T22:14:01.815 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:14:01 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:14:01.546+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:14:02.128 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:14:01 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:14:01.870+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:10:43.026858+0000 (oldest deadline 2023-12-16T22:11:03.026858+0000) 2023-12-16T22:14:02.129 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:14:01 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:14:01.870+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:10:43.026858+0000 (oldest deadline 2023-12-16T22:11:03.026858+0000) 2023-12-16T22:14:02.129 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:14:01 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:14:01.870+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:10:43.026858+0000 (oldest deadline 2023-12-16T22:11:03.026858+0000) 2023-12-16T22:14:02.129 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:14:01 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:14:01.870+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:14:02.129 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:14:02 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:14:02.082+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:11:07.849373+0000 (oldest deadline 2023-12-16T22:11:27.849373+0000) 2023-12-16T22:14:02.129 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:14:02 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:14:02.082+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:11:07.849373+0000 (oldest deadline 2023-12-16T22:11:27.849373+0000) 2023-12-16T22:14:02.129 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:14:02 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:14:02.082+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:11:07.849373+0000 (oldest deadline 2023-12-16T22:11:27.849373+0000) 2023-12-16T22:14:02.130 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:14:02 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:14:02.084+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:14:02.531 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Dec 16 22:14:02 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-mon-smithi045[80759]: 2023-12-16T22:14:02.127+0000 7f45a1f8f700 -1 mon.smithi045@0(probing) e2 get_health_metrics reporting 34581 slow ops, oldest is monmgrreport(gid 14214, 0 checks, 1 progress events) 2023-12-16T22:14:02.814 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:14:02 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:14:02.530+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:10:42.731761+0000 (oldest deadline 2023-12-16T22:11:02.731761+0000) 2023-12-16T22:14:02.814 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:14:02 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:14:02.530+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:10:42.731761+0000 (oldest deadline 2023-12-16T22:11:02.731761+0000) 2023-12-16T22:14:02.815 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:14:02 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:14:02.530+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:10:42.731761+0000 (oldest deadline 2023-12-16T22:11:02.731761+0000) 2023-12-16T22:14:02.815 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:14:02 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:14:02.531+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:14:03.314 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:14:02 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:14:02.919+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:10:43.026858+0000 (oldest deadline 2023-12-16T22:11:03.026858+0000) 2023-12-16T22:14:03.315 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:14:02 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:14:02.919+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:10:43.026858+0000 (oldest deadline 2023-12-16T22:11:03.026858+0000) 2023-12-16T22:14:03.315 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:14:02 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:14:02.919+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:10:43.026858+0000 (oldest deadline 2023-12-16T22:11:03.026858+0000) 2023-12-16T22:14:03.315 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:14:02 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:14:02.920+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:14:03.315 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:14:03 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:14:03.107+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:11:07.849373+0000 (oldest deadline 2023-12-16T22:11:27.849373+0000) 2023-12-16T22:14:03.315 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:14:03 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:14:03.107+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:11:07.849373+0000 (oldest deadline 2023-12-16T22:11:27.849373+0000) 2023-12-16T22:14:03.316 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:14:03 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:14:03.107+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:11:07.849373+0000 (oldest deadline 2023-12-16T22:11:27.849373+0000) 2023-12-16T22:14:03.316 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:14:03 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:14:03.108+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:14:03.932 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:14:03 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:14:03.564+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:10:42.731761+0000 (oldest deadline 2023-12-16T22:11:02.731761+0000) 2023-12-16T22:14:03.932 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:14:03 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:14:03.564+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:10:42.731761+0000 (oldest deadline 2023-12-16T22:11:02.731761+0000) 2023-12-16T22:14:03.933 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:14:03 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:14:03.564+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:10:42.731761+0000 (oldest deadline 2023-12-16T22:11:02.731761+0000) 2023-12-16T22:14:03.933 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:14:03 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:14:03.565+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:14:04.303 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:14:04 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:14:04.109+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:11:07.849373+0000 (oldest deadline 2023-12-16T22:11:27.849373+0000) 2023-12-16T22:14:04.303 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:14:04 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:14:04.109+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:11:07.849373+0000 (oldest deadline 2023-12-16T22:11:27.849373+0000) 2023-12-16T22:14:04.304 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:14:04 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:14:04.109+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:11:07.849373+0000 (oldest deadline 2023-12-16T22:11:27.849373+0000) 2023-12-16T22:14:04.304 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:14:04 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:14:04.110+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:14:04.304 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:14:03 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:14:03.931+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:10:43.026858+0000 (oldest deadline 2023-12-16T22:11:03.026858+0000) 2023-12-16T22:14:04.304 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:14:03 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:14:03.931+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:10:43.026858+0000 (oldest deadline 2023-12-16T22:11:03.026858+0000) 2023-12-16T22:14:04.305 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:14:03 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:14:03.931+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:10:43.026858+0000 (oldest deadline 2023-12-16T22:11:03.026858+0000) 2023-12-16T22:14:04.305 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:14:03 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:14:03.932+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:14:04.902 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:14:04 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:14:04.590+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:10:42.731761+0000 (oldest deadline 2023-12-16T22:11:02.731761+0000) 2023-12-16T22:14:04.902 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:14:04 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:14:04.590+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:10:42.731761+0000 (oldest deadline 2023-12-16T22:11:02.731761+0000) 2023-12-16T22:14:04.902 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:14:04 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:14:04.590+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:10:42.731761+0000 (oldest deadline 2023-12-16T22:11:02.731761+0000) 2023-12-16T22:14:04.903 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:14:04 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:14:04.590+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:14:05.314 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:14:04 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:14:04.901+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:10:43.026858+0000 (oldest deadline 2023-12-16T22:11:03.026858+0000) 2023-12-16T22:14:05.315 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:14:04 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:14:04.901+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:10:43.026858+0000 (oldest deadline 2023-12-16T22:11:03.026858+0000) 2023-12-16T22:14:05.315 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:14:04 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:14:04.901+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:10:43.026858+0000 (oldest deadline 2023-12-16T22:11:03.026858+0000) 2023-12-16T22:14:05.315 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:14:04 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:14:04.902+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:14:05.315 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:14:05 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:14:05.131+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:11:07.849373+0000 (oldest deadline 2023-12-16T22:11:27.849373+0000) 2023-12-16T22:14:05.315 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:14:05 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:14:05.131+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:11:07.849373+0000 (oldest deadline 2023-12-16T22:11:27.849373+0000) 2023-12-16T22:14:05.315 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:14:05 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:14:05.131+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:11:07.849373+0000 (oldest deadline 2023-12-16T22:11:27.849373+0000) 2023-12-16T22:14:05.316 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:14:05 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:14:05.132+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:14:05.920 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:14:05 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:14:05.611+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:10:42.731761+0000 (oldest deadline 2023-12-16T22:11:02.731761+0000) 2023-12-16T22:14:05.920 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:14:05 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:14:05.611+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:10:42.731761+0000 (oldest deadline 2023-12-16T22:11:02.731761+0000) 2023-12-16T22:14:05.921 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:14:05 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:14:05.611+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:10:42.731761+0000 (oldest deadline 2023-12-16T22:11:02.731761+0000) 2023-12-16T22:14:05.921 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:14:05 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:14:05.612+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:14:06.106 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi169.front.sepia.ceph.com' 2023-12-16T22:14:06.107 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi169.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-12-16T22:14:06.314 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:14:05 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:14:05.919+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:10:43.026858+0000 (oldest deadline 2023-12-16T22:11:03.026858+0000) 2023-12-16T22:14:06.314 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:14:05 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:14:05.919+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:10:43.026858+0000 (oldest deadline 2023-12-16T22:11:03.026858+0000) 2023-12-16T22:14:06.315 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:14:05 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:14:05.919+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:10:43.026858+0000 (oldest deadline 2023-12-16T22:11:03.026858+0000) 2023-12-16T22:14:06.315 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:14:05 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:14:05.920+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:14:06.315 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:14:06 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:14:06.142+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:11:07.849373+0000 (oldest deadline 2023-12-16T22:11:27.849373+0000) 2023-12-16T22:14:06.315 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:14:06 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:14:06.142+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:11:07.849373+0000 (oldest deadline 2023-12-16T22:11:27.849373+0000) 2023-12-16T22:14:06.315 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:14:06 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:14:06.142+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:11:07.849373+0000 (oldest deadline 2023-12-16T22:11:27.849373+0000) 2023-12-16T22:14:06.315 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:14:06 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:14:06.143+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:14:06.878 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:14:06 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:14:06.587+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:10:42.731761+0000 (oldest deadline 2023-12-16T22:11:02.731761+0000) 2023-12-16T22:14:06.878 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:14:06 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:14:06.587+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:10:42.731761+0000 (oldest deadline 2023-12-16T22:11:02.731761+0000) 2023-12-16T22:14:06.879 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:14:06 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:14:06.587+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:10:42.731761+0000 (oldest deadline 2023-12-16T22:11:02.731761+0000) 2023-12-16T22:14:06.879 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:14:06 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:14:06.588+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:14:07.132 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:14:06 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:14:06.877+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:10:43.026858+0000 (oldest deadline 2023-12-16T22:11:03.026858+0000) 2023-12-16T22:14:07.132 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:14:06 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:14:06.877+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:10:43.026858+0000 (oldest deadline 2023-12-16T22:11:03.026858+0000) 2023-12-16T22:14:07.133 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:14:06 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:14:06.877+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:10:43.026858+0000 (oldest deadline 2023-12-16T22:11:03.026858+0000) 2023-12-16T22:14:07.133 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:14:06 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:14:06.878+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:14:07.552 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Dec 16 22:14:07 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-mon-smithi045[80759]: 2023-12-16T22:14:07.131+0000 7f45a1f8f700 -1 mon.smithi045@0(probing) e2 get_health_metrics reporting 34671 slow ops, oldest is monmgrreport(gid 14214, 0 checks, 1 progress events) 2023-12-16T22:14:07.553 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:14:07 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:14:07.135+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:11:07.849373+0000 (oldest deadline 2023-12-16T22:11:27.849373+0000) 2023-12-16T22:14:07.553 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:14:07 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:14:07.135+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:11:07.849373+0000 (oldest deadline 2023-12-16T22:11:27.849373+0000) 2023-12-16T22:14:07.553 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:14:07 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:14:07.135+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:11:07.849373+0000 (oldest deadline 2023-12-16T22:11:27.849373+0000) 2023-12-16T22:14:07.553 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:14:07 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:14:07.136+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:14:07.814 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:14:07 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:14:07.551+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:10:42.731761+0000 (oldest deadline 2023-12-16T22:11:02.731761+0000) 2023-12-16T22:14:07.814 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:14:07 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:14:07.551+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:10:42.731761+0000 (oldest deadline 2023-12-16T22:11:02.731761+0000) 2023-12-16T22:14:07.814 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:14:07 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:14:07.551+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:10:42.731761+0000 (oldest deadline 2023-12-16T22:11:02.731761+0000) 2023-12-16T22:14:07.815 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:14:07 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:14:07.552+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:14:08.165 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:14:07 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:14:07.858+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:10:43.026858+0000 (oldest deadline 2023-12-16T22:11:03.026858+0000) 2023-12-16T22:14:08.166 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:14:07 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:14:07.858+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:10:43.026858+0000 (oldest deadline 2023-12-16T22:11:03.026858+0000) 2023-12-16T22:14:08.166 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:14:07 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:14:07.858+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:10:43.026858+0000 (oldest deadline 2023-12-16T22:11:03.026858+0000) 2023-12-16T22:14:08.166 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:14:07 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:14:07.859+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:14:08.564 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:14:08 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:14:08.164+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:11:07.849373+0000 (oldest deadline 2023-12-16T22:11:27.849373+0000) 2023-12-16T22:14:08.564 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:14:08 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:14:08.164+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:11:07.849373+0000 (oldest deadline 2023-12-16T22:11:27.849373+0000) 2023-12-16T22:14:08.564 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:14:08 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:14:08.164+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:11:07.849373+0000 (oldest deadline 2023-12-16T22:11:27.849373+0000) 2023-12-16T22:14:08.565 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:14:08 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:14:08.165+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:14:08.901 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:14:08 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:14:08.577+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:10:42.731761+0000 (oldest deadline 2023-12-16T22:11:02.731761+0000) 2023-12-16T22:14:08.901 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:14:08 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:14:08.577+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:10:42.731761+0000 (oldest deadline 2023-12-16T22:11:02.731761+0000) 2023-12-16T22:14:08.901 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:14:08 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:14:08.577+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:10:42.731761+0000 (oldest deadline 2023-12-16T22:11:02.731761+0000) 2023-12-16T22:14:08.901 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:14:08 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:14:08.578+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:14:09.169 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.169 2023-12-16T22:14:09.314 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:14:08 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:14:08.900+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:10:43.026858+0000 (oldest deadline 2023-12-16T22:11:03.026858+0000) 2023-12-16T22:14:09.314 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:14:08 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:14:08.900+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:10:43.026858+0000 (oldest deadline 2023-12-16T22:11:03.026858+0000) 2023-12-16T22:14:09.315 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:14:08 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:14:08.900+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:10:43.026858+0000 (oldest deadline 2023-12-16T22:11:03.026858+0000) 2023-12-16T22:14:09.315 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:14:08 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:14:08.901+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:14:09.315 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:14:09 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:14:09.122+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:11:07.849373+0000 (oldest deadline 2023-12-16T22:11:27.849373+0000) 2023-12-16T22:14:09.315 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:14:09 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:14:09.122+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:11:07.849373+0000 (oldest deadline 2023-12-16T22:11:27.849373+0000) 2023-12-16T22:14:09.315 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:14:09 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:14:09.122+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:11:07.849373+0000 (oldest deadline 2023-12-16T22:11:27.849373+0000) 2023-12-16T22:14:09.315 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:14:09 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:14:09.123+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:14:09.884 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:14:09 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:14:09.623+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:10:42.731761+0000 (oldest deadline 2023-12-16T22:11:02.731761+0000) 2023-12-16T22:14:09.884 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:14:09 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:14:09.623+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:10:42.731761+0000 (oldest deadline 2023-12-16T22:11:02.731761+0000) 2023-12-16T22:14:09.885 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:14:09 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:14:09.623+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:10:42.731761+0000 (oldest deadline 2023-12-16T22:11:02.731761+0000) 2023-12-16T22:14:09.885 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:14:09 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:14:09.623+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:14:10.228 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:14:10 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:14:10.091+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:11:07.849373+0000 (oldest deadline 2023-12-16T22:11:27.849373+0000) 2023-12-16T22:14:10.229 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:14:10 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:14:10.091+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:11:07.849373+0000 (oldest deadline 2023-12-16T22:11:27.849373+0000) 2023-12-16T22:14:10.229 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:14:10 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:14:10.091+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:11:07.849373+0000 (oldest deadline 2023-12-16T22:11:27.849373+0000) 2023-12-16T22:14:10.229 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:14:10 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:14:10.092+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:14:10.229 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:14:09 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:14:09.883+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:10:43.026858+0000 (oldest deadline 2023-12-16T22:11:03.026858+0000) 2023-12-16T22:14:10.229 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:14:09 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:14:09.883+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:10:43.026858+0000 (oldest deadline 2023-12-16T22:11:03.026858+0000) 2023-12-16T22:14:10.229 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:14:09 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:14:09.883+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:10:43.026858+0000 (oldest deadline 2023-12-16T22:11:03.026858+0000) 2023-12-16T22:14:10.229 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:14:09 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:14:09.884+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:14:11.050 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:14:10 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:14:10.874+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:10:43.026858+0000 (oldest deadline 2023-12-16T22:11:03.026858+0000) 2023-12-16T22:14:11.050 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:14:10 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:14:10.874+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:10:43.026858+0000 (oldest deadline 2023-12-16T22:11:03.026858+0000) 2023-12-16T22:14:11.051 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:14:10 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:14:10.874+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:10:43.026858+0000 (oldest deadline 2023-12-16T22:11:03.026858+0000) 2023-12-16T22:14:11.051 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:14:10 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:14:10.875+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:14:11.051 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:14:10 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:14:10.652+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:10:42.731761+0000 (oldest deadline 2023-12-16T22:11:02.731761+0000) 2023-12-16T22:14:11.051 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:14:10 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:14:10.652+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:10:42.731761+0000 (oldest deadline 2023-12-16T22:11:02.731761+0000) 2023-12-16T22:14:11.051 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:14:10 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:14:10.652+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:10:42.731761+0000 (oldest deadline 2023-12-16T22:11:02.731761+0000) 2023-12-16T22:14:11.051 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:14:10 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:14:10.652+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:14:11.314 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:14:11 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:14:11.049+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:11:07.849373+0000 (oldest deadline 2023-12-16T22:11:27.849373+0000) 2023-12-16T22:14:11.314 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:14:11 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:14:11.049+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:11:07.849373+0000 (oldest deadline 2023-12-16T22:11:27.849373+0000) 2023-12-16T22:14:11.315 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:14:11 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:14:11.049+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:11:07.849373+0000 (oldest deadline 2023-12-16T22:11:27.849373+0000) 2023-12-16T22:14:11.315 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:14:11 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:14:11.050+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:14:12.064 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:14:11 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:14:11.679+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:10:42.731761+0000 (oldest deadline 2023-12-16T22:11:02.731761+0000) 2023-12-16T22:14:12.064 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:14:11 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:14:11.679+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:10:42.731761+0000 (oldest deadline 2023-12-16T22:11:02.731761+0000) 2023-12-16T22:14:12.065 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:14:11 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:14:11.679+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:10:42.731761+0000 (oldest deadline 2023-12-16T22:11:02.731761+0000) 2023-12-16T22:14:12.065 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:14:11 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:14:11.680+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:14:12.065 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:14:11 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:14:11.859+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:10:43.026858+0000 (oldest deadline 2023-12-16T22:11:03.026858+0000) 2023-12-16T22:14:12.065 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:14:11 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:14:11.859+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:10:43.026858+0000 (oldest deadline 2023-12-16T22:11:03.026858+0000) 2023-12-16T22:14:12.065 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:14:11 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:14:11.859+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:10:43.026858+0000 (oldest deadline 2023-12-16T22:11:03.026858+0000) 2023-12-16T22:14:12.065 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:14:11 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:14:11.860+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:14:12.564 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Dec 16 22:14:12 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-mon-smithi045[80759]: 2023-12-16T22:14:12.136+0000 7f45a1f8f700 -1 mon.smithi045@0(probing) e2 get_health_metrics reporting 34761 slow ops, oldest is monmgrreport(gid 14214, 0 checks, 1 progress events) 2023-12-16T22:14:12.565 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:14:12 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:14:12.069+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:11:07.849373+0000 (oldest deadline 2023-12-16T22:11:27.849373+0000) 2023-12-16T22:14:12.565 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:14:12 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:14:12.069+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:11:07.849373+0000 (oldest deadline 2023-12-16T22:11:27.849373+0000) 2023-12-16T22:14:12.565 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:14:12 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:14:12.069+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:11:07.849373+0000 (oldest deadline 2023-12-16T22:11:27.849373+0000) 2023-12-16T22:14:12.565 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:14:12 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:14:12.070+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:14:13.064 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:14:12 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:14:12.641+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:10:42.731761+0000 (oldest deadline 2023-12-16T22:11:02.731761+0000) 2023-12-16T22:14:13.064 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:14:12 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:14:12.641+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:10:42.731761+0000 (oldest deadline 2023-12-16T22:11:02.731761+0000) 2023-12-16T22:14:13.065 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:14:12 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:14:12.641+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:10:42.731761+0000 (oldest deadline 2023-12-16T22:11:02.731761+0000) 2023-12-16T22:14:13.065 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:14:12 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:14:12.642+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:14:13.065 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:14:12 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:14:12.873+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:10:43.026858+0000 (oldest deadline 2023-12-16T22:11:03.026858+0000) 2023-12-16T22:14:13.065 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:14:12 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:14:12.873+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:10:43.026858+0000 (oldest deadline 2023-12-16T22:11:03.026858+0000) 2023-12-16T22:14:13.065 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:14:12 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:14:12.873+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:10:43.026858+0000 (oldest deadline 2023-12-16T22:11:03.026858+0000) 2023-12-16T22:14:13.065 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:14:12 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:14:12.874+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:14:13.521 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:14:13 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:14:13.097+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:11:07.849373+0000 (oldest deadline 2023-12-16T22:11:27.849373+0000) 2023-12-16T22:14:13.521 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:14:13 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:14:13.097+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:11:07.849373+0000 (oldest deadline 2023-12-16T22:11:27.849373+0000) 2023-12-16T22:14:13.521 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:14:13 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:14:13.097+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:11:07.849373+0000 (oldest deadline 2023-12-16T22:11:27.849373+0000) 2023-12-16T22:14:13.521 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:14:13 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:14:13.097+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:14:13.814 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:14:13 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:14:13.594+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:10:42.731761+0000 (oldest deadline 2023-12-16T22:11:02.731761+0000) 2023-12-16T22:14:13.815 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:14:13 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:14:13.594+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:10:42.731761+0000 (oldest deadline 2023-12-16T22:11:02.731761+0000) 2023-12-16T22:14:13.815 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:14:13 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:14:13.594+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:10:42.731761+0000 (oldest deadline 2023-12-16T22:11:02.731761+0000) 2023-12-16T22:14:13.815 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:14:13 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:14:13.596+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:14:14.142 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:14:13 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:14:13.846+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:10:43.026858+0000 (oldest deadline 2023-12-16T22:11:03.026858+0000) 2023-12-16T22:14:14.142 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:14:13 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:14:13.846+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:10:43.026858+0000 (oldest deadline 2023-12-16T22:11:03.026858+0000) 2023-12-16T22:14:14.142 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:14:13 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:14:13.846+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:10:43.026858+0000 (oldest deadline 2023-12-16T22:11:03.026858+0000) 2023-12-16T22:14:14.142 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:14:13 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:14:13.847+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:14:14.564 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:14:14 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:14:14.140+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:11:07.849373+0000 (oldest deadline 2023-12-16T22:11:27.849373+0000) 2023-12-16T22:14:14.564 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:14:14 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:14:14.140+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:11:07.849373+0000 (oldest deadline 2023-12-16T22:11:27.849373+0000) 2023-12-16T22:14:14.564 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:14:14 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:14:14.140+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:11:07.849373+0000 (oldest deadline 2023-12-16T22:11:27.849373+0000) 2023-12-16T22:14:14.565 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:14:14 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:14:14.140+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:14:15.064 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:14:14 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:14:14.876+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:10:43.026858+0000 (oldest deadline 2023-12-16T22:11:03.026858+0000) 2023-12-16T22:14:15.064 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:14:14 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:14:14.876+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:10:43.026858+0000 (oldest deadline 2023-12-16T22:11:03.026858+0000) 2023-12-16T22:14:15.065 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:14:14 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:14:14.876+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:10:43.026858+0000 (oldest deadline 2023-12-16T22:11:03.026858+0000) 2023-12-16T22:14:15.065 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:14:14 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:14:14.877+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:14:15.065 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:14:14 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:14:14.643+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:10:42.731761+0000 (oldest deadline 2023-12-16T22:11:02.731761+0000) 2023-12-16T22:14:15.065 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:14:14 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:14:14.643+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:10:42.731761+0000 (oldest deadline 2023-12-16T22:11:02.731761+0000) 2023-12-16T22:14:15.065 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:14:14 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:14:14.643+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:10:42.731761+0000 (oldest deadline 2023-12-16T22:11:02.731761+0000) 2023-12-16T22:14:15.065 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:14:14 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:14:14.645+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:14:15.564 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:14:15 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:14:15.152+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:11:07.849373+0000 (oldest deadline 2023-12-16T22:11:27.849373+0000) 2023-12-16T22:14:15.564 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:14:15 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:14:15.152+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:11:07.849373+0000 (oldest deadline 2023-12-16T22:11:27.849373+0000) 2023-12-16T22:14:15.564 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:14:15 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:14:15.152+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:11:07.849373+0000 (oldest deadline 2023-12-16T22:11:27.849373+0000) 2023-12-16T22:14:15.565 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:14:15 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:14:15.152+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:14:16.064 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:14:15 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:14:15.910+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:10:43.026858+0000 (oldest deadline 2023-12-16T22:11:03.026858+0000) 2023-12-16T22:14:16.064 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:14:15 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:14:15.910+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:10:43.026858+0000 (oldest deadline 2023-12-16T22:11:03.026858+0000) 2023-12-16T22:14:16.065 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:14:15 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:14:15.910+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:10:43.026858+0000 (oldest deadline 2023-12-16T22:11:03.026858+0000) 2023-12-16T22:14:16.065 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:14:15 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:14:15.911+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:14:16.065 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:14:15 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:14:15.666+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:10:42.731761+0000 (oldest deadline 2023-12-16T22:11:02.731761+0000) 2023-12-16T22:14:16.065 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:14:15 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:14:15.666+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:10:42.731761+0000 (oldest deadline 2023-12-16T22:11:02.731761+0000) 2023-12-16T22:14:16.065 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:14:15 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:14:15.666+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:10:42.731761+0000 (oldest deadline 2023-12-16T22:11:02.731761+0000) 2023-12-16T22:14:16.065 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:14:15 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:14:15.666+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:14:16.564 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:14:16 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:14:16.172+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:11:07.849373+0000 (oldest deadline 2023-12-16T22:11:27.849373+0000) 2023-12-16T22:14:16.565 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:14:16 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:14:16.172+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:11:07.849373+0000 (oldest deadline 2023-12-16T22:11:27.849373+0000) 2023-12-16T22:14:16.565 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:14:16 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:14:16.172+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:11:07.849373+0000 (oldest deadline 2023-12-16T22:11:27.849373+0000) 2023-12-16T22:14:16.565 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:14:16 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:14:16.172+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:14:16.904 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:14:16 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:14:16.629+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:10:42.731761+0000 (oldest deadline 2023-12-16T22:11:02.731761+0000) 2023-12-16T22:14:16.905 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:14:16 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:14:16.629+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:10:42.731761+0000 (oldest deadline 2023-12-16T22:11:02.731761+0000) 2023-12-16T22:14:16.905 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:14:16 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:14:16.629+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:10:42.731761+0000 (oldest deadline 2023-12-16T22:11:02.731761+0000) 2023-12-16T22:14:16.905 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:14:16 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:14:16.630+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:14:17.164 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Dec 16 22:14:17 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-mon-smithi045[80759]: 2023-12-16T22:14:17.140+0000 7f45a1f8f700 -1 mon.smithi045@0(probing) e2 get_health_metrics reporting 34852 slow ops, oldest is monmgrreport(gid 14214, 0 checks, 1 progress events) 2023-12-16T22:14:17.164 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:14:16 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:14:16.903+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:10:43.026858+0000 (oldest deadline 2023-12-16T22:11:03.026858+0000) 2023-12-16T22:14:17.165 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:14:16 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:14:16.903+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:10:43.026858+0000 (oldest deadline 2023-12-16T22:11:03.026858+0000) 2023-12-16T22:14:17.165 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:14:16 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:14:16.903+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:10:43.026858+0000 (oldest deadline 2023-12-16T22:11:03.026858+0000) 2023-12-16T22:14:17.165 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:14:16 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:14:16.905+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:14:17.564 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:14:17 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:14:17.183+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:11:07.849373+0000 (oldest deadline 2023-12-16T22:11:27.849373+0000) 2023-12-16T22:14:17.564 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:14:17 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:14:17.183+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:11:07.849373+0000 (oldest deadline 2023-12-16T22:11:27.849373+0000) 2023-12-16T22:14:17.564 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:14:17 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:14:17.183+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:11:07.849373+0000 (oldest deadline 2023-12-16T22:11:27.849373+0000) 2023-12-16T22:14:17.565 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:14:17 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:14:17.184+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:14:17.943 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:14:17 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:14:17.581+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:10:42.731761+0000 (oldest deadline 2023-12-16T22:11:02.731761+0000) 2023-12-16T22:14:17.943 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:14:17 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:14:17.581+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:10:42.731761+0000 (oldest deadline 2023-12-16T22:11:02.731761+0000) 2023-12-16T22:14:17.943 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:14:17 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:14:17.581+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:10:42.731761+0000 (oldest deadline 2023-12-16T22:11:02.731761+0000) 2023-12-16T22:14:17.943 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:14:17 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:14:17.582+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:14:18.314 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:14:17 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:14:17.942+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:10:43.026858+0000 (oldest deadline 2023-12-16T22:11:03.026858+0000) 2023-12-16T22:14:18.314 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:14:17 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:14:17.942+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:10:43.026858+0000 (oldest deadline 2023-12-16T22:11:03.026858+0000) 2023-12-16T22:14:18.315 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:14:17 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:14:17.942+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:10:43.026858+0000 (oldest deadline 2023-12-16T22:11:03.026858+0000) 2023-12-16T22:14:18.315 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:14:17 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:14:17.943+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:14:18.315 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:14:18 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:14:18.170+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:11:07.849373+0000 (oldest deadline 2023-12-16T22:11:27.849373+0000) 2023-12-16T22:14:18.315 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:14:18 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:14:18.170+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:11:07.849373+0000 (oldest deadline 2023-12-16T22:11:27.849373+0000) 2023-12-16T22:14:18.315 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:14:18 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:14:18.170+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:11:07.849373+0000 (oldest deadline 2023-12-16T22:11:27.849373+0000) 2023-12-16T22:14:18.316 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:14:18 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:14:18.171+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:14:18.904 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:14:18 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:14:18.601+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:10:42.731761+0000 (oldest deadline 2023-12-16T22:11:02.731761+0000) 2023-12-16T22:14:18.904 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:14:18 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:14:18.601+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:10:42.731761+0000 (oldest deadline 2023-12-16T22:11:02.731761+0000) 2023-12-16T22:14:18.904 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:14:18 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:14:18.601+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:10:42.731761+0000 (oldest deadline 2023-12-16T22:11:02.731761+0000) 2023-12-16T22:14:18.904 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:14:18 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:14:18.602+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:14:19.314 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:14:19 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:14:19.144+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:11:07.849373+0000 (oldest deadline 2023-12-16T22:11:27.849373+0000) 2023-12-16T22:14:19.314 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:14:19 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:14:19.144+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:11:07.849373+0000 (oldest deadline 2023-12-16T22:11:27.849373+0000) 2023-12-16T22:14:19.314 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:14:19 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:14:19.144+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:11:07.849373+0000 (oldest deadline 2023-12-16T22:11:27.849373+0000) 2023-12-16T22:14:19.315 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:14:19 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:14:19.145+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:14:19.315 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:14:18 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:14:18.903+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:10:43.026858+0000 (oldest deadline 2023-12-16T22:11:03.026858+0000) 2023-12-16T22:14:19.315 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:14:18 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:14:18.903+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:10:43.026858+0000 (oldest deadline 2023-12-16T22:11:03.026858+0000) 2023-12-16T22:14:19.315 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:14:18 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:14:18.903+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:10:43.026858+0000 (oldest deadline 2023-12-16T22:11:03.026858+0000) 2023-12-16T22:14:19.315 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:14:18 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:14:18.904+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:14:19.880 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:14:19 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:14:19.622+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:10:42.731761+0000 (oldest deadline 2023-12-16T22:11:02.731761+0000) 2023-12-16T22:14:19.881 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:14:19 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:14:19.622+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:10:42.731761+0000 (oldest deadline 2023-12-16T22:11:02.731761+0000) 2023-12-16T22:14:19.881 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:14:19 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:14:19.622+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:10:42.731761+0000 (oldest deadline 2023-12-16T22:11:02.731761+0000) 2023-12-16T22:14:19.881 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:14:19 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:14:19.622+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:14:20.135 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:14:19 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:14:19.879+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:10:43.026858+0000 (oldest deadline 2023-12-16T22:11:03.026858+0000) 2023-12-16T22:14:20.135 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:14:19 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:14:19.879+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:10:43.026858+0000 (oldest deadline 2023-12-16T22:11:03.026858+0000) 2023-12-16T22:14:20.135 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:14:19 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:14:19.879+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:10:43.026858+0000 (oldest deadline 2023-12-16T22:11:03.026858+0000) 2023-12-16T22:14:20.135 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:14:19 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:14:19.881+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:14:20.564 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:14:20 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:14:20.134+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:11:07.849373+0000 (oldest deadline 2023-12-16T22:11:27.849373+0000) 2023-12-16T22:14:20.564 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:14:20 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:14:20.134+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:11:07.849373+0000 (oldest deadline 2023-12-16T22:11:27.849373+0000) 2023-12-16T22:14:20.564 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:14:20 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:14:20.134+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:11:07.849373+0000 (oldest deadline 2023-12-16T22:11:27.849373+0000) 2023-12-16T22:14:20.565 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:14:20 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:14:20.135+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:14:20.918 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:14:20 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:14:20.615+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:10:42.731761+0000 (oldest deadline 2023-12-16T22:11:02.731761+0000) 2023-12-16T22:14:20.918 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:14:20 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:14:20.615+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:10:42.731761+0000 (oldest deadline 2023-12-16T22:11:02.731761+0000) 2023-12-16T22:14:20.918 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:14:20 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:14:20.615+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:10:42.731761+0000 (oldest deadline 2023-12-16T22:11:02.731761+0000) 2023-12-16T22:14:20.918 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:14:20 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:14:20.617+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:14:21.170 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi169.front.sepia.ceph.com' 2023-12-16T22:14:21.171 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi169.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-12-16T22:14:21.314 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:14:21 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:14:21.110+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:11:07.849373+0000 (oldest deadline 2023-12-16T22:11:27.849373+0000) 2023-12-16T22:14:21.314 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:14:21 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:14:21.110+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:11:07.849373+0000 (oldest deadline 2023-12-16T22:11:27.849373+0000) 2023-12-16T22:14:21.315 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:14:21 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:14:21.110+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:11:07.849373+0000 (oldest deadline 2023-12-16T22:11:27.849373+0000) 2023-12-16T22:14:21.315 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:14:21 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:14:21.111+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:14:21.315 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:14:20 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:14:20.915+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:10:43.026858+0000 (oldest deadline 2023-12-16T22:11:03.026858+0000) 2023-12-16T22:14:21.315 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:14:20 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:14:20.915+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:10:43.026858+0000 (oldest deadline 2023-12-16T22:11:03.026858+0000) 2023-12-16T22:14:21.315 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:14:20 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:14:20.915+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:10:43.026858+0000 (oldest deadline 2023-12-16T22:11:03.026858+0000) 2023-12-16T22:14:21.315 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:14:20 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:14:20.917+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:14:21.905 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:14:21 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:14:21.570+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:10:42.731761+0000 (oldest deadline 2023-12-16T22:11:02.731761+0000) 2023-12-16T22:14:21.905 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:14:21 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:14:21.570+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:10:42.731761+0000 (oldest deadline 2023-12-16T22:11:02.731761+0000) 2023-12-16T22:14:21.905 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:14:21 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:14:21.570+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:10:42.731761+0000 (oldest deadline 2023-12-16T22:11:02.731761+0000) 2023-12-16T22:14:21.906 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:14:21 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:14:21.571+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:14:22.155 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Dec 16 22:14:22 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-mon-smithi045[80759]: 2023-12-16T22:14:22.145+0000 7f45a1f8f700 -1 mon.smithi045@0(probing) e2 get_health_metrics reporting 34941 slow ops, oldest is monmgrreport(gid 14214, 0 checks, 1 progress events) 2023-12-16T22:14:22.156 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:14:21 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:14:21.904+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:10:43.026858+0000 (oldest deadline 2023-12-16T22:11:03.026858+0000) 2023-12-16T22:14:22.156 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:14:21 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:14:21.904+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:10:43.026858+0000 (oldest deadline 2023-12-16T22:11:03.026858+0000) 2023-12-16T22:14:22.156 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:14:21 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:14:21.904+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:10:43.026858+0000 (oldest deadline 2023-12-16T22:11:03.026858+0000) 2023-12-16T22:14:22.156 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:14:21 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:14:21.905+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:14:22.156 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:14:22 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:14:22.084+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:11:07.849373+0000 (oldest deadline 2023-12-16T22:11:27.849373+0000) 2023-12-16T22:14:22.157 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:14:22 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:14:22.084+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:11:07.849373+0000 (oldest deadline 2023-12-16T22:11:27.849373+0000) 2023-12-16T22:14:22.157 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:14:22 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:14:22.084+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:11:07.849373+0000 (oldest deadline 2023-12-16T22:11:27.849373+0000) 2023-12-16T22:14:22.157 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:14:22 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:14:22.084+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:14:22.814 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:14:22 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:14:22.539+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:10:42.731761+0000 (oldest deadline 2023-12-16T22:11:02.731761+0000) 2023-12-16T22:14:22.815 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:14:22 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:14:22.539+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:10:42.731761+0000 (oldest deadline 2023-12-16T22:11:02.731761+0000) 2023-12-16T22:14:22.815 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:14:22 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:14:22.539+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:10:42.731761+0000 (oldest deadline 2023-12-16T22:11:02.731761+0000) 2023-12-16T22:14:22.816 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:14:22 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:14:22.540+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:14:23.315 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:14:23 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:14:23.082+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:11:07.849373+0000 (oldest deadline 2023-12-16T22:11:27.849373+0000) 2023-12-16T22:14:23.315 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:14:23 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:14:23.082+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:11:07.849373+0000 (oldest deadline 2023-12-16T22:11:27.849373+0000) 2023-12-16T22:14:23.315 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:14:23 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:14:23.082+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:11:07.849373+0000 (oldest deadline 2023-12-16T22:11:27.849373+0000) 2023-12-16T22:14:23.315 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:14:23 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:14:23.082+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:14:23.315 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:14:22 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:14:22.908+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:10:43.026858+0000 (oldest deadline 2023-12-16T22:11:03.026858+0000) 2023-12-16T22:14:23.316 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:14:22 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:14:22.908+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:10:43.026858+0000 (oldest deadline 2023-12-16T22:11:03.026858+0000) 2023-12-16T22:14:23.316 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:14:22 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:14:22.908+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:10:43.026858+0000 (oldest deadline 2023-12-16T22:11:03.026858+0000) 2023-12-16T22:14:23.316 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:14:22 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:14:22.909+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:14:23.814 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:14:23 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:14:23.552+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:10:42.731761+0000 (oldest deadline 2023-12-16T22:11:02.731761+0000) 2023-12-16T22:14:23.814 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:14:23 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:14:23.552+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:10:42.731761+0000 (oldest deadline 2023-12-16T22:11:02.731761+0000) 2023-12-16T22:14:23.815 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:14:23 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:14:23.552+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:10:42.731761+0000 (oldest deadline 2023-12-16T22:11:02.731761+0000) 2023-12-16T22:14:23.815 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:14:23 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:14:23.553+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:14:24.245 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.169 2023-12-16T22:14:24.314 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:14:24 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:14:24.097+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:11:07.849373+0000 (oldest deadline 2023-12-16T22:11:27.849373+0000) 2023-12-16T22:14:24.314 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:14:24 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:14:24.097+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:11:07.849373+0000 (oldest deadline 2023-12-16T22:11:27.849373+0000) 2023-12-16T22:14:24.315 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:14:24 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:14:24.097+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:11:07.849373+0000 (oldest deadline 2023-12-16T22:11:27.849373+0000) 2023-12-16T22:14:24.315 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:14:24 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:14:24.097+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:14:24.315 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:14:23 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:14:23.883+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:10:43.026858+0000 (oldest deadline 2023-12-16T22:11:03.026858+0000) 2023-12-16T22:14:24.315 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:14:23 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:14:23.883+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:10:43.026858+0000 (oldest deadline 2023-12-16T22:11:03.026858+0000) 2023-12-16T22:14:24.315 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:14:23 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:14:23.883+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:10:43.026858+0000 (oldest deadline 2023-12-16T22:11:03.026858+0000) 2023-12-16T22:14:24.315 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:14:23 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:14:23.884+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:14:24.891 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:14:24 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:14:24.590+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:10:42.731761+0000 (oldest deadline 2023-12-16T22:11:02.731761+0000) 2023-12-16T22:14:24.891 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:14:24 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:14:24.590+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:10:42.731761+0000 (oldest deadline 2023-12-16T22:11:02.731761+0000) 2023-12-16T22:14:24.891 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:14:24 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:14:24.590+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:10:42.731761+0000 (oldest deadline 2023-12-16T22:11:02.731761+0000) 2023-12-16T22:14:24.892 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:14:24 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:14:24.591+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:14:25.314 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:14:24 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:14:24.890+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:10:43.026858+0000 (oldest deadline 2023-12-16T22:11:03.026858+0000) 2023-12-16T22:14:25.315 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:14:24 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:14:24.890+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:10:43.026858+0000 (oldest deadline 2023-12-16T22:11:03.026858+0000) 2023-12-16T22:14:25.315 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:14:24 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:14:24.890+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:10:43.026858+0000 (oldest deadline 2023-12-16T22:11:03.026858+0000) 2023-12-16T22:14:25.315 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:14:24 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:14:24.891+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:14:25.315 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:14:25 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:14:25.134+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:11:07.849373+0000 (oldest deadline 2023-12-16T22:11:27.849373+0000) 2023-12-16T22:14:25.315 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:14:25 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:14:25.134+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:11:07.849373+0000 (oldest deadline 2023-12-16T22:11:27.849373+0000) 2023-12-16T22:14:25.316 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:14:25 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:14:25.134+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:11:07.849373+0000 (oldest deadline 2023-12-16T22:11:27.849373+0000) 2023-12-16T22:14:25.316 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:14:25 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:14:25.135+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:14:25.906 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:14:25 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:14:25.580+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:10:42.731761+0000 (oldest deadline 2023-12-16T22:11:02.731761+0000) 2023-12-16T22:14:25.906 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:14:25 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:14:25.580+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:10:42.731761+0000 (oldest deadline 2023-12-16T22:11:02.731761+0000) 2023-12-16T22:14:25.906 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:14:25 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:14:25.580+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:10:42.731761+0000 (oldest deadline 2023-12-16T22:11:02.731761+0000) 2023-12-16T22:14:25.907 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:14:25 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:14:25.581+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:14:26.314 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:14:26 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:14:26.114+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:11:07.849373+0000 (oldest deadline 2023-12-16T22:11:27.849373+0000) 2023-12-16T22:14:26.314 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:14:26 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:14:26.114+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:11:07.849373+0000 (oldest deadline 2023-12-16T22:11:27.849373+0000) 2023-12-16T22:14:26.315 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:14:26 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:14:26.114+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:11:07.849373+0000 (oldest deadline 2023-12-16T22:11:27.849373+0000) 2023-12-16T22:14:26.315 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:14:26 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:14:26.115+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:14:26.315 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:14:25 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:14:25.904+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:10:43.026858+0000 (oldest deadline 2023-12-16T22:11:03.026858+0000) 2023-12-16T22:14:26.315 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:14:25 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:14:25.904+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:10:43.026858+0000 (oldest deadline 2023-12-16T22:11:03.026858+0000) 2023-12-16T22:14:26.315 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:14:25 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:14:25.905+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:10:43.026858+0000 (oldest deadline 2023-12-16T22:11:03.026858+0000) 2023-12-16T22:14:26.315 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:14:25 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:14:25.905+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:14:26.939 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:14:26 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:14:26.568+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:10:42.731761+0000 (oldest deadline 2023-12-16T22:11:02.731761+0000) 2023-12-16T22:14:26.939 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:14:26 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:14:26.568+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:10:42.731761+0000 (oldest deadline 2023-12-16T22:11:02.731761+0000) 2023-12-16T22:14:26.939 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:14:26 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:14:26.568+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:10:42.731761+0000 (oldest deadline 2023-12-16T22:11:02.731761+0000) 2023-12-16T22:14:26.940 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:14:26 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:14:26.569+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:14:27.246 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi169.front.sepia.ceph.com' 2023-12-16T22:14:27.247 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi169.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-12-16T22:14:27.277 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:14:26 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:14:26.937+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:10:43.026858+0000 (oldest deadline 2023-12-16T22:11:03.026858+0000) 2023-12-16T22:14:27.277 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:14:26 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:14:26.937+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:10:43.026858+0000 (oldest deadline 2023-12-16T22:11:03.026858+0000) 2023-12-16T22:14:27.277 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:14:26 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:14:26.937+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:10:43.026858+0000 (oldest deadline 2023-12-16T22:11:03.026858+0000) 2023-12-16T22:14:27.278 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:14:26 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:14:26.938+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:14:27.278 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:14:27 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:14:27.111+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:11:07.849373+0000 (oldest deadline 2023-12-16T22:11:27.849373+0000) 2023-12-16T22:14:27.278 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:14:27 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:14:27.111+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:11:07.849373+0000 (oldest deadline 2023-12-16T22:11:27.849373+0000) 2023-12-16T22:14:27.278 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:14:27 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:14:27.111+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:11:07.849373+0000 (oldest deadline 2023-12-16T22:11:27.849373+0000) 2023-12-16T22:14:27.279 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:14:27 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:14:27.113+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:14:27.279 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Dec 16 22:14:27 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-mon-smithi045[80759]: 2023-12-16T22:14:27.149+0000 7f45a1f8f700 -1 mon.smithi045@0(probing) e2 get_health_metrics reporting 35032 slow ops, oldest is monmgrreport(gid 14214, 0 checks, 1 progress events) 2023-12-16T22:14:27.313 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.169 2023-12-16T22:14:27.314 WARNING:teuthology.contextutil:'reconnect to {self.shortname}' reached maximum tries (5) after waiting for 30 seconds 2023-12-16T22:14:27.314 INFO:tasks.cephadm:Cleaning up testdir ceph.* files... 2023-12-16T22:14:27.314 DEBUG:teuthology.orchestra.run.smithi045:> rm -f /home/ubuntu/cephtest/seed.ceph.conf /home/ubuntu/cephtest/ceph.pub 2023-12-16T22:14:27.331 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi169.front.sepia.ceph.com' 2023-12-16T22:14:27.332 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi169.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-12-16T22:14:27.814 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:14:27 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:14:27.528+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:10:42.731761+0000 (oldest deadline 2023-12-16T22:11:02.731761+0000) 2023-12-16T22:14:27.815 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:14:27 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:14:27.528+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:10:42.731761+0000 (oldest deadline 2023-12-16T22:11:02.731761+0000) 2023-12-16T22:14:27.815 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:14:27 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:14:27.528+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:10:42.731761+0000 (oldest deadline 2023-12-16T22:11:02.731761+0000) 2023-12-16T22:14:27.815 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:14:27 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:14:27.529+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:14:28.314 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:14:27 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:14:27.922+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:10:43.026858+0000 (oldest deadline 2023-12-16T22:11:03.026858+0000) 2023-12-16T22:14:28.315 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:14:27 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:14:27.922+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:10:43.026858+0000 (oldest deadline 2023-12-16T22:11:03.026858+0000) 2023-12-16T22:14:28.315 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:14:27 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:14:27.922+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:10:43.026858+0000 (oldest deadline 2023-12-16T22:11:03.026858+0000) 2023-12-16T22:14:28.315 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:14:27 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:14:27.923+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:14:28.315 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:14:28 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:14:28.106+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:11:07.849373+0000 (oldest deadline 2023-12-16T22:11:27.849373+0000) 2023-12-16T22:14:28.316 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:14:28 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:14:28.106+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:11:07.849373+0000 (oldest deadline 2023-12-16T22:11:27.849373+0000) 2023-12-16T22:14:28.316 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:14:28 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:14:28.106+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:11:07.849373+0000 (oldest deadline 2023-12-16T22:11:27.849373+0000) 2023-12-16T22:14:28.316 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:14:28 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:14:28.107+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:14:28.931 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:14:28 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:14:28.571+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:10:42.731761+0000 (oldest deadline 2023-12-16T22:11:02.731761+0000) 2023-12-16T22:14:28.932 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:14:28 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:14:28.571+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:10:42.731761+0000 (oldest deadline 2023-12-16T22:11:02.731761+0000) 2023-12-16T22:14:28.932 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:14:28 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:14:28.571+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:10:42.731761+0000 (oldest deadline 2023-12-16T22:11:02.731761+0000) 2023-12-16T22:14:28.932 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:14:28 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:14:28.572+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:14:29.314 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:14:29 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:14:29.087+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:11:07.849373+0000 (oldest deadline 2023-12-16T22:11:27.849373+0000) 2023-12-16T22:14:29.314 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:14:29 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:14:29.087+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:11:07.849373+0000 (oldest deadline 2023-12-16T22:11:27.849373+0000) 2023-12-16T22:14:29.315 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:14:29 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:14:29.087+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:11:07.849373+0000 (oldest deadline 2023-12-16T22:11:27.849373+0000) 2023-12-16T22:14:29.315 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:14:29 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:14:29.088+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:14:29.315 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:14:28 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:14:28.930+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:10:43.026858+0000 (oldest deadline 2023-12-16T22:11:03.026858+0000) 2023-12-16T22:14:29.315 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:14:28 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:14:28.930+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:10:43.026858+0000 (oldest deadline 2023-12-16T22:11:03.026858+0000) 2023-12-16T22:14:29.315 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:14:28 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:14:28.930+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:10:43.026858+0000 (oldest deadline 2023-12-16T22:11:03.026858+0000) 2023-12-16T22:14:29.315 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:14:28 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:14:28.931+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:14:29.814 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:14:29 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:14:29.554+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:10:42.731761+0000 (oldest deadline 2023-12-16T22:11:02.731761+0000) 2023-12-16T22:14:29.814 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:14:29 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:14:29.554+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:10:42.731761+0000 (oldest deadline 2023-12-16T22:11:02.731761+0000) 2023-12-16T22:14:29.814 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:14:29 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:14:29.554+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:10:42.731761+0000 (oldest deadline 2023-12-16T22:11:02.731761+0000) 2023-12-16T22:14:29.815 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:14:29 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:14:29.555+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:14:30.229 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:14:30 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:14:30.095+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:11:07.849373+0000 (oldest deadline 2023-12-16T22:11:27.849373+0000) 2023-12-16T22:14:30.229 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:14:30 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:14:30.095+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:11:07.849373+0000 (oldest deadline 2023-12-16T22:11:27.849373+0000) 2023-12-16T22:14:30.229 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:14:30 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:14:30.095+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:11:07.849373+0000 (oldest deadline 2023-12-16T22:11:27.849373+0000) 2023-12-16T22:14:30.229 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:14:30 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:14:30.096+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:14:30.230 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:14:29 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:14:29.977+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:10:43.026858+0000 (oldest deadline 2023-12-16T22:11:03.026858+0000) 2023-12-16T22:14:30.230 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:14:29 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:14:29.977+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:10:43.026858+0000 (oldest deadline 2023-12-16T22:11:03.026858+0000) 2023-12-16T22:14:30.230 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:14:29 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:14:29.977+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:10:43.026858+0000 (oldest deadline 2023-12-16T22:11:03.026858+0000) 2023-12-16T22:14:30.230 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:14:29 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:14:29.978+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:14:30.389 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.169 2023-12-16T22:14:30.975 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:14:30 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:14:30.605+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:10:42.731761+0000 (oldest deadline 2023-12-16T22:11:02.731761+0000) 2023-12-16T22:14:30.975 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:14:30 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:14:30.605+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:10:42.731761+0000 (oldest deadline 2023-12-16T22:11:02.731761+0000) 2023-12-16T22:14:30.975 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:14:30 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:14:30.605+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:10:42.731761+0000 (oldest deadline 2023-12-16T22:11:02.731761+0000) 2023-12-16T22:14:30.976 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:14:30 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:14:30.606+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:14:31.314 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:14:31 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:14:31.075+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:11:07.849373+0000 (oldest deadline 2023-12-16T22:11:27.849373+0000) 2023-12-16T22:14:31.315 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:14:31 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:14:31.075+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:11:07.849373+0000 (oldest deadline 2023-12-16T22:11:27.849373+0000) 2023-12-16T22:14:31.315 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:14:31 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:14:31.075+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:11:07.849373+0000 (oldest deadline 2023-12-16T22:11:27.849373+0000) 2023-12-16T22:14:31.315 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:14:31 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:14:31.076+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:14:31.315 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:14:30 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:14:30.974+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:10:43.026858+0000 (oldest deadline 2023-12-16T22:11:03.026858+0000) 2023-12-16T22:14:31.315 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:14:30 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:14:30.974+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:10:43.026858+0000 (oldest deadline 2023-12-16T22:11:03.026858+0000) 2023-12-16T22:14:31.315 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:14:30 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:14:30.974+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:10:43.026858+0000 (oldest deadline 2023-12-16T22:11:03.026858+0000) 2023-12-16T22:14:31.316 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:14:30 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:14:30.975+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:14:31.951 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:14:31 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:14:31.596+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:10:42.731761+0000 (oldest deadline 2023-12-16T22:11:02.731761+0000) 2023-12-16T22:14:31.951 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:14:31 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:14:31.596+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:10:42.731761+0000 (oldest deadline 2023-12-16T22:11:02.731761+0000) 2023-12-16T22:14:31.951 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:14:31 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:14:31.596+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:10:42.731761+0000 (oldest deadline 2023-12-16T22:11:02.731761+0000) 2023-12-16T22:14:31.951 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:14:31 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:14:31.597+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:14:32.276 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Dec 16 22:14:32 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-mon-smithi045[80759]: 2023-12-16T22:14:32.153+0000 7f45a1f8f700 -1 mon.smithi045@0(probing) e2 get_health_metrics reporting 35122 slow ops, oldest is monmgrreport(gid 14214, 0 checks, 1 progress events) 2023-12-16T22:14:32.277 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:14:31 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:14:31.957+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:10:43.026858+0000 (oldest deadline 2023-12-16T22:11:03.026858+0000) 2023-12-16T22:14:32.277 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:14:31 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:14:31.958+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:10:43.026858+0000 (oldest deadline 2023-12-16T22:11:03.026858+0000) 2023-12-16T22:14:32.277 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:14:31 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:14:31.958+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:10:43.026858+0000 (oldest deadline 2023-12-16T22:11:03.026858+0000) 2023-12-16T22:14:32.277 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:14:31 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:14:31.958+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:14:32.278 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:14:32 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:14:32.096+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:11:07.849373+0000 (oldest deadline 2023-12-16T22:11:27.849373+0000) 2023-12-16T22:14:32.278 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:14:32 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:14:32.096+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:11:07.849373+0000 (oldest deadline 2023-12-16T22:11:27.849373+0000) 2023-12-16T22:14:32.278 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:14:32 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:14:32.096+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:11:07.849373+0000 (oldest deadline 2023-12-16T22:11:27.849373+0000) 2023-12-16T22:14:32.278 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:14:32 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:14:32.097+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:14:32.815 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:14:32 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:14:32.556+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:10:42.731761+0000 (oldest deadline 2023-12-16T22:11:02.731761+0000) 2023-12-16T22:14:32.815 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:14:32 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:14:32.556+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:10:42.731761+0000 (oldest deadline 2023-12-16T22:11:02.731761+0000) 2023-12-16T22:14:32.815 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:14:32 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:14:32.556+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:10:42.731761+0000 (oldest deadline 2023-12-16T22:11:02.731761+0000) 2023-12-16T22:14:32.815 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:14:32 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:14:32.558+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:14:33.314 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:14:32 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:14:32.914+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:10:43.026858+0000 (oldest deadline 2023-12-16T22:11:03.026858+0000) 2023-12-16T22:14:33.315 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:14:32 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:14:32.914+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:10:43.026858+0000 (oldest deadline 2023-12-16T22:11:03.026858+0000) 2023-12-16T22:14:33.315 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:14:32 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:14:32.914+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:10:43.026858+0000 (oldest deadline 2023-12-16T22:11:03.026858+0000) 2023-12-16T22:14:33.315 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:14:32 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:14:32.914+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:14:33.316 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:14:33 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:14:33.053+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:11:07.849373+0000 (oldest deadline 2023-12-16T22:11:27.849373+0000) 2023-12-16T22:14:33.316 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:14:33 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:14:33.053+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:11:07.849373+0000 (oldest deadline 2023-12-16T22:11:27.849373+0000) 2023-12-16T22:14:33.316 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:14:33 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:14:33.053+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:11:07.849373+0000 (oldest deadline 2023-12-16T22:11:27.849373+0000) 2023-12-16T22:14:33.316 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:14:33 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:14:33.054+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:14:33.894 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:14:33 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:14:33.577+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:10:42.731761+0000 (oldest deadline 2023-12-16T22:11:02.731761+0000) 2023-12-16T22:14:33.894 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:14:33 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:14:33.577+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:10:42.731761+0000 (oldest deadline 2023-12-16T22:11:02.731761+0000) 2023-12-16T22:14:33.894 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:14:33 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:14:33.577+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:10:42.731761+0000 (oldest deadline 2023-12-16T22:11:02.731761+0000) 2023-12-16T22:14:33.895 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:14:33 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:14:33.578+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:14:34.314 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:14:33 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:14:33.893+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:10:43.026858+0000 (oldest deadline 2023-12-16T22:11:03.026858+0000) 2023-12-16T22:14:34.314 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:14:33 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:14:33.893+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:10:43.026858+0000 (oldest deadline 2023-12-16T22:11:03.026858+0000) 2023-12-16T22:14:34.315 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:14:33 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:14:33.893+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:10:43.026858+0000 (oldest deadline 2023-12-16T22:11:03.026858+0000) 2023-12-16T22:14:34.315 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:14:33 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:14:33.894+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:14:34.315 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:14:34 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:14:34.029+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:11:07.849373+0000 (oldest deadline 2023-12-16T22:11:27.849373+0000) 2023-12-16T22:14:34.315 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:14:34 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:14:34.029+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:11:07.849373+0000 (oldest deadline 2023-12-16T22:11:27.849373+0000) 2023-12-16T22:14:34.315 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:14:34 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:14:34.029+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:11:07.849373+0000 (oldest deadline 2023-12-16T22:11:27.849373+0000) 2023-12-16T22:14:34.315 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:14:34 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:14:34.030+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:14:34.814 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:14:34 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:14:34.549+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:10:42.731761+0000 (oldest deadline 2023-12-16T22:11:02.731761+0000) 2023-12-16T22:14:34.814 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:14:34 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:14:34.549+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:10:42.731761+0000 (oldest deadline 2023-12-16T22:11:02.731761+0000) 2023-12-16T22:14:34.815 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:14:34 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:14:34.549+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:10:42.731761+0000 (oldest deadline 2023-12-16T22:11:02.731761+0000) 2023-12-16T22:14:34.815 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:14:34 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:14:34.550+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:14:35.314 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:14:34 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:14:34.866+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:10:43.026858+0000 (oldest deadline 2023-12-16T22:11:03.026858+0000) 2023-12-16T22:14:35.314 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:14:34 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:14:34.866+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:10:43.026858+0000 (oldest deadline 2023-12-16T22:11:03.026858+0000) 2023-12-16T22:14:35.315 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:14:34 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:14:34.866+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:10:43.026858+0000 (oldest deadline 2023-12-16T22:11:03.026858+0000) 2023-12-16T22:14:35.315 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:14:34 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:14:34.867+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:14:35.315 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:14:35 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:14:35.017+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:11:07.849373+0000 (oldest deadline 2023-12-16T22:11:27.849373+0000) 2023-12-16T22:14:35.315 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:14:35 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:14:35.017+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:11:07.849373+0000 (oldest deadline 2023-12-16T22:11:27.849373+0000) 2023-12-16T22:14:35.315 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:14:35 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:14:35.017+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:11:07.849373+0000 (oldest deadline 2023-12-16T22:11:27.849373+0000) 2023-12-16T22:14:35.315 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:14:35 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:14:35.018+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:14:35.814 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:14:35 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:14:35.548+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:10:42.731761+0000 (oldest deadline 2023-12-16T22:11:02.731761+0000) 2023-12-16T22:14:35.814 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:14:35 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:14:35.548+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:10:42.731761+0000 (oldest deadline 2023-12-16T22:11:02.731761+0000) 2023-12-16T22:14:35.814 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:14:35 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:14:35.548+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:10:42.731761+0000 (oldest deadline 2023-12-16T22:11:02.731761+0000) 2023-12-16T22:14:35.814 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:14:35 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:14:35.550+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:14:36.314 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:14:36 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:14:36.004+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:11:07.849373+0000 (oldest deadline 2023-12-16T22:11:27.849373+0000) 2023-12-16T22:14:36.314 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:14:36 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:14:36.004+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:11:07.849373+0000 (oldest deadline 2023-12-16T22:11:27.849373+0000) 2023-12-16T22:14:36.315 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:14:36 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:14:36.004+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:11:07.849373+0000 (oldest deadline 2023-12-16T22:11:27.849373+0000) 2023-12-16T22:14:36.315 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:14:36 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:14:36.005+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:14:36.315 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:14:35 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:14:35.908+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:10:43.026858+0000 (oldest deadline 2023-12-16T22:11:03.026858+0000) 2023-12-16T22:14:36.315 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:14:35 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:14:35.908+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:10:43.026858+0000 (oldest deadline 2023-12-16T22:11:03.026858+0000) 2023-12-16T22:14:36.315 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:14:35 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:14:35.908+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:10:43.026858+0000 (oldest deadline 2023-12-16T22:11:03.026858+0000) 2023-12-16T22:14:36.315 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:14:35 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:14:35.910+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:14:36.936 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:14:36 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:14:36.598+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:10:42.731761+0000 (oldest deadline 2023-12-16T22:11:02.731761+0000) 2023-12-16T22:14:36.937 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:14:36 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:14:36.598+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:10:42.731761+0000 (oldest deadline 2023-12-16T22:11:02.731761+0000) 2023-12-16T22:14:36.937 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:14:36 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:14:36.598+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:10:42.731761+0000 (oldest deadline 2023-12-16T22:11:02.731761+0000) 2023-12-16T22:14:36.937 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:14:36 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:14:36.598+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:14:37.284 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Dec 16 22:14:37 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-mon-smithi045[80759]: 2023-12-16T22:14:37.158+0000 7f45a1f8f700 -1 mon.smithi045@0(probing) e2 get_health_metrics reporting 35214 slow ops, oldest is monmgrreport(gid 14214, 0 checks, 1 progress events) 2023-12-16T22:14:37.285 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:14:36 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:14:36.935+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:10:43.026858+0000 (oldest deadline 2023-12-16T22:11:03.026858+0000) 2023-12-16T22:14:37.285 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:14:36 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:14:36.935+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:10:43.026858+0000 (oldest deadline 2023-12-16T22:11:03.026858+0000) 2023-12-16T22:14:37.285 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:14:36 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:14:36.935+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:10:43.026858+0000 (oldest deadline 2023-12-16T22:11:03.026858+0000) 2023-12-16T22:14:37.285 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:14:36 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:14:36.936+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:14:37.286 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:14:36 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:14:36.983+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:11:07.849373+0000 (oldest deadline 2023-12-16T22:11:27.849373+0000) 2023-12-16T22:14:37.286 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:14:36 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:14:36.983+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:11:07.849373+0000 (oldest deadline 2023-12-16T22:11:27.849373+0000) 2023-12-16T22:14:37.286 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:14:36 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:14:36.983+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:11:07.849373+0000 (oldest deadline 2023-12-16T22:11:27.849373+0000) 2023-12-16T22:14:37.286 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:14:36 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:14:36.984+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:14:37.889 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:14:37 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:14:37.632+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:10:42.731761+0000 (oldest deadline 2023-12-16T22:11:02.731761+0000) 2023-12-16T22:14:37.889 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:14:37 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:14:37.632+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:10:42.731761+0000 (oldest deadline 2023-12-16T22:11:02.731761+0000) 2023-12-16T22:14:37.890 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:14:37 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:14:37.632+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:10:42.731761+0000 (oldest deadline 2023-12-16T22:11:02.731761+0000) 2023-12-16T22:14:37.890 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:14:37 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:14:37.633+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:14:38.314 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:14:37 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:14:37.953+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:11:07.849373+0000 (oldest deadline 2023-12-16T22:11:27.849373+0000) 2023-12-16T22:14:38.314 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:14:37 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:14:37.953+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:11:07.849373+0000 (oldest deadline 2023-12-16T22:11:27.849373+0000) 2023-12-16T22:14:38.315 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:14:37 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:14:37.953+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:11:07.849373+0000 (oldest deadline 2023-12-16T22:11:27.849373+0000) 2023-12-16T22:14:38.315 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:14:37 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:14:37.954+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:14:38.315 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:14:37 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:14:37.887+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:10:43.026858+0000 (oldest deadline 2023-12-16T22:11:03.026858+0000) 2023-12-16T22:14:38.315 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:14:37 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:14:37.887+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:10:43.026858+0000 (oldest deadline 2023-12-16T22:11:03.026858+0000) 2023-12-16T22:14:38.315 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:14:37 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:14:37.887+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:10:43.026858+0000 (oldest deadline 2023-12-16T22:11:03.026858+0000) 2023-12-16T22:14:38.315 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:14:37 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:14:37.888+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:14:38.993 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:14:38 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:14:38.676+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:10:42.731761+0000 (oldest deadline 2023-12-16T22:11:02.731761+0000) 2023-12-16T22:14:38.993 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:14:38 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:14:38.676+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:10:42.731761+0000 (oldest deadline 2023-12-16T22:11:02.731761+0000) 2023-12-16T22:14:38.993 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:14:38 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:14:38.676+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:10:42.731761+0000 (oldest deadline 2023-12-16T22:11:02.731761+0000) 2023-12-16T22:14:38.994 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:14:38 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:14:38.677+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:14:38.994 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:14:38 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:14:38.842+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:10:43.026858+0000 (oldest deadline 2023-12-16T22:11:03.026858+0000) 2023-12-16T22:14:38.994 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:14:38 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:14:38.842+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:10:43.026858+0000 (oldest deadline 2023-12-16T22:11:03.026858+0000) 2023-12-16T22:14:38.994 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:14:38 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:14:38.842+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:10:43.026858+0000 (oldest deadline 2023-12-16T22:11:03.026858+0000) 2023-12-16T22:14:38.994 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:14:38 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:14:38.843+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:14:39.314 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:14:38 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:14:38.992+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:11:07.849373+0000 (oldest deadline 2023-12-16T22:11:27.849373+0000) 2023-12-16T22:14:39.314 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:14:38 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:14:38.992+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:11:07.849373+0000 (oldest deadline 2023-12-16T22:11:27.849373+0000) 2023-12-16T22:14:39.314 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:14:38 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:14:38.992+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:11:07.849373+0000 (oldest deadline 2023-12-16T22:11:27.849373+0000) 2023-12-16T22:14:39.315 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:14:38 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:14:38.993+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:14:39.390 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi169.front.sepia.ceph.com' 2023-12-16T22:14:39.391 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi169.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-12-16T22:14:40.064 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:14:39 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:14:39.821+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:10:43.026858+0000 (oldest deadline 2023-12-16T22:11:03.026858+0000) 2023-12-16T22:14:40.065 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:14:39 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:14:39.821+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:10:43.026858+0000 (oldest deadline 2023-12-16T22:11:03.026858+0000) 2023-12-16T22:14:40.065 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:14:39 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:14:39.821+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:10:43.026858+0000 (oldest deadline 2023-12-16T22:11:03.026858+0000) 2023-12-16T22:14:40.065 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:14:39 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:14:39.822+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:14:40.065 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:14:39 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:14:39.945+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:11:07.849373+0000 (oldest deadline 2023-12-16T22:11:27.849373+0000) 2023-12-16T22:14:40.065 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:14:39 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:14:39.945+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:11:07.849373+0000 (oldest deadline 2023-12-16T22:11:27.849373+0000) 2023-12-16T22:14:40.066 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:14:39 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:14:39.945+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:11:07.849373+0000 (oldest deadline 2023-12-16T22:11:27.849373+0000) 2023-12-16T22:14:40.066 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:14:39 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:14:39.946+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:14:40.066 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:14:39 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:14:39.725+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:10:42.731761+0000 (oldest deadline 2023-12-16T22:11:02.731761+0000) 2023-12-16T22:14:40.066 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:14:39 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:14:39.725+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:10:42.731761+0000 (oldest deadline 2023-12-16T22:11:02.731761+0000) 2023-12-16T22:14:40.066 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:14:39 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:14:39.725+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:10:42.731761+0000 (oldest deadline 2023-12-16T22:11:02.731761+0000) 2023-12-16T22:14:40.066 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:14:39 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:14:39.726+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:14:41.064 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:14:40 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:14:40.944+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:11:07.849373+0000 (oldest deadline 2023-12-16T22:11:27.849373+0000) 2023-12-16T22:14:41.065 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:14:40 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:14:40.944+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:11:07.849373+0000 (oldest deadline 2023-12-16T22:11:27.849373+0000) 2023-12-16T22:14:41.065 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:14:40 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:14:40.944+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:11:07.849373+0000 (oldest deadline 2023-12-16T22:11:27.849373+0000) 2023-12-16T22:14:41.065 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:14:40 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:14:40.945+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:14:41.065 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:14:40 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:14:40.831+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:10:43.026858+0000 (oldest deadline 2023-12-16T22:11:03.026858+0000) 2023-12-16T22:14:41.065 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:14:40 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:14:40.831+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:10:43.026858+0000 (oldest deadline 2023-12-16T22:11:03.026858+0000) 2023-12-16T22:14:41.065 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:14:40 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:14:40.831+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:10:43.026858+0000 (oldest deadline 2023-12-16T22:11:03.026858+0000) 2023-12-16T22:14:41.066 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:14:40 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:14:40.832+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:14:41.066 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:14:40 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:14:40.750+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:10:42.731761+0000 (oldest deadline 2023-12-16T22:11:02.731761+0000) 2023-12-16T22:14:41.066 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:14:40 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:14:40.750+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:10:42.731761+0000 (oldest deadline 2023-12-16T22:11:02.731761+0000) 2023-12-16T22:14:41.066 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:14:40 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:14:40.750+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:10:42.731761+0000 (oldest deadline 2023-12-16T22:11:02.731761+0000) 2023-12-16T22:14:41.066 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:14:40 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:14:40.750+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:14:42.064 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:14:41 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:14:41.878+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:10:43.026858+0000 (oldest deadline 2023-12-16T22:11:03.026858+0000) 2023-12-16T22:14:42.064 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:14:41 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:14:41.878+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:10:43.026858+0000 (oldest deadline 2023-12-16T22:11:03.026858+0000) 2023-12-16T22:14:42.065 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:14:41 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:14:41.878+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:10:43.026858+0000 (oldest deadline 2023-12-16T22:11:03.026858+0000) 2023-12-16T22:14:42.065 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:14:41 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:14:41.879+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:14:42.065 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:14:41 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:14:41.900+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:11:07.849373+0000 (oldest deadline 2023-12-16T22:11:27.849373+0000) 2023-12-16T22:14:42.065 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:14:41 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:14:41.900+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:11:07.849373+0000 (oldest deadline 2023-12-16T22:11:27.849373+0000) 2023-12-16T22:14:42.065 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:14:41 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:14:41.900+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:11:07.849373+0000 (oldest deadline 2023-12-16T22:11:27.849373+0000) 2023-12-16T22:14:42.066 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:14:41 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:14:41.901+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:14:42.066 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:14:41 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:14:41.743+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:10:42.731761+0000 (oldest deadline 2023-12-16T22:11:02.731761+0000) 2023-12-16T22:14:42.066 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:14:41 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:14:41.743+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:10:42.731761+0000 (oldest deadline 2023-12-16T22:11:02.731761+0000) 2023-12-16T22:14:42.066 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:14:41 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:14:41.743+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:10:42.731761+0000 (oldest deadline 2023-12-16T22:11:02.731761+0000) 2023-12-16T22:14:42.066 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:14:41 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:14:41.743+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:14:42.449 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.169 2023-12-16T22:14:42.564 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Dec 16 22:14:42 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-mon-smithi045[80759]: 2023-12-16T22:14:42.162+0000 7f45a1f8f700 -1 mon.smithi045@0(probing) e2 get_health_metrics reporting 35304 slow ops, oldest is monmgrreport(gid 14214, 0 checks, 1 progress events) 2023-12-16T22:14:43.065 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:14:42 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:14:42.752+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:10:42.731761+0000 (oldest deadline 2023-12-16T22:11:02.731761+0000) 2023-12-16T22:14:43.065 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:14:42 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:14:42.752+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:10:42.731761+0000 (oldest deadline 2023-12-16T22:11:02.731761+0000) 2023-12-16T22:14:43.065 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:14:42 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:14:42.752+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:10:42.731761+0000 (oldest deadline 2023-12-16T22:11:02.731761+0000) 2023-12-16T22:14:43.065 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:14:42 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:14:42.753+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:14:43.066 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:14:42 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:14:42.916+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:10:43.026858+0000 (oldest deadline 2023-12-16T22:11:03.026858+0000) 2023-12-16T22:14:43.066 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:14:42 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:14:42.916+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:10:43.026858+0000 (oldest deadline 2023-12-16T22:11:03.026858+0000) 2023-12-16T22:14:43.066 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:14:42 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:14:42.916+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:10:43.026858+0000 (oldest deadline 2023-12-16T22:11:03.026858+0000) 2023-12-16T22:14:43.066 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:14:42 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:14:42.917+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:14:43.066 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:14:42 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:14:42.948+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:11:07.849373+0000 (oldest deadline 2023-12-16T22:11:27.849373+0000) 2023-12-16T22:14:43.066 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:14:42 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:14:42.948+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:11:07.849373+0000 (oldest deadline 2023-12-16T22:11:27.849373+0000) 2023-12-16T22:14:43.067 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:14:42 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:14:42.948+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:11:07.849373+0000 (oldest deadline 2023-12-16T22:11:27.849373+0000) 2023-12-16T22:14:43.067 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:14:42 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:14:42.949+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:14:43.983 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:14:43 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:14:43.879+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:10:43.026858+0000 (oldest deadline 2023-12-16T22:11:03.026858+0000) 2023-12-16T22:14:43.983 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:14:43 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:14:43.879+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:10:43.026858+0000 (oldest deadline 2023-12-16T22:11:03.026858+0000) 2023-12-16T22:14:43.983 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:14:43 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:14:43.879+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:10:43.026858+0000 (oldest deadline 2023-12-16T22:11:03.026858+0000) 2023-12-16T22:14:43.983 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:14:43 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:14:43.879+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:14:43.983 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:14:43 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:14:43.745+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:10:42.731761+0000 (oldest deadline 2023-12-16T22:11:02.731761+0000) 2023-12-16T22:14:43.984 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:14:43 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:14:43.745+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:10:42.731761+0000 (oldest deadline 2023-12-16T22:11:02.731761+0000) 2023-12-16T22:14:43.984 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:14:43 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:14:43.745+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:10:42.731761+0000 (oldest deadline 2023-12-16T22:11:02.731761+0000) 2023-12-16T22:14:43.984 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:14:43 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:14:43.745+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:14:44.239 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:14:43 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:14:43.980+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:11:07.849373+0000 (oldest deadline 2023-12-16T22:11:27.849373+0000) 2023-12-16T22:14:44.239 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:14:43 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:14:43.980+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:11:07.849373+0000 (oldest deadline 2023-12-16T22:11:27.849373+0000) 2023-12-16T22:14:44.239 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:14:43 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:14:43.980+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:11:07.849373+0000 (oldest deadline 2023-12-16T22:11:27.849373+0000) 2023-12-16T22:14:44.239 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:14:43 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:14:43.982+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:14:45.064 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:14:44 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:14:44.919+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:10:43.026858+0000 (oldest deadline 2023-12-16T22:11:03.026858+0000) 2023-12-16T22:14:45.065 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:14:44 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:14:44.919+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:10:43.026858+0000 (oldest deadline 2023-12-16T22:11:03.026858+0000) 2023-12-16T22:14:45.065 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:14:44 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:14:44.919+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:10:43.026858+0000 (oldest deadline 2023-12-16T22:11:03.026858+0000) 2023-12-16T22:14:45.065 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:14:44 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:14:44.919+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:14:45.065 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:14:44 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:14:44.777+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:10:42.731761+0000 (oldest deadline 2023-12-16T22:11:02.731761+0000) 2023-12-16T22:14:45.065 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:14:44 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:14:44.777+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:10:42.731761+0000 (oldest deadline 2023-12-16T22:11:02.731761+0000) 2023-12-16T22:14:45.065 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:14:44 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:14:44.777+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:10:42.731761+0000 (oldest deadline 2023-12-16T22:11:02.731761+0000) 2023-12-16T22:14:45.066 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:14:44 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:14:44.777+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:14:45.066 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:14:44 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:14:44.946+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:11:07.849373+0000 (oldest deadline 2023-12-16T22:11:27.849373+0000) 2023-12-16T22:14:45.066 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:14:44 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:14:44.946+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:11:07.849373+0000 (oldest deadline 2023-12-16T22:11:27.849373+0000) 2023-12-16T22:14:45.066 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:14:44 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:14:44.946+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:11:07.849373+0000 (oldest deadline 2023-12-16T22:11:27.849373+0000) 2023-12-16T22:14:45.066 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:14:44 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:14:44.947+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:14:46.064 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:14:45 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:14:45.942+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:11:07.849373+0000 (oldest deadline 2023-12-16T22:11:27.849373+0000) 2023-12-16T22:14:46.064 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:14:45 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:14:45.942+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:11:07.849373+0000 (oldest deadline 2023-12-16T22:11:27.849373+0000) 2023-12-16T22:14:46.065 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:14:45 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:14:45.942+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:11:07.849373+0000 (oldest deadline 2023-12-16T22:11:27.849373+0000) 2023-12-16T22:14:46.065 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:14:45 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:14:45.942+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:14:46.065 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:14:45 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:14:45.912+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:10:43.026858+0000 (oldest deadline 2023-12-16T22:11:03.026858+0000) 2023-12-16T22:14:46.065 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:14:45 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:14:45.912+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:10:43.026858+0000 (oldest deadline 2023-12-16T22:11:03.026858+0000) 2023-12-16T22:14:46.065 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:14:45 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:14:45.912+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:10:43.026858+0000 (oldest deadline 2023-12-16T22:11:03.026858+0000) 2023-12-16T22:14:46.066 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:14:45 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:14:45.912+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:14:46.066 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:14:45 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:14:45.797+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:10:42.731761+0000 (oldest deadline 2023-12-16T22:11:02.731761+0000) 2023-12-16T22:14:46.066 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:14:45 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:14:45.797+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:10:42.731761+0000 (oldest deadline 2023-12-16T22:11:02.731761+0000) 2023-12-16T22:14:46.066 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:14:45 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:14:45.797+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:10:42.731761+0000 (oldest deadline 2023-12-16T22:11:02.731761+0000) 2023-12-16T22:14:46.066 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:14:45 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:14:45.797+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:14:47.169 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:14:46 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:14:46.923+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:11:07.849373+0000 (oldest deadline 2023-12-16T22:11:27.849373+0000) 2023-12-16T22:14:47.169 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:14:46 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:14:46.923+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:11:07.849373+0000 (oldest deadline 2023-12-16T22:11:27.849373+0000) 2023-12-16T22:14:47.169 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:14:46 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:14:46.923+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:11:07.849373+0000 (oldest deadline 2023-12-16T22:11:27.849373+0000) 2023-12-16T22:14:47.169 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:14:46 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:14:46.924+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:14:47.170 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:14:46 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:14:46.926+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:10:43.026858+0000 (oldest deadline 2023-12-16T22:11:03.026858+0000) 2023-12-16T22:14:47.170 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:14:46 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:14:46.926+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:10:43.026858+0000 (oldest deadline 2023-12-16T22:11:03.026858+0000) 2023-12-16T22:14:47.170 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:14:46 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:14:46.926+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:10:43.026858+0000 (oldest deadline 2023-12-16T22:11:03.026858+0000) 2023-12-16T22:14:47.170 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:14:46 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:14:46.926+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:14:47.170 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:14:46 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:14:46.841+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:10:42.731761+0000 (oldest deadline 2023-12-16T22:11:02.731761+0000) 2023-12-16T22:14:47.171 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:14:46 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:14:46.841+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:10:42.731761+0000 (oldest deadline 2023-12-16T22:11:02.731761+0000) 2023-12-16T22:14:47.171 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:14:46 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:14:46.841+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:10:42.731761+0000 (oldest deadline 2023-12-16T22:11:02.731761+0000) 2023-12-16T22:14:47.171 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:14:46 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:14:46.842+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:14:47.564 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Dec 16 22:14:47 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-mon-smithi045[80759]: 2023-12-16T22:14:47.168+0000 7f45a1f8f700 -1 mon.smithi045@0(probing) e2 get_health_metrics reporting 35396 slow ops, oldest is monmgrreport(gid 14214, 0 checks, 1 progress events) 2023-12-16T22:14:48.064 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:14:47 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:14:47.813+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:10:42.731761+0000 (oldest deadline 2023-12-16T22:11:02.731761+0000) 2023-12-16T22:14:48.065 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:14:47 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:14:47.813+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:10:42.731761+0000 (oldest deadline 2023-12-16T22:11:02.731761+0000) 2023-12-16T22:14:48.065 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:14:47 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:14:47.813+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:10:42.731761+0000 (oldest deadline 2023-12-16T22:11:02.731761+0000) 2023-12-16T22:14:48.065 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:14:47 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:14:47.814+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:14:48.065 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:14:47 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:14:47.905+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:11:07.849373+0000 (oldest deadline 2023-12-16T22:11:27.849373+0000) 2023-12-16T22:14:48.065 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:14:47 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:14:47.905+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:11:07.849373+0000 (oldest deadline 2023-12-16T22:11:27.849373+0000) 2023-12-16T22:14:48.065 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:14:47 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:14:47.905+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:11:07.849373+0000 (oldest deadline 2023-12-16T22:11:27.849373+0000) 2023-12-16T22:14:48.066 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:14:47 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:14:47.906+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:14:48.066 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:14:47 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:14:47.974+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:10:43.026858+0000 (oldest deadline 2023-12-16T22:11:03.026858+0000) 2023-12-16T22:14:48.066 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:14:47 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:14:47.974+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:10:43.026858+0000 (oldest deadline 2023-12-16T22:11:03.026858+0000) 2023-12-16T22:14:48.066 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:14:47 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:14:47.974+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:10:43.026858+0000 (oldest deadline 2023-12-16T22:11:03.026858+0000) 2023-12-16T22:14:48.066 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:14:47 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:14:47.975+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:14:48.979 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:14:48 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:14:48.815+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:10:42.731761+0000 (oldest deadline 2023-12-16T22:11:02.731761+0000) 2023-12-16T22:14:48.979 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:14:48 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:14:48.815+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:10:42.731761+0000 (oldest deadline 2023-12-16T22:11:02.731761+0000) 2023-12-16T22:14:48.979 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:14:48 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:14:48.815+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:10:42.731761+0000 (oldest deadline 2023-12-16T22:11:02.731761+0000) 2023-12-16T22:14:48.979 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:14:48 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:14:48.816+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:14:48.980 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:14:48 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:14:48.912+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:11:07.849373+0000 (oldest deadline 2023-12-16T22:11:27.849373+0000) 2023-12-16T22:14:48.980 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:14:48 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:14:48.913+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:11:07.849373+0000 (oldest deadline 2023-12-16T22:11:27.849373+0000) 2023-12-16T22:14:48.980 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:14:48 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:14:48.913+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:11:07.849373+0000 (oldest deadline 2023-12-16T22:11:27.849373+0000) 2023-12-16T22:14:48.980 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:14:48 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:14:48.914+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:14:49.314 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:14:48 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:14:48.977+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:10:43.026858+0000 (oldest deadline 2023-12-16T22:11:03.026858+0000) 2023-12-16T22:14:49.314 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:14:48 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:14:48.978+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:10:43.026858+0000 (oldest deadline 2023-12-16T22:11:03.026858+0000) 2023-12-16T22:14:49.315 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:14:48 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:14:48.978+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:10:43.026858+0000 (oldest deadline 2023-12-16T22:11:03.026858+0000) 2023-12-16T22:14:49.315 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:14:48 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:14:48.978+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:14:50.064 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:14:49 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:14:49.789+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:10:42.731761+0000 (oldest deadline 2023-12-16T22:11:02.731761+0000) 2023-12-16T22:14:50.065 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:14:49 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:14:49.789+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:10:42.731761+0000 (oldest deadline 2023-12-16T22:11:02.731761+0000) 2023-12-16T22:14:50.065 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:14:49 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:14:49.789+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:10:42.731761+0000 (oldest deadline 2023-12-16T22:11:02.731761+0000) 2023-12-16T22:14:50.065 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:14:49 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:14:49.790+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:14:50.065 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:14:49 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:14:49.869+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:11:07.849373+0000 (oldest deadline 2023-12-16T22:11:27.849373+0000) 2023-12-16T22:14:50.065 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:14:49 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:14:49.869+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:11:07.849373+0000 (oldest deadline 2023-12-16T22:11:27.849373+0000) 2023-12-16T22:14:50.065 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:14:49 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:14:49.869+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:11:07.849373+0000 (oldest deadline 2023-12-16T22:11:27.849373+0000) 2023-12-16T22:14:50.066 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:14:49 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:14:49.870+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:14:50.066 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:14:49 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:14:49.953+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:10:43.026858+0000 (oldest deadline 2023-12-16T22:11:03.026858+0000) 2023-12-16T22:14:50.066 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:14:49 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:14:49.953+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:10:43.026858+0000 (oldest deadline 2023-12-16T22:11:03.026858+0000) 2023-12-16T22:14:50.066 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:14:49 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:14:49.953+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:10:43.026858+0000 (oldest deadline 2023-12-16T22:11:03.026858+0000) 2023-12-16T22:14:50.066 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:14:49 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:14:49.954+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:14:51.064 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:14:50 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:14:50.886+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:11:07.849373+0000 (oldest deadline 2023-12-16T22:11:27.849373+0000) 2023-12-16T22:14:51.064 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:14:50 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:14:50.886+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:11:07.849373+0000 (oldest deadline 2023-12-16T22:11:27.849373+0000) 2023-12-16T22:14:51.065 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:14:50 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:14:50.886+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:11:07.849373+0000 (oldest deadline 2023-12-16T22:11:27.849373+0000) 2023-12-16T22:14:51.065 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:14:50 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:14:50.887+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:14:51.065 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:14:50 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:14:50.930+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:10:43.026858+0000 (oldest deadline 2023-12-16T22:11:03.026858+0000) 2023-12-16T22:14:51.065 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:14:50 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:14:50.930+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:10:43.026858+0000 (oldest deadline 2023-12-16T22:11:03.026858+0000) 2023-12-16T22:14:51.065 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:14:50 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:14:50.930+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:10:43.026858+0000 (oldest deadline 2023-12-16T22:11:03.026858+0000) 2023-12-16T22:14:51.066 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:14:50 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:14:50.931+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:14:51.066 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:14:50 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:14:50.794+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:10:42.731761+0000 (oldest deadline 2023-12-16T22:11:02.731761+0000) 2023-12-16T22:14:51.066 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:14:50 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:14:50.794+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:10:42.731761+0000 (oldest deadline 2023-12-16T22:11:02.731761+0000) 2023-12-16T22:14:51.066 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:14:50 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:14:50.794+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:10:42.731761+0000 (oldest deadline 2023-12-16T22:11:02.731761+0000) 2023-12-16T22:14:51.066 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:14:50 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:14:50.796+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:14:52.064 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:14:51 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:14:51.842+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:10:42.731761+0000 (oldest deadline 2023-12-16T22:11:02.731761+0000) 2023-12-16T22:14:52.064 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:14:51 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:14:51.842+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:10:42.731761+0000 (oldest deadline 2023-12-16T22:11:02.731761+0000) 2023-12-16T22:14:52.065 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:14:51 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:14:51.842+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:10:42.731761+0000 (oldest deadline 2023-12-16T22:11:02.731761+0000) 2023-12-16T22:14:52.065 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:14:51 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:14:51.843+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:14:52.065 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:14:51 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:14:51.952+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:10:43.026858+0000 (oldest deadline 2023-12-16T22:11:03.026858+0000) 2023-12-16T22:14:52.065 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:14:51 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:14:51.952+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:10:43.026858+0000 (oldest deadline 2023-12-16T22:11:03.026858+0000) 2023-12-16T22:14:52.065 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:14:51 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:14:51.952+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:10:43.026858+0000 (oldest deadline 2023-12-16T22:11:03.026858+0000) 2023-12-16T22:14:52.065 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:14:51 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:14:51.953+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:14:52.066 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:14:51 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:14:51.844+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:11:07.849373+0000 (oldest deadline 2023-12-16T22:11:27.849373+0000) 2023-12-16T22:14:52.066 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:14:51 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:14:51.844+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:11:07.849373+0000 (oldest deadline 2023-12-16T22:11:27.849373+0000) 2023-12-16T22:14:52.066 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:14:51 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:14:51.844+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:11:07.849373+0000 (oldest deadline 2023-12-16T22:11:27.849373+0000) 2023-12-16T22:14:52.066 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:14:51 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:14:51.845+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:14:52.564 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Dec 16 22:14:52 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-mon-smithi045[80759]: 2023-12-16T22:14:52.172+0000 7f45a1f8f700 -1 mon.smithi045@0(probing) e2 get_health_metrics reporting 35487 slow ops, oldest is monmgrreport(gid 14214, 0 checks, 1 progress events) 2023-12-16T22:14:53.314 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:14:52 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:14:52.815+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:11:07.849373+0000 (oldest deadline 2023-12-16T22:11:27.849373+0000) 2023-12-16T22:14:53.314 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:14:52 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:14:52.815+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:11:07.849373+0000 (oldest deadline 2023-12-16T22:11:27.849373+0000) 2023-12-16T22:14:53.315 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:14:52 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:14:52.815+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:11:07.849373+0000 (oldest deadline 2023-12-16T22:11:27.849373+0000) 2023-12-16T22:14:53.315 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:14:52 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:14:52.815+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:14:53.315 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:14:52 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:14:52.921+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:10:43.026858+0000 (oldest deadline 2023-12-16T22:11:03.026858+0000) 2023-12-16T22:14:53.315 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:14:52 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:14:52.921+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:10:43.026858+0000 (oldest deadline 2023-12-16T22:11:03.026858+0000) 2023-12-16T22:14:53.315 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:14:52 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:14:52.921+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:10:43.026858+0000 (oldest deadline 2023-12-16T22:11:03.026858+0000) 2023-12-16T22:14:53.316 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:14:52 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:14:52.922+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:14:53.316 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:14:52 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:14:52.831+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:10:42.731761+0000 (oldest deadline 2023-12-16T22:11:02.731761+0000) 2023-12-16T22:14:53.316 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:14:52 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:14:52.831+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:10:42.731761+0000 (oldest deadline 2023-12-16T22:11:02.731761+0000) 2023-12-16T22:14:53.316 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:14:52 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:14:52.831+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:10:42.731761+0000 (oldest deadline 2023-12-16T22:11:02.731761+0000) 2023-12-16T22:14:53.316 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:14:52 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:14:52.832+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:14:54.064 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:14:53 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:14:53.808+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:11:07.849373+0000 (oldest deadline 2023-12-16T22:11:27.849373+0000) 2023-12-16T22:14:54.064 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:14:53 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:14:53.808+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:11:07.849373+0000 (oldest deadline 2023-12-16T22:11:27.849373+0000) 2023-12-16T22:14:54.065 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:14:53 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:14:53.808+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:11:07.849373+0000 (oldest deadline 2023-12-16T22:11:27.849373+0000) 2023-12-16T22:14:54.065 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:14:53 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:14:53.809+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:14:54.065 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:14:53 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:14:53.857+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:10:42.731761+0000 (oldest deadline 2023-12-16T22:11:02.731761+0000) 2023-12-16T22:14:54.065 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:14:53 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:14:53.857+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:10:42.731761+0000 (oldest deadline 2023-12-16T22:11:02.731761+0000) 2023-12-16T22:14:54.065 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:14:53 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:14:53.857+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:10:42.731761+0000 (oldest deadline 2023-12-16T22:11:02.731761+0000) 2023-12-16T22:14:54.066 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:14:53 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:14:53.859+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:14:54.066 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:14:53 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:14:53.954+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:10:43.026858+0000 (oldest deadline 2023-12-16T22:11:03.026858+0000) 2023-12-16T22:14:54.066 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:14:53 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:14:53.954+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:10:43.026858+0000 (oldest deadline 2023-12-16T22:11:03.026858+0000) 2023-12-16T22:14:54.066 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:14:53 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:14:53.954+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:10:43.026858+0000 (oldest deadline 2023-12-16T22:11:03.026858+0000) 2023-12-16T22:14:54.066 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:14:53 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:14:53.955+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:14:54.450 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi169.front.sepia.ceph.com' 2023-12-16T22:14:54.451 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi169.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-12-16T22:14:55.209 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:14:54 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:14:54.957+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:10:43.026858+0000 (oldest deadline 2023-12-16T22:11:03.026858+0000) 2023-12-16T22:14:55.209 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:14:54 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:14:54.957+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:10:43.026858+0000 (oldest deadline 2023-12-16T22:11:03.026858+0000) 2023-12-16T22:14:55.210 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:14:54 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:14:54.957+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:10:43.026858+0000 (oldest deadline 2023-12-16T22:11:03.026858+0000) 2023-12-16T22:14:55.210 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:14:54 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:14:54.957+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:14:55.210 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:14:54 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:14:54.839+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:11:07.849373+0000 (oldest deadline 2023-12-16T22:11:27.849373+0000) 2023-12-16T22:14:55.210 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:14:54 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:14:54.839+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:11:07.849373+0000 (oldest deadline 2023-12-16T22:11:27.849373+0000) 2023-12-16T22:14:55.210 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:14:54 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:14:54.839+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:11:07.849373+0000 (oldest deadline 2023-12-16T22:11:27.849373+0000) 2023-12-16T22:14:55.211 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:14:54 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:14:54.840+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:14:55.211 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:14:54 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:14:54.839+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:10:42.731761+0000 (oldest deadline 2023-12-16T22:11:02.731761+0000) 2023-12-16T22:14:55.211 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:14:54 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:14:54.839+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:10:42.731761+0000 (oldest deadline 2023-12-16T22:11:02.731761+0000) 2023-12-16T22:14:55.211 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:14:54 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:14:54.839+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:10:42.731761+0000 (oldest deadline 2023-12-16T22:11:02.731761+0000) 2023-12-16T22:14:55.211 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:14:54 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:14:54.839+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:14:56.315 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:14:55 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:14:55.919+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:10:43.026858+0000 (oldest deadline 2023-12-16T22:11:03.026858+0000) 2023-12-16T22:14:56.315 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:14:55 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:14:55.919+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:10:43.026858+0000 (oldest deadline 2023-12-16T22:11:03.026858+0000) 2023-12-16T22:14:56.315 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:14:55 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:14:55.919+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:10:43.026858+0000 (oldest deadline 2023-12-16T22:11:03.026858+0000) 2023-12-16T22:14:56.315 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:14:55 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:14:55.919+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:14:56.316 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:14:55 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:14:55.820+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:10:42.731761+0000 (oldest deadline 2023-12-16T22:11:02.731761+0000) 2023-12-16T22:14:56.316 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:14:55 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:14:55.820+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:10:42.731761+0000 (oldest deadline 2023-12-16T22:11:02.731761+0000) 2023-12-16T22:14:56.316 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:14:55 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:14:55.820+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:10:42.731761+0000 (oldest deadline 2023-12-16T22:11:02.731761+0000) 2023-12-16T22:14:56.316 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:14:55 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:14:55.820+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:14:56.316 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:14:55 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:14:55.870+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:11:07.849373+0000 (oldest deadline 2023-12-16T22:11:27.849373+0000) 2023-12-16T22:14:56.316 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:14:55 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:14:55.870+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:11:07.849373+0000 (oldest deadline 2023-12-16T22:11:27.849373+0000) 2023-12-16T22:14:56.317 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:14:55 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:14:55.870+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:11:07.849373+0000 (oldest deadline 2023-12-16T22:11:27.849373+0000) 2023-12-16T22:14:56.317 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:14:55 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:14:55.871+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:14:57.178 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:14:56 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:14:56.834+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:10:42.731761+0000 (oldest deadline 2023-12-16T22:11:02.731761+0000) 2023-12-16T22:14:57.178 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:14:56 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:14:56.834+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:10:42.731761+0000 (oldest deadline 2023-12-16T22:11:02.731761+0000) 2023-12-16T22:14:57.178 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:14:56 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:14:56.834+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:10:42.731761+0000 (oldest deadline 2023-12-16T22:11:02.731761+0000) 2023-12-16T22:14:57.178 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:14:56 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:14:56.835+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:14:57.179 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:14:56 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:14:56.893+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:11:07.849373+0000 (oldest deadline 2023-12-16T22:11:27.849373+0000) 2023-12-16T22:14:57.179 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:14:56 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:14:56.893+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:11:07.849373+0000 (oldest deadline 2023-12-16T22:11:27.849373+0000) 2023-12-16T22:14:57.179 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:14:56 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:14:56.893+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:11:07.849373+0000 (oldest deadline 2023-12-16T22:11:27.849373+0000) 2023-12-16T22:14:57.179 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:14:56 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:14:56.895+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:14:57.179 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:14:56 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:14:56.912+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:10:43.026858+0000 (oldest deadline 2023-12-16T22:11:03.026858+0000) 2023-12-16T22:14:57.179 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:14:56 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:14:56.912+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:10:43.026858+0000 (oldest deadline 2023-12-16T22:11:03.026858+0000) 2023-12-16T22:14:57.179 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:14:56 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:14:56.912+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:10:43.026858+0000 (oldest deadline 2023-12-16T22:11:03.026858+0000) 2023-12-16T22:14:57.180 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:14:56 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:14:56.913+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:14:57.521 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.169 2023-12-16T22:14:57.564 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Dec 16 22:14:57 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-mon-smithi045[80759]: 2023-12-16T22:14:57.176+0000 7f45a1f8f700 -1 mon.smithi045@0(probing) e2 get_health_metrics reporting 35579 slow ops, oldest is monmgrreport(gid 14214, 0 checks, 1 progress events) 2023-12-16T22:14:58.314 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:14:57 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:14:57.894+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:10:43.026858+0000 (oldest deadline 2023-12-16T22:11:03.026858+0000) 2023-12-16T22:14:58.315 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:14:57 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:14:57.894+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:10:43.026858+0000 (oldest deadline 2023-12-16T22:11:03.026858+0000) 2023-12-16T22:14:58.315 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:14:57 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:14:57.894+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:10:43.026858+0000 (oldest deadline 2023-12-16T22:11:03.026858+0000) 2023-12-16T22:14:58.315 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:14:57 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:14:57.896+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:14:58.315 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:14:57 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:14:57.842+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:10:42.731761+0000 (oldest deadline 2023-12-16T22:11:02.731761+0000) 2023-12-16T22:14:58.315 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:14:57 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:14:57.842+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:10:42.731761+0000 (oldest deadline 2023-12-16T22:11:02.731761+0000) 2023-12-16T22:14:58.316 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:14:57 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:14:57.842+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:10:42.731761+0000 (oldest deadline 2023-12-16T22:11:02.731761+0000) 2023-12-16T22:14:58.316 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:14:57 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:14:57.843+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:14:58.316 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:14:57 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:14:57.902+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:11:07.849373+0000 (oldest deadline 2023-12-16T22:11:27.849373+0000) 2023-12-16T22:14:58.316 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:14:57 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:14:57.902+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:11:07.849373+0000 (oldest deadline 2023-12-16T22:11:27.849373+0000) 2023-12-16T22:14:58.316 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:14:57 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:14:57.902+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:11:07.849373+0000 (oldest deadline 2023-12-16T22:11:27.849373+0000) 2023-12-16T22:14:58.316 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:14:57 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:14:57.903+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:14:59.064 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:14:58 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:14:58.911+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:11:07.849373+0000 (oldest deadline 2023-12-16T22:11:27.849373+0000) 2023-12-16T22:14:59.065 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:14:58 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:14:58.911+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:11:07.849373+0000 (oldest deadline 2023-12-16T22:11:27.849373+0000) 2023-12-16T22:14:59.065 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:14:58 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:14:58.911+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:11:07.849373+0000 (oldest deadline 2023-12-16T22:11:27.849373+0000) 2023-12-16T22:14:59.065 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:14:58 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:14:58.911+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:14:59.065 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:14:58 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:14:58.867+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:10:43.026858+0000 (oldest deadline 2023-12-16T22:11:03.026858+0000) 2023-12-16T22:14:59.065 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:14:58 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:14:58.867+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:10:43.026858+0000 (oldest deadline 2023-12-16T22:11:03.026858+0000) 2023-12-16T22:14:59.066 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:14:58 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:14:58.867+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:10:43.026858+0000 (oldest deadline 2023-12-16T22:11:03.026858+0000) 2023-12-16T22:14:59.066 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:14:58 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:14:58.867+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:14:59.066 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:14:58 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:14:58.821+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:10:42.731761+0000 (oldest deadline 2023-12-16T22:11:02.731761+0000) 2023-12-16T22:14:59.066 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:14:58 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:14:58.821+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:10:42.731761+0000 (oldest deadline 2023-12-16T22:11:02.731761+0000) 2023-12-16T22:14:59.066 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:14:58 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:14:58.821+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:10:42.731761+0000 (oldest deadline 2023-12-16T22:11:02.731761+0000) 2023-12-16T22:14:59.066 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:14:58 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:14:58.822+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:15:00.231 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:14:59 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:14:59.829+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:10:42.731761+0000 (oldest deadline 2023-12-16T22:11:02.731761+0000) 2023-12-16T22:15:00.231 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:14:59 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:14:59.829+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:10:42.731761+0000 (oldest deadline 2023-12-16T22:11:02.731761+0000) 2023-12-16T22:15:00.231 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:14:59 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:14:59.829+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:10:42.731761+0000 (oldest deadline 2023-12-16T22:11:02.731761+0000) 2023-12-16T22:15:00.231 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:14:59 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:14:59.831+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:15:00.232 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:14:59 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:14:59.915+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:11:07.849373+0000 (oldest deadline 2023-12-16T22:11:27.849373+0000) 2023-12-16T22:15:00.232 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:14:59 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:14:59.915+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:11:07.849373+0000 (oldest deadline 2023-12-16T22:11:27.849373+0000) 2023-12-16T22:15:00.232 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:14:59 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:14:59.915+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:11:07.849373+0000 (oldest deadline 2023-12-16T22:11:27.849373+0000) 2023-12-16T22:15:00.233 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:14:59 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:14:59.917+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:15:00.233 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:14:59 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:14:59.867+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:10:43.026858+0000 (oldest deadline 2023-12-16T22:11:03.026858+0000) 2023-12-16T22:15:00.233 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:14:59 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:14:59.867+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:10:43.026858+0000 (oldest deadline 2023-12-16T22:11:03.026858+0000) 2023-12-16T22:15:00.234 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:14:59 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:14:59.867+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:10:43.026858+0000 (oldest deadline 2023-12-16T22:11:03.026858+0000) 2023-12-16T22:15:00.234 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:14:59 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:14:59.867+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:15:00.522 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi169.front.sepia.ceph.com' 2023-12-16T22:15:00.523 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi169.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-12-16T22:15:00.597 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.169 2023-12-16T22:15:00.598 WARNING:teuthology.contextutil:'reconnect to {self.shortname}' reached maximum tries (5) after waiting for 30 seconds 2023-12-16T22:15:00.598 INFO:tasks.cephadm:Archiving crash dumps... 2023-12-16T22:15:00.600 DEBUG:teuthology.misc:Transferring archived files from smithi045:/var/lib/ceph/88ff363c-9c5a-11ee-95a5-87774f69a715/crash to /home/teuthworker/archive/yuriw-2023-12-16_15:45:14-fs-wip-yuri8-testing-2023-12-15-0911-distro-default-smithi/7494858/remote/smithi045/crash 2023-12-16T22:15:00.601 DEBUG:teuthology.orchestra.run.smithi045:> sudo tar c -f - -C /var/lib/ceph/88ff363c-9c5a-11ee-95a5-87774f69a715/crash -- . 2023-12-16T22:15:00.673 DEBUG:teuthology.misc:Transferring archived files from smithi169:/var/lib/ceph/88ff363c-9c5a-11ee-95a5-87774f69a715/crash to /home/teuthworker/archive/yuriw-2023-12-16_15:45:14-fs-wip-yuri8-testing-2023-12-15-0911-distro-default-smithi/7494858/remote/smithi169/crash 2023-12-16T22:15:00.674 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi169.front.sepia.ceph.com' 2023-12-16T22:15:00.674 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi169.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-12-16T22:15:00.951 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:15:00 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:15:00.811+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:10:42.731761+0000 (oldest deadline 2023-12-16T22:11:02.731761+0000) 2023-12-16T22:15:00.951 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:15:00 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:15:00.811+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:10:42.731761+0000 (oldest deadline 2023-12-16T22:11:02.731761+0000) 2023-12-16T22:15:00.951 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:15:00 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:15:00.811+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:10:42.731761+0000 (oldest deadline 2023-12-16T22:11:02.731761+0000) 2023-12-16T22:15:00.951 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:15:00 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:15:00.812+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:15:00.952 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:15:00 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:15:00.860+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:10:43.026858+0000 (oldest deadline 2023-12-16T22:11:03.026858+0000) 2023-12-16T22:15:00.952 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:15:00 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:15:00.860+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:10:43.026858+0000 (oldest deadline 2023-12-16T22:11:03.026858+0000) 2023-12-16T22:15:00.952 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:15:00 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:15:00.860+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:10:43.026858+0000 (oldest deadline 2023-12-16T22:11:03.026858+0000) 2023-12-16T22:15:00.952 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:15:00 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:15:00.861+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:15:01.314 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:15:00 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:15:00.950+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:11:07.849373+0000 (oldest deadline 2023-12-16T22:11:27.849373+0000) 2023-12-16T22:15:01.315 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:15:00 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:15:00.950+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:11:07.849373+0000 (oldest deadline 2023-12-16T22:11:27.849373+0000) 2023-12-16T22:15:01.315 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:15:00 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:15:00.950+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:11:07.849373+0000 (oldest deadline 2023-12-16T22:11:27.849373+0000) 2023-12-16T22:15:01.315 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:15:00 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:15:00.951+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:15:02.064 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:15:01 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:15:01.967+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:11:07.849373+0000 (oldest deadline 2023-12-16T22:11:27.849373+0000) 2023-12-16T22:15:02.065 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:15:01 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:15:01.967+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:11:07.849373+0000 (oldest deadline 2023-12-16T22:11:27.849373+0000) 2023-12-16T22:15:02.065 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:15:01 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:15:01.967+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:11:07.849373+0000 (oldest deadline 2023-12-16T22:11:27.849373+0000) 2023-12-16T22:15:02.065 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:15:01 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:15:01.968+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:15:02.065 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:15:01 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:15:01.806+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:10:42.731761+0000 (oldest deadline 2023-12-16T22:11:02.731761+0000) 2023-12-16T22:15:02.065 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:15:01 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:15:01.806+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:10:42.731761+0000 (oldest deadline 2023-12-16T22:11:02.731761+0000) 2023-12-16T22:15:02.065 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:15:01 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:15:01.806+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:10:42.731761+0000 (oldest deadline 2023-12-16T22:11:02.731761+0000) 2023-12-16T22:15:02.066 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:15:01 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:15:01.807+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:15:02.066 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:15:01 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:15:01.875+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:10:43.026858+0000 (oldest deadline 2023-12-16T22:11:03.026858+0000) 2023-12-16T22:15:02.066 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:15:01 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:15:01.875+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:10:43.026858+0000 (oldest deadline 2023-12-16T22:11:03.026858+0000) 2023-12-16T22:15:02.066 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:15:01 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:15:01.875+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:10:43.026858+0000 (oldest deadline 2023-12-16T22:11:03.026858+0000) 2023-12-16T22:15:02.066 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:15:01 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:15:01.876+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:15:02.564 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Dec 16 22:15:02 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-mon-smithi045[80759]: 2023-12-16T22:15:02.181+0000 7f45a1f8f700 -1 mon.smithi045@0(probing) e2 get_health_metrics reporting 35669 slow ops, oldest is monmgrreport(gid 14214, 0 checks, 1 progress events) 2023-12-16T22:15:02.967 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:15:02 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:15:02.787+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:10:42.731761+0000 (oldest deadline 2023-12-16T22:11:02.731761+0000) 2023-12-16T22:15:02.967 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:15:02 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:15:02.787+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:10:42.731761+0000 (oldest deadline 2023-12-16T22:11:02.731761+0000) 2023-12-16T22:15:02.967 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:15:02 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:15:02.787+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:10:42.731761+0000 (oldest deadline 2023-12-16T22:11:02.731761+0000) 2023-12-16T22:15:02.967 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:15:02 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:15:02.787+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:15:02.968 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:15:02 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:15:02.838+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:10:43.026858+0000 (oldest deadline 2023-12-16T22:11:03.026858+0000) 2023-12-16T22:15:02.968 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:15:02 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:15:02.838+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:10:43.026858+0000 (oldest deadline 2023-12-16T22:11:03.026858+0000) 2023-12-16T22:15:02.968 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:15:02 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:15:02.838+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:10:43.026858+0000 (oldest deadline 2023-12-16T22:11:03.026858+0000) 2023-12-16T22:15:02.968 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:15:02 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:15:02.838+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:15:03.314 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:15:02 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:15:02.966+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:11:07.849373+0000 (oldest deadline 2023-12-16T22:11:27.849373+0000) 2023-12-16T22:15:03.315 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:15:02 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:15:02.966+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:11:07.849373+0000 (oldest deadline 2023-12-16T22:11:27.849373+0000) 2023-12-16T22:15:03.315 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:15:02 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:15:02.966+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:11:07.849373+0000 (oldest deadline 2023-12-16T22:11:27.849373+0000) 2023-12-16T22:15:03.315 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:15:02 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:15:02.967+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:15:03.729 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.169 2023-12-16T22:15:03.943 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:15:03 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:15:03.766+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:10:42.731761+0000 (oldest deadline 2023-12-16T22:11:02.731761+0000) 2023-12-16T22:15:03.943 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:15:03 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:15:03.766+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:10:42.731761+0000 (oldest deadline 2023-12-16T22:11:02.731761+0000) 2023-12-16T22:15:03.943 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:15:03 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:15:03.766+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:10:42.731761+0000 (oldest deadline 2023-12-16T22:11:02.731761+0000) 2023-12-16T22:15:03.943 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:15:03 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:15:03.768+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:15:03.944 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:15:03 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:15:03.836+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:10:43.026858+0000 (oldest deadline 2023-12-16T22:11:03.026858+0000) 2023-12-16T22:15:03.944 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:15:03 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:15:03.836+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:10:43.026858+0000 (oldest deadline 2023-12-16T22:11:03.026858+0000) 2023-12-16T22:15:03.944 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:15:03 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:15:03.836+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:10:43.026858+0000 (oldest deadline 2023-12-16T22:11:03.026858+0000) 2023-12-16T22:15:03.944 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:15:03 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:15:03.836+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:15:04.314 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:15:03 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:15:03.942+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:11:07.849373+0000 (oldest deadline 2023-12-16T22:11:27.849373+0000) 2023-12-16T22:15:04.314 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:15:03 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:15:03.942+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:11:07.849373+0000 (oldest deadline 2023-12-16T22:11:27.849373+0000) 2023-12-16T22:15:04.314 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:15:03 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:15:03.942+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:11:07.849373+0000 (oldest deadline 2023-12-16T22:11:27.849373+0000) 2023-12-16T22:15:04.315 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:15:03 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:15:03.943+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:15:05.064 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:15:04 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:15:04.794+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:10:42.731761+0000 (oldest deadline 2023-12-16T22:11:02.731761+0000) 2023-12-16T22:15:05.064 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:15:04 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:15:04.794+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:10:42.731761+0000 (oldest deadline 2023-12-16T22:11:02.731761+0000) 2023-12-16T22:15:05.065 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:15:04 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:15:04.794+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:10:42.731761+0000 (oldest deadline 2023-12-16T22:11:02.731761+0000) 2023-12-16T22:15:05.065 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:15:04 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:15:04.796+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:15:05.065 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:15:04 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:15:04.923+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:11:07.849373+0000 (oldest deadline 2023-12-16T22:11:27.849373+0000) 2023-12-16T22:15:05.065 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:15:04 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:15:04.923+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:11:07.849373+0000 (oldest deadline 2023-12-16T22:11:27.849373+0000) 2023-12-16T22:15:05.065 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:15:04 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:15:04.923+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:11:07.849373+0000 (oldest deadline 2023-12-16T22:11:27.849373+0000) 2023-12-16T22:15:05.065 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:15:04 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:15:04.924+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:15:05.066 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:15:04 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:15:04.859+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:10:43.026858+0000 (oldest deadline 2023-12-16T22:11:03.026858+0000) 2023-12-16T22:15:05.066 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:15:04 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:15:04.859+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:10:43.026858+0000 (oldest deadline 2023-12-16T22:11:03.026858+0000) 2023-12-16T22:15:05.066 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:15:04 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:15:04.859+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:10:43.026858+0000 (oldest deadline 2023-12-16T22:11:03.026858+0000) 2023-12-16T22:15:05.066 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:15:04 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:15:04.860+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:15:05.822 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:15:05 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:15:05.772+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:10:42.731761+0000 (oldest deadline 2023-12-16T22:11:02.731761+0000) 2023-12-16T22:15:05.822 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:15:05 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:15:05.772+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:10:42.731761+0000 (oldest deadline 2023-12-16T22:11:02.731761+0000) 2023-12-16T22:15:05.822 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:15:05 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:15:05.772+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:10:42.731761+0000 (oldest deadline 2023-12-16T22:11:02.731761+0000) 2023-12-16T22:15:05.822 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:15:05 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:15:05.773+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:15:06.314 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:15:05 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:15:05.969+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:11:07.849373+0000 (oldest deadline 2023-12-16T22:11:27.849373+0000) 2023-12-16T22:15:06.314 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:15:05 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:15:05.969+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:11:07.849373+0000 (oldest deadline 2023-12-16T22:11:27.849373+0000) 2023-12-16T22:15:06.315 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:15:05 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:15:05.969+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:11:07.849373+0000 (oldest deadline 2023-12-16T22:11:27.849373+0000) 2023-12-16T22:15:06.315 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:15:05 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:15:05.970+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:15:06.315 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:15:05 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:15:05.859+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:10:43.026858+0000 (oldest deadline 2023-12-16T22:11:03.026858+0000) 2023-12-16T22:15:06.315 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:15:05 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:15:05.859+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:10:43.026858+0000 (oldest deadline 2023-12-16T22:11:03.026858+0000) 2023-12-16T22:15:06.315 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:15:05 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:15:05.859+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:10:43.026858+0000 (oldest deadline 2023-12-16T22:11:03.026858+0000) 2023-12-16T22:15:06.315 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:15:05 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:15:05.860+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:15:07.064 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:15:06 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:15:06.943+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:11:07.849373+0000 (oldest deadline 2023-12-16T22:11:27.849373+0000) 2023-12-16T22:15:07.064 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:15:06 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:15:06.943+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:11:07.849373+0000 (oldest deadline 2023-12-16T22:11:27.849373+0000) 2023-12-16T22:15:07.065 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:15:06 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:15:06.943+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:11:07.849373+0000 (oldest deadline 2023-12-16T22:11:27.849373+0000) 2023-12-16T22:15:07.065 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:15:06 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:15:06.944+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:15:07.065 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:15:06 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:15:06.909+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:10:43.026858+0000 (oldest deadline 2023-12-16T22:11:03.026858+0000) 2023-12-16T22:15:07.065 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:15:06 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:15:06.909+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:10:43.026858+0000 (oldest deadline 2023-12-16T22:11:03.026858+0000) 2023-12-16T22:15:07.065 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:15:06 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:15:06.909+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:10:43.026858+0000 (oldest deadline 2023-12-16T22:11:03.026858+0000) 2023-12-16T22:15:07.066 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:15:06 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:15:06.910+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:15:07.066 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:15:06 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:15:06.744+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:10:42.731761+0000 (oldest deadline 2023-12-16T22:11:02.731761+0000) 2023-12-16T22:15:07.066 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:15:06 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:15:06.744+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:10:42.731761+0000 (oldest deadline 2023-12-16T22:11:02.731761+0000) 2023-12-16T22:15:07.066 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:15:06 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:15:06.744+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:10:42.731761+0000 (oldest deadline 2023-12-16T22:11:02.731761+0000) 2023-12-16T22:15:07.066 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:15:06 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:15:06.745+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:15:07.564 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Dec 16 22:15:07 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-mon-smithi045[80759]: 2023-12-16T22:15:07.185+0000 7f45a1f8f700 -1 mon.smithi045@0(probing) e2 get_health_metrics reporting 35761 slow ops, oldest is monmgrreport(gid 14214, 0 checks, 1 progress events) 2023-12-16T22:15:08.064 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:15:07 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:15:07.746+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:10:42.731761+0000 (oldest deadline 2023-12-16T22:11:02.731761+0000) 2023-12-16T22:15:08.064 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:15:07 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:15:07.746+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:10:42.731761+0000 (oldest deadline 2023-12-16T22:11:02.731761+0000) 2023-12-16T22:15:08.065 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:15:07 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:15:07.746+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:10:42.731761+0000 (oldest deadline 2023-12-16T22:11:02.731761+0000) 2023-12-16T22:15:08.065 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:15:07 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:15:07.747+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:15:08.065 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:15:07 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:15:07.929+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:11:07.849373+0000 (oldest deadline 2023-12-16T22:11:27.849373+0000) 2023-12-16T22:15:08.065 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:15:07 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:15:07.929+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:11:07.849373+0000 (oldest deadline 2023-12-16T22:11:27.849373+0000) 2023-12-16T22:15:08.065 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:15:07 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:15:07.929+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:11:07.849373+0000 (oldest deadline 2023-12-16T22:11:27.849373+0000) 2023-12-16T22:15:08.066 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:15:07 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:15:07.930+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:15:08.066 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:15:07 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:15:07.868+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:10:43.026858+0000 (oldest deadline 2023-12-16T22:11:03.026858+0000) 2023-12-16T22:15:08.066 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:15:07 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:15:07.868+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:10:43.026858+0000 (oldest deadline 2023-12-16T22:11:03.026858+0000) 2023-12-16T22:15:08.066 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:15:07 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:15:07.868+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:10:43.026858+0000 (oldest deadline 2023-12-16T22:11:03.026858+0000) 2023-12-16T22:15:08.066 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:15:07 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:15:07.869+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:15:09.064 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:15:08 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:15:08.884+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:11:07.849373+0000 (oldest deadline 2023-12-16T22:11:27.849373+0000) 2023-12-16T22:15:09.065 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:15:08 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:15:08.884+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:11:07.849373+0000 (oldest deadline 2023-12-16T22:11:27.849373+0000) 2023-12-16T22:15:09.065 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:15:08 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:15:08.884+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:11:07.849373+0000 (oldest deadline 2023-12-16T22:11:27.849373+0000) 2023-12-16T22:15:09.065 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:15:08 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:15:08.884+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:15:09.065 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:15:08 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:15:08.843+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:10:43.026858+0000 (oldest deadline 2023-12-16T22:11:03.026858+0000) 2023-12-16T22:15:09.066 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:15:08 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:15:08.843+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:10:43.026858+0000 (oldest deadline 2023-12-16T22:11:03.026858+0000) 2023-12-16T22:15:09.066 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:15:08 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:15:08.843+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:10:43.026858+0000 (oldest deadline 2023-12-16T22:11:03.026858+0000) 2023-12-16T22:15:09.066 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:15:08 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:15:08.843+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:15:09.066 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:15:08 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:15:08.735+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:10:42.731761+0000 (oldest deadline 2023-12-16T22:11:02.731761+0000) 2023-12-16T22:15:09.066 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:15:08 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:15:08.735+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:10:42.731761+0000 (oldest deadline 2023-12-16T22:11:02.731761+0000) 2023-12-16T22:15:09.067 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:15:08 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:15:08.735+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:10:42.731761+0000 (oldest deadline 2023-12-16T22:11:02.731761+0000) 2023-12-16T22:15:09.067 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:15:08 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:15:08.736+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:15:10.064 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:15:09 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:15:09.878+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:10:43.026858+0000 (oldest deadline 2023-12-16T22:11:03.026858+0000) 2023-12-16T22:15:10.064 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:15:09 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:15:09.878+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:10:43.026858+0000 (oldest deadline 2023-12-16T22:11:03.026858+0000) 2023-12-16T22:15:10.065 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:15:09 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:15:09.878+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:10:43.026858+0000 (oldest deadline 2023-12-16T22:11:03.026858+0000) 2023-12-16T22:15:10.065 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:15:09 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:15:09.879+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:15:10.065 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:15:09 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:15:09.715+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:10:42.731761+0000 (oldest deadline 2023-12-16T22:11:02.731761+0000) 2023-12-16T22:15:10.065 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:15:09 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:15:09.715+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:10:42.731761+0000 (oldest deadline 2023-12-16T22:11:02.731761+0000) 2023-12-16T22:15:10.065 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:15:09 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:15:09.715+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:10:42.731761+0000 (oldest deadline 2023-12-16T22:11:02.731761+0000) 2023-12-16T22:15:10.066 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:15:09 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:15:09.716+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:15:10.066 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:15:09 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:15:09.877+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:11:07.849373+0000 (oldest deadline 2023-12-16T22:11:27.849373+0000) 2023-12-16T22:15:10.066 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:15:09 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:15:09.877+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:11:07.849373+0000 (oldest deadline 2023-12-16T22:11:27.849373+0000) 2023-12-16T22:15:10.066 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:15:09 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:15:09.877+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:11:07.849373+0000 (oldest deadline 2023-12-16T22:11:27.849373+0000) 2023-12-16T22:15:10.066 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:15:09 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:15:09.878+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:15:11.064 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:15:10 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:15:10.716+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:10:42.731761+0000 (oldest deadline 2023-12-16T22:11:02.731761+0000) 2023-12-16T22:15:11.064 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:15:10 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:15:10.716+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:10:42.731761+0000 (oldest deadline 2023-12-16T22:11:02.731761+0000) 2023-12-16T22:15:11.065 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:15:10 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:15:10.716+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:10:42.731761+0000 (oldest deadline 2023-12-16T22:11:02.731761+0000) 2023-12-16T22:15:11.065 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:15:10 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:15:10.717+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:15:11.065 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:15:10 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:15:10.842+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:11:07.849373+0000 (oldest deadline 2023-12-16T22:11:27.849373+0000) 2023-12-16T22:15:11.065 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:15:10 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:15:10.843+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:11:07.849373+0000 (oldest deadline 2023-12-16T22:11:27.849373+0000) 2023-12-16T22:15:11.065 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:15:10 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:15:10.843+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:11:07.849373+0000 (oldest deadline 2023-12-16T22:11:27.849373+0000) 2023-12-16T22:15:11.065 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:15:10 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:15:10.844+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:15:11.066 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:15:10 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:15:10.849+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:10:43.026858+0000 (oldest deadline 2023-12-16T22:11:03.026858+0000) 2023-12-16T22:15:11.066 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:15:10 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:15:10.849+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:10:43.026858+0000 (oldest deadline 2023-12-16T22:11:03.026858+0000) 2023-12-16T22:15:11.066 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:15:10 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:15:10.849+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:10:43.026858+0000 (oldest deadline 2023-12-16T22:11:03.026858+0000) 2023-12-16T22:15:11.066 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:15:10 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:15:10.849+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:15:12.064 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:15:11 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:15:11.723+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:10:42.731761+0000 (oldest deadline 2023-12-16T22:11:02.731761+0000) 2023-12-16T22:15:12.064 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:15:11 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:15:11.723+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:10:42.731761+0000 (oldest deadline 2023-12-16T22:11:02.731761+0000) 2023-12-16T22:15:12.065 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:15:11 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:15:11.723+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:10:42.731761+0000 (oldest deadline 2023-12-16T22:11:02.731761+0000) 2023-12-16T22:15:12.065 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:15:11 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:15:11.724+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:15:12.065 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:15:11 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:15:11.839+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:10:43.026858+0000 (oldest deadline 2023-12-16T22:11:03.026858+0000) 2023-12-16T22:15:12.065 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:15:11 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:15:11.839+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:10:43.026858+0000 (oldest deadline 2023-12-16T22:11:03.026858+0000) 2023-12-16T22:15:12.065 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:15:11 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:15:11.839+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:10:43.026858+0000 (oldest deadline 2023-12-16T22:11:03.026858+0000) 2023-12-16T22:15:12.066 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:15:11 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:15:11.840+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:15:12.066 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:15:11 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:15:11.886+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:11:07.849373+0000 (oldest deadline 2023-12-16T22:11:27.849373+0000) 2023-12-16T22:15:12.066 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:15:11 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:15:11.886+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:11:07.849373+0000 (oldest deadline 2023-12-16T22:11:27.849373+0000) 2023-12-16T22:15:12.066 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:15:11 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:15:11.886+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:11:07.849373+0000 (oldest deadline 2023-12-16T22:11:27.849373+0000) 2023-12-16T22:15:12.066 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:15:11 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:15:11.887+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:15:12.564 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Dec 16 22:15:12 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-mon-smithi045[80759]: 2023-12-16T22:15:12.190+0000 7f45a1f8f700 -1 mon.smithi045@0(probing) e2 get_health_metrics reporting 35852 slow ops, oldest is monmgrreport(gid 14214, 0 checks, 1 progress events) 2023-12-16T22:15:12.730 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi169.front.sepia.ceph.com' 2023-12-16T22:15:12.731 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi169.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-12-16T22:15:13.064 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:15:12 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:15:12.824+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:10:43.026858+0000 (oldest deadline 2023-12-16T22:11:03.026858+0000) 2023-12-16T22:15:13.065 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:15:12 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:15:12.824+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:10:43.026858+0000 (oldest deadline 2023-12-16T22:11:03.026858+0000) 2023-12-16T22:15:13.065 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:15:12 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:15:12.824+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:10:43.026858+0000 (oldest deadline 2023-12-16T22:11:03.026858+0000) 2023-12-16T22:15:13.065 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:15:12 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:15:12.826+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:15:13.065 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:15:12 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:15:12.688+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:10:42.731761+0000 (oldest deadline 2023-12-16T22:11:02.731761+0000) 2023-12-16T22:15:13.066 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:15:12 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:15:12.688+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:10:42.731761+0000 (oldest deadline 2023-12-16T22:11:02.731761+0000) 2023-12-16T22:15:13.066 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:15:12 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:15:12.688+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:10:42.731761+0000 (oldest deadline 2023-12-16T22:11:02.731761+0000) 2023-12-16T22:15:13.066 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:15:12 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:15:12.689+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:15:13.066 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:15:12 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:15:12.924+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:11:07.849373+0000 (oldest deadline 2023-12-16T22:11:27.849373+0000) 2023-12-16T22:15:13.066 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:15:12 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:15:12.924+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:11:07.849373+0000 (oldest deadline 2023-12-16T22:11:27.849373+0000) 2023-12-16T22:15:13.066 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:15:12 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:15:12.924+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:11:07.849373+0000 (oldest deadline 2023-12-16T22:11:27.849373+0000) 2023-12-16T22:15:13.067 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:15:12 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:15:12.925+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:15:13.968 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:15:13 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:15:13.855+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:10:43.026858+0000 (oldest deadline 2023-12-16T22:11:03.026858+0000) 2023-12-16T22:15:13.969 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:15:13 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:15:13.855+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:10:43.026858+0000 (oldest deadline 2023-12-16T22:11:03.026858+0000) 2023-12-16T22:15:13.969 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:15:13 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:15:13.855+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:10:43.026858+0000 (oldest deadline 2023-12-16T22:11:03.026858+0000) 2023-12-16T22:15:13.969 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:15:13 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:15:13.856+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:15:13.970 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:15:13 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:15:13.695+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:10:42.731761+0000 (oldest deadline 2023-12-16T22:11:02.731761+0000) 2023-12-16T22:15:13.970 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:15:13 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:15:13.695+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:10:42.731761+0000 (oldest deadline 2023-12-16T22:11:02.731761+0000) 2023-12-16T22:15:13.970 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:15:13 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:15:13.695+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:10:42.731761+0000 (oldest deadline 2023-12-16T22:11:02.731761+0000) 2023-12-16T22:15:13.970 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:15:13 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:15:13.696+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:15:14.314 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:15:13 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:15:13.966+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:11:07.849373+0000 (oldest deadline 2023-12-16T22:11:27.849373+0000) 2023-12-16T22:15:14.314 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:15:13 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:15:13.966+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:11:07.849373+0000 (oldest deadline 2023-12-16T22:11:27.849373+0000) 2023-12-16T22:15:14.315 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:15:13 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:15:13.966+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:11:07.849373+0000 (oldest deadline 2023-12-16T22:11:27.849373+0000) 2023-12-16T22:15:14.315 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:15:13 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:15:13.967+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:15:14.959 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:15:14 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:15:14.812+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:10:43.026858+0000 (oldest deadline 2023-12-16T22:11:03.026858+0000) 2023-12-16T22:15:14.960 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:15:14 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:15:14.812+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:10:43.026858+0000 (oldest deadline 2023-12-16T22:11:03.026858+0000) 2023-12-16T22:15:14.960 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:15:14 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:15:14.812+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:10:43.026858+0000 (oldest deadline 2023-12-16T22:11:03.026858+0000) 2023-12-16T22:15:14.960 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:15:14 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:15:14.813+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:15:14.960 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:15:14 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:15:14.685+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:10:42.731761+0000 (oldest deadline 2023-12-16T22:11:02.731761+0000) 2023-12-16T22:15:14.960 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:15:14 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:15:14.685+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:10:42.731761+0000 (oldest deadline 2023-12-16T22:11:02.731761+0000) 2023-12-16T22:15:14.961 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:15:14 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:15:14.685+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:10:42.731761+0000 (oldest deadline 2023-12-16T22:11:02.731761+0000) 2023-12-16T22:15:14.961 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:15:14 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:15:14.686+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:15:15.314 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:15:14 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:15:14.992+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:11:07.849373+0000 (oldest deadline 2023-12-16T22:11:27.849373+0000) 2023-12-16T22:15:15.314 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:15:14 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:15:14.992+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:11:07.849373+0000 (oldest deadline 2023-12-16T22:11:27.849373+0000) 2023-12-16T22:15:15.315 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:15:14 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:15:14.992+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:11:07.849373+0000 (oldest deadline 2023-12-16T22:11:27.849373+0000) 2023-12-16T22:15:15.315 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:15:14 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:15:14.993+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:15:15.798 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.169 2023-12-16T22:15:16.030 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:15:15 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:15:15.802+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:10:43.026858+0000 (oldest deadline 2023-12-16T22:11:03.026858+0000) 2023-12-16T22:15:16.031 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:15:15 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:15:15.802+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:10:43.026858+0000 (oldest deadline 2023-12-16T22:11:03.026858+0000) 2023-12-16T22:15:16.031 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:15:15 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:15:15.802+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:10:43.026858+0000 (oldest deadline 2023-12-16T22:11:03.026858+0000) 2023-12-16T22:15:16.031 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:15:15 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:15:15.803+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:15:16.031 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:15:15 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:15:15.646+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:10:42.731761+0000 (oldest deadline 2023-12-16T22:11:02.731761+0000) 2023-12-16T22:15:16.031 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:15:15 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:15:15.646+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:10:42.731761+0000 (oldest deadline 2023-12-16T22:11:02.731761+0000) 2023-12-16T22:15:16.032 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:15:15 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:15:15.646+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:10:42.731761+0000 (oldest deadline 2023-12-16T22:11:02.731761+0000) 2023-12-16T22:15:16.032 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:15:15 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:15:15.647+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:15:16.314 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:15:16 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:15:16.029+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:11:07.849373+0000 (oldest deadline 2023-12-16T22:11:27.849373+0000) 2023-12-16T22:15:16.314 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:15:16 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:15:16.029+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:11:07.849373+0000 (oldest deadline 2023-12-16T22:11:27.849373+0000) 2023-12-16T22:15:16.315 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:15:16 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:15:16.029+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:11:07.849373+0000 (oldest deadline 2023-12-16T22:11:27.849373+0000) 2023-12-16T22:15:16.315 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:15:16 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:15:16.031+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:15:17.042 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:15:16 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:15:16.817+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:10:43.026858+0000 (oldest deadline 2023-12-16T22:11:03.026858+0000) 2023-12-16T22:15:17.043 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:15:16 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:15:16.817+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:10:43.026858+0000 (oldest deadline 2023-12-16T22:11:03.026858+0000) 2023-12-16T22:15:17.043 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:15:16 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:15:16.817+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:10:43.026858+0000 (oldest deadline 2023-12-16T22:11:03.026858+0000) 2023-12-16T22:15:17.043 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:15:16 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:15:16.819+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:15:17.043 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:15:16 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:15:16.646+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:10:42.731761+0000 (oldest deadline 2023-12-16T22:11:02.731761+0000) 2023-12-16T22:15:17.043 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:15:16 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:15:16.646+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:10:42.731761+0000 (oldest deadline 2023-12-16T22:11:02.731761+0000) 2023-12-16T22:15:17.043 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:15:16 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:15:16.646+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:10:42.731761+0000 (oldest deadline 2023-12-16T22:11:02.731761+0000) 2023-12-16T22:15:17.043 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:15:16 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:15:16.646+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:15:17.314 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Dec 16 22:15:17 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-mon-smithi045[80759]: 2023-12-16T22:15:17.195+0000 7f45a1f8f700 -1 mon.smithi045@0(probing) e2 get_health_metrics reporting 35944 slow ops, oldest is monmgrreport(gid 14214, 0 checks, 1 progress events) 2023-12-16T22:15:17.314 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:15:17 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:15:17.041+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:11:07.849373+0000 (oldest deadline 2023-12-16T22:11:27.849373+0000) 2023-12-16T22:15:17.315 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:15:17 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:15:17.041+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:11:07.849373+0000 (oldest deadline 2023-12-16T22:11:27.849373+0000) 2023-12-16T22:15:17.315 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:15:17 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:15:17.041+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:11:07.849373+0000 (oldest deadline 2023-12-16T22:11:27.849373+0000) 2023-12-16T22:15:17.315 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:15:17 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:15:17.042+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:15:18.050 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:15:17 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:15:17.778+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:10:43.026858+0000 (oldest deadline 2023-12-16T22:11:03.026858+0000) 2023-12-16T22:15:18.051 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:15:17 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:15:17.778+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:10:43.026858+0000 (oldest deadline 2023-12-16T22:11:03.026858+0000) 2023-12-16T22:15:18.051 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:15:17 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:15:17.778+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:10:43.026858+0000 (oldest deadline 2023-12-16T22:11:03.026858+0000) 2023-12-16T22:15:18.051 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:15:17 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:15:17.779+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:15:18.051 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:15:17 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:15:17.620+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:10:42.731761+0000 (oldest deadline 2023-12-16T22:11:02.731761+0000) 2023-12-16T22:15:18.051 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:15:17 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:15:17.620+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:10:42.731761+0000 (oldest deadline 2023-12-16T22:11:02.731761+0000) 2023-12-16T22:15:18.051 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:15:17 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:15:17.621+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:10:42.731761+0000 (oldest deadline 2023-12-16T22:11:02.731761+0000) 2023-12-16T22:15:18.052 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:15:17 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:15:17.621+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:15:18.314 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:15:18 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:15:18.049+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:11:07.849373+0000 (oldest deadline 2023-12-16T22:11:27.849373+0000) 2023-12-16T22:15:18.314 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:15:18 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:15:18.049+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:11:07.849373+0000 (oldest deadline 2023-12-16T22:11:27.849373+0000) 2023-12-16T22:15:18.314 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:15:18 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:15:18.049+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:11:07.849373+0000 (oldest deadline 2023-12-16T22:11:27.849373+0000) 2023-12-16T22:15:18.315 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:15:18 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:15:18.050+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:15:19.064 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:15:18 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:15:18.632+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:10:42.731761+0000 (oldest deadline 2023-12-16T22:11:02.731761+0000) 2023-12-16T22:15:19.065 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:15:18 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:15:18.633+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:10:42.731761+0000 (oldest deadline 2023-12-16T22:11:02.731761+0000) 2023-12-16T22:15:19.065 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:15:18 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:15:18.633+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:10:42.731761+0000 (oldest deadline 2023-12-16T22:11:02.731761+0000) 2023-12-16T22:15:19.065 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:15:18 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:15:18.633+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:15:19.065 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:15:18 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:15:18.804+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:10:43.026858+0000 (oldest deadline 2023-12-16T22:11:03.026858+0000) 2023-12-16T22:15:19.065 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:15:18 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:15:18.804+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:10:43.026858+0000 (oldest deadline 2023-12-16T22:11:03.026858+0000) 2023-12-16T22:15:19.065 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:15:18 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:15:18.804+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:10:43.026858+0000 (oldest deadline 2023-12-16T22:11:03.026858+0000) 2023-12-16T22:15:19.066 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:15:18 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:15:18.804+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:15:19.565 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:15:19 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:15:19.072+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:11:07.849373+0000 (oldest deadline 2023-12-16T22:11:27.849373+0000) 2023-12-16T22:15:19.565 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:15:19 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:15:19.072+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:11:07.849373+0000 (oldest deadline 2023-12-16T22:11:27.849373+0000) 2023-12-16T22:15:19.566 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:15:19 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:15:19.072+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:11:07.849373+0000 (oldest deadline 2023-12-16T22:11:27.849373+0000) 2023-12-16T22:15:19.566 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:15:19 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:15:19.073+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:15:20.064 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:15:19 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:15:19.844+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:10:43.026858+0000 (oldest deadline 2023-12-16T22:11:03.026858+0000) 2023-12-16T22:15:20.065 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:15:19 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:15:19.844+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:10:43.026858+0000 (oldest deadline 2023-12-16T22:11:03.026858+0000) 2023-12-16T22:15:20.065 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:15:19 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:15:19.844+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:10:43.026858+0000 (oldest deadline 2023-12-16T22:11:03.026858+0000) 2023-12-16T22:15:20.065 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:15:19 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:15:19.845+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:15:20.065 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:15:19 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:15:19.614+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:10:42.731761+0000 (oldest deadline 2023-12-16T22:11:02.731761+0000) 2023-12-16T22:15:20.065 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:15:19 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:15:19.614+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:10:42.731761+0000 (oldest deadline 2023-12-16T22:11:02.731761+0000) 2023-12-16T22:15:20.065 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:15:19 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:15:19.614+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:10:42.731761+0000 (oldest deadline 2023-12-16T22:11:02.731761+0000) 2023-12-16T22:15:20.066 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:15:19 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:15:19.615+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:15:20.564 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:15:20 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:15:20.068+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:11:07.849373+0000 (oldest deadline 2023-12-16T22:11:27.849373+0000) 2023-12-16T22:15:20.565 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:15:20 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:15:20.068+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:11:07.849373+0000 (oldest deadline 2023-12-16T22:11:27.849373+0000) 2023-12-16T22:15:20.565 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:15:20 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:15:20.068+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:11:07.849373+0000 (oldest deadline 2023-12-16T22:11:27.849373+0000) 2023-12-16T22:15:20.565 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:15:20 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:15:20.069+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:15:20.851 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:15:20 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:15:20.573+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:10:42.731761+0000 (oldest deadline 2023-12-16T22:11:02.731761+0000) 2023-12-16T22:15:20.851 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:15:20 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:15:20.573+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:10:42.731761+0000 (oldest deadline 2023-12-16T22:11:02.731761+0000) 2023-12-16T22:15:20.852 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:15:20 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:15:20.573+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:10:42.731761+0000 (oldest deadline 2023-12-16T22:11:02.731761+0000) 2023-12-16T22:15:20.852 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:15:20 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:15:20.574+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:15:21.104 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:15:20 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:15:20.850+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:10:43.026858+0000 (oldest deadline 2023-12-16T22:11:03.026858+0000) 2023-12-16T22:15:21.104 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:15:20 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:15:20.850+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:10:43.026858+0000 (oldest deadline 2023-12-16T22:11:03.026858+0000) 2023-12-16T22:15:21.105 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:15:20 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:15:20.850+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:10:43.026858+0000 (oldest deadline 2023-12-16T22:11:03.026858+0000) 2023-12-16T22:15:21.105 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:15:20 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:15:20.851+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:15:21.105 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:15:21 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:15:21.078+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:11:07.849373+0000 (oldest deadline 2023-12-16T22:11:27.849373+0000) 2023-12-16T22:15:21.105 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:15:21 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:15:21.078+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:11:07.849373+0000 (oldest deadline 2023-12-16T22:11:27.849373+0000) 2023-12-16T22:15:21.105 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:15:21 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:15:21.078+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:11:07.849373+0000 (oldest deadline 2023-12-16T22:11:27.849373+0000) 2023-12-16T22:15:21.105 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:15:21 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:15:21.079+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:15:21.814 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:15:21 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:15:21.560+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:10:42.731761+0000 (oldest deadline 2023-12-16T22:11:02.731761+0000) 2023-12-16T22:15:21.814 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:15:21 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:15:21.560+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:10:42.731761+0000 (oldest deadline 2023-12-16T22:11:02.731761+0000) 2023-12-16T22:15:21.814 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:15:21 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:15:21.560+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:10:42.731761+0000 (oldest deadline 2023-12-16T22:11:02.731761+0000) 2023-12-16T22:15:21.815 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:15:21 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:15:21.561+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:15:22.092 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:15:21 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:15:21.832+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:10:43.026858+0000 (oldest deadline 2023-12-16T22:11:03.026858+0000) 2023-12-16T22:15:22.093 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:15:21 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:15:21.832+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:10:43.026858+0000 (oldest deadline 2023-12-16T22:11:03.026858+0000) 2023-12-16T22:15:22.093 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:15:21 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:15:21.832+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:10:43.026858+0000 (oldest deadline 2023-12-16T22:11:03.026858+0000) 2023-12-16T22:15:22.093 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:15:21 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:15:21.833+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:15:22.564 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Dec 16 22:15:22 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-mon-smithi045[80759]: 2023-12-16T22:15:22.199+0000 7f45a1f8f700 -1 mon.smithi045@0(probing) e2 get_health_metrics reporting 36034 slow ops, oldest is monmgrreport(gid 14214, 0 checks, 1 progress events) 2023-12-16T22:15:22.565 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:15:22 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:15:22.091+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:11:07.849373+0000 (oldest deadline 2023-12-16T22:11:27.849373+0000) 2023-12-16T22:15:22.565 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:15:22 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:15:22.091+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:11:07.849373+0000 (oldest deadline 2023-12-16T22:11:27.849373+0000) 2023-12-16T22:15:22.565 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:15:22 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:15:22.091+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:11:07.849373+0000 (oldest deadline 2023-12-16T22:11:27.849373+0000) 2023-12-16T22:15:22.565 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:15:22 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:15:22.093+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:15:23.064 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:15:22 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:15:22.801+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:10:43.026858+0000 (oldest deadline 2023-12-16T22:11:03.026858+0000) 2023-12-16T22:15:23.065 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:15:22 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:15:22.801+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:10:43.026858+0000 (oldest deadline 2023-12-16T22:11:03.026858+0000) 2023-12-16T22:15:23.065 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:15:22 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:15:22.801+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:10:43.026858+0000 (oldest deadline 2023-12-16T22:11:03.026858+0000) 2023-12-16T22:15:23.065 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:15:22 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:15:22.802+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:15:23.066 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:15:22 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:15:22.588+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:10:42.731761+0000 (oldest deadline 2023-12-16T22:11:02.731761+0000) 2023-12-16T22:15:23.066 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:15:22 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:15:22.588+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:10:42.731761+0000 (oldest deadline 2023-12-16T22:11:02.731761+0000) 2023-12-16T22:15:23.066 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:15:22 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:15:22.588+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:10:42.731761+0000 (oldest deadline 2023-12-16T22:11:02.731761+0000) 2023-12-16T22:15:23.066 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:15:22 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:15:22.589+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:15:23.564 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:15:23 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:15:23.124+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:11:07.849373+0000 (oldest deadline 2023-12-16T22:11:27.849373+0000) 2023-12-16T22:15:23.565 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:15:23 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:15:23.124+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:11:07.849373+0000 (oldest deadline 2023-12-16T22:11:27.849373+0000) 2023-12-16T22:15:23.565 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:15:23 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:15:23.124+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:11:07.849373+0000 (oldest deadline 2023-12-16T22:11:27.849373+0000) 2023-12-16T22:15:23.565 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:15:23 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:15:23.125+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:15:24.064 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:15:23 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:15:23.778+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:10:43.026858+0000 (oldest deadline 2023-12-16T22:11:03.026858+0000) 2023-12-16T22:15:24.065 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:15:23 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:15:23.778+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:10:43.026858+0000 (oldest deadline 2023-12-16T22:11:03.026858+0000) 2023-12-16T22:15:24.065 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:15:23 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:15:23.778+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:10:43.026858+0000 (oldest deadline 2023-12-16T22:11:03.026858+0000) 2023-12-16T22:15:24.065 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:15:23 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:15:23.779+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:15:24.065 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:15:23 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:15:23.603+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:10:42.731761+0000 (oldest deadline 2023-12-16T22:11:02.731761+0000) 2023-12-16T22:15:24.065 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:15:23 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:15:23.603+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:10:42.731761+0000 (oldest deadline 2023-12-16T22:11:02.731761+0000) 2023-12-16T22:15:24.066 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:15:23 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:15:23.603+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:10:42.731761+0000 (oldest deadline 2023-12-16T22:11:02.731761+0000) 2023-12-16T22:15:24.066 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:15:23 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:15:23.604+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:15:24.564 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:15:24 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:15:24.076+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:11:07.849373+0000 (oldest deadline 2023-12-16T22:11:27.849373+0000) 2023-12-16T22:15:24.564 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:15:24 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:15:24.076+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:11:07.849373+0000 (oldest deadline 2023-12-16T22:11:27.849373+0000) 2023-12-16T22:15:24.564 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:15:24 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:15:24.076+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:11:07.849373+0000 (oldest deadline 2023-12-16T22:11:27.849373+0000) 2023-12-16T22:15:24.565 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:15:24 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:15:24.077+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:15:25.042 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:15:24 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:15:24.789+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:10:43.026858+0000 (oldest deadline 2023-12-16T22:11:03.026858+0000) 2023-12-16T22:15:25.042 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:15:24 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:15:24.790+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:10:43.026858+0000 (oldest deadline 2023-12-16T22:11:03.026858+0000) 2023-12-16T22:15:25.042 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:15:24 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:15:24.790+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:10:43.026858+0000 (oldest deadline 2023-12-16T22:11:03.026858+0000) 2023-12-16T22:15:25.042 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:15:24 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:15:24.791+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:15:25.042 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:15:24 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:15:24.599+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:10:42.731761+0000 (oldest deadline 2023-12-16T22:11:02.731761+0000) 2023-12-16T22:15:25.043 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:15:24 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:15:24.600+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:10:42.731761+0000 (oldest deadline 2023-12-16T22:11:02.731761+0000) 2023-12-16T22:15:25.043 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:15:24 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:15:24.600+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:10:42.731761+0000 (oldest deadline 2023-12-16T22:11:02.731761+0000) 2023-12-16T22:15:25.043 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:15:24 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:15:24.601+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:15:25.314 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:15:25 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:15:25.040+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:11:07.849373+0000 (oldest deadline 2023-12-16T22:11:27.849373+0000) 2023-12-16T22:15:25.314 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:15:25 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:15:25.040+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:11:07.849373+0000 (oldest deadline 2023-12-16T22:11:27.849373+0000) 2023-12-16T22:15:25.314 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:15:25 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:15:25.040+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:11:07.849373+0000 (oldest deadline 2023-12-16T22:11:27.849373+0000) 2023-12-16T22:15:25.315 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:15:25 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:15:25.041+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:15:26.003 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:15:25 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:15:25.600+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:10:42.731761+0000 (oldest deadline 2023-12-16T22:11:02.731761+0000) 2023-12-16T22:15:26.004 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:15:25 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:15:25.600+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:10:42.731761+0000 (oldest deadline 2023-12-16T22:11:02.731761+0000) 2023-12-16T22:15:26.004 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:15:25 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:15:25.600+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:10:42.731761+0000 (oldest deadline 2023-12-16T22:11:02.731761+0000) 2023-12-16T22:15:26.004 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:15:25 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:15:25.601+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:15:26.004 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:15:25 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:15:25.789+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:10:43.026858+0000 (oldest deadline 2023-12-16T22:11:03.026858+0000) 2023-12-16T22:15:26.004 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:15:25 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:15:25.789+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:10:43.026858+0000 (oldest deadline 2023-12-16T22:11:03.026858+0000) 2023-12-16T22:15:26.005 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:15:25 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:15:25.789+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:10:43.026858+0000 (oldest deadline 2023-12-16T22:11:03.026858+0000) 2023-12-16T22:15:26.005 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:15:25 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:15:25.790+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:15:26.314 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:15:26 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:15:26.002+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:11:07.849373+0000 (oldest deadline 2023-12-16T22:11:27.849373+0000) 2023-12-16T22:15:26.315 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:15:26 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:15:26.002+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:11:07.849373+0000 (oldest deadline 2023-12-16T22:11:27.849373+0000) 2023-12-16T22:15:26.315 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:15:26 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:15:26.002+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:11:07.849373+0000 (oldest deadline 2023-12-16T22:11:27.849373+0000) 2023-12-16T22:15:26.315 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:15:26 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:15:26.003+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:15:27.033 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:15:26 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:15:26.766+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:10:43.026858+0000 (oldest deadline 2023-12-16T22:11:03.026858+0000) 2023-12-16T22:15:27.034 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:15:26 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:15:26.766+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:10:43.026858+0000 (oldest deadline 2023-12-16T22:11:03.026858+0000) 2023-12-16T22:15:27.034 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:15:26 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:15:26.766+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:10:43.026858+0000 (oldest deadline 2023-12-16T22:11:03.026858+0000) 2023-12-16T22:15:27.034 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:15:26 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:15:26.767+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:15:27.034 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:15:26 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:15:26.564+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:10:42.731761+0000 (oldest deadline 2023-12-16T22:11:02.731761+0000) 2023-12-16T22:15:27.035 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:15:26 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:15:26.564+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:10:42.731761+0000 (oldest deadline 2023-12-16T22:11:02.731761+0000) 2023-12-16T22:15:27.035 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:15:26 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:15:26.564+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:10:42.731761+0000 (oldest deadline 2023-12-16T22:11:02.731761+0000) 2023-12-16T22:15:27.035 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:15:26 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:15:26.565+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:15:27.314 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Dec 16 22:15:27 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-mon-smithi045[80759]: 2023-12-16T22:15:27.204+0000 7f45a1f8f700 -1 mon.smithi045@0(probing) e2 get_health_metrics reporting 36126 slow ops, oldest is monmgrreport(gid 14214, 0 checks, 1 progress events) 2023-12-16T22:15:27.315 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:15:27 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:15:27.032+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:11:07.849373+0000 (oldest deadline 2023-12-16T22:11:27.849373+0000) 2023-12-16T22:15:27.315 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:15:27 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:15:27.032+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:11:07.849373+0000 (oldest deadline 2023-12-16T22:11:27.849373+0000) 2023-12-16T22:15:27.315 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:15:27 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:15:27.032+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:11:07.849373+0000 (oldest deadline 2023-12-16T22:11:27.849373+0000) 2023-12-16T22:15:27.315 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:15:27 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:15:27.033+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:15:27.721 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:15:27 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:15:27.524+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:10:42.731761+0000 (oldest deadline 2023-12-16T22:11:02.731761+0000) 2023-12-16T22:15:27.721 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:15:27 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:15:27.524+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:10:42.731761+0000 (oldest deadline 2023-12-16T22:11:02.731761+0000) 2023-12-16T22:15:27.721 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:15:27 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:15:27.524+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:10:42.731761+0000 (oldest deadline 2023-12-16T22:11:02.731761+0000) 2023-12-16T22:15:27.721 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:15:27 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:15:27.525+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:15:27.799 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi169.front.sepia.ceph.com' 2023-12-16T22:15:27.800 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi169.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-12-16T22:15:28.052 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:15:27 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:15:27.720+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:10:43.026858+0000 (oldest deadline 2023-12-16T22:11:03.026858+0000) 2023-12-16T22:15:28.052 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:15:27 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:15:27.720+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:10:43.026858+0000 (oldest deadline 2023-12-16T22:11:03.026858+0000) 2023-12-16T22:15:28.052 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:15:27 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:15:27.720+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:10:43.026858+0000 (oldest deadline 2023-12-16T22:11:03.026858+0000) 2023-12-16T22:15:28.053 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:15:27 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:15:27.721+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:15:28.314 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:15:28 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:15:28.051+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:11:07.849373+0000 (oldest deadline 2023-12-16T22:11:27.849373+0000) 2023-12-16T22:15:28.314 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:15:28 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:15:28.051+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:11:07.849373+0000 (oldest deadline 2023-12-16T22:11:27.849373+0000) 2023-12-16T22:15:28.314 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:15:28 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:15:28.051+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:11:07.849373+0000 (oldest deadline 2023-12-16T22:11:27.849373+0000) 2023-12-16T22:15:28.315 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:15:28 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:15:28.052+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:15:28.814 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:15:28 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:15:28.712+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:10:43.026858+0000 (oldest deadline 2023-12-16T22:11:03.026858+0000) 2023-12-16T22:15:28.815 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:15:28 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:15:28.712+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:10:43.026858+0000 (oldest deadline 2023-12-16T22:11:03.026858+0000) 2023-12-16T22:15:28.815 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:15:28 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:15:28.712+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:10:43.026858+0000 (oldest deadline 2023-12-16T22:11:03.026858+0000) 2023-12-16T22:15:28.815 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:15:28 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:15:28.712+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:15:28.815 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:15:28 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:15:28.514+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:10:42.731761+0000 (oldest deadline 2023-12-16T22:11:02.731761+0000) 2023-12-16T22:15:28.815 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:15:28 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:15:28.514+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:10:42.731761+0000 (oldest deadline 2023-12-16T22:11:02.731761+0000) 2023-12-16T22:15:28.816 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:15:28 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:15:28.514+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:10:42.731761+0000 (oldest deadline 2023-12-16T22:11:02.731761+0000) 2023-12-16T22:15:28.816 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:15:28 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:15:28.516+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:15:29.314 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:15:29 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:15:29.036+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:11:07.849373+0000 (oldest deadline 2023-12-16T22:11:27.849373+0000) 2023-12-16T22:15:29.314 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:15:29 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:15:29.036+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:11:07.849373+0000 (oldest deadline 2023-12-16T22:11:27.849373+0000) 2023-12-16T22:15:29.315 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:15:29 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:15:29.036+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:11:07.849373+0000 (oldest deadline 2023-12-16T22:11:27.849373+0000) 2023-12-16T22:15:29.315 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:15:29 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:15:29.037+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:15:29.814 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:15:29 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:15:29.664+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:10:43.026858+0000 (oldest deadline 2023-12-16T22:11:03.026858+0000) 2023-12-16T22:15:29.815 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:15:29 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:15:29.664+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:10:43.026858+0000 (oldest deadline 2023-12-16T22:11:03.026858+0000) 2023-12-16T22:15:29.815 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:15:29 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:15:29.665+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:10:43.026858+0000 (oldest deadline 2023-12-16T22:11:03.026858+0000) 2023-12-16T22:15:29.815 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:15:29 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:15:29.666+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:15:29.815 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:15:29 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:15:29.550+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:10:42.731761+0000 (oldest deadline 2023-12-16T22:11:02.731761+0000) 2023-12-16T22:15:29.815 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:15:29 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:15:29.550+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:10:42.731761+0000 (oldest deadline 2023-12-16T22:11:02.731761+0000) 2023-12-16T22:15:29.815 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:15:29 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:15:29.550+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:10:42.731761+0000 (oldest deadline 2023-12-16T22:11:02.731761+0000) 2023-12-16T22:15:29.815 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:15:29 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:15:29.551+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:15:30.317 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:15:29 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:15:29.992+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:11:07.849373+0000 (oldest deadline 2023-12-16T22:11:27.849373+0000) 2023-12-16T22:15:30.317 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:15:29 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:15:29.992+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:11:07.849373+0000 (oldest deadline 2023-12-16T22:11:27.849373+0000) 2023-12-16T22:15:30.317 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:15:29 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:15:29.992+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:11:07.849373+0000 (oldest deadline 2023-12-16T22:11:27.849373+0000) 2023-12-16T22:15:30.317 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:15:29 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:15:29.993+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:15:30.714 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:15:30 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:15:30.531+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:10:42.731761+0000 (oldest deadline 2023-12-16T22:11:02.731761+0000) 2023-12-16T22:15:30.714 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:15:30 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:15:30.531+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:10:42.731761+0000 (oldest deadline 2023-12-16T22:11:02.731761+0000) 2023-12-16T22:15:30.714 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:15:30 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:15:30.531+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:10:42.731761+0000 (oldest deadline 2023-12-16T22:11:02.731761+0000) 2023-12-16T22:15:30.714 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:15:30 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:15:30.532+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:15:30.865 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.169 2023-12-16T22:15:30.965 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:15:30 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:15:30.713+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:10:43.026858+0000 (oldest deadline 2023-12-16T22:11:03.026858+0000) 2023-12-16T22:15:30.966 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:15:30 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:15:30.713+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:10:43.026858+0000 (oldest deadline 2023-12-16T22:11:03.026858+0000) 2023-12-16T22:15:30.966 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:15:30 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:15:30.713+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:10:43.026858+0000 (oldest deadline 2023-12-16T22:11:03.026858+0000) 2023-12-16T22:15:30.966 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:15:30 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:15:30.714+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:15:31.314 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:15:30 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:15:30.964+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:11:07.849373+0000 (oldest deadline 2023-12-16T22:11:27.849373+0000) 2023-12-16T22:15:31.314 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:15:30 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:15:30.964+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:11:07.849373+0000 (oldest deadline 2023-12-16T22:11:27.849373+0000) 2023-12-16T22:15:31.314 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:15:30 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:15:30.964+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:11:07.849373+0000 (oldest deadline 2023-12-16T22:11:27.849373+0000) 2023-12-16T22:15:31.315 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:15:30 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:15:30.965+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:15:31.814 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:15:31 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:15:31.738+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:10:43.026858+0000 (oldest deadline 2023-12-16T22:11:03.026858+0000) 2023-12-16T22:15:31.815 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:15:31 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:15:31.738+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:10:43.026858+0000 (oldest deadline 2023-12-16T22:11:03.026858+0000) 2023-12-16T22:15:31.815 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:15:31 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:15:31.738+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:10:43.026858+0000 (oldest deadline 2023-12-16T22:11:03.026858+0000) 2023-12-16T22:15:31.815 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:15:31 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:15:31.739+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:15:31.815 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:15:31 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:15:31.557+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:10:42.731761+0000 (oldest deadline 2023-12-16T22:11:02.731761+0000) 2023-12-16T22:15:31.815 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:15:31 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:15:31.557+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:10:42.731761+0000 (oldest deadline 2023-12-16T22:11:02.731761+0000) 2023-12-16T22:15:31.816 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:15:31 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:15:31.557+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:10:42.731761+0000 (oldest deadline 2023-12-16T22:11:02.731761+0000) 2023-12-16T22:15:31.816 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:15:31 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:15:31.558+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:15:32.221 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Dec 16 22:15:32 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-mon-smithi045[80759]: 2023-12-16T22:15:32.209+0000 7f45a1f8f700 -1 mon.smithi045@0(probing) e2 get_health_metrics reporting 36217 slow ops, oldest is monmgrreport(gid 14214, 0 checks, 1 progress events) 2023-12-16T22:15:32.221 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:15:31 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:15:31.968+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:11:07.849373+0000 (oldest deadline 2023-12-16T22:11:27.849373+0000) 2023-12-16T22:15:32.222 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:15:31 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:15:31.968+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:11:07.849373+0000 (oldest deadline 2023-12-16T22:11:27.849373+0000) 2023-12-16T22:15:32.222 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:15:31 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:15:31.968+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:11:07.849373+0000 (oldest deadline 2023-12-16T22:11:27.849373+0000) 2023-12-16T22:15:32.222 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:15:31 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:15:31.969+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:15:32.814 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:15:32 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:15:32.550+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:10:42.731761+0000 (oldest deadline 2023-12-16T22:11:02.731761+0000) 2023-12-16T22:15:32.814 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:15:32 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:15:32.550+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:10:42.731761+0000 (oldest deadline 2023-12-16T22:11:02.731761+0000) 2023-12-16T22:15:32.815 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:15:32 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:15:32.550+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:10:42.731761+0000 (oldest deadline 2023-12-16T22:11:02.731761+0000) 2023-12-16T22:15:32.815 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:15:32 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:15:32.551+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:15:32.815 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:15:32 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:15:32.750+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:10:43.026858+0000 (oldest deadline 2023-12-16T22:11:03.026858+0000) 2023-12-16T22:15:32.815 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:15:32 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:15:32.750+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:10:43.026858+0000 (oldest deadline 2023-12-16T22:11:03.026858+0000) 2023-12-16T22:15:32.815 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:15:32 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:15:32.750+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:10:43.026858+0000 (oldest deadline 2023-12-16T22:11:03.026858+0000) 2023-12-16T22:15:32.815 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:15:32 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:15:32.750+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:15:33.314 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:15:32 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:15:32.992+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:11:07.849373+0000 (oldest deadline 2023-12-16T22:11:27.849373+0000) 2023-12-16T22:15:33.314 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:15:32 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:15:32.992+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:11:07.849373+0000 (oldest deadline 2023-12-16T22:11:27.849373+0000) 2023-12-16T22:15:33.315 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:15:32 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:15:32.992+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:11:07.849373+0000 (oldest deadline 2023-12-16T22:11:27.849373+0000) 2023-12-16T22:15:33.315 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:15:32 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:15:32.992+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:15:33.655 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:15:33 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:15:33.516+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:10:42.731761+0000 (oldest deadline 2023-12-16T22:11:02.731761+0000) 2023-12-16T22:15:33.655 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:15:33 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:15:33.516+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:10:42.731761+0000 (oldest deadline 2023-12-16T22:11:02.731761+0000) 2023-12-16T22:15:33.655 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:15:33 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:15:33.516+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:10:42.731761+0000 (oldest deadline 2023-12-16T22:11:02.731761+0000) 2023-12-16T22:15:33.655 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:15:33 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:15:33.516+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:15:33.866 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi169.front.sepia.ceph.com' 2023-12-16T22:15:33.867 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi169.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-12-16T22:15:33.937 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.169 2023-12-16T22:15:33.938 WARNING:teuthology.contextutil:'reconnect to {self.shortname}' reached maximum tries (5) after waiting for 30 seconds 2023-12-16T22:15:33.938 INFO:tasks.cephadm:Checking cluster log for badness... 2023-12-16T22:15:33.939 DEBUG:teuthology.orchestra.run.smithi045:> sudo egrep '\[ERR\]|\[WRN\]|\[SEC\]' /var/log/ceph/88ff363c-9c5a-11ee-95a5-87774f69a715/ceph.log | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | egrep -v 'overall HEALTH_' | egrep -v '\(FS_DEGRADED\)' | egrep -v '\(MDS_FAILED\)' | egrep -v '\(MDS_DEGRADED\)' | egrep -v '\(FS_WITH_FAILED_MDS\)' | egrep -v '\(MDS_DAMAGE\)' | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | egrep -v '\(FS_INLINE_DATA_DEPRECATED\)' | egrep -v '\(POOL_APP_NOT_ENABLED\)' | egrep -v 'overall HEALTH_' | egrep -v '\(OSD_DOWN\)' | egrep -v '\(OSD_' | egrep -v 'but it is still running' | egrep -v 'is not responding' | head -n 1 2023-12-16T22:15:33.972 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:15:33 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:15:33.752+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:10:43.026858+0000 (oldest deadline 2023-12-16T22:11:03.026858+0000) 2023-12-16T22:15:33.972 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:15:33 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:15:33.752+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:10:43.026858+0000 (oldest deadline 2023-12-16T22:11:03.026858+0000) 2023-12-16T22:15:33.972 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:15:33 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:15:33.752+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:10:43.026858+0000 (oldest deadline 2023-12-16T22:11:03.026858+0000) 2023-12-16T22:15:33.972 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:15:33 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:15:33.753+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:15:33.975 INFO:teuthology.orchestra.run.smithi045.stderr:grep: /var/log/ceph/88ff363c-9c5a-11ee-95a5-87774f69a715/ceph.log: No such file or directory 2023-12-16T22:15:33.977 INFO:tasks.cephadm:Compressing logs... 2023-12-16T22:15:33.978 DEBUG:teuthology.orchestra.run.smithi045:> time sudo find /var/log/ceph /var/log/rbd-target-api -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2023-12-16T22:15:34.020 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi169.front.sepia.ceph.com' 2023-12-16T22:15:34.020 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi169.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-12-16T22:15:34.051 INFO:teuthology.orchestra.run.smithi045.stderr:find: ‘/var/log/rbd-target-api’gzip -5 --verbose -- /var/log/ceph/cephadm.log 2023-12-16T22:15:34.051 INFO:teuthology.orchestra.run.smithi045.stderr:: No such file or directory 2023-12-16T22:15:34.051 INFO:teuthology.orchestra.run.smithi045.stderr:gzip -5 --verbose -- /var/log/ceph/88ff363c-9c5a-11ee-95a5-87774f69a715/ceph-osd.1.log 2023-12-16T22:15:34.052 INFO:teuthology.orchestra.run.smithi045.stderr:gzip -5 --verbose -- /var/log/ceph/88ff363c-9c5a-11ee-95a5-87774f69a715/ceph-mds.cephfs.smithi045.hkjgzw.log 2023-12-16T22:15:34.053 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/cephadm.log: gzip -5 --verbose -- /var/log/ceph/88ff363c-9c5a-11ee-95a5-87774f69a715/ceph-mon.smithi045.log 2023-12-16T22:15:34.053 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/88ff363c-9c5a-11ee-95a5-87774f69a715/ceph-osd.1.log: /var/log/ceph/88ff363c-9c5a-11ee-95a5-87774f69a715/ceph-mds.cephfs.smithi045.hkjgzw.log: gzip -5 --verbose -- /var/log/ceph/88ff363c-9c5a-11ee-95a5-87774f69a715/ceph-mds.cephfs.smithi045.fgsbgq.log 2023-12-16T22:15:34.054 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/88ff363c-9c5a-11ee-95a5-87774f69a715/ceph-mon.smithi045.log: /var/log/ceph/88ff363c-9c5a-11ee-95a5-87774f69a715/ceph-mds.cephfs.smithi045.fgsbgq.log: gzip -5 --verbose -- /var/log/ceph/88ff363c-9c5a-11ee-95a5-87774f69a715/ceph-osd.0.log 2023-12-16T22:15:34.054 INFO:teuthology.orchestra.run.smithi045.stderr:gzip -5 --verbose -- /var/log/ceph/88ff363c-9c5a-11ee-95a5-87774f69a715/ceph-osd.2.log 2023-12-16T22:15:34.055 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/88ff363c-9c5a-11ee-95a5-87774f69a715/ceph-osd.0.log: gzip -5 --verbose -- /var/log/ceph/88ff363c-9c5a-11ee-95a5-87774f69a715/ceph-volume.log 2023-12-16T22:15:34.060 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/88ff363c-9c5a-11ee-95a5-87774f69a715/ceph-osd.2.log: 93.3% -- replaced with /var/log/ceph/cephadm.log.gz 2023-12-16T22:15:34.061 INFO:teuthology.orchestra.run.smithi045.stderr:gzip -5 --verbose -- /var/log/ceph/88ff363c-9c5a-11ee-95a5-87774f69a715/ceph-mgr.smithi045.nhtmwc.log 2023-12-16T22:15:34.068 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/88ff363c-9c5a-11ee-95a5-87774f69a715/ceph-volume.log: /var/log/ceph/88ff363c-9c5a-11ee-95a5-87774f69a715/ceph-mgr.smithi045.nhtmwc.log: 90.8% -- replaced with /var/log/ceph/88ff363c-9c5a-11ee-95a5-87774f69a715/ceph-volume.log.gz 2023-12-16T22:15:34.093 INFO:teuthology.orchestra.run.smithi045.stderr: 95.1% -- replaced with /var/log/ceph/88ff363c-9c5a-11ee-95a5-87774f69a715/ceph-mds.cephfs.smithi045.fgsbgq.log.gz 2023-12-16T22:15:34.314 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:15:33 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:15:33.981+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:11:07.849373+0000 (oldest deadline 2023-12-16T22:11:27.849373+0000) 2023-12-16T22:15:34.315 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:15:33 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:15:33.981+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:11:07.849373+0000 (oldest deadline 2023-12-16T22:11:27.849373+0000) 2023-12-16T22:15:34.315 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:15:33 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:15:33.981+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:11:07.849373+0000 (oldest deadline 2023-12-16T22:11:27.849373+0000) 2023-12-16T22:15:34.315 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:15:33 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:15:33.982+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:15:34.882 INFO:teuthology.orchestra.run.smithi045.stderr:gzip: /var/log/ceph/88ff363c-9c5a-11ee-95a5-87774f69a715/ceph-mgr.smithi045.nhtmwc.log: file size changed while zipping 2023-12-16T22:15:34.882 INFO:teuthology.orchestra.run.smithi045.stderr: 90.0% -- replaced with /var/log/ceph/88ff363c-9c5a-11ee-95a5-87774f69a715/ceph-mgr.smithi045.nhtmwc.log.gz 2023-12-16T22:15:35.021 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:15:34 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:15:34.711+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:10:43.026858+0000 (oldest deadline 2023-12-16T22:11:03.026858+0000) 2023-12-16T22:15:35.021 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:15:34 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:15:34.711+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:10:43.026858+0000 (oldest deadline 2023-12-16T22:11:03.026858+0000) 2023-12-16T22:15:35.022 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:15:34 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:15:34.711+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:10:43.026858+0000 (oldest deadline 2023-12-16T22:11:03.026858+0000) 2023-12-16T22:15:35.022 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:15:34 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:15:34.712+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:15:35.022 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:15:34 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:15:34.562+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:10:42.731761+0000 (oldest deadline 2023-12-16T22:11:02.731761+0000) 2023-12-16T22:15:35.022 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:15:34 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:15:34.562+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:10:42.731761+0000 (oldest deadline 2023-12-16T22:11:02.731761+0000) 2023-12-16T22:15:35.022 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:15:34 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:15:34.562+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:10:42.731761+0000 (oldest deadline 2023-12-16T22:11:02.731761+0000) 2023-12-16T22:15:35.022 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:15:34 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:15:34.563+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:15:35.314 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:15:35 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:15:35.018+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:11:07.849373+0000 (oldest deadline 2023-12-16T22:11:27.849373+0000) 2023-12-16T22:15:35.314 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:15:35 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:15:35.018+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:11:07.849373+0000 (oldest deadline 2023-12-16T22:11:27.849373+0000) 2023-12-16T22:15:35.315 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:15:35 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:15:35.018+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:11:07.849373+0000 (oldest deadline 2023-12-16T22:11:27.849373+0000) 2023-12-16T22:15:35.315 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:15:35 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:15:35.019+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:15:35.454 INFO:teuthology.orchestra.run.smithi045.stderr:gzip: /var/log/ceph/88ff363c-9c5a-11ee-95a5-87774f69a715/ceph-mon.smithi045.log: file size changed while zipping 2023-12-16T22:15:35.454 INFO:teuthology.orchestra.run.smithi045.stderr: 92.7% -- replaced with /var/log/ceph/88ff363c-9c5a-11ee-95a5-87774f69a715/ceph-mon.smithi045.log.gz 2023-12-16T22:15:36.042 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:15:35 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:15:35.758+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:10:43.026858+0000 (oldest deadline 2023-12-16T22:11:03.026858+0000) 2023-12-16T22:15:36.042 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:15:35 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:15:35.758+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:10:43.026858+0000 (oldest deadline 2023-12-16T22:11:03.026858+0000) 2023-12-16T22:15:36.042 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:15:35 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:15:35.758+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:10:43.026858+0000 (oldest deadline 2023-12-16T22:11:03.026858+0000) 2023-12-16T22:15:36.042 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:15:35 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:15:35.759+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:15:36.043 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:15:35 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:15:35.602+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:10:42.731761+0000 (oldest deadline 2023-12-16T22:11:02.731761+0000) 2023-12-16T22:15:36.043 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:15:35 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:15:35.602+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:10:42.731761+0000 (oldest deadline 2023-12-16T22:11:02.731761+0000) 2023-12-16T22:15:36.043 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:15:35 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:15:35.602+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:10:42.731761+0000 (oldest deadline 2023-12-16T22:11:02.731761+0000) 2023-12-16T22:15:36.043 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:15:35 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:15:35.603+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:15:36.315 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:15:36 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:15:36.041+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:11:07.849373+0000 (oldest deadline 2023-12-16T22:11:27.849373+0000) 2023-12-16T22:15:36.315 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:15:36 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:15:36.041+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:11:07.849373+0000 (oldest deadline 2023-12-16T22:11:27.849373+0000) 2023-12-16T22:15:36.316 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:15:36 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:15:36.041+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:11:07.849373+0000 (oldest deadline 2023-12-16T22:11:27.849373+0000) 2023-12-16T22:15:36.316 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:15:36 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:15:36.042+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:15:36.718 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:15:36 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:15:36.626+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:10:42.731761+0000 (oldest deadline 2023-12-16T22:11:02.731761+0000) 2023-12-16T22:15:36.718 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:15:36 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:15:36.626+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:10:42.731761+0000 (oldest deadline 2023-12-16T22:11:02.731761+0000) 2023-12-16T22:15:36.718 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:15:36 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:15:36.626+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:10:42.731761+0000 (oldest deadline 2023-12-16T22:11:02.731761+0000) 2023-12-16T22:15:36.718 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:15:36 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:15:36.627+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:15:36.720 INFO:teuthology.orchestra.run.smithi045.stderr: 95.2% -- replaced with /var/log/ceph/88ff363c-9c5a-11ee-95a5-87774f69a715/ceph-mds.cephfs.smithi045.hkjgzw.log.gz 2023-12-16T22:15:37.064 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:15:36 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:15:36.716+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:10:43.026858+0000 (oldest deadline 2023-12-16T22:11:03.026858+0000) 2023-12-16T22:15:37.064 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:15:36 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:15:36.716+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:10:43.026858+0000 (oldest deadline 2023-12-16T22:11:03.026858+0000) 2023-12-16T22:15:37.065 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:15:36 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:15:36.716+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:10:43.026858+0000 (oldest deadline 2023-12-16T22:11:03.026858+0000) 2023-12-16T22:15:37.065 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:15:36 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:15:36.717+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:15:37.077 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.169 2023-12-16T22:15:37.346 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:15:37 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:15:37.072+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:11:07.849373+0000 (oldest deadline 2023-12-16T22:11:27.849373+0000) 2023-12-16T22:15:37.346 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:15:37 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:15:37.072+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:11:07.849373+0000 (oldest deadline 2023-12-16T22:11:27.849373+0000) 2023-12-16T22:15:37.347 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:15:37 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:15:37.072+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:11:07.849373+0000 (oldest deadline 2023-12-16T22:11:27.849373+0000) 2023-12-16T22:15:37.347 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:15:37 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:15:37.073+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:15:37.347 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Dec 16 22:15:37 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-mon-smithi045[80759]: 2023-12-16T22:15:37.213+0000 7f45a1f8f700 -1 mon.smithi045@0(probing) e2 get_health_metrics reporting 36309 slow ops, oldest is monmgrreport(gid 14214, 0 checks, 1 progress events) 2023-12-16T22:15:38.064 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:15:37 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:15:37.609+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:10:42.731761+0000 (oldest deadline 2023-12-16T22:11:02.731761+0000) 2023-12-16T22:15:38.064 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:15:37 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:15:37.609+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:10:42.731761+0000 (oldest deadline 2023-12-16T22:11:02.731761+0000) 2023-12-16T22:15:38.065 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:15:37 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:15:37.609+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:10:42.731761+0000 (oldest deadline 2023-12-16T22:11:02.731761+0000) 2023-12-16T22:15:38.065 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:15:37 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:15:37.610+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:15:38.065 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:15:37 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:15:37.746+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:10:43.026858+0000 (oldest deadline 2023-12-16T22:11:03.026858+0000) 2023-12-16T22:15:38.065 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:15:37 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:15:37.746+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:10:43.026858+0000 (oldest deadline 2023-12-16T22:11:03.026858+0000) 2023-12-16T22:15:38.065 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:15:37 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:15:37.746+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:10:43.026858+0000 (oldest deadline 2023-12-16T22:11:03.026858+0000) 2023-12-16T22:15:38.065 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:15:37 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:15:37.747+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:15:38.564 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:15:38 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:15:38.079+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:11:07.849373+0000 (oldest deadline 2023-12-16T22:11:27.849373+0000) 2023-12-16T22:15:38.564 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:15:38 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:15:38.079+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:11:07.849373+0000 (oldest deadline 2023-12-16T22:11:27.849373+0000) 2023-12-16T22:15:38.565 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:15:38 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:15:38.079+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:11:07.849373+0000 (oldest deadline 2023-12-16T22:11:27.849373+0000) 2023-12-16T22:15:38.565 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:15:38 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:15:38.080+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:15:39.064 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:15:38 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:15:38.627+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:10:42.731761+0000 (oldest deadline 2023-12-16T22:11:02.731761+0000) 2023-12-16T22:15:39.064 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:15:38 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:15:38.627+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:10:42.731761+0000 (oldest deadline 2023-12-16T22:11:02.731761+0000) 2023-12-16T22:15:39.065 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:15:38 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:15:38.627+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:10:42.731761+0000 (oldest deadline 2023-12-16T22:11:02.731761+0000) 2023-12-16T22:15:39.065 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:15:38 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:15:38.628+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:15:39.065 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:15:38 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:15:38.780+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:10:43.026858+0000 (oldest deadline 2023-12-16T22:11:03.026858+0000) 2023-12-16T22:15:39.065 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:15:38 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:15:38.780+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:10:43.026858+0000 (oldest deadline 2023-12-16T22:11:03.026858+0000) 2023-12-16T22:15:39.065 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:15:38 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:15:38.780+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:10:43.026858+0000 (oldest deadline 2023-12-16T22:11:03.026858+0000) 2023-12-16T22:15:39.065 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:15:38 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:15:38.781+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:15:39.564 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:15:39 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:15:39.078+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:11:07.849373+0000 (oldest deadline 2023-12-16T22:11:27.849373+0000) 2023-12-16T22:15:39.564 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:15:39 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:15:39.078+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:11:07.849373+0000 (oldest deadline 2023-12-16T22:11:27.849373+0000) 2023-12-16T22:15:39.565 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:15:39 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:15:39.078+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:11:07.849373+0000 (oldest deadline 2023-12-16T22:11:27.849373+0000) 2023-12-16T22:15:39.565 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:15:39 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:15:39.079+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:15:39.654 INFO:teuthology.orchestra.run.smithi045.stderr:gzip: /var/log/ceph/88ff363c-9c5a-11ee-95a5-87774f69a715/ceph-osd.2.log: file size changed while zipping 2023-12-16T22:15:39.654 INFO:teuthology.orchestra.run.smithi045.stderr: 94.9% -- replaced with /var/log/ceph/88ff363c-9c5a-11ee-95a5-87774f69a715/ceph-osd.2.log.gz 2023-12-16T22:15:40.064 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:15:39 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:15:39.789+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:10:43.026858+0000 (oldest deadline 2023-12-16T22:11:03.026858+0000) 2023-12-16T22:15:40.065 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:15:39 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:15:39.789+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:10:43.026858+0000 (oldest deadline 2023-12-16T22:11:03.026858+0000) 2023-12-16T22:15:40.065 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:15:39 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:15:39.789+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:10:43.026858+0000 (oldest deadline 2023-12-16T22:11:03.026858+0000) 2023-12-16T22:15:40.065 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:15:39 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:15:39.790+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:15:40.065 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:15:39 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:15:39.648+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:10:42.731761+0000 (oldest deadline 2023-12-16T22:11:02.731761+0000) 2023-12-16T22:15:40.065 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:15:39 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:15:39.648+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:10:42.731761+0000 (oldest deadline 2023-12-16T22:11:02.731761+0000) 2023-12-16T22:15:40.065 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:15:39 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:15:39.648+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:10:42.731761+0000 (oldest deadline 2023-12-16T22:11:02.731761+0000) 2023-12-16T22:15:40.065 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:15:39 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:15:39.649+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:15:40.130 INFO:teuthology.orchestra.run.smithi045.stderr:gzip: /var/log/ceph/88ff363c-9c5a-11ee-95a5-87774f69a715/ceph-osd.0.log: file size changed while zipping 2023-12-16T22:15:40.130 INFO:teuthology.orchestra.run.smithi045.stderr: 94.9% -- replaced with /var/log/ceph/88ff363c-9c5a-11ee-95a5-87774f69a715/ceph-osd.0.log.gz 2023-12-16T22:15:40.292 INFO:teuthology.orchestra.run.smithi045.stderr:gzip: /var/log/ceph/88ff363c-9c5a-11ee-95a5-87774f69a715/ceph-osd.1.log: file size changed while zipping 2023-12-16T22:15:40.292 INFO:teuthology.orchestra.run.smithi045.stderr: 94.9% -- replaced with /var/log/ceph/88ff363c-9c5a-11ee-95a5-87774f69a715/ceph-osd.1.log.gz 2023-12-16T22:15:40.294 INFO:teuthology.orchestra.run.smithi045.stderr: 2023-12-16T22:15:40.294 INFO:teuthology.orchestra.run.smithi045.stderr:real 0m6.261s 2023-12-16T22:15:40.294 INFO:teuthology.orchestra.run.smithi045.stderr:user 0m22.332s 2023-12-16T22:15:40.295 INFO:teuthology.orchestra.run.smithi045.stderr:sys 0m0.483s 2023-12-16T22:15:40.564 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:15:40 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:15:40.116+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:11:07.849373+0000 (oldest deadline 2023-12-16T22:11:27.849373+0000) 2023-12-16T22:15:40.564 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:15:40 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:15:40.116+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:11:07.849373+0000 (oldest deadline 2023-12-16T22:11:27.849373+0000) 2023-12-16T22:15:40.564 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:15:40 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:15:40.116+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:11:07.849373+0000 (oldest deadline 2023-12-16T22:11:27.849373+0000) 2023-12-16T22:15:40.565 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:15:40 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:15:40.117+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:15:41.064 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:15:40 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:15:40.745+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:10:43.026858+0000 (oldest deadline 2023-12-16T22:11:03.026858+0000) 2023-12-16T22:15:41.065 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:15:40 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:15:40.745+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:10:43.026858+0000 (oldest deadline 2023-12-16T22:11:03.026858+0000) 2023-12-16T22:15:41.065 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:15:40 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:15:40.745+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:10:43.026858+0000 (oldest deadline 2023-12-16T22:11:03.026858+0000) 2023-12-16T22:15:41.065 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:15:40 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:15:40.746+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:15:41.065 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:15:40 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:15:40.681+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:10:42.731761+0000 (oldest deadline 2023-12-16T22:11:02.731761+0000) 2023-12-16T22:15:41.065 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:15:40 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:15:40.681+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:10:42.731761+0000 (oldest deadline 2023-12-16T22:11:02.731761+0000) 2023-12-16T22:15:41.066 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:15:40 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:15:40.681+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:10:42.731761+0000 (oldest deadline 2023-12-16T22:11:02.731761+0000) 2023-12-16T22:15:41.066 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:15:40 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:15:40.682+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:15:41.564 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:15:41 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:15:41.107+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:11:07.849373+0000 (oldest deadline 2023-12-16T22:11:27.849373+0000) 2023-12-16T22:15:41.564 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:15:41 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:15:41.107+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:11:07.849373+0000 (oldest deadline 2023-12-16T22:11:27.849373+0000) 2023-12-16T22:15:41.564 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:15:41 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:15:41.107+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:11:07.849373+0000 (oldest deadline 2023-12-16T22:11:27.849373+0000) 2023-12-16T22:15:41.565 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:15:41 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:15:41.108+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:15:42.064 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:15:41 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:15:41.715+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:10:43.026858+0000 (oldest deadline 2023-12-16T22:11:03.026858+0000) 2023-12-16T22:15:42.065 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:15:41 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:15:41.715+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:10:43.026858+0000 (oldest deadline 2023-12-16T22:11:03.026858+0000) 2023-12-16T22:15:42.065 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:15:41 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:15:41.715+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:10:43.026858+0000 (oldest deadline 2023-12-16T22:11:03.026858+0000) 2023-12-16T22:15:42.065 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:15:41 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:15:41.716+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:15:42.065 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:15:41 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:15:41.663+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:10:42.731761+0000 (oldest deadline 2023-12-16T22:11:02.731761+0000) 2023-12-16T22:15:42.066 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:15:41 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:15:41.663+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:10:42.731761+0000 (oldest deadline 2023-12-16T22:11:02.731761+0000) 2023-12-16T22:15:42.066 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:15:41 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:15:41.663+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:10:42.731761+0000 (oldest deadline 2023-12-16T22:11:02.731761+0000) 2023-12-16T22:15:42.066 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:15:41 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:15:41.663+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:15:42.345 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:15:42 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:15:42.084+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:11:07.849373+0000 (oldest deadline 2023-12-16T22:11:27.849373+0000) 2023-12-16T22:15:42.345 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:15:42 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:15:42.084+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:11:07.849373+0000 (oldest deadline 2023-12-16T22:11:27.849373+0000) 2023-12-16T22:15:42.346 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:15:42 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:15:42.084+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:11:07.849373+0000 (oldest deadline 2023-12-16T22:11:27.849373+0000) 2023-12-16T22:15:42.346 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:15:42 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:15:42.085+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:15:42.346 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Dec 16 22:15:42 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-mon-smithi045[80759]: 2023-12-16T22:15:42.218+0000 7f45a1f8f700 -1 mon.smithi045@0(probing) e2 get_health_metrics reporting 36399 slow ops, oldest is monmgrreport(gid 14214, 0 checks, 1 progress events) 2023-12-16T22:15:43.057 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:15:42 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:15:42.725+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:10:43.026858+0000 (oldest deadline 2023-12-16T22:11:03.026858+0000) 2023-12-16T22:15:43.057 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:15:42 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:15:42.725+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:10:43.026858+0000 (oldest deadline 2023-12-16T22:11:03.026858+0000) 2023-12-16T22:15:43.058 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:15:42 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:15:42.725+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:10:43.026858+0000 (oldest deadline 2023-12-16T22:11:03.026858+0000) 2023-12-16T22:15:43.058 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:15:42 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:15:42.726+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:15:43.058 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:15:42 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:15:42.668+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:10:42.731761+0000 (oldest deadline 2023-12-16T22:11:02.731761+0000) 2023-12-16T22:15:43.059 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:15:42 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:15:42.668+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:10:42.731761+0000 (oldest deadline 2023-12-16T22:11:02.731761+0000) 2023-12-16T22:15:43.059 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:15:42 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:15:42.668+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:10:42.731761+0000 (oldest deadline 2023-12-16T22:11:02.731761+0000) 2023-12-16T22:15:43.059 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:15:42 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:15:42.669+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:15:43.314 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:15:43 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:15:43.056+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:11:07.849373+0000 (oldest deadline 2023-12-16T22:11:27.849373+0000) 2023-12-16T22:15:43.314 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:15:43 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:15:43.056+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:11:07.849373+0000 (oldest deadline 2023-12-16T22:11:27.849373+0000) 2023-12-16T22:15:43.314 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:15:43 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:15:43.056+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:11:07.849373+0000 (oldest deadline 2023-12-16T22:11:27.849373+0000) 2023-12-16T22:15:43.315 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:15:43 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:15:43.057+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:15:44.064 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:15:43 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:15:43.765+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:10:43.026858+0000 (oldest deadline 2023-12-16T22:11:03.026858+0000) 2023-12-16T22:15:44.065 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:15:43 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:15:43.765+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:10:43.026858+0000 (oldest deadline 2023-12-16T22:11:03.026858+0000) 2023-12-16T22:15:44.065 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:15:43 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:15:43.765+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:10:43.026858+0000 (oldest deadline 2023-12-16T22:11:03.026858+0000) 2023-12-16T22:15:44.065 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:15:43 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:15:43.766+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:15:44.065 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:15:43 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:15:43.695+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:10:42.731761+0000 (oldest deadline 2023-12-16T22:11:02.731761+0000) 2023-12-16T22:15:44.065 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:15:43 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:15:43.695+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:10:42.731761+0000 (oldest deadline 2023-12-16T22:11:02.731761+0000) 2023-12-16T22:15:44.065 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:15:43 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:15:43.695+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:10:42.731761+0000 (oldest deadline 2023-12-16T22:11:02.731761+0000) 2023-12-16T22:15:44.065 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:15:43 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:15:43.696+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:15:44.564 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:15:44 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:15:44.066+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:11:07.849373+0000 (oldest deadline 2023-12-16T22:11:27.849373+0000) 2023-12-16T22:15:44.564 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:15:44 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:15:44.066+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:11:07.849373+0000 (oldest deadline 2023-12-16T22:11:27.849373+0000) 2023-12-16T22:15:44.564 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:15:44 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:15:44.066+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:11:07.849373+0000 (oldest deadline 2023-12-16T22:11:27.849373+0000) 2023-12-16T22:15:44.565 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:15:44 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:15:44.067+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:15:45.064 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:15:44 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:15:44.802+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:10:43.026858+0000 (oldest deadline 2023-12-16T22:11:03.026858+0000) 2023-12-16T22:15:45.065 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:15:44 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:15:44.802+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:10:43.026858+0000 (oldest deadline 2023-12-16T22:11:03.026858+0000) 2023-12-16T22:15:45.065 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:15:44 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:15:44.802+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:10:43.026858+0000 (oldest deadline 2023-12-16T22:11:03.026858+0000) 2023-12-16T22:15:45.065 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:15:44 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:15:44.803+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:15:45.065 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:15:44 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:15:44.673+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:10:42.731761+0000 (oldest deadline 2023-12-16T22:11:02.731761+0000) 2023-12-16T22:15:45.065 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:15:44 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:15:44.673+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:10:42.731761+0000 (oldest deadline 2023-12-16T22:11:02.731761+0000) 2023-12-16T22:15:45.066 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:15:44 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:15:44.673+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:10:42.731761+0000 (oldest deadline 2023-12-16T22:11:02.731761+0000) 2023-12-16T22:15:45.066 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:15:44 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:15:44.675+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:15:45.564 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:15:45 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:15:45.075+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:11:07.849373+0000 (oldest deadline 2023-12-16T22:11:27.849373+0000) 2023-12-16T22:15:45.564 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:15:45 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:15:45.075+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:11:07.849373+0000 (oldest deadline 2023-12-16T22:11:27.849373+0000) 2023-12-16T22:15:45.565 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:15:45 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:15:45.075+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:11:07.849373+0000 (oldest deadline 2023-12-16T22:11:27.849373+0000) 2023-12-16T22:15:45.565 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:15:45 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:15:45.076+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:15:46.056 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:15:45 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:15:45.761+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:10:43.026858+0000 (oldest deadline 2023-12-16T22:11:03.026858+0000) 2023-12-16T22:15:46.057 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:15:45 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:15:45.761+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:10:43.026858+0000 (oldest deadline 2023-12-16T22:11:03.026858+0000) 2023-12-16T22:15:46.057 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:15:45 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:15:45.761+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:10:43.026858+0000 (oldest deadline 2023-12-16T22:11:03.026858+0000) 2023-12-16T22:15:46.057 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:15:45 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:15:45.762+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:15:46.057 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:15:45 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:15:45.721+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:10:42.731761+0000 (oldest deadline 2023-12-16T22:11:02.731761+0000) 2023-12-16T22:15:46.058 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:15:45 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:15:45.721+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:10:42.731761+0000 (oldest deadline 2023-12-16T22:11:02.731761+0000) 2023-12-16T22:15:46.058 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:15:45 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:15:45.721+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:10:42.731761+0000 (oldest deadline 2023-12-16T22:11:02.731761+0000) 2023-12-16T22:15:46.058 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:15:45 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:15:45.721+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:15:46.078 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi169.front.sepia.ceph.com' 2023-12-16T22:15:46.079 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi169.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-12-16T22:15:46.314 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:15:46 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:15:46.055+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:11:07.849373+0000 (oldest deadline 2023-12-16T22:11:27.849373+0000) 2023-12-16T22:15:46.315 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:15:46 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:15:46.055+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:11:07.849373+0000 (oldest deadline 2023-12-16T22:11:27.849373+0000) 2023-12-16T22:15:46.315 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:15:46 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:15:46.055+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:11:07.849373+0000 (oldest deadline 2023-12-16T22:11:27.849373+0000) 2023-12-16T22:15:46.315 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:15:46 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:15:46.056+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:15:47.064 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:15:46 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:15:46.701+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:10:42.731761+0000 (oldest deadline 2023-12-16T22:11:02.731761+0000) 2023-12-16T22:15:47.064 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:15:46 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:15:46.701+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:10:42.731761+0000 (oldest deadline 2023-12-16T22:11:02.731761+0000) 2023-12-16T22:15:47.065 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:15:46 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:15:46.701+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:10:42.731761+0000 (oldest deadline 2023-12-16T22:11:02.731761+0000) 2023-12-16T22:15:47.065 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:15:46 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:15:46.702+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:15:47.065 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:15:46 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:15:46.810+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:10:43.026858+0000 (oldest deadline 2023-12-16T22:11:03.026858+0000) 2023-12-16T22:15:47.065 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:15:46 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:15:46.810+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:10:43.026858+0000 (oldest deadline 2023-12-16T22:11:03.026858+0000) 2023-12-16T22:15:47.065 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:15:46 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:15:46.810+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:10:43.026858+0000 (oldest deadline 2023-12-16T22:11:03.026858+0000) 2023-12-16T22:15:47.066 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:15:46 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:15:46.812+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:15:47.350 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Dec 16 22:15:47 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-mon-smithi045[80759]: 2023-12-16T22:15:47.222+0000 7f45a1f8f700 -1 mon.smithi045@0(probing) e2 get_health_metrics reporting 36490 slow ops, oldest is monmgrreport(gid 14214, 0 checks, 1 progress events) 2023-12-16T22:15:47.351 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:15:47 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:15:47.064+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:11:07.849373+0000 (oldest deadline 2023-12-16T22:11:27.849373+0000) 2023-12-16T22:15:47.351 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:15:47 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:15:47.064+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:11:07.849373+0000 (oldest deadline 2023-12-16T22:11:27.849373+0000) 2023-12-16T22:15:47.351 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:15:47 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:15:47.064+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:11:07.849373+0000 (oldest deadline 2023-12-16T22:11:27.849373+0000) 2023-12-16T22:15:47.352 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:15:47 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:15:47.065+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:15:48.064 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:15:47 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:15:47.685+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:10:42.731761+0000 (oldest deadline 2023-12-16T22:11:02.731761+0000) 2023-12-16T22:15:48.065 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:15:47 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:15:47.685+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:10:42.731761+0000 (oldest deadline 2023-12-16T22:11:02.731761+0000) 2023-12-16T22:15:48.065 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:15:47 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:15:47.685+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:10:42.731761+0000 (oldest deadline 2023-12-16T22:11:02.731761+0000) 2023-12-16T22:15:48.065 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:15:47 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:15:47.686+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:15:48.065 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:15:47 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:15:47.833+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:10:43.026858+0000 (oldest deadline 2023-12-16T22:11:03.026858+0000) 2023-12-16T22:15:48.065 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:15:47 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:15:47.833+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:10:43.026858+0000 (oldest deadline 2023-12-16T22:11:03.026858+0000) 2023-12-16T22:15:48.065 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:15:47 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:15:47.833+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:10:43.026858+0000 (oldest deadline 2023-12-16T22:11:03.026858+0000) 2023-12-16T22:15:48.066 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:15:47 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:15:47.834+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:15:48.564 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:15:48 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:15:48.113+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:11:07.849373+0000 (oldest deadline 2023-12-16T22:11:27.849373+0000) 2023-12-16T22:15:48.564 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:15:48 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:15:48.113+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:11:07.849373+0000 (oldest deadline 2023-12-16T22:11:27.849373+0000) 2023-12-16T22:15:48.564 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:15:48 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:15:48.113+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:11:07.849373+0000 (oldest deadline 2023-12-16T22:11:27.849373+0000) 2023-12-16T22:15:48.565 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:15:48 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:15:48.114+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:15:49.064 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:15:48 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:15:48.657+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:10:42.731761+0000 (oldest deadline 2023-12-16T22:11:02.731761+0000) 2023-12-16T22:15:49.065 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:15:48 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:15:48.657+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:10:42.731761+0000 (oldest deadline 2023-12-16T22:11:02.731761+0000) 2023-12-16T22:15:49.065 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:15:48 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:15:48.657+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:10:42.731761+0000 (oldest deadline 2023-12-16T22:11:02.731761+0000) 2023-12-16T22:15:49.065 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:15:48 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:15:48.658+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:15:49.065 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:15:48 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:15:48.797+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:10:43.026858+0000 (oldest deadline 2023-12-16T22:11:03.026858+0000) 2023-12-16T22:15:49.065 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:15:48 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:15:48.797+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:10:43.026858+0000 (oldest deadline 2023-12-16T22:11:03.026858+0000) 2023-12-16T22:15:49.065 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:15:48 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:15:48.797+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:10:43.026858+0000 (oldest deadline 2023-12-16T22:11:03.026858+0000) 2023-12-16T22:15:49.066 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:15:48 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:15:48.798+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:15:49.137 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.169 2023-12-16T22:15:49.564 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:15:49 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:15:49.138+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:11:07.849373+0000 (oldest deadline 2023-12-16T22:11:27.849373+0000) 2023-12-16T22:15:49.564 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:15:49 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:15:49.138+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:11:07.849373+0000 (oldest deadline 2023-12-16T22:11:27.849373+0000) 2023-12-16T22:15:49.565 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:15:49 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:15:49.138+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:11:07.849373+0000 (oldest deadline 2023-12-16T22:11:27.849373+0000) 2023-12-16T22:15:49.565 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:15:49 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:15:49.139+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:15:50.064 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:15:49 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:15:49.771+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:10:43.026858+0000 (oldest deadline 2023-12-16T22:11:03.026858+0000) 2023-12-16T22:15:50.065 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:15:49 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:15:49.771+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:10:43.026858+0000 (oldest deadline 2023-12-16T22:11:03.026858+0000) 2023-12-16T22:15:50.065 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:15:49 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:15:49.771+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:10:43.026858+0000 (oldest deadline 2023-12-16T22:11:03.026858+0000) 2023-12-16T22:15:50.065 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:15:49 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:15:49.772+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:15:50.066 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:15:49 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:15:49.610+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:10:42.731761+0000 (oldest deadline 2023-12-16T22:11:02.731761+0000) 2023-12-16T22:15:50.066 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:15:49 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:15:49.610+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:10:42.731761+0000 (oldest deadline 2023-12-16T22:11:02.731761+0000) 2023-12-16T22:15:50.066 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:15:49 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:15:49.610+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:10:42.731761+0000 (oldest deadline 2023-12-16T22:11:02.731761+0000) 2023-12-16T22:15:50.066 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:15:49 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:15:49.611+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:15:50.566 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:15:50 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:15:50.128+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:11:07.849373+0000 (oldest deadline 2023-12-16T22:11:27.849373+0000) 2023-12-16T22:15:50.566 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:15:50 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:15:50.128+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:11:07.849373+0000 (oldest deadline 2023-12-16T22:11:27.849373+0000) 2023-12-16T22:15:50.566 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:15:50 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:15:50.128+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:11:07.849373+0000 (oldest deadline 2023-12-16T22:11:27.849373+0000) 2023-12-16T22:15:50.567 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:15:50 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:15:50.128+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:15:51.064 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:15:50 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:15:50.769+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:10:43.026858+0000 (oldest deadline 2023-12-16T22:11:03.026858+0000) 2023-12-16T22:15:51.065 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:15:50 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:15:50.769+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:10:43.026858+0000 (oldest deadline 2023-12-16T22:11:03.026858+0000) 2023-12-16T22:15:51.065 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:15:50 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:15:50.769+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:10:43.026858+0000 (oldest deadline 2023-12-16T22:11:03.026858+0000) 2023-12-16T22:15:51.065 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:15:50 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:15:50.769+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:15:51.065 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:15:50 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:15:50.597+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:10:42.731761+0000 (oldest deadline 2023-12-16T22:11:02.731761+0000) 2023-12-16T22:15:51.065 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:15:50 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:15:50.597+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:10:42.731761+0000 (oldest deadline 2023-12-16T22:11:02.731761+0000) 2023-12-16T22:15:51.066 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:15:50 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:15:50.597+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:10:42.731761+0000 (oldest deadline 2023-12-16T22:11:02.731761+0000) 2023-12-16T22:15:51.066 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:15:50 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:15:50.598+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:15:51.564 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:15:51 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:15:51.135+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:11:07.849373+0000 (oldest deadline 2023-12-16T22:11:27.849373+0000) 2023-12-16T22:15:51.565 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:15:51 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:15:51.135+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:11:07.849373+0000 (oldest deadline 2023-12-16T22:11:27.849373+0000) 2023-12-16T22:15:51.565 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:15:51 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:15:51.135+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:11:07.849373+0000 (oldest deadline 2023-12-16T22:11:27.849373+0000) 2023-12-16T22:15:51.565 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:15:51 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:15:51.135+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:15:52.065 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:15:51 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:15:51.776+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:10:43.026858+0000 (oldest deadline 2023-12-16T22:11:03.026858+0000) 2023-12-16T22:15:52.065 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:15:51 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:15:51.777+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:10:43.026858+0000 (oldest deadline 2023-12-16T22:11:03.026858+0000) 2023-12-16T22:15:52.065 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:15:51 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:15:51.777+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:10:43.026858+0000 (oldest deadline 2023-12-16T22:11:03.026858+0000) 2023-12-16T22:15:52.065 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:15:51 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:15:51.777+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:15:52.066 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:15:51 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:15:51.591+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:10:42.731761+0000 (oldest deadline 2023-12-16T22:11:02.731761+0000) 2023-12-16T22:15:52.066 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:15:51 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:15:51.591+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:10:42.731761+0000 (oldest deadline 2023-12-16T22:11:02.731761+0000) 2023-12-16T22:15:52.066 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:15:51 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:15:51.591+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:10:42.731761+0000 (oldest deadline 2023-12-16T22:11:02.731761+0000) 2023-12-16T22:15:52.066 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:15:51 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:15:51.591+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:15:52.564 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Dec 16 22:15:52 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-mon-smithi045[80759]: 2023-12-16T22:15:52.228+0000 7f45a1f8f700 -1 mon.smithi045@0(probing) e2 get_health_metrics reporting 36582 slow ops, oldest is monmgrreport(gid 14214, 0 checks, 1 progress events) 2023-12-16T22:15:52.564 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:15:52 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:15:52.110+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:11:07.849373+0000 (oldest deadline 2023-12-16T22:11:27.849373+0000) 2023-12-16T22:15:52.565 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:15:52 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:15:52.110+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:11:07.849373+0000 (oldest deadline 2023-12-16T22:11:27.849373+0000) 2023-12-16T22:15:52.565 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:15:52 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:15:52.110+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:11:07.849373+0000 (oldest deadline 2023-12-16T22:11:27.849373+0000) 2023-12-16T22:15:52.565 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:15:52 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:15:52.111+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:15:53.065 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:15:52 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:15:52.639+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:10:42.731761+0000 (oldest deadline 2023-12-16T22:11:02.731761+0000) 2023-12-16T22:15:53.065 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:15:52 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:15:52.639+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:10:42.731761+0000 (oldest deadline 2023-12-16T22:11:02.731761+0000) 2023-12-16T22:15:53.066 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:15:52 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:15:52.639+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:10:42.731761+0000 (oldest deadline 2023-12-16T22:11:02.731761+0000) 2023-12-16T22:15:53.066 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:15:52 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:15:52.639+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:15:53.066 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:15:52 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:15:52.806+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:10:43.026858+0000 (oldest deadline 2023-12-16T22:11:03.026858+0000) 2023-12-16T22:15:53.066 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:15:52 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:15:52.806+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:10:43.026858+0000 (oldest deadline 2023-12-16T22:11:03.026858+0000) 2023-12-16T22:15:53.066 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:15:52 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:15:52.806+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:10:43.026858+0000 (oldest deadline 2023-12-16T22:11:03.026858+0000) 2023-12-16T22:15:53.066 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:15:52 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:15:52.807+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:15:53.564 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:15:53 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:15:53.125+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:11:07.849373+0000 (oldest deadline 2023-12-16T22:11:27.849373+0000) 2023-12-16T22:15:53.565 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:15:53 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:15:53.125+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:11:07.849373+0000 (oldest deadline 2023-12-16T22:11:27.849373+0000) 2023-12-16T22:15:53.565 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:15:53 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:15:53.125+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:11:07.849373+0000 (oldest deadline 2023-12-16T22:11:27.849373+0000) 2023-12-16T22:15:53.565 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:15:53 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:15:53.126+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:15:54.064 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:15:53 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:15:53.758+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:10:43.026858+0000 (oldest deadline 2023-12-16T22:11:03.026858+0000) 2023-12-16T22:15:54.065 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:15:53 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:15:53.758+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:10:43.026858+0000 (oldest deadline 2023-12-16T22:11:03.026858+0000) 2023-12-16T22:15:54.065 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:15:53 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:15:53.758+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:10:43.026858+0000 (oldest deadline 2023-12-16T22:11:03.026858+0000) 2023-12-16T22:15:54.065 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:15:53 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:15:53.759+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:15:54.065 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:15:53 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:15:53.640+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:10:42.731761+0000 (oldest deadline 2023-12-16T22:11:02.731761+0000) 2023-12-16T22:15:54.065 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:15:53 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:15:53.640+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:10:42.731761+0000 (oldest deadline 2023-12-16T22:11:02.731761+0000) 2023-12-16T22:15:54.065 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:15:53 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:15:53.640+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:10:42.731761+0000 (oldest deadline 2023-12-16T22:11:02.731761+0000) 2023-12-16T22:15:54.066 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:15:53 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:15:53.641+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:15:54.564 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:15:54 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:15:54.168+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:11:07.849373+0000 (oldest deadline 2023-12-16T22:11:27.849373+0000) 2023-12-16T22:15:54.565 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:15:54 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:15:54.168+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:11:07.849373+0000 (oldest deadline 2023-12-16T22:11:27.849373+0000) 2023-12-16T22:15:54.565 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:15:54 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:15:54.168+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:11:07.849373+0000 (oldest deadline 2023-12-16T22:11:27.849373+0000) 2023-12-16T22:15:54.565 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:15:54 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:15:54.169+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:15:55.064 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:15:54 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:15:54.722+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:10:43.026858+0000 (oldest deadline 2023-12-16T22:11:03.026858+0000) 2023-12-16T22:15:55.064 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:15:54 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:15:54.722+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:10:43.026858+0000 (oldest deadline 2023-12-16T22:11:03.026858+0000) 2023-12-16T22:15:55.065 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:15:54 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:15:54.722+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:10:43.026858+0000 (oldest deadline 2023-12-16T22:11:03.026858+0000) 2023-12-16T22:15:55.065 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:15:54 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:15:54.723+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:15:55.065 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:15:54 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:15:54.648+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:10:42.731761+0000 (oldest deadline 2023-12-16T22:11:02.731761+0000) 2023-12-16T22:15:55.065 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:15:54 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:15:54.648+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:10:42.731761+0000 (oldest deadline 2023-12-16T22:11:02.731761+0000) 2023-12-16T22:15:55.065 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:15:54 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:15:54.648+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:10:42.731761+0000 (oldest deadline 2023-12-16T22:11:02.731761+0000) 2023-12-16T22:15:55.065 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:15:54 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:15:54.649+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:15:55.564 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:15:55 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:15:55.163+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:11:07.849373+0000 (oldest deadline 2023-12-16T22:11:27.849373+0000) 2023-12-16T22:15:55.564 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:15:55 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:15:55.163+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:11:07.849373+0000 (oldest deadline 2023-12-16T22:11:27.849373+0000) 2023-12-16T22:15:55.565 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:15:55 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:15:55.163+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:11:07.849373+0000 (oldest deadline 2023-12-16T22:11:27.849373+0000) 2023-12-16T22:15:55.565 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:15:55 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:15:55.163+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:15:56.064 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:15:55 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:15:55.737+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:10:43.026858+0000 (oldest deadline 2023-12-16T22:11:03.026858+0000) 2023-12-16T22:15:56.065 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:15:55 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:15:55.737+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:10:43.026858+0000 (oldest deadline 2023-12-16T22:11:03.026858+0000) 2023-12-16T22:15:56.065 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:15:55 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:15:55.737+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:10:43.026858+0000 (oldest deadline 2023-12-16T22:11:03.026858+0000) 2023-12-16T22:15:56.065 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:15:55 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:15:55.738+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:15:56.065 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:15:55 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:15:55.656+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:10:42.731761+0000 (oldest deadline 2023-12-16T22:11:02.731761+0000) 2023-12-16T22:15:56.065 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:15:55 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:15:55.656+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:10:42.731761+0000 (oldest deadline 2023-12-16T22:11:02.731761+0000) 2023-12-16T22:15:56.066 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:15:55 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:15:55.656+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:10:42.731761+0000 (oldest deadline 2023-12-16T22:11:02.731761+0000) 2023-12-16T22:15:56.066 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:15:55 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:15:55.657+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:15:56.564 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:15:56 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:15:56.115+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:11:07.849373+0000 (oldest deadline 2023-12-16T22:11:27.849373+0000) 2023-12-16T22:15:56.564 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:15:56 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:15:56.115+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:11:07.849373+0000 (oldest deadline 2023-12-16T22:11:27.849373+0000) 2023-12-16T22:15:56.564 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:15:56 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:15:56.115+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:11:07.849373+0000 (oldest deadline 2023-12-16T22:11:27.849373+0000) 2023-12-16T22:15:56.565 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:15:56 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:15:56.115+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:15:57.064 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:15:56 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:15:56.722+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:10:43.026858+0000 (oldest deadline 2023-12-16T22:11:03.026858+0000) 2023-12-16T22:15:57.065 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:15:56 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:15:56.722+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:10:43.026858+0000 (oldest deadline 2023-12-16T22:11:03.026858+0000) 2023-12-16T22:15:57.065 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:15:56 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:15:56.722+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:10:43.026858+0000 (oldest deadline 2023-12-16T22:11:03.026858+0000) 2023-12-16T22:15:57.065 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:15:56 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:15:56.723+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:15:57.065 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:15:56 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:15:56.632+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:10:42.731761+0000 (oldest deadline 2023-12-16T22:11:02.731761+0000) 2023-12-16T22:15:57.065 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:15:56 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:15:56.632+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:10:42.731761+0000 (oldest deadline 2023-12-16T22:11:02.731761+0000) 2023-12-16T22:15:57.065 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:15:56 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:15:56.632+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:10:42.731761+0000 (oldest deadline 2023-12-16T22:11:02.731761+0000) 2023-12-16T22:15:57.066 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:15:56 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:15:56.633+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:15:57.564 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Dec 16 22:15:57 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-mon-smithi045[80759]: 2023-12-16T22:15:57.233+0000 7f45a1f8f700 -1 mon.smithi045@0(probing) e2 get_health_metrics reporting 36673 slow ops, oldest is monmgrreport(gid 14214, 0 checks, 1 progress events) 2023-12-16T22:15:57.564 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:15:57 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:15:57.123+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:11:07.849373+0000 (oldest deadline 2023-12-16T22:11:27.849373+0000) 2023-12-16T22:15:57.565 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:15:57 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:15:57.123+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:11:07.849373+0000 (oldest deadline 2023-12-16T22:11:27.849373+0000) 2023-12-16T22:15:57.565 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:15:57 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:15:57.123+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:11:07.849373+0000 (oldest deadline 2023-12-16T22:11:27.849373+0000) 2023-12-16T22:15:57.565 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:15:57 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:15:57.125+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:15:58.064 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:15:57 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:15:57.701+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:10:43.026858+0000 (oldest deadline 2023-12-16T22:11:03.026858+0000) 2023-12-16T22:15:58.065 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:15:57 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:15:57.701+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:10:43.026858+0000 (oldest deadline 2023-12-16T22:11:03.026858+0000) 2023-12-16T22:15:58.065 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:15:57 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:15:57.701+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:10:43.026858+0000 (oldest deadline 2023-12-16T22:11:03.026858+0000) 2023-12-16T22:15:58.065 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:15:57 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:15:57.702+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:15:58.065 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:15:57 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:15:57.605+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:10:42.731761+0000 (oldest deadline 2023-12-16T22:11:02.731761+0000) 2023-12-16T22:15:58.065 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:15:57 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:15:57.605+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:10:42.731761+0000 (oldest deadline 2023-12-16T22:11:02.731761+0000) 2023-12-16T22:15:58.066 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:15:57 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:15:57.605+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:10:42.731761+0000 (oldest deadline 2023-12-16T22:11:02.731761+0000) 2023-12-16T22:15:58.066 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:15:57 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:15:57.606+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:15:58.564 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:15:58 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:15:58.105+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:11:07.849373+0000 (oldest deadline 2023-12-16T22:11:27.849373+0000) 2023-12-16T22:15:58.565 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:15:58 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:15:58.105+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:11:07.849373+0000 (oldest deadline 2023-12-16T22:11:27.849373+0000) 2023-12-16T22:15:58.565 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:15:58 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:15:58.105+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:11:07.849373+0000 (oldest deadline 2023-12-16T22:11:27.849373+0000) 2023-12-16T22:15:58.565 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:15:58 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:15:58.107+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:15:59.064 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:15:58 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:15:58.696+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:10:43.026858+0000 (oldest deadline 2023-12-16T22:11:03.026858+0000) 2023-12-16T22:15:59.065 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:15:58 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:15:58.696+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:10:43.026858+0000 (oldest deadline 2023-12-16T22:11:03.026858+0000) 2023-12-16T22:15:59.065 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:15:58 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:15:58.696+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:10:43.026858+0000 (oldest deadline 2023-12-16T22:11:03.026858+0000) 2023-12-16T22:15:59.065 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:15:58 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:15:58.697+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:15:59.065 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:15:58 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:15:58.656+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:10:42.731761+0000 (oldest deadline 2023-12-16T22:11:02.731761+0000) 2023-12-16T22:15:59.065 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:15:58 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:15:58.656+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:10:42.731761+0000 (oldest deadline 2023-12-16T22:11:02.731761+0000) 2023-12-16T22:15:59.065 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:15:58 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:15:58.656+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:10:42.731761+0000 (oldest deadline 2023-12-16T22:11:02.731761+0000) 2023-12-16T22:15:59.066 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:15:58 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:15:58.657+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:15:59.564 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:15:59 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:15:59.075+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:11:07.849373+0000 (oldest deadline 2023-12-16T22:11:27.849373+0000) 2023-12-16T22:15:59.565 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:15:59 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:15:59.075+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:11:07.849373+0000 (oldest deadline 2023-12-16T22:11:27.849373+0000) 2023-12-16T22:15:59.565 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:15:59 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:15:59.075+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:11:07.849373+0000 (oldest deadline 2023-12-16T22:11:27.849373+0000) 2023-12-16T22:15:59.565 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:15:59 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:15:59.076+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:16:00.064 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:15:59 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:15:59.652+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:10:43.026858+0000 (oldest deadline 2023-12-16T22:11:03.026858+0000) 2023-12-16T22:16:00.065 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:15:59 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:15:59.652+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:10:43.026858+0000 (oldest deadline 2023-12-16T22:11:03.026858+0000) 2023-12-16T22:16:00.065 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:15:59 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:15:59.652+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:10:43.026858+0000 (oldest deadline 2023-12-16T22:11:03.026858+0000) 2023-12-16T22:16:00.065 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:15:59 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:15:59.653+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:16:00.065 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:15:59 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:15:59.643+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:10:42.731761+0000 (oldest deadline 2023-12-16T22:11:02.731761+0000) 2023-12-16T22:16:00.065 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:15:59 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:15:59.643+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:10:42.731761+0000 (oldest deadline 2023-12-16T22:11:02.731761+0000) 2023-12-16T22:16:00.066 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:15:59 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:15:59.643+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:10:42.731761+0000 (oldest deadline 2023-12-16T22:11:02.731761+0000) 2023-12-16T22:16:00.066 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:15:59 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:15:59.643+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:16:00.567 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:16:00 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:16:00.065+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:11:07.849373+0000 (oldest deadline 2023-12-16T22:11:27.849373+0000) 2023-12-16T22:16:00.567 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:16:00 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:16:00.065+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:11:07.849373+0000 (oldest deadline 2023-12-16T22:11:27.849373+0000) 2023-12-16T22:16:00.567 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:16:00 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:16:00.065+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:11:07.849373+0000 (oldest deadline 2023-12-16T22:11:27.849373+0000) 2023-12-16T22:16:00.567 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:16:00 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:16:00.067+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:16:01.039 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:16:00 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:16:00.623+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:10:43.026858+0000 (oldest deadline 2023-12-16T22:11:03.026858+0000) 2023-12-16T22:16:01.039 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:16:00 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:16:00.623+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:10:43.026858+0000 (oldest deadline 2023-12-16T22:11:03.026858+0000) 2023-12-16T22:16:01.040 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:16:00 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:16:00.623+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:10:43.026858+0000 (oldest deadline 2023-12-16T22:11:03.026858+0000) 2023-12-16T22:16:01.040 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:16:00 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:16:00.624+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:16:01.040 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:16:00 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:16:00.689+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:10:42.731761+0000 (oldest deadline 2023-12-16T22:11:02.731761+0000) 2023-12-16T22:16:01.041 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:16:00 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:16:00.689+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:10:42.731761+0000 (oldest deadline 2023-12-16T22:11:02.731761+0000) 2023-12-16T22:16:01.041 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:16:00 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:16:00.689+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:10:42.731761+0000 (oldest deadline 2023-12-16T22:11:02.731761+0000) 2023-12-16T22:16:01.041 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:16:00 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:16:00.690+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:16:01.139 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi169.front.sepia.ceph.com' 2023-12-16T22:16:01.139 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi169.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-12-16T22:16:01.314 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:16:01 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:16:01.042+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:11:07.849373+0000 (oldest deadline 2023-12-16T22:11:27.849373+0000) 2023-12-16T22:16:01.314 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:16:01 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:16:01.042+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:11:07.849373+0000 (oldest deadline 2023-12-16T22:11:27.849373+0000) 2023-12-16T22:16:01.314 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:16:01 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:16:01.042+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:11:07.849373+0000 (oldest deadline 2023-12-16T22:11:27.849373+0000) 2023-12-16T22:16:01.315 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:16:01 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:16:01.043+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:16:02.043 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:16:01 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:16:01.652+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:10:43.026858+0000 (oldest deadline 2023-12-16T22:11:03.026858+0000) 2023-12-16T22:16:02.044 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:16:01 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:16:01.652+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:10:43.026858+0000 (oldest deadline 2023-12-16T22:11:03.026858+0000) 2023-12-16T22:16:02.044 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:16:01 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:16:01.652+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:10:43.026858+0000 (oldest deadline 2023-12-16T22:11:03.026858+0000) 2023-12-16T22:16:02.044 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:16:01 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:16:01.652+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:16:02.045 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:16:01 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:16:01.654+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:10:42.731761+0000 (oldest deadline 2023-12-16T22:11:02.731761+0000) 2023-12-16T22:16:02.045 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:16:01 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:16:01.654+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:10:42.731761+0000 (oldest deadline 2023-12-16T22:11:02.731761+0000) 2023-12-16T22:16:02.046 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:16:01 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:16:01.654+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:10:42.731761+0000 (oldest deadline 2023-12-16T22:11:02.731761+0000) 2023-12-16T22:16:02.046 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:16:01 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:16:01.655+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:16:02.314 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Dec 16 22:16:02 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-mon-smithi045[80759]: 2023-12-16T22:16:02.237+0000 7f45a1f8f700 -1 mon.smithi045@0(probing) e2 get_health_metrics reporting 36764 slow ops, oldest is monmgrreport(gid 14214, 0 checks, 1 progress events) 2023-12-16T22:16:02.315 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:16:02 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:16:02.042+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:11:07.849373+0000 (oldest deadline 2023-12-16T22:11:27.849373+0000) 2023-12-16T22:16:02.315 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:16:02 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:16:02.042+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:11:07.849373+0000 (oldest deadline 2023-12-16T22:11:27.849373+0000) 2023-12-16T22:16:02.315 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:16:02 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:16:02.042+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:11:07.849373+0000 (oldest deadline 2023-12-16T22:11:27.849373+0000) 2023-12-16T22:16:02.315 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:16:02 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:16:02.043+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:16:03.044 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:16:02 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:16:02.631+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:10:42.731761+0000 (oldest deadline 2023-12-16T22:11:02.731761+0000) 2023-12-16T22:16:03.044 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:16:02 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:16:02.631+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:10:42.731761+0000 (oldest deadline 2023-12-16T22:11:02.731761+0000) 2023-12-16T22:16:03.045 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:16:02 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:16:02.631+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:10:42.731761+0000 (oldest deadline 2023-12-16T22:11:02.731761+0000) 2023-12-16T22:16:03.045 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:16:02 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:16:02.632+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:16:03.045 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:16:02 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:16:02.679+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:10:43.026858+0000 (oldest deadline 2023-12-16T22:11:03.026858+0000) 2023-12-16T22:16:03.045 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:16:02 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:16:02.679+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:10:43.026858+0000 (oldest deadline 2023-12-16T22:11:03.026858+0000) 2023-12-16T22:16:03.046 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:16:02 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:16:02.679+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:10:43.026858+0000 (oldest deadline 2023-12-16T22:11:03.026858+0000) 2023-12-16T22:16:03.046 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:16:02 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:16:02.680+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:16:03.318 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:16:03 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:16:03.042+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:11:07.849373+0000 (oldest deadline 2023-12-16T22:11:27.849373+0000) 2023-12-16T22:16:03.318 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:16:03 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:16:03.042+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:11:07.849373+0000 (oldest deadline 2023-12-16T22:11:27.849373+0000) 2023-12-16T22:16:03.319 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:16:03 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:16:03.042+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:11:07.849373+0000 (oldest deadline 2023-12-16T22:11:27.849373+0000) 2023-12-16T22:16:03.319 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:16:03 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:16:03.043+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:16:03.998 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:16:03 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:16:03.730+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:10:43.026858+0000 (oldest deadline 2023-12-16T22:11:03.026858+0000) 2023-12-16T22:16:03.998 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:16:03 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:16:03.730+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:10:43.026858+0000 (oldest deadline 2023-12-16T22:11:03.026858+0000) 2023-12-16T22:16:03.998 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:16:03 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:16:03.730+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:10:43.026858+0000 (oldest deadline 2023-12-16T22:11:03.026858+0000) 2023-12-16T22:16:03.998 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:16:03 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:16:03.731+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:16:03.999 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:16:03 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:16:03.646+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:10:42.731761+0000 (oldest deadline 2023-12-16T22:11:02.731761+0000) 2023-12-16T22:16:03.999 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:16:03 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:16:03.646+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:10:42.731761+0000 (oldest deadline 2023-12-16T22:11:02.731761+0000) 2023-12-16T22:16:03.999 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:16:03 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:16:03.646+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:10:42.731761+0000 (oldest deadline 2023-12-16T22:11:02.731761+0000) 2023-12-16T22:16:03.999 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:16:03 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:16:03.646+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:16:04.209 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.169 2023-12-16T22:16:04.314 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:16:03 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:16:03.997+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:11:07.849373+0000 (oldest deadline 2023-12-16T22:11:27.849373+0000) 2023-12-16T22:16:04.314 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:16:03 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:16:03.997+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:11:07.849373+0000 (oldest deadline 2023-12-16T22:11:27.849373+0000) 2023-12-16T22:16:04.315 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:16:03 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:16:03.997+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:11:07.849373+0000 (oldest deadline 2023-12-16T22:11:27.849373+0000) 2023-12-16T22:16:04.315 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:16:03 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:16:03.998+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:16:05.019 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:16:04 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:16:04.642+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:10:42.731761+0000 (oldest deadline 2023-12-16T22:11:02.731761+0000) 2023-12-16T22:16:05.020 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:16:04 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:16:04.642+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:10:42.731761+0000 (oldest deadline 2023-12-16T22:11:02.731761+0000) 2023-12-16T22:16:05.020 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:16:04 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:16:04.642+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:10:42.731761+0000 (oldest deadline 2023-12-16T22:11:02.731761+0000) 2023-12-16T22:16:05.020 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:16:04 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:16:04.643+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:16:05.020 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:16:04 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:16:04.751+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:10:43.026858+0000 (oldest deadline 2023-12-16T22:11:03.026858+0000) 2023-12-16T22:16:05.020 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:16:04 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:16:04.751+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:10:43.026858+0000 (oldest deadline 2023-12-16T22:11:03.026858+0000) 2023-12-16T22:16:05.020 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:16:04 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:16:04.751+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:10:43.026858+0000 (oldest deadline 2023-12-16T22:11:03.026858+0000) 2023-12-16T22:16:05.021 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:16:04 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:16:04.752+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:16:05.314 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:16:05 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:16:05.018+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:11:07.849373+0000 (oldest deadline 2023-12-16T22:11:27.849373+0000) 2023-12-16T22:16:05.314 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:16:05 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:16:05.019+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:11:07.849373+0000 (oldest deadline 2023-12-16T22:11:27.849373+0000) 2023-12-16T22:16:05.314 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:16:05 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:16:05.019+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:11:07.849373+0000 (oldest deadline 2023-12-16T22:11:27.849373+0000) 2023-12-16T22:16:05.315 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:16:05 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:16:05.020+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:16:05.992 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:16:05 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:16:05.756+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:10:43.026858+0000 (oldest deadline 2023-12-16T22:11:03.026858+0000) 2023-12-16T22:16:05.992 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:16:05 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:16:05.756+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:10:43.026858+0000 (oldest deadline 2023-12-16T22:11:03.026858+0000) 2023-12-16T22:16:05.992 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:16:05 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:16:05.756+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:10:43.026858+0000 (oldest deadline 2023-12-16T22:11:03.026858+0000) 2023-12-16T22:16:05.992 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:16:05 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:16:05.757+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:16:05.993 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:16:05 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:16:05.623+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:10:42.731761+0000 (oldest deadline 2023-12-16T22:11:02.731761+0000) 2023-12-16T22:16:05.993 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:16:05 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:16:05.623+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:10:42.731761+0000 (oldest deadline 2023-12-16T22:11:02.731761+0000) 2023-12-16T22:16:05.993 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:16:05 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:16:05.623+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:10:42.731761+0000 (oldest deadline 2023-12-16T22:11:02.731761+0000) 2023-12-16T22:16:05.993 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:16:05 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:16:05.623+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:16:06.314 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:16:05 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:16:05.991+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:11:07.849373+0000 (oldest deadline 2023-12-16T22:11:27.849373+0000) 2023-12-16T22:16:06.314 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:16:05 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:16:05.991+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:11:07.849373+0000 (oldest deadline 2023-12-16T22:11:27.849373+0000) 2023-12-16T22:16:06.314 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:16:05 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:16:05.991+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:11:07.849373+0000 (oldest deadline 2023-12-16T22:11:27.849373+0000) 2023-12-16T22:16:06.315 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:16:05 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:16:05.991+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:16:06.969 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:16:06 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:16:06.709+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:10:43.026858+0000 (oldest deadline 2023-12-16T22:11:03.026858+0000) 2023-12-16T22:16:06.969 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:16:06 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:16:06.709+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:10:43.026858+0000 (oldest deadline 2023-12-16T22:11:03.026858+0000) 2023-12-16T22:16:06.970 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:16:06 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:16:06.709+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:10:43.026858+0000 (oldest deadline 2023-12-16T22:11:03.026858+0000) 2023-12-16T22:16:06.970 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:16:06 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:16:06.709+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:16:06.970 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:16:06 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:16:06.631+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:10:42.731761+0000 (oldest deadline 2023-12-16T22:11:02.731761+0000) 2023-12-16T22:16:06.970 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:16:06 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:16:06.631+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:10:42.731761+0000 (oldest deadline 2023-12-16T22:11:02.731761+0000) 2023-12-16T22:16:06.970 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:16:06 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:16:06.631+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:10:42.731761+0000 (oldest deadline 2023-12-16T22:11:02.731761+0000) 2023-12-16T22:16:06.970 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:16:06 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:16:06.632+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:16:07.210 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi169.front.sepia.ceph.com' 2023-12-16T22:16:07.211 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi169.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-12-16T22:16:07.243 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:16:06 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:16:06.968+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:11:07.849373+0000 (oldest deadline 2023-12-16T22:11:27.849373+0000) 2023-12-16T22:16:07.244 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:16:06 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:16:06.968+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:11:07.849373+0000 (oldest deadline 2023-12-16T22:11:27.849373+0000) 2023-12-16T22:16:07.244 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:16:06 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:16:06.968+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:11:07.849373+0000 (oldest deadline 2023-12-16T22:11:27.849373+0000) 2023-12-16T22:16:07.245 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:16:06 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:16:06.969+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:16:07.281 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.169 2023-12-16T22:16:07.282 WARNING:teuthology.contextutil:'reconnect to {self.shortname}' reached maximum tries (5) after waiting for 30 seconds 2023-12-16T22:16:07.282 INFO:tasks.cephadm:Removing cluster... 2023-12-16T22:16:07.282 DEBUG:teuthology.orchestra.run.smithi045:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid 88ff363c-9c5a-11ee-95a5-87774f69a715 --force 2023-12-16T22:16:07.564 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Dec 16 22:16:07 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-mon-smithi045[80759]: 2023-12-16T22:16:07.242+0000 7f45a1f8f700 -1 mon.smithi045@0(probing) e2 get_health_metrics reporting 36855 slow ops, oldest is monmgrreport(gid 14214, 0 checks, 1 progress events) 2023-12-16T22:16:07.935 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:16:07 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:16:07.680+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:10:43.026858+0000 (oldest deadline 2023-12-16T22:11:03.026858+0000) 2023-12-16T22:16:07.935 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:16:07 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:16:07.680+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:10:43.026858+0000 (oldest deadline 2023-12-16T22:11:03.026858+0000) 2023-12-16T22:16:07.935 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:16:07 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:16:07.680+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:10:43.026858+0000 (oldest deadline 2023-12-16T22:11:03.026858+0000) 2023-12-16T22:16:07.935 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:16:07 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:16:07.681+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:16:07.936 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:16:07 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:16:07.636+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:10:42.731761+0000 (oldest deadline 2023-12-16T22:11:02.731761+0000) 2023-12-16T22:16:07.936 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:16:07 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:16:07.636+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:10:42.731761+0000 (oldest deadline 2023-12-16T22:11:02.731761+0000) 2023-12-16T22:16:07.936 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:16:07 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:16:07.636+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:10:42.731761+0000 (oldest deadline 2023-12-16T22:11:02.731761+0000) 2023-12-16T22:16:07.936 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:16:07 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:16:07.637+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:16:07.937 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Dec 16 22:16:07 smithi045 systemd[1]: Stopping Ceph mon.smithi045 for 88ff363c-9c5a-11ee-95a5-87774f69a715... 2023-12-16T22:16:07.937 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Dec 16 22:16:07 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-mon-smithi045[80759]: 2023-12-16T22:16:07.834+0000 7f45a4f95700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mon -n mon.smithi045 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-journald=true --default-mon-cluster-log-to-stderr=false (PID: 1) UID: 0 2023-12-16T22:16:07.937 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Dec 16 22:16:07 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-mon-smithi045[80759]: 2023-12-16T22:16:07.834+0000 7f45a4f95700 -1 mon.smithi045@0(probing) e2 *** Got Signal Terminated *** 2023-12-16T22:16:08.272 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:16:07 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:16:07.933+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:11:07.849373+0000 (oldest deadline 2023-12-16T22:11:27.849373+0000) 2023-12-16T22:16:08.272 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:16:07 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:16:07.933+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:11:07.849373+0000 (oldest deadline 2023-12-16T22:11:27.849373+0000) 2023-12-16T22:16:08.272 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:16:07 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:16:07.933+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:11:07.849373+0000 (oldest deadline 2023-12-16T22:11:27.849373+0000) 2023-12-16T22:16:08.272 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:16:07 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:16:07.935+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:16:08.564 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Dec 16 22:16:08 smithi045 podman[136940]: 2023-12-16 22:16:08.272228002 +0000 UTC m=+0.499262002 container died 1cddbf5059f8aabb81ecf60d6ebf743c70edae445f2da2f20f0280c5672e87b1 (image=quay.ceph.io/ceph-ci/ceph:quincy, name=ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-mon-smithi045, RELEASE=quincy-1f2f551, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, GIT_COMMIT=0396eef90bef641b676c164ec7a3876f45010308, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20231212, io.buildah.version=1.31.3, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD) 2023-12-16T22:16:08.564 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Dec 16 22:16:08 smithi045 podman[136940]: 2023-12-16 22:16:08.415309523 +0000 UTC m=+0.642343509 container cleanup 1cddbf5059f8aabb81ecf60d6ebf743c70edae445f2da2f20f0280c5672e87b1 (image=quay.ceph.io/ceph-ci/ceph:quincy, name=ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-mon-smithi045, org.label-schema.build-date=20231212, ceph=True, org.label-schema.license=GPLv2, RELEASE=quincy-1f2f551, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.31.3, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, GIT_COMMIT=0396eef90bef641b676c164ec7a3876f45010308, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=) 2023-12-16T22:16:08.564 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Dec 16 22:16:08 smithi045 bash[136940]: ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-mon-smithi045 2023-12-16T22:16:08.919 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:16:08 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:16:08.686+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:10:43.026858+0000 (oldest deadline 2023-12-16T22:11:03.026858+0000) 2023-12-16T22:16:08.919 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:16:08 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:16:08.686+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:10:43.026858+0000 (oldest deadline 2023-12-16T22:11:03.026858+0000) 2023-12-16T22:16:08.919 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:16:08 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:16:08.686+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:10:43.026858+0000 (oldest deadline 2023-12-16T22:11:03.026858+0000) 2023-12-16T22:16:08.920 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:16:08 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:16:08.687+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:16:08.920 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:16:08 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:16:08.630+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:10:42.731761+0000 (oldest deadline 2023-12-16T22:11:02.731761+0000) 2023-12-16T22:16:08.920 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:16:08 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:16:08.630+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:10:42.731761+0000 (oldest deadline 2023-12-16T22:11:02.731761+0000) 2023-12-16T22:16:08.920 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:16:08 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:16:08.630+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:10:42.731761+0000 (oldest deadline 2023-12-16T22:11:02.731761+0000) 2023-12-16T22:16:08.920 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:16:08 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:16:08.631+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:16:08.921 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Dec 16 22:16:08 smithi045 podman[136954]: 2023-12-16 22:16:08.565183538 +0000 UTC m=+0.291198144 container remove 1cddbf5059f8aabb81ecf60d6ebf743c70edae445f2da2f20f0280c5672e87b1 (image=quay.ceph.io/ceph-ci/ceph:quincy, name=ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-mon-smithi045, ceph=True, org.label-schema.build-date=20231212, org.label-schema.license=GPLv2, GIT_COMMIT=0396eef90bef641b676c164ec7a3876f45010308, io.buildah.version=1.31.3, org.label-schema.vendor=CentOS, GIT_CLEAN=True, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=quincy-1f2f551, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image) 2023-12-16T22:16:08.921 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Dec 16 22:16:08 smithi045 systemd[1]: ceph-88ff363c-9c5a-11ee-95a5-87774f69a715@mon.smithi045.service: Deactivated successfully. 2023-12-16T22:16:08.921 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Dec 16 22:16:08 smithi045 systemd[1]: Stopped Ceph mon.smithi045 for 88ff363c-9c5a-11ee-95a5-87774f69a715. 2023-12-16T22:16:08.921 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Dec 16 22:16:08 smithi045 systemd[1]: ceph-88ff363c-9c5a-11ee-95a5-87774f69a715@mon.smithi045.service: Consumed 25.650s CPU time. 2023-12-16T22:16:09.170 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:16:08 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:16:08.953+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:11:07.849373+0000 (oldest deadline 2023-12-16T22:11:27.849373+0000) 2023-12-16T22:16:09.170 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:16:08 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:16:08.953+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:11:07.849373+0000 (oldest deadline 2023-12-16T22:11:27.849373+0000) 2023-12-16T22:16:09.170 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:16:08 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:16:08.953+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:11:07.849373+0000 (oldest deadline 2023-12-16T22:11:27.849373+0000) 2023-12-16T22:16:09.170 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:16:08 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:16:08.954+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:16:09.726 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:16:09 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:16:09.641+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:10:42.731761+0000 (oldest deadline 2023-12-16T22:11:02.731761+0000) 2023-12-16T22:16:09.726 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:16:09 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:16:09.641+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:10:42.731761+0000 (oldest deadline 2023-12-16T22:11:02.731761+0000) 2023-12-16T22:16:09.726 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:16:09 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:16:09.641+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:10:42.731761+0000 (oldest deadline 2023-12-16T22:11:02.731761+0000) 2023-12-16T22:16:09.726 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:16:09 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:16:09.642+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:16:09.727 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:16:09 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:16:09.660+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:10:43.026858+0000 (oldest deadline 2023-12-16T22:11:03.026858+0000) 2023-12-16T22:16:09.727 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:16:09 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:16:09.660+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:10:43.026858+0000 (oldest deadline 2023-12-16T22:11:03.026858+0000) 2023-12-16T22:16:09.727 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:16:09 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:16:09.660+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:10:43.026858+0000 (oldest deadline 2023-12-16T22:11:03.026858+0000) 2023-12-16T22:16:09.727 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:16:09 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:16:09.661+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:16:10.245 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:16:09 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:16:09.992+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:11:07.849373+0000 (oldest deadline 2023-12-16T22:11:27.849373+0000) 2023-12-16T22:16:10.246 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:16:09 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:16:09.992+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:11:07.849373+0000 (oldest deadline 2023-12-16T22:11:27.849373+0000) 2023-12-16T22:16:10.246 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:16:09 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:16:09.992+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:11:07.849373+0000 (oldest deadline 2023-12-16T22:11:27.849373+0000) 2023-12-16T22:16:10.246 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:16:09 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:16:09.993+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:16:10.815 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:16:10 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:16:10.626+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:10:43.026858+0000 (oldest deadline 2023-12-16T22:11:03.026858+0000) 2023-12-16T22:16:10.816 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:16:10 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:16:10.626+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:10:43.026858+0000 (oldest deadline 2023-12-16T22:11:03.026858+0000) 2023-12-16T22:16:10.816 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:16:10 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:16:10.626+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:10:43.026858+0000 (oldest deadline 2023-12-16T22:11:03.026858+0000) 2023-12-16T22:16:10.817 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:16:10 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:16:10.627+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:16:10.818 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:16:10 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:16:10.658+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:10:42.731761+0000 (oldest deadline 2023-12-16T22:11:02.731761+0000) 2023-12-16T22:16:10.818 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:16:10 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:16:10.658+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:10:42.731761+0000 (oldest deadline 2023-12-16T22:11:02.731761+0000) 2023-12-16T22:16:10.818 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:16:10 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:16:10.658+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:10:42.731761+0000 (oldest deadline 2023-12-16T22:11:02.731761+0000) 2023-12-16T22:16:10.819 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:16:10 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:16:10.659+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:16:11.234 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:16:11 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:16:11.035+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:11:07.849373+0000 (oldest deadline 2023-12-16T22:11:27.849373+0000) 2023-12-16T22:16:11.234 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:16:11 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:16:11.035+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:11:07.849373+0000 (oldest deadline 2023-12-16T22:11:27.849373+0000) 2023-12-16T22:16:11.234 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:16:11 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:16:11.035+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:11:07.849373+0000 (oldest deadline 2023-12-16T22:11:27.849373+0000) 2023-12-16T22:16:11.235 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:16:11 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:16:11.036+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:16:11.814 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:16:11 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:16:11.658+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:10:43.026858+0000 (oldest deadline 2023-12-16T22:11:03.026858+0000) 2023-12-16T22:16:11.814 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:16:11 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:16:11.658+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:10:43.026858+0000 (oldest deadline 2023-12-16T22:11:03.026858+0000) 2023-12-16T22:16:11.815 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:16:11 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:16:11.658+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:10:43.026858+0000 (oldest deadline 2023-12-16T22:11:03.026858+0000) 2023-12-16T22:16:11.815 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:16:11 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:16:11.659+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:16:11.815 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:16:11 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:16:11.621+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:10:42.731761+0000 (oldest deadline 2023-12-16T22:11:02.731761+0000) 2023-12-16T22:16:11.815 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:16:11 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:16:11.621+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:10:42.731761+0000 (oldest deadline 2023-12-16T22:11:02.731761+0000) 2023-12-16T22:16:11.815 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:16:11 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:16:11.621+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:10:42.731761+0000 (oldest deadline 2023-12-16T22:11:02.731761+0000) 2023-12-16T22:16:11.815 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:16:11 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:16:11.622+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:16:12.231 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:16:12 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:16:12.025+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:11:07.849373+0000 (oldest deadline 2023-12-16T22:11:27.849373+0000) 2023-12-16T22:16:12.231 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:16:12 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:16:12.025+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:11:07.849373+0000 (oldest deadline 2023-12-16T22:11:27.849373+0000) 2023-12-16T22:16:12.231 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:16:12 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:16:12.025+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:11:07.849373+0000 (oldest deadline 2023-12-16T22:11:27.849373+0000) 2023-12-16T22:16:12.231 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:16:12 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:16:12.027+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:16:12.815 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:16:12 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:16:12.676+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:10:43.026858+0000 (oldest deadline 2023-12-16T22:11:03.026858+0000) 2023-12-16T22:16:12.815 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:16:12 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:16:12.676+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:10:43.026858+0000 (oldest deadline 2023-12-16T22:11:03.026858+0000) 2023-12-16T22:16:12.815 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:16:12 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:16:12.676+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:10:43.026858+0000 (oldest deadline 2023-12-16T22:11:03.026858+0000) 2023-12-16T22:16:12.815 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:16:12 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:16:12.676+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:16:12.815 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:16:12 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:16:12.641+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:10:42.731761+0000 (oldest deadline 2023-12-16T22:11:02.731761+0000) 2023-12-16T22:16:12.816 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:16:12 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:16:12.641+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:10:42.731761+0000 (oldest deadline 2023-12-16T22:11:02.731761+0000) 2023-12-16T22:16:12.816 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:16:12 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:16:12.641+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:10:42.731761+0000 (oldest deadline 2023-12-16T22:11:02.731761+0000) 2023-12-16T22:16:12.816 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:16:12 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:16:12.642+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:16:13.314 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:16:13 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:16:13.046+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:11:07.849373+0000 (oldest deadline 2023-12-16T22:11:27.849373+0000) 2023-12-16T22:16:13.315 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:16:13 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:16:13.046+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:11:07.849373+0000 (oldest deadline 2023-12-16T22:11:27.849373+0000) 2023-12-16T22:16:13.315 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:16:13 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:16:13.046+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:11:07.849373+0000 (oldest deadline 2023-12-16T22:11:27.849373+0000) 2023-12-16T22:16:13.315 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:16:13 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:16:13.047+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:16:13.652 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:16:13 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:16:13.600+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:10:42.731761+0000 (oldest deadline 2023-12-16T22:11:02.731761+0000) 2023-12-16T22:16:13.653 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:16:13 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:16:13.600+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:10:42.731761+0000 (oldest deadline 2023-12-16T22:11:02.731761+0000) 2023-12-16T22:16:13.653 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:16:13 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:16:13.600+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:10:42.731761+0000 (oldest deadline 2023-12-16T22:11:02.731761+0000) 2023-12-16T22:16:13.653 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:16:13 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:16:13.601+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:16:13.947 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:16:13 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:16:13.651+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:10:43.026858+0000 (oldest deadline 2023-12-16T22:11:03.026858+0000) 2023-12-16T22:16:13.948 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:16:13 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:16:13.651+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:10:43.026858+0000 (oldest deadline 2023-12-16T22:11:03.026858+0000) 2023-12-16T22:16:13.948 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:16:13 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:16:13.651+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:10:43.026858+0000 (oldest deadline 2023-12-16T22:11:03.026858+0000) 2023-12-16T22:16:13.948 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:16:13 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:16:13.652+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:16:14.305 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:16:14 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:16:13.999+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:11:07.849373+0000 (oldest deadline 2023-12-16T22:11:27.849373+0000) 2023-12-16T22:16:14.305 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:16:14 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:16:13.999+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:11:07.849373+0000 (oldest deadline 2023-12-16T22:11:27.849373+0000) 2023-12-16T22:16:14.306 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:16:14 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:16:13.999+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:11:07.849373+0000 (oldest deadline 2023-12-16T22:11:27.849373+0000) 2023-12-16T22:16:14.306 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:16:14 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:16:14.000+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:16:14.564 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:16:14 smithi045 systemd[1]: Stopping Ceph osd.0 for 88ff363c-9c5a-11ee-95a5-87774f69a715... 2023-12-16T22:16:14.987 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:16:14 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:16:14.669+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:10:43.026858+0000 (oldest deadline 2023-12-16T22:11:03.026858+0000) 2023-12-16T22:16:14.988 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:16:14 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:16:14.669+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:10:43.026858+0000 (oldest deadline 2023-12-16T22:11:03.026858+0000) 2023-12-16T22:16:14.988 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:16:14 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:16:14.669+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:10:43.026858+0000 (oldest deadline 2023-12-16T22:11:03.026858+0000) 2023-12-16T22:16:14.988 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:16:14 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:16:14.670+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:16:14.988 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:16:14 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:16:14.725+0000 7f2bc0c3a700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.0 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2023-12-16T22:16:14.988 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:16:14 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:16:14.725+0000 7f2bc0c3a700 -1 osd.0 43 *** Got signal Terminated *** 2023-12-16T22:16:14.988 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:16:14 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:16:14.725+0000 7f2bc0c3a700 -1 osd.0 43 *** Immediate shutdown (osd_fast_shutdown=true) *** 2023-12-16T22:16:14.988 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:16:14 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:16:14.601+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:10:42.731761+0000 (oldest deadline 2023-12-16T22:11:02.731761+0000) 2023-12-16T22:16:14.989 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:16:14 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:16:14.601+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:10:42.731761+0000 (oldest deadline 2023-12-16T22:11:02.731761+0000) 2023-12-16T22:16:14.989 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:16:14 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:16:14.601+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:10:42.731761+0000 (oldest deadline 2023-12-16T22:11:02.731761+0000) 2023-12-16T22:16:14.989 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:16:14 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:16:14.602+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:16:15.314 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:16:14 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:16:14.986+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:11:07.849373+0000 (oldest deadline 2023-12-16T22:11:27.849373+0000) 2023-12-16T22:16:15.315 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:16:14 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:16:14.986+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:11:07.849373+0000 (oldest deadline 2023-12-16T22:11:27.849373+0000) 2023-12-16T22:16:15.315 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:16:14 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:16:14.986+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:11:07.849373+0000 (oldest deadline 2023-12-16T22:11:27.849373+0000) 2023-12-16T22:16:15.315 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:16:14 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:16:14.987+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:16:15.939 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:16:15 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:16:15.613+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:10:42.731761+0000 (oldest deadline 2023-12-16T22:11:02.731761+0000) 2023-12-16T22:16:15.939 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:16:15 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:16:15.613+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:10:42.731761+0000 (oldest deadline 2023-12-16T22:11:02.731761+0000) 2023-12-16T22:16:15.939 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:16:15 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:16:15.613+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:10:42.731761+0000 (oldest deadline 2023-12-16T22:11:02.731761+0000) 2023-12-16T22:16:15.939 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:16:15 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:16:15.613+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:16:15.940 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:16:15 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:16:15.680+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:10:43.026858+0000 (oldest deadline 2023-12-16T22:11:03.026858+0000) 2023-12-16T22:16:15.940 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:16:15 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:16:15.680+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:10:43.026858+0000 (oldest deadline 2023-12-16T22:11:03.026858+0000) 2023-12-16T22:16:15.940 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:16:15 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:16:15.680+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:10:43.026858+0000 (oldest deadline 2023-12-16T22:11:03.026858+0000) 2023-12-16T22:16:15.940 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:16:15 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:16:15.681+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:16:16.314 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:16:15 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:16:15.938+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:11:07.849373+0000 (oldest deadline 2023-12-16T22:11:27.849373+0000) 2023-12-16T22:16:16.315 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:16:15 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:16:15.938+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:11:07.849373+0000 (oldest deadline 2023-12-16T22:11:27.849373+0000) 2023-12-16T22:16:16.315 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:16:15 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:16:15.938+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:11:07.849373+0000 (oldest deadline 2023-12-16T22:11:27.849373+0000) 2023-12-16T22:16:16.315 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:16:15 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:16:15.939+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:16:16.965 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:16:16 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:16:16.680+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:10:43.026858+0000 (oldest deadline 2023-12-16T22:11:03.026858+0000) 2023-12-16T22:16:16.966 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:16:16 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:16:16.680+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:10:43.026858+0000 (oldest deadline 2023-12-16T22:11:03.026858+0000) 2023-12-16T22:16:16.966 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:16:16 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:16:16.680+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:10:43.026858+0000 (oldest deadline 2023-12-16T22:11:03.026858+0000) 2023-12-16T22:16:16.966 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:16:16 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:16:16.681+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:16:16.966 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:16:16 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:16:16.637+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:10:42.731761+0000 (oldest deadline 2023-12-16T22:11:02.731761+0000) 2023-12-16T22:16:16.966 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:16:16 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:16:16.637+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:10:42.731761+0000 (oldest deadline 2023-12-16T22:11:02.731761+0000) 2023-12-16T22:16:16.967 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:16:16 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:16:16.637+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:10:42.731761+0000 (oldest deadline 2023-12-16T22:11:02.731761+0000) 2023-12-16T22:16:16.967 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:16:16 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:16:16.638+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:16:17.314 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:16:16 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:16:16.964+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:11:07.849373+0000 (oldest deadline 2023-12-16T22:11:27.849373+0000) 2023-12-16T22:16:17.315 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:16:16 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:16:16.964+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:11:07.849373+0000 (oldest deadline 2023-12-16T22:11:27.849373+0000) 2023-12-16T22:16:17.315 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:16:16 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:16:16.964+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:11:07.849373+0000 (oldest deadline 2023-12-16T22:11:27.849373+0000) 2023-12-16T22:16:17.315 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:16:16 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:16:16.966+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:16:18.011 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:16:17 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:16:17.711+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:10:43.026858+0000 (oldest deadline 2023-12-16T22:11:03.026858+0000) 2023-12-16T22:16:18.012 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:16:17 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:16:17.711+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:10:43.026858+0000 (oldest deadline 2023-12-16T22:11:03.026858+0000) 2023-12-16T22:16:18.012 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:16:17 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:16:17.711+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:10:43.026858+0000 (oldest deadline 2023-12-16T22:11:03.026858+0000) 2023-12-16T22:16:18.012 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:16:17 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:16:17.712+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:16:18.012 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:16:17 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:16:17.686+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:10:42.731761+0000 (oldest deadline 2023-12-16T22:11:02.731761+0000) 2023-12-16T22:16:18.012 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:16:17 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:16:17.686+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:10:42.731761+0000 (oldest deadline 2023-12-16T22:11:02.731761+0000) 2023-12-16T22:16:18.012 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:16:17 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:16:17.686+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:10:42.731761+0000 (oldest deadline 2023-12-16T22:11:02.731761+0000) 2023-12-16T22:16:18.013 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:16:17 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:16:17.687+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:16:18.314 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:16:18 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:16:18.010+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:11:07.849373+0000 (oldest deadline 2023-12-16T22:11:27.849373+0000) 2023-12-16T22:16:18.314 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:16:18 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:16:18.010+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:11:07.849373+0000 (oldest deadline 2023-12-16T22:11:27.849373+0000) 2023-12-16T22:16:18.315 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:16:18 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:16:18.010+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:11:07.849373+0000 (oldest deadline 2023-12-16T22:11:27.849373+0000) 2023-12-16T22:16:18.315 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:16:18 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:16:18.011+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:16:19.050 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:16:18 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:16:18.724+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:10:43.026858+0000 (oldest deadline 2023-12-16T22:11:03.026858+0000) 2023-12-16T22:16:19.050 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:16:18 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:16:18.724+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:10:43.026858+0000 (oldest deadline 2023-12-16T22:11:03.026858+0000) 2023-12-16T22:16:19.050 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:16:18 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:16:18.724+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:10:43.026858+0000 (oldest deadline 2023-12-16T22:11:03.026858+0000) 2023-12-16T22:16:19.051 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:16:18 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:16:18.725+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:16:19.051 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:16:18 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:16:18.713+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:10:42.731761+0000 (oldest deadline 2023-12-16T22:11:02.731761+0000) 2023-12-16T22:16:19.051 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:16:18 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:16:18.713+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:10:42.731761+0000 (oldest deadline 2023-12-16T22:11:02.731761+0000) 2023-12-16T22:16:19.051 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:16:18 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:16:18.713+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:10:42.731761+0000 (oldest deadline 2023-12-16T22:11:02.731761+0000) 2023-12-16T22:16:19.051 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:16:18 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:16:18.715+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:16:19.314 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:16:19 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:16:19.049+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:11:07.849373+0000 (oldest deadline 2023-12-16T22:11:27.849373+0000) 2023-12-16T22:16:19.315 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:16:19 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:16:19.049+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:11:07.849373+0000 (oldest deadline 2023-12-16T22:11:27.849373+0000) 2023-12-16T22:16:19.315 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:16:19 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:16:19.049+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:11:07.849373+0000 (oldest deadline 2023-12-16T22:11:27.849373+0000) 2023-12-16T22:16:19.315 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:16:19 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:16:19.050+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:16:20.065 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:16:19 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:16:19.720+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:10:43.026858+0000 (oldest deadline 2023-12-16T22:11:03.026858+0000) 2023-12-16T22:16:20.065 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:16:19 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:16:19.720+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:10:43.026858+0000 (oldest deadline 2023-12-16T22:11:03.026858+0000) 2023-12-16T22:16:20.065 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:16:19 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:16:19.720+0000 7f2bbc236700 -1 osd.0 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:10:43.026858+0000 (oldest deadline 2023-12-16T22:11:03.026858+0000) 2023-12-16T22:16:20.065 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:16:19 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0[99349]: 2023-12-16T22:16:19.721+0000 7f2bbc236700 -1 osd.0 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6173 3.1f 3:f932ace6:::10000003704.00000000:head [write 0~5985 in=5985b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:16:20.066 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:16:19 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:16:19.747+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:10:42.731761+0000 (oldest deadline 2023-12-16T22:11:02.731761+0000) 2023-12-16T22:16:20.066 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:16:19 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:16:19.747+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:10:42.731761+0000 (oldest deadline 2023-12-16T22:11:02.731761+0000) 2023-12-16T22:16:20.066 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:16:19 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:16:19.747+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:10:42.731761+0000 (oldest deadline 2023-12-16T22:11:02.731761+0000) 2023-12-16T22:16:20.066 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:16:19 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:16:19.748+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:16:20.381 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:16:20 smithi045 podman[137788]: 2023-12-16 22:16:20.310026149 +0000 UTC m=+5.647278349 container died eeea01051a628cc089dbc907b4a3d21e87039cb987305f7d725700895f79ba88 (image=quay.ceph.io/ceph-ci/ceph@sha256:7a1644eb327b921af6e6cf30118d696e8368bd632346b48049fba15207e60112, name=ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.31.3, CEPH_POINT_RELEASE=, GIT_COMMIT=0396eef90bef641b676c164ec7a3876f45010308, org.label-schema.vendor=CentOS, GIT_CLEAN=True, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, org.label-schema.build-date=20231212, RELEASE=quincy-1f2f551, org.label-schema.license=GPLv2) 2023-12-16T22:16:20.382 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:16:20 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:16:20.067+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:11:07.849373+0000 (oldest deadline 2023-12-16T22:11:27.849373+0000) 2023-12-16T22:16:20.382 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:16:20 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:16:20.067+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:11:07.849373+0000 (oldest deadline 2023-12-16T22:11:27.849373+0000) 2023-12-16T22:16:20.382 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:16:20 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:16:20.067+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:11:07.849373+0000 (oldest deadline 2023-12-16T22:11:27.849373+0000) 2023-12-16T22:16:20.382 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:16:20 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:16:20.069+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:16:20.768 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:16:20 smithi045 podman[137788]: 2023-12-16 22:16:20.421752493 +0000 UTC m=+5.759004682 container cleanup eeea01051a628cc089dbc907b4a3d21e87039cb987305f7d725700895f79ba88 (image=quay.ceph.io/ceph-ci/ceph@sha256:7a1644eb327b921af6e6cf30118d696e8368bd632346b48049fba15207e60112, name=ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0, RELEASE=quincy-1f2f551, io.buildah.version=1.31.3, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, org.label-schema.build-date=20231212, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=0396eef90bef641b676c164ec7a3876f45010308, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, org.label-schema.license=GPLv2, GIT_CLEAN=True) 2023-12-16T22:16:20.768 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:16:20 smithi045 bash[137788]: ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0 2023-12-16T22:16:20.768 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:16:20 smithi045 podman[137807]: 2023-12-16 22:16:20.513369742 +0000 UTC m=+0.205819465 container remove eeea01051a628cc089dbc907b4a3d21e87039cb987305f7d725700895f79ba88 (image=quay.ceph.io/ceph-ci/ceph@sha256:7a1644eb327b921af6e6cf30118d696e8368bd632346b48049fba15207e60112, name=ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0, org.label-schema.schema-version=1.0, io.buildah.version=1.31.3, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, GIT_CLEAN=True, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, org.label-schema.build-date=20231212, ceph=True, RELEASE=quincy-1f2f551, maintainer=Guillaume Abrioux , GIT_COMMIT=0396eef90bef641b676c164ec7a3876f45010308, GIT_REPO=git@github.com:ceph/ceph-container.git) 2023-12-16T22:16:21.021 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:16:20 smithi045 podman[137898]: 2023-12-16 22:16:20.92996506 +0000 UTC m=+0.063825747 container create 0124f3595b2bba9926521477b3466913f244c771e4188fa180c1a13c7a9cbc93 (image=quay.ceph.io/ceph-ci/ceph@sha256:7a1644eb327b921af6e6cf30118d696e8368bd632346b48049fba15207e60112, name=ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0-deactivate, org.label-schema.build-date=20231212, org.label-schema.license=GPLv2, GIT_COMMIT=0396eef90bef641b676c164ec7a3876f45010308, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , GIT_CLEAN=True, RELEASE=quincy-1f2f551, ceph=True, org.label-schema.schema-version=1.0, io.buildah.version=1.31.3) 2023-12-16T22:16:21.022 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:16:20 smithi045 podman[137898]: 2023-12-16 22:16:20.884389064 +0000 UTC m=+0.018249751 image pull 60cc83a9c8f10fdde41a08d59bce905c1a01fc7389b91726758c2505caa3f0c5 quay.ceph.io/ceph-ci/ceph@sha256:7a1644eb327b921af6e6cf30118d696e8368bd632346b48049fba15207e60112 2023-12-16T22:16:21.022 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:16:20 smithi045 podman[137898]: 2023-12-16 22:16:20.997196445 +0000 UTC m=+0.131057133 container init 0124f3595b2bba9926521477b3466913f244c771e4188fa180c1a13c7a9cbc93 (image=quay.ceph.io/ceph-ci/ceph@sha256:7a1644eb327b921af6e6cf30118d696e8368bd632346b48049fba15207e60112, name=ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0-deactivate, maintainer=Guillaume Abrioux , GIT_COMMIT=0396eef90bef641b676c164ec7a3876f45010308, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, io.buildah.version=1.31.3, ceph=True, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, org.label-schema.build-date=20231212, RELEASE=quincy-1f2f551, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=) 2023-12-16T22:16:21.022 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:16:21 smithi045 podman[137898]: 2023-12-16 22:16:21.015290417 +0000 UTC m=+0.149151109 container start 0124f3595b2bba9926521477b3466913f244c771e4188fa180c1a13c7a9cbc93 (image=quay.ceph.io/ceph-ci/ceph@sha256:7a1644eb327b921af6e6cf30118d696e8368bd632346b48049fba15207e60112, name=ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0-deactivate, io.buildah.version=1.31.3, CEPH_POINT_RELEASE=, ceph=True, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, GIT_COMMIT=0396eef90bef641b676c164ec7a3876f45010308, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, RELEASE=quincy-1f2f551, org.label-schema.build-date=20231212, maintainer=Guillaume Abrioux ) 2023-12-16T22:16:21.022 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:16:20 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:16:20.767+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:10:42.731761+0000 (oldest deadline 2023-12-16T22:11:02.731761+0000) 2023-12-16T22:16:21.022 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:16:20 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:16:20.767+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:10:42.731761+0000 (oldest deadline 2023-12-16T22:11:02.731761+0000) 2023-12-16T22:16:21.022 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:16:20 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:16:20.767+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:10:42.731761+0000 (oldest deadline 2023-12-16T22:11:02.731761+0000) 2023-12-16T22:16:21.023 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:16:20 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:16:20.768+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:16:21.315 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:16:21 smithi045 podman[137898]: 2023-12-16 22:16:21.043453345 +0000 UTC m=+0.177314039 container attach 0124f3595b2bba9926521477b3466913f244c771e4188fa180c1a13c7a9cbc93 (image=quay.ceph.io/ceph-ci/ceph@sha256:7a1644eb327b921af6e6cf30118d696e8368bd632346b48049fba15207e60112, name=ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0-deactivate, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, ceph=True, RELEASE=quincy-1f2f551, org.label-schema.build-date=20231212, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, GIT_CLEAN=True, GIT_COMMIT=0396eef90bef641b676c164ec7a3876f45010308, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, io.buildah.version=1.31.3) 2023-12-16T22:16:21.315 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:16:21 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:16:21.020+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:11:07.849373+0000 (oldest deadline 2023-12-16T22:11:27.849373+0000) 2023-12-16T22:16:21.315 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:16:21 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:16:21.020+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:11:07.849373+0000 (oldest deadline 2023-12-16T22:11:27.849373+0000) 2023-12-16T22:16:21.316 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:16:21 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:16:21.020+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:11:07.849373+0000 (oldest deadline 2023-12-16T22:11:27.849373+0000) 2023-12-16T22:16:21.316 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:16:21 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:16:21.021+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:16:21.743 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:16:21 smithi045 podman[137898]: 2023-12-16 22:16:21.326899341 +0000 UTC m=+0.460760040 container died 0124f3595b2bba9926521477b3466913f244c771e4188fa180c1a13c7a9cbc93 (image=quay.ceph.io/ceph-ci/ceph@sha256:7a1644eb327b921af6e6cf30118d696e8368bd632346b48049fba15207e60112, name=ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0-deactivate, CEPH_POINT_RELEASE=, org.label-schema.build-date=20231212, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, io.buildah.version=1.31.3, ceph=True, RELEASE=quincy-1f2f551, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, GIT_COMMIT=0396eef90bef641b676c164ec7a3876f45010308, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS) 2023-12-16T22:16:21.744 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:16:21 smithi045 podman[137928]: 2023-12-16 22:16:21.538200916 +0000 UTC m=+0.205875072 container remove 0124f3595b2bba9926521477b3466913f244c771e4188fa180c1a13c7a9cbc93 (image=quay.ceph.io/ceph-ci/ceph@sha256:7a1644eb327b921af6e6cf30118d696e8368bd632346b48049fba15207e60112, name=ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-0-deactivate, RELEASE=quincy-1f2f551, GIT_CLEAN=True, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, GIT_COMMIT=0396eef90bef641b676c164ec7a3876f45010308, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, io.buildah.version=1.31.3, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , org.label-schema.build-date=20231212, ceph=True, org.label-schema.schema-version=1.0) 2023-12-16T22:16:21.744 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:16:21 smithi045 systemd[1]: ceph-88ff363c-9c5a-11ee-95a5-87774f69a715@osd.0.service: Deactivated successfully. 2023-12-16T22:16:21.744 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:16:21 smithi045 systemd[1]: Stopped Ceph osd.0 for 88ff363c-9c5a-11ee-95a5-87774f69a715. 2023-12-16T22:16:21.744 INFO:journalctl@ceph.osd.0.smithi045.stdout:Dec 16 22:16:21 smithi045 systemd[1]: ceph-88ff363c-9c5a-11ee-95a5-87774f69a715@osd.0.service: Consumed 18min 7.441s CPU time. 2023-12-16T22:16:22.035 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:16:21 smithi045 systemd[1]: Stopping Ceph osd.1 for 88ff363c-9c5a-11ee-95a5-87774f69a715... 2023-12-16T22:16:22.035 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:16:21 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:16:21.788+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:10:42.731761+0000 (oldest deadline 2023-12-16T22:11:02.731761+0000) 2023-12-16T22:16:22.035 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:16:21 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:16:21.788+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:10:42.731761+0000 (oldest deadline 2023-12-16T22:11:02.731761+0000) 2023-12-16T22:16:22.036 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:16:21 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:16:21.788+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:10:42.731761+0000 (oldest deadline 2023-12-16T22:11:02.731761+0000) 2023-12-16T22:16:22.036 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:16:21 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:16:21.789+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:16:22.036 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:16:21 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:16:21.965+0000 7fa1b5f79700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.1 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2023-12-16T22:16:22.036 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:16:21 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:16:21.965+0000 7fa1b5f79700 -1 osd.1 43 *** Got signal Terminated *** 2023-12-16T22:16:22.037 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:16:21 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:16:21.965+0000 7fa1b5f79700 -1 osd.1 43 *** Immediate shutdown (osd_fast_shutdown=true) *** 2023-12-16T22:16:22.314 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:16:22 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:16:22.033+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:11:07.849373+0000 (oldest deadline 2023-12-16T22:11:27.849373+0000) 2023-12-16T22:16:22.314 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:16:22 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:16:22.033+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:11:07.849373+0000 (oldest deadline 2023-12-16T22:11:27.849373+0000) 2023-12-16T22:16:22.314 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:16:22 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:16:22.033+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:11:07.849373+0000 (oldest deadline 2023-12-16T22:11:27.849373+0000) 2023-12-16T22:16:22.315 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:16:22 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:16:22.034+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:16:23.064 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:16:22 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:16:22.758+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:10:42.731761+0000 (oldest deadline 2023-12-16T22:11:02.731761+0000) 2023-12-16T22:16:23.064 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:16:22 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:16:22.758+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:10:42.731761+0000 (oldest deadline 2023-12-16T22:11:02.731761+0000) 2023-12-16T22:16:23.065 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:16:22 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:16:22.758+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:10:42.731761+0000 (oldest deadline 2023-12-16T22:11:02.731761+0000) 2023-12-16T22:16:23.065 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:16:22 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:16:22.759+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:16:23.065 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:16:23 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:16:23.006+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:11:07.849373+0000 (oldest deadline 2023-12-16T22:11:27.849373+0000) 2023-12-16T22:16:23.065 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:16:23 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:16:23.006+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:11:07.849373+0000 (oldest deadline 2023-12-16T22:11:27.849373+0000) 2023-12-16T22:16:23.065 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:16:23 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:16:23.006+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:11:07.849373+0000 (oldest deadline 2023-12-16T22:11:27.849373+0000) 2023-12-16T22:16:23.065 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:16:23 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:16:23.007+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:16:23.988 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:16:23 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:16:23.730+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:10:42.731761+0000 (oldest deadline 2023-12-16T22:11:02.731761+0000) 2023-12-16T22:16:23.989 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:16:23 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:16:23.730+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:10:42.731761+0000 (oldest deadline 2023-12-16T22:11:02.731761+0000) 2023-12-16T22:16:23.989 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:16:23 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:16:23.730+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:10:42.731761+0000 (oldest deadline 2023-12-16T22:11:02.731761+0000) 2023-12-16T22:16:23.989 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:16:23 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:16:23.731+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:16:24.314 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:16:23 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:16:23.988+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:11:07.849373+0000 (oldest deadline 2023-12-16T22:11:27.849373+0000) 2023-12-16T22:16:24.314 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:16:23 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:16:23.988+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:11:07.849373+0000 (oldest deadline 2023-12-16T22:11:27.849373+0000) 2023-12-16T22:16:24.315 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:16:23 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:16:23.988+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:11:07.849373+0000 (oldest deadline 2023-12-16T22:11:27.849373+0000) 2023-12-16T22:16:24.315 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:16:23 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:16:23.989+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:16:25.065 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:16:24 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:16:24.729+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:10:42.731761+0000 (oldest deadline 2023-12-16T22:11:02.731761+0000) 2023-12-16T22:16:25.065 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:16:24 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:16:24.729+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:10:42.731761+0000 (oldest deadline 2023-12-16T22:11:02.731761+0000) 2023-12-16T22:16:25.065 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:16:24 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:16:24.729+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:10:42.731761+0000 (oldest deadline 2023-12-16T22:11:02.731761+0000) 2023-12-16T22:16:25.066 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:16:24 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:16:24.730+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:16:25.066 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:16:24 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:16:24.964+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:11:07.849373+0000 (oldest deadline 2023-12-16T22:11:27.849373+0000) 2023-12-16T22:16:25.066 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:16:24 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:16:24.964+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:11:07.849373+0000 (oldest deadline 2023-12-16T22:11:27.849373+0000) 2023-12-16T22:16:25.066 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:16:24 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:16:24.964+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:11:07.849373+0000 (oldest deadline 2023-12-16T22:11:27.849373+0000) 2023-12-16T22:16:25.066 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:16:24 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:16:24.965+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:16:26.064 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:16:25 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:16:25.919+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:11:07.849373+0000 (oldest deadline 2023-12-16T22:11:27.849373+0000) 2023-12-16T22:16:26.064 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:16:25 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:16:25.919+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:11:07.849373+0000 (oldest deadline 2023-12-16T22:11:27.849373+0000) 2023-12-16T22:16:26.064 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:16:25 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:16:25.919+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:11:07.849373+0000 (oldest deadline 2023-12-16T22:11:27.849373+0000) 2023-12-16T22:16:26.065 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:16:25 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:16:25.920+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:16:26.065 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:16:25 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:16:25.759+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:10:42.731761+0000 (oldest deadline 2023-12-16T22:11:02.731761+0000) 2023-12-16T22:16:26.065 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:16:25 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:16:25.759+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:10:42.731761+0000 (oldest deadline 2023-12-16T22:11:02.731761+0000) 2023-12-16T22:16:26.065 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:16:25 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:16:25.759+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:10:42.731761+0000 (oldest deadline 2023-12-16T22:11:02.731761+0000) 2023-12-16T22:16:26.065 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:16:25 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:16:25.760+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:16:27.064 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:16:26 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:16:26.800+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:10:42.731761+0000 (oldest deadline 2023-12-16T22:11:02.731761+0000) 2023-12-16T22:16:27.064 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:16:26 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:16:26.800+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:10:42.731761+0000 (oldest deadline 2023-12-16T22:11:02.731761+0000) 2023-12-16T22:16:27.065 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:16:26 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:16:26.800+0000 7fa1b3579700 -1 osd.1 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:10:42.731761+0000 (oldest deadline 2023-12-16T22:11:02.731761+0000) 2023-12-16T22:16:27.065 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:16:26 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1[107440]: 2023-12-16T22:16:26.800+0000 7fa1b3579700 -1 osd.1 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6168 3.19 3:9d6673d1:::10000003700.00000000:head [write 0~1881 in=1881b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:16:27.065 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:16:26 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:16:26.916+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:11:07.849373+0000 (oldest deadline 2023-12-16T22:11:27.849373+0000) 2023-12-16T22:16:27.065 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:16:26 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:16:26.916+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:11:07.849373+0000 (oldest deadline 2023-12-16T22:11:27.849373+0000) 2023-12-16T22:16:27.065 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:16:26 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:16:26.916+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:11:07.849373+0000 (oldest deadline 2023-12-16T22:11:27.849373+0000) 2023-12-16T22:16:27.065 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:16:26 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:16:26.916+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:16:27.814 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:16:27 smithi045 podman[138014]: 2023-12-16 22:16:27.547914601 +0000 UTC m=+5.631863442 container died 94ded6ce4ce12b2507f26394a1373bb36052ae7878bf570e019f41c9b1d7b254 (image=quay.ceph.io/ceph-ci/ceph@sha256:7a1644eb327b921af6e6cf30118d696e8368bd632346b48049fba15207e60112, name=ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1, maintainer=Guillaume Abrioux , GIT_CLEAN=True, org.label-schema.vendor=CentOS, GIT_COMMIT=0396eef90bef641b676c164ec7a3876f45010308, org.label-schema.build-date=20231212, org.label-schema.license=GPLv2, io.buildah.version=1.31.3, GIT_BRANCH=HEAD, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, RELEASE=quincy-1f2f551) 2023-12-16T22:16:27.814 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:16:27 smithi045 podman[138014]: 2023-12-16 22:16:27.645701666 +0000 UTC m=+5.729650486 container cleanup 94ded6ce4ce12b2507f26394a1373bb36052ae7878bf570e019f41c9b1d7b254 (image=quay.ceph.io/ceph-ci/ceph@sha256:7a1644eb327b921af6e6cf30118d696e8368bd632346b48049fba15207e60112, name=ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.31.3, GIT_CLEAN=True, ceph=True, GIT_BRANCH=HEAD, org.label-schema.build-date=20231212, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, RELEASE=quincy-1f2f551, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=0396eef90bef641b676c164ec7a3876f45010308) 2023-12-16T22:16:27.815 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:16:27 smithi045 bash[138014]: ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1 2023-12-16T22:16:27.815 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:16:27 smithi045 podman[138033]: 2023-12-16 22:16:27.7552847 +0000 UTC m=+0.201997310 container remove 94ded6ce4ce12b2507f26394a1373bb36052ae7878bf570e019f41c9b1d7b254 (image=quay.ceph.io/ceph-ci/ceph@sha256:7a1644eb327b921af6e6cf30118d696e8368bd632346b48049fba15207e60112, name=ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1, GIT_CLEAN=True, org.label-schema.schema-version=1.0, ceph=True, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , org.label-schema.build-date=20231212, RELEASE=quincy-1f2f551, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.31.3, GIT_COMMIT=0396eef90bef641b676c164ec7a3876f45010308, org.label-schema.license=GPLv2) 2023-12-16T22:16:28.222 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:16:27 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:16:27.962+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:11:07.849373+0000 (oldest deadline 2023-12-16T22:11:27.849373+0000) 2023-12-16T22:16:28.223 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:16:27 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:16:27.962+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:11:07.849373+0000 (oldest deadline 2023-12-16T22:11:27.849373+0000) 2023-12-16T22:16:28.223 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:16:27 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:16:27.962+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:11:07.849373+0000 (oldest deadline 2023-12-16T22:11:27.849373+0000) 2023-12-16T22:16:28.223 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:16:27 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:16:27.963+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:16:28.223 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:16:28 smithi045 podman[138124]: 2023-12-16 22:16:28.120670442 +0000 UTC m=+0.060629935 container create 138f4eb1bed4a62de741e43d047d5202ba08de3a15d3b325e00d2e05c30b0f39 (image=quay.ceph.io/ceph-ci/ceph@sha256:7a1644eb327b921af6e6cf30118d696e8368bd632346b48049fba15207e60112, name=ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1-deactivate, ceph=True, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=quincy-1f2f551, CEPH_POINT_RELEASE=, org.label-schema.build-date=20231212, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, io.buildah.version=1.31.3, GIT_CLEAN=True, GIT_COMMIT=0396eef90bef641b676c164ec7a3876f45010308, org.label-schema.schema-version=1.0) 2023-12-16T22:16:28.224 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:16:28 smithi045 podman[138124]: 2023-12-16 22:16:28.078108938 +0000 UTC m=+0.018068403 image pull 60cc83a9c8f10fdde41a08d59bce905c1a01fc7389b91726758c2505caa3f0c5 quay.ceph.io/ceph-ci/ceph@sha256:7a1644eb327b921af6e6cf30118d696e8368bd632346b48049fba15207e60112 2023-12-16T22:16:28.224 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:16:28 smithi045 podman[138124]: 2023-12-16 22:16:28.20445204 +0000 UTC m=+0.144411492 container init 138f4eb1bed4a62de741e43d047d5202ba08de3a15d3b325e00d2e05c30b0f39 (image=quay.ceph.io/ceph-ci/ceph@sha256:7a1644eb327b921af6e6cf30118d696e8368bd632346b48049fba15207e60112, name=ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1-deactivate, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, GIT_COMMIT=0396eef90bef641b676c164ec7a3876f45010308, RELEASE=quincy-1f2f551, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, GIT_CLEAN=True, ceph=True, org.label-schema.build-date=20231212, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.31.3) 2023-12-16T22:16:28.224 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:16:28 smithi045 podman[138124]: 2023-12-16 22:16:28.222721052 +0000 UTC m=+0.162680496 container start 138f4eb1bed4a62de741e43d047d5202ba08de3a15d3b325e00d2e05c30b0f39 (image=quay.ceph.io/ceph-ci/ceph@sha256:7a1644eb327b921af6e6cf30118d696e8368bd632346b48049fba15207e60112, name=ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1-deactivate, org.label-schema.vendor=CentOS, org.label-schema.build-date=20231212, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, io.buildah.version=1.31.3, ceph=True, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, GIT_COMMIT=0396eef90bef641b676c164ec7a3876f45010308, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, RELEASE=quincy-1f2f551) 2023-12-16T22:16:28.510 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:16:28 smithi045 podman[138124]: 2023-12-16 22:16:28.272197758 +0000 UTC m=+0.212157209 container attach 138f4eb1bed4a62de741e43d047d5202ba08de3a15d3b325e00d2e05c30b0f39 (image=quay.ceph.io/ceph-ci/ceph@sha256:7a1644eb327b921af6e6cf30118d696e8368bd632346b48049fba15207e60112, name=ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1-deactivate, io.buildah.version=1.31.3, maintainer=Guillaume Abrioux , org.label-schema.build-date=20231212, GIT_COMMIT=0396eef90bef641b676c164ec7a3876f45010308, GIT_BRANCH=HEAD, RELEASE=quincy-1f2f551, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, GIT_CLEAN=True, CEPH_POINT_RELEASE=) 2023-12-16T22:16:28.814 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:16:28 smithi045 podman[138124]: 2023-12-16 22:16:28.510668882 +0000 UTC m=+0.450628360 container died 138f4eb1bed4a62de741e43d047d5202ba08de3a15d3b325e00d2e05c30b0f39 (image=quay.ceph.io/ceph-ci/ceph@sha256:7a1644eb327b921af6e6cf30118d696e8368bd632346b48049fba15207e60112, name=ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1-deactivate, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=0396eef90bef641b676c164ec7a3876f45010308, RELEASE=quincy-1f2f551, io.buildah.version=1.31.3, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, ceph=True, org.label-schema.schema-version=1.0, org.label-schema.build-date=20231212, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, maintainer=Guillaume Abrioux ) 2023-12-16T22:16:28.814 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:16:28 smithi045 podman[138151]: 2023-12-16 22:16:28.728907273 +0000 UTC m=+0.208802057 container remove 138f4eb1bed4a62de741e43d047d5202ba08de3a15d3b325e00d2e05c30b0f39 (image=quay.ceph.io/ceph-ci/ceph@sha256:7a1644eb327b921af6e6cf30118d696e8368bd632346b48049fba15207e60112, name=ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-1-deactivate, RELEASE=quincy-1f2f551, CEPH_POINT_RELEASE=, io.buildah.version=1.31.3, ceph=True, GIT_COMMIT=0396eef90bef641b676c164ec7a3876f45010308, org.label-schema.build-date=20231212, org.label-schema.vendor=CentOS, GIT_CLEAN=True, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git) 2023-12-16T22:16:28.815 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:16:28 smithi045 systemd[1]: ceph-88ff363c-9c5a-11ee-95a5-87774f69a715@osd.1.service: Deactivated successfully. 2023-12-16T22:16:28.815 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:16:28 smithi045 systemd[1]: Stopped Ceph osd.1 for 88ff363c-9c5a-11ee-95a5-87774f69a715. 2023-12-16T22:16:28.815 INFO:journalctl@ceph.osd.1.smithi045.stdout:Dec 16 22:16:28 smithi045 systemd[1]: ceph-88ff363c-9c5a-11ee-95a5-87774f69a715@osd.1.service: Consumed 18min 11.145s CPU time. 2023-12-16T22:16:29.314 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:16:28 smithi045 systemd[1]: Stopping Ceph osd.2 for 88ff363c-9c5a-11ee-95a5-87774f69a715... 2023-12-16T22:16:29.315 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:16:29 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:16:29.010+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:11:07.849373+0000 (oldest deadline 2023-12-16T22:11:27.849373+0000) 2023-12-16T22:16:29.315 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:16:29 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:16:29.010+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:11:07.849373+0000 (oldest deadline 2023-12-16T22:11:27.849373+0000) 2023-12-16T22:16:29.315 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:16:29 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:16:29.010+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:11:07.849373+0000 (oldest deadline 2023-12-16T22:11:27.849373+0000) 2023-12-16T22:16:29.315 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:16:29 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:16:29.011+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:16:29.315 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:16:29 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:16:29.122+0000 7f950f48a700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.2 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2023-12-16T22:16:29.315 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:16:29 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:16:29.122+0000 7f950f48a700 -1 osd.2 43 *** Got signal Terminated *** 2023-12-16T22:16:29.316 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:16:29 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:16:29.122+0000 7f950f48a700 -1 osd.2 43 *** Immediate shutdown (osd_fast_shutdown=true) *** 2023-12-16T22:16:30.314 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:16:30 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:16:30.018+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:11:07.849373+0000 (oldest deadline 2023-12-16T22:11:27.849373+0000) 2023-12-16T22:16:30.314 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:16:30 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:16:30.018+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:11:07.849373+0000 (oldest deadline 2023-12-16T22:11:27.849373+0000) 2023-12-16T22:16:30.314 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:16:30 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:16:30.018+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:11:07.849373+0000 (oldest deadline 2023-12-16T22:11:27.849373+0000) 2023-12-16T22:16:30.314 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:16:30 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:16:30.019+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:16:31.314 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:16:30 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:16:30.994+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:11:07.849373+0000 (oldest deadline 2023-12-16T22:11:27.849373+0000) 2023-12-16T22:16:31.315 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:16:30 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:16:30.994+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:11:07.849373+0000 (oldest deadline 2023-12-16T22:11:27.849373+0000) 2023-12-16T22:16:31.315 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:16:30 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:16:30.994+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:11:07.849373+0000 (oldest deadline 2023-12-16T22:11:27.849373+0000) 2023-12-16T22:16:31.315 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:16:30 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:16:30.995+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:16:32.314 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:16:31 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:16:31.961+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:11:07.849373+0000 (oldest deadline 2023-12-16T22:11:27.849373+0000) 2023-12-16T22:16:32.314 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:16:31 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:16:31.961+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:11:07.849373+0000 (oldest deadline 2023-12-16T22:11:27.849373+0000) 2023-12-16T22:16:32.315 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:16:31 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:16:31.961+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:11:07.849373+0000 (oldest deadline 2023-12-16T22:11:27.849373+0000) 2023-12-16T22:16:32.315 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:16:31 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:16:31.962+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:16:33.314 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:16:32 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:16:32.944+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:11:07.849373+0000 (oldest deadline 2023-12-16T22:11:27.849373+0000) 2023-12-16T22:16:33.314 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:16:32 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:16:32.944+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:11:07.849373+0000 (oldest deadline 2023-12-16T22:11:27.849373+0000) 2023-12-16T22:16:33.314 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:16:32 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:16:32.944+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:11:07.849373+0000 (oldest deadline 2023-12-16T22:11:27.849373+0000) 2023-12-16T22:16:33.315 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:16:32 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:16:32.944+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:16:34.314 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:16:33 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:16:33.989+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6804 osd.3 ever on either front or back, first ping sent 2023-12-16T22:11:07.849373+0000 (oldest deadline 2023-12-16T22:11:27.849373+0000) 2023-12-16T22:16:34.314 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:16:33 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:16:33.989+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6812 osd.4 ever on either front or back, first ping sent 2023-12-16T22:11:07.849373+0000 (oldest deadline 2023-12-16T22:11:27.849373+0000) 2023-12-16T22:16:34.314 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:16:33 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:16:33.989+0000 7f950d28b700 -1 osd.2 43 heartbeat_check: no reply from 172.21.15.169:6820 osd.5 ever on either front or back, first ping sent 2023-12-16T22:11:07.849373+0000 (oldest deadline 2023-12-16T22:11:27.849373+0000) 2023-12-16T22:16:34.315 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:16:33 smithi045 ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2[114867]: 2023-12-16T22:16:33.990+0000 7f950d28b700 -1 osd.2 43 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14530.0:6165 3.7 3:e784fc71:::10000003afa.00000000:head [write 0~128 in=128b] snapc 1=[] ondisk+write+known_if_redirected e43) 2023-12-16T22:16:34.814 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:16:34 smithi045 podman[138230]: 2023-12-16 22:16:34.562353704 +0000 UTC m=+5.487730671 container died 5fc744d14d11edb2a6ad2aca1dff96033a46a358f9d1516dbab4ee2689db4e1d (image=quay.ceph.io/ceph-ci/ceph@sha256:7a1644eb327b921af6e6cf30118d696e8368bd632346b48049fba15207e60112, name=ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2, RELEASE=quincy-1f2f551, org.label-schema.build-date=20231212, org.label-schema.license=GPLv2, GIT_COMMIT=0396eef90bef641b676c164ec7a3876f45010308, io.buildah.version=1.31.3, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, ceph=True, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image) 2023-12-16T22:16:34.815 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:16:34 smithi045 podman[138230]: 2023-12-16 22:16:34.678226429 +0000 UTC m=+5.603603382 container cleanup 5fc744d14d11edb2a6ad2aca1dff96033a46a358f9d1516dbab4ee2689db4e1d (image=quay.ceph.io/ceph-ci/ceph@sha256:7a1644eb327b921af6e6cf30118d696e8368bd632346b48049fba15207e60112, name=ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2, org.label-schema.schema-version=1.0, GIT_CLEAN=True, maintainer=Guillaume Abrioux , io.buildah.version=1.31.3, org.label-schema.build-date=20231212, CEPH_POINT_RELEASE=, ceph=True, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, GIT_COMMIT=0396eef90bef641b676c164ec7a3876f45010308, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=quincy-1f2f551, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git) 2023-12-16T22:16:34.815 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:16:34 smithi045 bash[138230]: ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2 2023-12-16T22:16:35.103 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:16:34 smithi045 podman[138250]: 2023-12-16 22:16:34.81965089 +0000 UTC m=+0.260096027 container remove 5fc744d14d11edb2a6ad2aca1dff96033a46a358f9d1516dbab4ee2689db4e1d (image=quay.ceph.io/ceph-ci/ceph@sha256:7a1644eb327b921af6e6cf30118d696e8368bd632346b48049fba15207e60112, name=ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, io.buildah.version=1.31.3, org.label-schema.build-date=20231212, GIT_COMMIT=0396eef90bef641b676c164ec7a3876f45010308, RELEASE=quincy-1f2f551, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, GIT_CLEAN=True, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=) 2023-12-16T22:16:35.476 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:16:35 smithi045 podman[138343]: 2023-12-16 22:16:35.102963901 +0000 UTC m=+0.063792490 container create 900d26d2a74e3481d2dc10685ecac34210750765a8249f2ec0efe74deea886c6 (image=quay.ceph.io/ceph-ci/ceph@sha256:7a1644eb327b921af6e6cf30118d696e8368bd632346b48049fba15207e60112, name=ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2-deactivate, io.buildah.version=1.31.3, GIT_CLEAN=True, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, ceph=True, GIT_COMMIT=0396eef90bef641b676c164ec7a3876f45010308, RELEASE=quincy-1f2f551, org.label-schema.license=GPLv2, org.label-schema.build-date=20231212) 2023-12-16T22:16:35.476 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:16:35 smithi045 podman[138343]: 2023-12-16 22:16:35.05728094 +0000 UTC m=+0.018109543 image pull 60cc83a9c8f10fdde41a08d59bce905c1a01fc7389b91726758c2505caa3f0c5 quay.ceph.io/ceph-ci/ceph@sha256:7a1644eb327b921af6e6cf30118d696e8368bd632346b48049fba15207e60112 2023-12-16T22:16:35.476 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:16:35 smithi045 podman[138343]: 2023-12-16 22:16:35.169969269 +0000 UTC m=+0.130797871 container init 900d26d2a74e3481d2dc10685ecac34210750765a8249f2ec0efe74deea886c6 (image=quay.ceph.io/ceph-ci/ceph@sha256:7a1644eb327b921af6e6cf30118d696e8368bd632346b48049fba15207e60112, name=ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2-deactivate, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, ceph=True, org.label-schema.build-date=20231212, GIT_COMMIT=0396eef90bef641b676c164ec7a3876f45010308, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.31.3, RELEASE=quincy-1f2f551, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS) 2023-12-16T22:16:35.476 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:16:35 smithi045 podman[138343]: 2023-12-16 22:16:35.184173117 +0000 UTC m=+0.145001698 container start 900d26d2a74e3481d2dc10685ecac34210750765a8249f2ec0efe74deea886c6 (image=quay.ceph.io/ceph-ci/ceph@sha256:7a1644eb327b921af6e6cf30118d696e8368bd632346b48049fba15207e60112, name=ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2-deactivate, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, org.label-schema.schema-version=1.0, RELEASE=quincy-1f2f551, GIT_BRANCH=HEAD, org.label-schema.build-date=20231212, GIT_COMMIT=0396eef90bef641b676c164ec7a3876f45010308, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, ceph=True, io.buildah.version=1.31.3) 2023-12-16T22:16:35.476 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:16:35 smithi045 podman[138343]: 2023-12-16 22:16:35.211191081 +0000 UTC m=+0.172019677 container attach 900d26d2a74e3481d2dc10685ecac34210750765a8249f2ec0efe74deea886c6 (image=quay.ceph.io/ceph-ci/ceph@sha256:7a1644eb327b921af6e6cf30118d696e8368bd632346b48049fba15207e60112, name=ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2-deactivate, RELEASE=quincy-1f2f551, org.label-schema.license=GPLv2, org.label-schema.build-date=20231212, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.31.3, org.label-schema.vendor=CentOS, GIT_CLEAN=True, GIT_COMMIT=0396eef90bef641b676c164ec7a3876f45010308, maintainer=Guillaume Abrioux , ceph=True) 2023-12-16T22:16:35.476 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:16:35 smithi045 podman[138343]: 2023-12-16 22:16:35.475526515 +0000 UTC m=+0.436355128 container died 900d26d2a74e3481d2dc10685ecac34210750765a8249f2ec0efe74deea886c6 (image=quay.ceph.io/ceph-ci/ceph@sha256:7a1644eb327b921af6e6cf30118d696e8368bd632346b48049fba15207e60112, name=ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2-deactivate, GIT_BRANCH=HEAD, ceph=True, GIT_COMMIT=0396eef90bef641b676c164ec7a3876f45010308, org.label-schema.schema-version=1.0, RELEASE=quincy-1f2f551, org.label-schema.vendor=CentOS, org.label-schema.build-date=20231212, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, io.buildah.version=1.31.3) 2023-12-16T22:16:35.814 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:16:35 smithi045 podman[138369]: 2023-12-16 22:16:35.694447326 +0000 UTC m=+0.213305775 container remove 900d26d2a74e3481d2dc10685ecac34210750765a8249f2ec0efe74deea886c6 (image=quay.ceph.io/ceph-ci/ceph@sha256:7a1644eb327b921af6e6cf30118d696e8368bd632346b48049fba15207e60112, name=ceph-88ff363c-9c5a-11ee-95a5-87774f69a715-osd-2-deactivate, GIT_BRANCH=HEAD, GIT_CLEAN=True, GIT_COMMIT=0396eef90bef641b676c164ec7a3876f45010308, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , RELEASE=quincy-1f2f551, io.buildah.version=1.31.3, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, org.label-schema.build-date=20231212, ceph=True, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS) 2023-12-16T22:16:35.815 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:16:35 smithi045 systemd[1]: ceph-88ff363c-9c5a-11ee-95a5-87774f69a715@osd.2.service: Deactivated successfully. 2023-12-16T22:16:35.815 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:16:35 smithi045 systemd[1]: Stopped Ceph osd.2 for 88ff363c-9c5a-11ee-95a5-87774f69a715. 2023-12-16T22:16:35.815 INFO:journalctl@ceph.osd.2.smithi045.stdout:Dec 16 22:16:35 smithi045 systemd[1]: ceph-88ff363c-9c5a-11ee-95a5-87774f69a715@osd.2.service: Consumed 18min 14.100s CPU time. 2023-12-16T22:16:41.392 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi169.front.sepia.ceph.com' 2023-12-16T22:16:41.393 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi169.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-12-16T22:16:44.465 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.169 2023-12-16T22:16:53.466 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi169.front.sepia.ceph.com' 2023-12-16T22:16:53.467 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi169.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-12-16T22:16:56.529 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.169 2023-12-16T22:17:08.533 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi169.front.sepia.ceph.com' 2023-12-16T22:17:08.534 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi169.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-12-16T22:17:11.602 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.169 2023-12-16T22:17:14.603 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi169.front.sepia.ceph.com' 2023-12-16T22:17:14.603 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi169.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-12-16T22:17:14.673 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.169 2023-12-16T22:17:14.674 WARNING:teuthology.contextutil:'reconnect to {self.shortname}' reached maximum tries (5) after waiting for 30 seconds 2023-12-16T22:17:14.674 INFO:tasks.cephadm:Teardown complete 2023-12-16T22:17:14.675 ERROR:teuthology.run_tasks:Manager failed: cephadm Traceback (most recent call last): File "/home/teuthworker/src/github.com_ceph_ceph-c_c7586e95519b7be76f3b6e3a4b333888baf46bc9/qa/tasks/cephadm.py", line 1540, in distribute_config_and_admin_keyring yield File "/home/teuthworker/src/git.ceph.com_teuthology_5f3731ee8689ef3b0ef7d571e4cc2cb968c78d73/teuthology/contextutil.py", line 32, in nested yield vars File "/home/teuthworker/src/github.com_ceph_ceph-c_c7586e95519b7be76f3b6e3a4b333888baf46bc9/qa/tasks/cephadm.py", line 1794, in task yield File "/home/teuthworker/src/git.ceph.com_teuthology_5f3731ee8689ef3b0ef7d571e4cc2cb968c78d73/teuthology/run_tasks.py", line 154, in run_tasks suppress = manager.__exit__(*exc_info) File "/usr/lib/python3.8/contextlib.py", line 120, in __exit__ next(self.gen) File "/home/teuthworker/src/github.com_ceph_ceph-c_c7586e95519b7be76f3b6e3a4b333888baf46bc9/qa/tasks/kclient.py", line 139, in task forced = umount_all() File "/home/teuthworker/src/github.com_ceph_ceph-c_c7586e95519b7be76f3b6e3a4b333888baf46bc9/qa/tasks/kclient.py", line 119, in umount_all if mount.is_mounted(): File "/home/teuthworker/src/github.com_ceph_ceph-c_c7586e95519b7be76f3b6e3a4b333888baf46bc9/qa/tasks/cephfs/mount.py", line 220, in is_mounted file = self.client_remote.read_file('/proc/self/mounts',stdout=StringIO()) File "/home/teuthworker/src/git.ceph.com_teuthology_5f3731ee8689ef3b0ef7d571e4cc2cb968c78d73/teuthology/orchestra/remote.py", line 254, in read_file proc = self.run(args=args, stdout=stdout, stderr=StringIO(), File "/home/teuthworker/src/git.ceph.com_teuthology_5f3731ee8689ef3b0ef7d571e4cc2cb968c78d73/teuthology/orchestra/remote.py", line 523, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_5f3731ee8689ef3b0ef7d571e4cc2cb968c78d73/teuthology/orchestra/run.py", line 450, in run r.execute() File "/home/teuthworker/src/git.ceph.com_teuthology_5f3731ee8689ef3b0ef7d571e4cc2cb968c78d73/teuthology/orchestra/run.py", line 100, in execute self.client.exec_command(self.command) File "/home/teuthworker/src/git.ceph.com_teuthology_5f3731ee8689ef3b0ef7d571e4cc2cb968c78d73/virtualenv/lib/python3.8/site-packages/paramiko/client.py", line 525, in exec_command chan = self._transport.open_session(timeout=timeout) File "/home/teuthworker/src/git.ceph.com_teuthology_5f3731ee8689ef3b0ef7d571e4cc2cb968c78d73/virtualenv/lib/python3.8/site-packages/paramiko/transport.py", line 920, in open_session return self.open_channel( File "/home/teuthworker/src/git.ceph.com_teuthology_5f3731ee8689ef3b0ef7d571e4cc2cb968c78d73/virtualenv/lib/python3.8/site-packages/paramiko/transport.py", line 1051, in open_channel raise e File "/home/teuthworker/src/git.ceph.com_teuthology_5f3731ee8689ef3b0ef7d571e4cc2cb968c78d73/virtualenv/lib/python3.8/site-packages/paramiko/transport.py", line 2110, in run ptype, m = self.packetizer.read_message() File "/home/teuthworker/src/git.ceph.com_teuthology_5f3731ee8689ef3b0ef7d571e4cc2cb968c78d73/virtualenv/lib/python3.8/site-packages/paramiko/packet.py", line 459, in read_message header = self.read_all(self.__block_size_in, check_rekey=True) File "/home/teuthworker/src/git.ceph.com_teuthology_5f3731ee8689ef3b0ef7d571e4cc2cb968c78d73/virtualenv/lib/python3.8/site-packages/paramiko/packet.py", line 301, in read_all x = self.__socket.recv(n) File "/home/teuthworker/src/git.ceph.com_teuthology_5f3731ee8689ef3b0ef7d571e4cc2cb968c78d73/virtualenv/lib/python3.8/site-packages/gevent/_socketcommon.py", line 657, in recv return self._sock.recv(*args) OSError: [Errno 113] No route to host During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_5f3731ee8689ef3b0ef7d571e4cc2cb968c78d73/teuthology/run_tasks.py", line 154, in run_tasks suppress = manager.__exit__(*exc_info) File "/usr/lib/python3.8/contextlib.py", line 131, in __exit__ self.gen.throw(type, value, traceback) File "/home/teuthworker/src/github.com_ceph_ceph-c_c7586e95519b7be76f3b6e3a4b333888baf46bc9/qa/tasks/cephadm.py", line 1797, in task log.info('Teardown begin') File "/usr/lib/python3.8/contextlib.py", line 131, in __exit__ self.gen.throw(type, value, traceback) File "/home/teuthworker/src/git.ceph.com_teuthology_5f3731ee8689ef3b0ef7d571e4cc2cb968c78d73/teuthology/contextutil.py", line 54, in nested raise exc[1] File "/usr/lib/python3.8/contextlib.py", line 131, in __exit__ self.gen.throw(type, value, traceback) File "/home/teuthworker/src/github.com_ceph_ceph-c_c7586e95519b7be76f3b6e3a4b333888baf46bc9/qa/tasks/cephadm.py", line 1673, in initialize_config yield File "/home/teuthworker/src/git.ceph.com_teuthology_5f3731ee8689ef3b0ef7d571e4cc2cb968c78d73/teuthology/contextutil.py", line 46, in nested if exit(*exc): File "/usr/lib/python3.8/contextlib.py", line 131, in __exit__ self.gen.throw(type, value, traceback) File "/home/teuthworker/src/github.com_ceph_ceph-c_c7586e95519b7be76f3b6e3a4b333888baf46bc9/qa/tasks/cephadm.py", line 178, in download_cephadm _rm_cluster(ctx, cluster_name) File "/home/teuthworker/src/github.com_ceph_ceph-c_c7586e95519b7be76f3b6e3a4b333888baf46bc9/qa/tasks/cephadm.py", line 297, in _rm_cluster ctx.cluster.run(args=[ File "/home/teuthworker/src/git.ceph.com_teuthology_5f3731ee8689ef3b0ef7d571e4cc2cb968c78d73/teuthology/orchestra/cluster.py", line 85, in run procs = [remote.run(**kwargs, wait=_wait) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_teuthology_5f3731ee8689ef3b0ef7d571e4cc2cb968c78d73/teuthology/orchestra/cluster.py", line 85, in procs = [remote.run(**kwargs, wait=_wait) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_teuthology_5f3731ee8689ef3b0ef7d571e4cc2cb968c78d73/teuthology/orchestra/remote.py", line 522, in run raise ConnectionError(f'Failed to reconnect to {self.shortname}') ConnectionError: Failed to reconnect to smithi169 2023-12-16T22:17:14.696 DEBUG:teuthology.run_tasks:Unwinding manager install 2023-12-16T22:17:14.707 ERROR:teuthology.contextutil:Saw exception from nested tasks Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_5f3731ee8689ef3b0ef7d571e4cc2cb968c78d73/teuthology/contextutil.py", line 32, in nested yield vars File "/home/teuthworker/src/git.ceph.com_teuthology_5f3731ee8689ef3b0ef7d571e4cc2cb968c78d73/teuthology/task/install/__init__.py", line 626, in task yield File "/home/teuthworker/src/git.ceph.com_teuthology_5f3731ee8689ef3b0ef7d571e4cc2cb968c78d73/teuthology/run_tasks.py", line 154, in run_tasks suppress = manager.__exit__(*exc_info) File "/usr/lib/python3.8/contextlib.py", line 120, in __exit__ next(self.gen) File "/home/teuthworker/src/github.com_ceph_ceph-c_c7586e95519b7be76f3b6e3a4b333888baf46bc9/qa/tasks/kclient.py", line 139, in task forced = umount_all() File "/home/teuthworker/src/github.com_ceph_ceph-c_c7586e95519b7be76f3b6e3a4b333888baf46bc9/qa/tasks/kclient.py", line 119, in umount_all if mount.is_mounted(): File "/home/teuthworker/src/github.com_ceph_ceph-c_c7586e95519b7be76f3b6e3a4b333888baf46bc9/qa/tasks/cephfs/mount.py", line 220, in is_mounted file = self.client_remote.read_file('/proc/self/mounts',stdout=StringIO()) File "/home/teuthworker/src/git.ceph.com_teuthology_5f3731ee8689ef3b0ef7d571e4cc2cb968c78d73/teuthology/orchestra/remote.py", line 254, in read_file proc = self.run(args=args, stdout=stdout, stderr=StringIO(), File "/home/teuthworker/src/git.ceph.com_teuthology_5f3731ee8689ef3b0ef7d571e4cc2cb968c78d73/teuthology/orchestra/remote.py", line 523, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_5f3731ee8689ef3b0ef7d571e4cc2cb968c78d73/teuthology/orchestra/run.py", line 450, in run r.execute() File "/home/teuthworker/src/git.ceph.com_teuthology_5f3731ee8689ef3b0ef7d571e4cc2cb968c78d73/teuthology/orchestra/run.py", line 100, in execute self.client.exec_command(self.command) File "/home/teuthworker/src/git.ceph.com_teuthology_5f3731ee8689ef3b0ef7d571e4cc2cb968c78d73/virtualenv/lib/python3.8/site-packages/paramiko/client.py", line 525, in exec_command chan = self._transport.open_session(timeout=timeout) File "/home/teuthworker/src/git.ceph.com_teuthology_5f3731ee8689ef3b0ef7d571e4cc2cb968c78d73/virtualenv/lib/python3.8/site-packages/paramiko/transport.py", line 920, in open_session return self.open_channel( File "/home/teuthworker/src/git.ceph.com_teuthology_5f3731ee8689ef3b0ef7d571e4cc2cb968c78d73/virtualenv/lib/python3.8/site-packages/paramiko/transport.py", line 1051, in open_channel raise e File "/home/teuthworker/src/git.ceph.com_teuthology_5f3731ee8689ef3b0ef7d571e4cc2cb968c78d73/virtualenv/lib/python3.8/site-packages/paramiko/transport.py", line 2110, in run ptype, m = self.packetizer.read_message() File "/home/teuthworker/src/git.ceph.com_teuthology_5f3731ee8689ef3b0ef7d571e4cc2cb968c78d73/virtualenv/lib/python3.8/site-packages/paramiko/packet.py", line 459, in read_message header = self.read_all(self.__block_size_in, check_rekey=True) File "/home/teuthworker/src/git.ceph.com_teuthology_5f3731ee8689ef3b0ef7d571e4cc2cb968c78d73/virtualenv/lib/python3.8/site-packages/paramiko/packet.py", line 301, in read_all x = self.__socket.recv(n) File "/home/teuthworker/src/git.ceph.com_teuthology_5f3731ee8689ef3b0ef7d571e4cc2cb968c78d73/virtualenv/lib/python3.8/site-packages/gevent/_socketcommon.py", line 657, in recv return self._sock.recv(*args) OSError: [Errno 113] No route to host 2023-12-16T22:17:14.723 INFO:teuthology.task.install.util:Removing shipped files: /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer... 2023-12-16T22:17:14.723 DEBUG:teuthology.orchestra.run.smithi045:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2023-12-16T22:17:14.734 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi169.front.sepia.ceph.com' 2023-12-16T22:17:14.735 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi169.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-12-16T22:17:17.813 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.169 2023-12-16T22:17:26.815 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi169.front.sepia.ceph.com' 2023-12-16T22:17:26.816 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi169.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-12-16T22:17:29.877 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.169 2023-12-16T22:17:41.879 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi169.front.sepia.ceph.com' 2023-12-16T22:17:41.880 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi169.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-12-16T22:17:44.949 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.169 2023-12-16T22:17:47.951 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi169.front.sepia.ceph.com' 2023-12-16T22:17:47.951 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi169.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-12-16T22:17:48.021 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.169 2023-12-16T22:17:48.022 WARNING:teuthology.contextutil:'reconnect to {self.shortname}' reached maximum tries (5) after waiting for 30 seconds 2023-12-16T22:17:48.071 ERROR:teuthology.run_tasks:Manager failed: install Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_5f3731ee8689ef3b0ef7d571e4cc2cb968c78d73/teuthology/task/install/util.py", line 148, in ship_utilities yield File "/home/teuthworker/src/git.ceph.com_teuthology_5f3731ee8689ef3b0ef7d571e4cc2cb968c78d73/teuthology/contextutil.py", line 32, in nested yield vars File "/home/teuthworker/src/git.ceph.com_teuthology_5f3731ee8689ef3b0ef7d571e4cc2cb968c78d73/teuthology/task/install/__init__.py", line 626, in task yield File "/home/teuthworker/src/git.ceph.com_teuthology_5f3731ee8689ef3b0ef7d571e4cc2cb968c78d73/teuthology/run_tasks.py", line 154, in run_tasks suppress = manager.__exit__(*exc_info) File "/usr/lib/python3.8/contextlib.py", line 120, in __exit__ next(self.gen) File "/home/teuthworker/src/github.com_ceph_ceph-c_c7586e95519b7be76f3b6e3a4b333888baf46bc9/qa/tasks/kclient.py", line 139, in task forced = umount_all() File "/home/teuthworker/src/github.com_ceph_ceph-c_c7586e95519b7be76f3b6e3a4b333888baf46bc9/qa/tasks/kclient.py", line 119, in umount_all if mount.is_mounted(): File "/home/teuthworker/src/github.com_ceph_ceph-c_c7586e95519b7be76f3b6e3a4b333888baf46bc9/qa/tasks/cephfs/mount.py", line 220, in is_mounted file = self.client_remote.read_file('/proc/self/mounts',stdout=StringIO()) File "/home/teuthworker/src/git.ceph.com_teuthology_5f3731ee8689ef3b0ef7d571e4cc2cb968c78d73/teuthology/orchestra/remote.py", line 254, in read_file proc = self.run(args=args, stdout=stdout, stderr=StringIO(), File "/home/teuthworker/src/git.ceph.com_teuthology_5f3731ee8689ef3b0ef7d571e4cc2cb968c78d73/teuthology/orchestra/remote.py", line 523, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_5f3731ee8689ef3b0ef7d571e4cc2cb968c78d73/teuthology/orchestra/run.py", line 450, in run r.execute() File "/home/teuthworker/src/git.ceph.com_teuthology_5f3731ee8689ef3b0ef7d571e4cc2cb968c78d73/teuthology/orchestra/run.py", line 100, in execute self.client.exec_command(self.command) File "/home/teuthworker/src/git.ceph.com_teuthology_5f3731ee8689ef3b0ef7d571e4cc2cb968c78d73/virtualenv/lib/python3.8/site-packages/paramiko/client.py", line 525, in exec_command chan = self._transport.open_session(timeout=timeout) File "/home/teuthworker/src/git.ceph.com_teuthology_5f3731ee8689ef3b0ef7d571e4cc2cb968c78d73/virtualenv/lib/python3.8/site-packages/paramiko/transport.py", line 920, in open_session return self.open_channel( File "/home/teuthworker/src/git.ceph.com_teuthology_5f3731ee8689ef3b0ef7d571e4cc2cb968c78d73/virtualenv/lib/python3.8/site-packages/paramiko/transport.py", line 1051, in open_channel raise e File "/home/teuthworker/src/git.ceph.com_teuthology_5f3731ee8689ef3b0ef7d571e4cc2cb968c78d73/virtualenv/lib/python3.8/site-packages/paramiko/transport.py", line 2110, in run ptype, m = self.packetizer.read_message() File "/home/teuthworker/src/git.ceph.com_teuthology_5f3731ee8689ef3b0ef7d571e4cc2cb968c78d73/virtualenv/lib/python3.8/site-packages/paramiko/packet.py", line 459, in read_message header = self.read_all(self.__block_size_in, check_rekey=True) File "/home/teuthworker/src/git.ceph.com_teuthology_5f3731ee8689ef3b0ef7d571e4cc2cb968c78d73/virtualenv/lib/python3.8/site-packages/paramiko/packet.py", line 301, in read_all x = self.__socket.recv(n) File "/home/teuthworker/src/git.ceph.com_teuthology_5f3731ee8689ef3b0ef7d571e4cc2cb968c78d73/virtualenv/lib/python3.8/site-packages/gevent/_socketcommon.py", line 657, in recv return self._sock.recv(*args) OSError: [Errno 113] No route to host During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_5f3731ee8689ef3b0ef7d571e4cc2cb968c78d73/teuthology/run_tasks.py", line 154, in run_tasks suppress = manager.__exit__(*exc_info) File "/usr/lib/python3.8/contextlib.py", line 131, in __exit__ self.gen.throw(type, value, traceback) File "/home/teuthworker/src/git.ceph.com_teuthology_5f3731ee8689ef3b0ef7d571e4cc2cb968c78d73/teuthology/task/install/__init__.py", line 626, in task yield File "/usr/lib/python3.8/contextlib.py", line 131, in __exit__ self.gen.throw(type, value, traceback) File "/home/teuthworker/src/git.ceph.com_teuthology_5f3731ee8689ef3b0ef7d571e4cc2cb968c78d73/teuthology/contextutil.py", line 54, in nested raise exc[1] File "/usr/lib/python3.8/contextlib.py", line 131, in __exit__ self.gen.throw(type, value, traceback) File "/home/teuthworker/src/git.ceph.com_teuthology_5f3731ee8689ef3b0ef7d571e4cc2cb968c78d73/teuthology/task/install/__init__.py", line 220, in install yield File "/home/teuthworker/src/git.ceph.com_teuthology_5f3731ee8689ef3b0ef7d571e4cc2cb968c78d73/teuthology/contextutil.py", line 46, in nested if exit(*exc): File "/usr/lib/python3.8/contextlib.py", line 131, in __exit__ self.gen.throw(type, value, traceback) File "/home/teuthworker/src/git.ceph.com_teuthology_5f3731ee8689ef3b0ef7d571e4cc2cb968c78d73/teuthology/task/install/util.py", line 150, in ship_utilities _remove_utilities(ctx, filenames) File "/home/teuthworker/src/git.ceph.com_teuthology_5f3731ee8689ef3b0ef7d571e4cc2cb968c78d73/teuthology/task/install/util.py", line 121, in _remove_utilities ctx.cluster.run( File "/home/teuthworker/src/git.ceph.com_teuthology_5f3731ee8689ef3b0ef7d571e4cc2cb968c78d73/teuthology/orchestra/cluster.py", line 85, in run procs = [remote.run(**kwargs, wait=_wait) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_teuthology_5f3731ee8689ef3b0ef7d571e4cc2cb968c78d73/teuthology/orchestra/cluster.py", line 85, in procs = [remote.run(**kwargs, wait=_wait) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_teuthology_5f3731ee8689ef3b0ef7d571e4cc2cb968c78d73/teuthology/orchestra/remote.py", line 522, in run raise ConnectionError(f'Failed to reconnect to {self.shortname}') ConnectionError: Failed to reconnect to smithi169 2023-12-16T22:17:48.090 DEBUG:teuthology.run_tasks:Unwinding manager clock 2023-12-16T22:17:48.103 INFO:teuthology.task.clock:Checking final clock skew... 2023-12-16T22:17:48.104 DEBUG:teuthology.orchestra.run.smithi045:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2023-12-16T22:17:48.115 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi169.front.sepia.ceph.com' 2023-12-16T22:17:48.115 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi169.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-12-16T22:17:48.131 INFO:teuthology.orchestra.run.smithi045.stderr:bash: line 1: ntpq: command not found 2023-12-16T22:17:48.135 INFO:teuthology.orchestra.run.smithi045.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2023-12-16T22:17:48.135 INFO:teuthology.orchestra.run.smithi045.stdout:=============================================================================== 2023-12-16T22:17:48.136 INFO:teuthology.orchestra.run.smithi045.stdout:^+ hv01.front.sepia.ceph.com 2 6 377 32 -7195ns[-7195ns] +/- 82ms 2023-12-16T22:17:48.136 INFO:teuthology.orchestra.run.smithi045.stdout:^* hv02.front.sepia.ceph.com 3 8 377 97 -152us[ -152us] +/- 77ms 2023-12-16T22:17:48.136 INFO:teuthology.orchestra.run.smithi045.stdout:^+ hv03.front.sepia.ceph.com 3 8 377 37 +171us[ +171us] +/- 66ms 2023-12-16T22:17:48.136 INFO:teuthology.orchestra.run.smithi045.stdout:^? hv04.front.sepia.ceph.com 0 9 0 - +0ns[ +0ns] +/- 0ns 2023-12-16T22:17:51.185 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.169 2023-12-16T22:18:00.187 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi169.front.sepia.ceph.com' 2023-12-16T22:18:00.188 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi169.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-12-16T22:18:03.249 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.169 2023-12-16T22:18:15.251 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi169.front.sepia.ceph.com' 2023-12-16T22:18:15.252 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi169.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-12-16T22:18:18.321 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.169 2023-12-16T22:18:21.322 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi169.front.sepia.ceph.com' 2023-12-16T22:18:21.323 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi169.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-12-16T22:18:21.397 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.169 2023-12-16T22:18:21.398 WARNING:teuthology.contextutil:'reconnect to {self.shortname}' reached maximum tries (5) after waiting for 30 seconds 2023-12-16T22:18:21.398 ERROR:teuthology.run_tasks:Manager failed: clock Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_5f3731ee8689ef3b0ef7d571e4cc2cb968c78d73/teuthology/task/clock.py", line 66, in task yield File "/home/teuthworker/src/git.ceph.com_teuthology_5f3731ee8689ef3b0ef7d571e4cc2cb968c78d73/teuthology/run_tasks.py", line 154, in run_tasks suppress = manager.__exit__(*exc_info) File "/usr/lib/python3.8/contextlib.py", line 120, in __exit__ next(self.gen) File "/home/teuthworker/src/github.com_ceph_ceph-c_c7586e95519b7be76f3b6e3a4b333888baf46bc9/qa/tasks/kclient.py", line 139, in task forced = umount_all() File "/home/teuthworker/src/github.com_ceph_ceph-c_c7586e95519b7be76f3b6e3a4b333888baf46bc9/qa/tasks/kclient.py", line 119, in umount_all if mount.is_mounted(): File "/home/teuthworker/src/github.com_ceph_ceph-c_c7586e95519b7be76f3b6e3a4b333888baf46bc9/qa/tasks/cephfs/mount.py", line 220, in is_mounted file = self.client_remote.read_file('/proc/self/mounts',stdout=StringIO()) File "/home/teuthworker/src/git.ceph.com_teuthology_5f3731ee8689ef3b0ef7d571e4cc2cb968c78d73/teuthology/orchestra/remote.py", line 254, in read_file proc = self.run(args=args, stdout=stdout, stderr=StringIO(), File "/home/teuthworker/src/git.ceph.com_teuthology_5f3731ee8689ef3b0ef7d571e4cc2cb968c78d73/teuthology/orchestra/remote.py", line 523, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_5f3731ee8689ef3b0ef7d571e4cc2cb968c78d73/teuthology/orchestra/run.py", line 450, in run r.execute() File "/home/teuthworker/src/git.ceph.com_teuthology_5f3731ee8689ef3b0ef7d571e4cc2cb968c78d73/teuthology/orchestra/run.py", line 100, in execute self.client.exec_command(self.command) File "/home/teuthworker/src/git.ceph.com_teuthology_5f3731ee8689ef3b0ef7d571e4cc2cb968c78d73/virtualenv/lib/python3.8/site-packages/paramiko/client.py", line 525, in exec_command chan = self._transport.open_session(timeout=timeout) File "/home/teuthworker/src/git.ceph.com_teuthology_5f3731ee8689ef3b0ef7d571e4cc2cb968c78d73/virtualenv/lib/python3.8/site-packages/paramiko/transport.py", line 920, in open_session return self.open_channel( File "/home/teuthworker/src/git.ceph.com_teuthology_5f3731ee8689ef3b0ef7d571e4cc2cb968c78d73/virtualenv/lib/python3.8/site-packages/paramiko/transport.py", line 1051, in open_channel raise e File "/home/teuthworker/src/git.ceph.com_teuthology_5f3731ee8689ef3b0ef7d571e4cc2cb968c78d73/virtualenv/lib/python3.8/site-packages/paramiko/transport.py", line 2110, in run ptype, m = self.packetizer.read_message() File "/home/teuthworker/src/git.ceph.com_teuthology_5f3731ee8689ef3b0ef7d571e4cc2cb968c78d73/virtualenv/lib/python3.8/site-packages/paramiko/packet.py", line 459, in read_message header = self.read_all(self.__block_size_in, check_rekey=True) File "/home/teuthworker/src/git.ceph.com_teuthology_5f3731ee8689ef3b0ef7d571e4cc2cb968c78d73/virtualenv/lib/python3.8/site-packages/paramiko/packet.py", line 301, in read_all x = self.__socket.recv(n) File "/home/teuthworker/src/git.ceph.com_teuthology_5f3731ee8689ef3b0ef7d571e4cc2cb968c78d73/virtualenv/lib/python3.8/site-packages/gevent/_socketcommon.py", line 657, in recv return self._sock.recv(*args) OSError: [Errno 113] No route to host During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_5f3731ee8689ef3b0ef7d571e4cc2cb968c78d73/teuthology/run_tasks.py", line 154, in run_tasks suppress = manager.__exit__(*exc_info) File "/usr/lib/python3.8/contextlib.py", line 131, in __exit__ self.gen.throw(type, value, traceback) File "/home/teuthworker/src/git.ceph.com_teuthology_5f3731ee8689ef3b0ef7d571e4cc2cb968c78d73/teuthology/task/clock.py", line 72, in task cluster.run( File "/home/teuthworker/src/git.ceph.com_teuthology_5f3731ee8689ef3b0ef7d571e4cc2cb968c78d73/teuthology/orchestra/cluster.py", line 85, in run procs = [remote.run(**kwargs, wait=_wait) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_teuthology_5f3731ee8689ef3b0ef7d571e4cc2cb968c78d73/teuthology/orchestra/cluster.py", line 85, in procs = [remote.run(**kwargs, wait=_wait) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_teuthology_5f3731ee8689ef3b0ef7d571e4cc2cb968c78d73/teuthology/orchestra/remote.py", line 522, in run raise ConnectionError(f'Failed to reconnect to {self.shortname}') ConnectionError: Failed to reconnect to smithi169 2023-12-16T22:18:21.412 DEBUG:teuthology.run_tasks:Unwinding manager ansible.cephlab 2023-12-16T22:18:21.424 INFO:teuthology.task.ansible:Skipping ansible cleanup... 2023-12-16T22:18:21.424 DEBUG:teuthology.run_tasks:Unwinding manager selinux 2023-12-16T22:18:21.438 DEBUG:teuthology.orchestra.run.smithi045:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2023-12-16T22:18:21.565 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi169.front.sepia.ceph.com' 2023-12-16T22:18:21.566 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi169.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-12-16T22:18:24.629 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.169 2023-12-16T22:18:33.630 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi169.front.sepia.ceph.com' 2023-12-16T22:18:33.631 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi169.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-12-16T22:18:36.689 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.169 2023-12-16T22:18:48.691 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi169.front.sepia.ceph.com' 2023-12-16T22:18:48.692 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi169.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-12-16T22:18:51.765 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.169 2023-12-16T22:18:54.767 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi169.front.sepia.ceph.com' 2023-12-16T22:18:54.767 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi169.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-12-16T22:18:54.833 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.169 2023-12-16T22:18:54.834 WARNING:teuthology.contextutil:'reconnect to {self.shortname}' reached maximum tries (5) after waiting for 30 seconds 2023-12-16T22:18:54.834 ERROR:teuthology.run_tasks:Manager failed: selinux Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_5f3731ee8689ef3b0ef7d571e4cc2cb968c78d73/teuthology/run_tasks.py", line 154, in run_tasks suppress = manager.__exit__(*exc_info) File "/home/teuthworker/src/git.ceph.com_teuthology_5f3731ee8689ef3b0ef7d571e4cc2cb968c78d73/teuthology/task/__init__.py", line 136, in __exit__ self.teardown() File "/home/teuthworker/src/git.ceph.com_teuthology_5f3731ee8689ef3b0ef7d571e4cc2cb968c78d73/teuthology/task/selinux.py", line 165, in teardown self.archive_log() File "/home/teuthworker/src/git.ceph.com_teuthology_5f3731ee8689ef3b0ef7d571e4cc2cb968c78d73/teuthology/task/selinux.py", line 195, in archive_log self.cluster.run( File "/home/teuthworker/src/git.ceph.com_teuthology_5f3731ee8689ef3b0ef7d571e4cc2cb968c78d73/teuthology/orchestra/cluster.py", line 85, in run procs = [remote.run(**kwargs, wait=_wait) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_teuthology_5f3731ee8689ef3b0ef7d571e4cc2cb968c78d73/teuthology/orchestra/cluster.py", line 85, in procs = [remote.run(**kwargs, wait=_wait) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_teuthology_5f3731ee8689ef3b0ef7d571e4cc2cb968c78d73/teuthology/orchestra/remote.py", line 522, in run raise ConnectionError(f'Failed to reconnect to {self.shortname}') ConnectionError: Failed to reconnect to smithi169 2023-12-16T22:18:54.842 DEBUG:teuthology.run_tasks:Unwinding manager pcp 2023-12-16T22:18:54.854 DEBUG:teuthology.run_tasks:Unwinding manager internal.timer 2023-12-16T22:18:54.880 INFO:teuthology.task.internal:Duration was 3304.527089 seconds 2023-12-16T22:18:54.881 DEBUG:teuthology.run_tasks:Unwinding manager internal.syslog 2023-12-16T22:18:54.891 INFO:teuthology.task.internal.syslog:Shutting down syslog monitoring... 2023-12-16T22:18:54.892 DEBUG:teuthology.orchestra.run.smithi045:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2023-12-16T22:18:54.898 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi169.front.sepia.ceph.com' 2023-12-16T22:18:54.898 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi169.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-12-16T22:18:54.955 INFO:teuthology.orchestra.run.smithi045.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2023-12-16T22:18:57.969 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.169 2023-12-16T22:19:06.970 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi169.front.sepia.ceph.com' 2023-12-16T22:19:06.971 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi169.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-12-16T22:19:10.037 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.169 2023-12-16T22:19:22.039 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi169.front.sepia.ceph.com' 2023-12-16T22:19:22.040 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi169.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-12-16T22:19:25.106 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.169 2023-12-16T22:19:28.107 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi169.front.sepia.ceph.com' 2023-12-16T22:19:28.108 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi169.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-12-16T22:19:28.177 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.169 2023-12-16T22:19:28.180 WARNING:teuthology.contextutil:'reconnect to {self.shortname}' reached maximum tries (5) after waiting for 30 seconds 2023-12-16T22:19:28.181 ERROR:teuthology.run_tasks:Manager failed: internal.syslog Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_5f3731ee8689ef3b0ef7d571e4cc2cb968c78d73/teuthology/task/internal/syslog.py", line 76, in syslog yield File "/home/teuthworker/src/git.ceph.com_teuthology_5f3731ee8689ef3b0ef7d571e4cc2cb968c78d73/teuthology/run_tasks.py", line 154, in run_tasks suppress = manager.__exit__(*exc_info) File "/usr/lib/python3.8/contextlib.py", line 120, in __exit__ next(self.gen) File "/home/teuthworker/src/github.com_ceph_ceph-c_c7586e95519b7be76f3b6e3a4b333888baf46bc9/qa/tasks/kclient.py", line 139, in task forced = umount_all() File "/home/teuthworker/src/github.com_ceph_ceph-c_c7586e95519b7be76f3b6e3a4b333888baf46bc9/qa/tasks/kclient.py", line 119, in umount_all if mount.is_mounted(): File "/home/teuthworker/src/github.com_ceph_ceph-c_c7586e95519b7be76f3b6e3a4b333888baf46bc9/qa/tasks/cephfs/mount.py", line 220, in is_mounted file = self.client_remote.read_file('/proc/self/mounts',stdout=StringIO()) File "/home/teuthworker/src/git.ceph.com_teuthology_5f3731ee8689ef3b0ef7d571e4cc2cb968c78d73/teuthology/orchestra/remote.py", line 254, in read_file proc = self.run(args=args, stdout=stdout, stderr=StringIO(), File "/home/teuthworker/src/git.ceph.com_teuthology_5f3731ee8689ef3b0ef7d571e4cc2cb968c78d73/teuthology/orchestra/remote.py", line 523, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_5f3731ee8689ef3b0ef7d571e4cc2cb968c78d73/teuthology/orchestra/run.py", line 450, in run r.execute() File "/home/teuthworker/src/git.ceph.com_teuthology_5f3731ee8689ef3b0ef7d571e4cc2cb968c78d73/teuthology/orchestra/run.py", line 100, in execute self.client.exec_command(self.command) File "/home/teuthworker/src/git.ceph.com_teuthology_5f3731ee8689ef3b0ef7d571e4cc2cb968c78d73/virtualenv/lib/python3.8/site-packages/paramiko/client.py", line 525, in exec_command chan = self._transport.open_session(timeout=timeout) File "/home/teuthworker/src/git.ceph.com_teuthology_5f3731ee8689ef3b0ef7d571e4cc2cb968c78d73/virtualenv/lib/python3.8/site-packages/paramiko/transport.py", line 920, in open_session return self.open_channel( File "/home/teuthworker/src/git.ceph.com_teuthology_5f3731ee8689ef3b0ef7d571e4cc2cb968c78d73/virtualenv/lib/python3.8/site-packages/paramiko/transport.py", line 1051, in open_channel raise e File "/home/teuthworker/src/git.ceph.com_teuthology_5f3731ee8689ef3b0ef7d571e4cc2cb968c78d73/virtualenv/lib/python3.8/site-packages/paramiko/transport.py", line 2110, in run ptype, m = self.packetizer.read_message() File "/home/teuthworker/src/git.ceph.com_teuthology_5f3731ee8689ef3b0ef7d571e4cc2cb968c78d73/virtualenv/lib/python3.8/site-packages/paramiko/packet.py", line 459, in read_message header = self.read_all(self.__block_size_in, check_rekey=True) File "/home/teuthworker/src/git.ceph.com_teuthology_5f3731ee8689ef3b0ef7d571e4cc2cb968c78d73/virtualenv/lib/python3.8/site-packages/paramiko/packet.py", line 301, in read_all x = self.__socket.recv(n) File "/home/teuthworker/src/git.ceph.com_teuthology_5f3731ee8689ef3b0ef7d571e4cc2cb968c78d73/virtualenv/lib/python3.8/site-packages/gevent/_socketcommon.py", line 657, in recv return self._sock.recv(*args) OSError: [Errno 113] No route to host During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_5f3731ee8689ef3b0ef7d571e4cc2cb968c78d73/teuthology/run_tasks.py", line 154, in run_tasks suppress = manager.__exit__(*exc_info) File "/usr/lib/python3.8/contextlib.py", line 131, in __exit__ self.gen.throw(type, value, traceback) File "/home/teuthworker/src/git.ceph.com_teuthology_5f3731ee8689ef3b0ef7d571e4cc2cb968c78d73/teuthology/task/internal/syslog.py", line 85, in syslog cluster.run( File "/home/teuthworker/src/git.ceph.com_teuthology_5f3731ee8689ef3b0ef7d571e4cc2cb968c78d73/teuthology/orchestra/cluster.py", line 85, in run procs = [remote.run(**kwargs, wait=_wait) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_teuthology_5f3731ee8689ef3b0ef7d571e4cc2cb968c78d73/teuthology/orchestra/cluster.py", line 85, in procs = [remote.run(**kwargs, wait=_wait) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_teuthology_5f3731ee8689ef3b0ef7d571e4cc2cb968c78d73/teuthology/orchestra/remote.py", line 522, in run raise ConnectionError(f'Failed to reconnect to {self.shortname}') ConnectionError: Failed to reconnect to smithi169 2023-12-16T22:19:28.199 DEBUG:teuthology.run_tasks:Unwinding manager internal.sudo 2023-12-16T22:19:28.220 INFO:teuthology.task.internal:Restoring /etc/sudoers... 2023-12-16T22:19:28.221 DEBUG:teuthology.orchestra.run.smithi045:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2023-12-16T22:19:28.263 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi169.front.sepia.ceph.com' 2023-12-16T22:19:28.264 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi169.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-12-16T22:19:31.349 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.169 2023-12-16T22:19:40.350 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi169.front.sepia.ceph.com' 2023-12-16T22:19:40.351 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi169.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-12-16T22:19:43.413 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.169 2023-12-16T22:19:55.414 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi169.front.sepia.ceph.com' 2023-12-16T22:19:55.415 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi169.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-12-16T22:19:58.481 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.169 2023-12-16T22:20:01.483 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi169.front.sepia.ceph.com' 2023-12-16T22:20:01.483 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi169.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-12-16T22:20:01.553 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.169 2023-12-16T22:20:01.554 WARNING:teuthology.contextutil:'reconnect to {self.shortname}' reached maximum tries (5) after waiting for 30 seconds 2023-12-16T22:20:01.554 ERROR:teuthology.run_tasks:Manager failed: internal.sudo Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_5f3731ee8689ef3b0ef7d571e4cc2cb968c78d73/teuthology/task/internal/__init__.py", line 452, in sudo yield File "/home/teuthworker/src/git.ceph.com_teuthology_5f3731ee8689ef3b0ef7d571e4cc2cb968c78d73/teuthology/run_tasks.py", line 154, in run_tasks suppress = manager.__exit__(*exc_info) File "/usr/lib/python3.8/contextlib.py", line 120, in __exit__ next(self.gen) File "/home/teuthworker/src/github.com_ceph_ceph-c_c7586e95519b7be76f3b6e3a4b333888baf46bc9/qa/tasks/kclient.py", line 139, in task forced = umount_all() File "/home/teuthworker/src/github.com_ceph_ceph-c_c7586e95519b7be76f3b6e3a4b333888baf46bc9/qa/tasks/kclient.py", line 119, in umount_all if mount.is_mounted(): File "/home/teuthworker/src/github.com_ceph_ceph-c_c7586e95519b7be76f3b6e3a4b333888baf46bc9/qa/tasks/cephfs/mount.py", line 220, in is_mounted file = self.client_remote.read_file('/proc/self/mounts',stdout=StringIO()) File "/home/teuthworker/src/git.ceph.com_teuthology_5f3731ee8689ef3b0ef7d571e4cc2cb968c78d73/teuthology/orchestra/remote.py", line 254, in read_file proc = self.run(args=args, stdout=stdout, stderr=StringIO(), File "/home/teuthworker/src/git.ceph.com_teuthology_5f3731ee8689ef3b0ef7d571e4cc2cb968c78d73/teuthology/orchestra/remote.py", line 523, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_5f3731ee8689ef3b0ef7d571e4cc2cb968c78d73/teuthology/orchestra/run.py", line 450, in run r.execute() File "/home/teuthworker/src/git.ceph.com_teuthology_5f3731ee8689ef3b0ef7d571e4cc2cb968c78d73/teuthology/orchestra/run.py", line 100, in execute self.client.exec_command(self.command) File "/home/teuthworker/src/git.ceph.com_teuthology_5f3731ee8689ef3b0ef7d571e4cc2cb968c78d73/virtualenv/lib/python3.8/site-packages/paramiko/client.py", line 525, in exec_command chan = self._transport.open_session(timeout=timeout) File "/home/teuthworker/src/git.ceph.com_teuthology_5f3731ee8689ef3b0ef7d571e4cc2cb968c78d73/virtualenv/lib/python3.8/site-packages/paramiko/transport.py", line 920, in open_session return self.open_channel( File "/home/teuthworker/src/git.ceph.com_teuthology_5f3731ee8689ef3b0ef7d571e4cc2cb968c78d73/virtualenv/lib/python3.8/site-packages/paramiko/transport.py", line 1051, in open_channel raise e File "/home/teuthworker/src/git.ceph.com_teuthology_5f3731ee8689ef3b0ef7d571e4cc2cb968c78d73/virtualenv/lib/python3.8/site-packages/paramiko/transport.py", line 2110, in run ptype, m = self.packetizer.read_message() File "/home/teuthworker/src/git.ceph.com_teuthology_5f3731ee8689ef3b0ef7d571e4cc2cb968c78d73/virtualenv/lib/python3.8/site-packages/paramiko/packet.py", line 459, in read_message header = self.read_all(self.__block_size_in, check_rekey=True) File "/home/teuthworker/src/git.ceph.com_teuthology_5f3731ee8689ef3b0ef7d571e4cc2cb968c78d73/virtualenv/lib/python3.8/site-packages/paramiko/packet.py", line 301, in read_all x = self.__socket.recv(n) File "/home/teuthworker/src/git.ceph.com_teuthology_5f3731ee8689ef3b0ef7d571e4cc2cb968c78d73/virtualenv/lib/python3.8/site-packages/gevent/_socketcommon.py", line 657, in recv return self._sock.recv(*args) OSError: [Errno 113] No route to host During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_5f3731ee8689ef3b0ef7d571e4cc2cb968c78d73/teuthology/run_tasks.py", line 154, in run_tasks suppress = manager.__exit__(*exc_info) File "/usr/lib/python3.8/contextlib.py", line 131, in __exit__ self.gen.throw(type, value, traceback) File "/home/teuthworker/src/git.ceph.com_teuthology_5f3731ee8689ef3b0ef7d571e4cc2cb968c78d73/teuthology/task/internal/__init__.py", line 455, in sudo ctx.cluster.run( File "/home/teuthworker/src/git.ceph.com_teuthology_5f3731ee8689ef3b0ef7d571e4cc2cb968c78d73/teuthology/orchestra/cluster.py", line 85, in run procs = [remote.run(**kwargs, wait=_wait) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_teuthology_5f3731ee8689ef3b0ef7d571e4cc2cb968c78d73/teuthology/orchestra/cluster.py", line 85, in procs = [remote.run(**kwargs, wait=_wait) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_teuthology_5f3731ee8689ef3b0ef7d571e4cc2cb968c78d73/teuthology/orchestra/remote.py", line 522, in run raise ConnectionError(f'Failed to reconnect to {self.shortname}') ConnectionError: Failed to reconnect to smithi169 2023-12-16T22:20:01.571 DEBUG:teuthology.run_tasks:Unwinding manager internal.coredump 2023-12-16T22:20:01.584 DEBUG:teuthology.orchestra.run.smithi045:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2023-12-16T22:20:01.594 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi169.front.sepia.ceph.com' 2023-12-16T22:20:01.594 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi169.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-12-16T22:20:01.627 INFO:teuthology.orchestra.run.smithi045.stdout:kernel.core_pattern = core 2023-12-16T22:20:04.657 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.169 2023-12-16T22:20:13.659 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi169.front.sepia.ceph.com' 2023-12-16T22:20:13.660 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi169.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-12-16T22:20:16.721 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.169 2023-12-16T22:20:28.722 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi169.front.sepia.ceph.com' 2023-12-16T22:20:28.723 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi169.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-12-16T22:20:31.793 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.169 2023-12-16T22:20:34.794 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi169.front.sepia.ceph.com' 2023-12-16T22:20:34.795 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi169.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-12-16T22:20:34.865 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.169 2023-12-16T22:20:34.866 WARNING:teuthology.contextutil:'reconnect to {self.shortname}' reached maximum tries (5) after waiting for 30 seconds 2023-12-16T22:20:34.866 ERROR:teuthology.run_tasks:Manager failed: internal.coredump Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_5f3731ee8689ef3b0ef7d571e4cc2cb968c78d73/teuthology/task/internal/__init__.py", line 488, in coredump yield File "/home/teuthworker/src/git.ceph.com_teuthology_5f3731ee8689ef3b0ef7d571e4cc2cb968c78d73/teuthology/run_tasks.py", line 154, in run_tasks suppress = manager.__exit__(*exc_info) File "/usr/lib/python3.8/contextlib.py", line 120, in __exit__ next(self.gen) File "/home/teuthworker/src/github.com_ceph_ceph-c_c7586e95519b7be76f3b6e3a4b333888baf46bc9/qa/tasks/kclient.py", line 139, in task forced = umount_all() File "/home/teuthworker/src/github.com_ceph_ceph-c_c7586e95519b7be76f3b6e3a4b333888baf46bc9/qa/tasks/kclient.py", line 119, in umount_all if mount.is_mounted(): File "/home/teuthworker/src/github.com_ceph_ceph-c_c7586e95519b7be76f3b6e3a4b333888baf46bc9/qa/tasks/cephfs/mount.py", line 220, in is_mounted file = self.client_remote.read_file('/proc/self/mounts',stdout=StringIO()) File "/home/teuthworker/src/git.ceph.com_teuthology_5f3731ee8689ef3b0ef7d571e4cc2cb968c78d73/teuthology/orchestra/remote.py", line 254, in read_file proc = self.run(args=args, stdout=stdout, stderr=StringIO(), File "/home/teuthworker/src/git.ceph.com_teuthology_5f3731ee8689ef3b0ef7d571e4cc2cb968c78d73/teuthology/orchestra/remote.py", line 523, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_5f3731ee8689ef3b0ef7d571e4cc2cb968c78d73/teuthology/orchestra/run.py", line 450, in run r.execute() File "/home/teuthworker/src/git.ceph.com_teuthology_5f3731ee8689ef3b0ef7d571e4cc2cb968c78d73/teuthology/orchestra/run.py", line 100, in execute self.client.exec_command(self.command) File "/home/teuthworker/src/git.ceph.com_teuthology_5f3731ee8689ef3b0ef7d571e4cc2cb968c78d73/virtualenv/lib/python3.8/site-packages/paramiko/client.py", line 525, in exec_command chan = self._transport.open_session(timeout=timeout) File "/home/teuthworker/src/git.ceph.com_teuthology_5f3731ee8689ef3b0ef7d571e4cc2cb968c78d73/virtualenv/lib/python3.8/site-packages/paramiko/transport.py", line 920, in open_session return self.open_channel( File "/home/teuthworker/src/git.ceph.com_teuthology_5f3731ee8689ef3b0ef7d571e4cc2cb968c78d73/virtualenv/lib/python3.8/site-packages/paramiko/transport.py", line 1051, in open_channel raise e File "/home/teuthworker/src/git.ceph.com_teuthology_5f3731ee8689ef3b0ef7d571e4cc2cb968c78d73/virtualenv/lib/python3.8/site-packages/paramiko/transport.py", line 2110, in run ptype, m = self.packetizer.read_message() File "/home/teuthworker/src/git.ceph.com_teuthology_5f3731ee8689ef3b0ef7d571e4cc2cb968c78d73/virtualenv/lib/python3.8/site-packages/paramiko/packet.py", line 459, in read_message header = self.read_all(self.__block_size_in, check_rekey=True) File "/home/teuthworker/src/git.ceph.com_teuthology_5f3731ee8689ef3b0ef7d571e4cc2cb968c78d73/virtualenv/lib/python3.8/site-packages/paramiko/packet.py", line 301, in read_all x = self.__socket.recv(n) File "/home/teuthworker/src/git.ceph.com_teuthology_5f3731ee8689ef3b0ef7d571e4cc2cb968c78d73/virtualenv/lib/python3.8/site-packages/gevent/_socketcommon.py", line 657, in recv return self._sock.recv(*args) OSError: [Errno 113] No route to host During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_5f3731ee8689ef3b0ef7d571e4cc2cb968c78d73/teuthology/run_tasks.py", line 154, in run_tasks suppress = manager.__exit__(*exc_info) File "/usr/lib/python3.8/contextlib.py", line 131, in __exit__ self.gen.throw(type, value, traceback) File "/home/teuthworker/src/git.ceph.com_teuthology_5f3731ee8689ef3b0ef7d571e4cc2cb968c78d73/teuthology/task/internal/__init__.py", line 492, in coredump cluster.run( File "/home/teuthworker/src/git.ceph.com_teuthology_5f3731ee8689ef3b0ef7d571e4cc2cb968c78d73/teuthology/orchestra/cluster.py", line 85, in run procs = [remote.run(**kwargs, wait=_wait) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_teuthology_5f3731ee8689ef3b0ef7d571e4cc2cb968c78d73/teuthology/orchestra/cluster.py", line 85, in procs = [remote.run(**kwargs, wait=_wait) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_teuthology_5f3731ee8689ef3b0ef7d571e4cc2cb968c78d73/teuthology/orchestra/remote.py", line 522, in run raise ConnectionError(f'Failed to reconnect to {self.shortname}') ConnectionError: Failed to reconnect to smithi169 2023-12-16T22:20:34.882 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive 2023-12-16T22:20:34.895 INFO:teuthology.task.internal:Transferring archived files... 2023-12-16T22:20:34.897 DEBUG:teuthology.misc:Transferring archived files from smithi045:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/yuriw-2023-12-16_15:45:14-fs-wip-yuri8-testing-2023-12-15-0911-distro-default-smithi/7494858/remote/smithi045 2023-12-16T22:20:34.897 DEBUG:teuthology.orchestra.run.smithi045:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2023-12-16T22:20:34.947 DEBUG:teuthology.misc:Transferring archived files from smithi169:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/yuriw-2023-12-16_15:45:14-fs-wip-yuri8-testing-2023-12-15-0911-distro-default-smithi/7494858/remote/smithi169 2023-12-16T22:20:34.947 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi169.front.sepia.ceph.com' 2023-12-16T22:20:34.948 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi169.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-12-16T22:20:38.005 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.169 2023-12-16T22:20:47.006 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi169.front.sepia.ceph.com' 2023-12-16T22:20:47.007 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi169.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-12-16T22:20:50.069 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.169 2023-12-16T22:21:02.070 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi169.front.sepia.ceph.com' 2023-12-16T22:21:02.071 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi169.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-12-16T22:21:05.137 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.169 2023-12-16T22:21:08.138 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi169.front.sepia.ceph.com' 2023-12-16T22:21:08.139 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi169.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-12-16T22:21:08.209 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.169 2023-12-16T22:21:08.210 WARNING:teuthology.contextutil:'reconnect to {self.shortname}' reached maximum tries (5) after waiting for 30 seconds 2023-12-16T22:21:08.210 ERROR:teuthology.run_tasks:Manager failed: internal.archive Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_5f3731ee8689ef3b0ef7d571e4cc2cb968c78d73/teuthology/task/internal/__init__.py", line 392, in archive yield File "/home/teuthworker/src/git.ceph.com_teuthology_5f3731ee8689ef3b0ef7d571e4cc2cb968c78d73/teuthology/run_tasks.py", line 154, in run_tasks suppress = manager.__exit__(*exc_info) File "/usr/lib/python3.8/contextlib.py", line 120, in __exit__ next(self.gen) File "/home/teuthworker/src/github.com_ceph_ceph-c_c7586e95519b7be76f3b6e3a4b333888baf46bc9/qa/tasks/kclient.py", line 139, in task forced = umount_all() File "/home/teuthworker/src/github.com_ceph_ceph-c_c7586e95519b7be76f3b6e3a4b333888baf46bc9/qa/tasks/kclient.py", line 119, in umount_all if mount.is_mounted(): File "/home/teuthworker/src/github.com_ceph_ceph-c_c7586e95519b7be76f3b6e3a4b333888baf46bc9/qa/tasks/cephfs/mount.py", line 220, in is_mounted file = self.client_remote.read_file('/proc/self/mounts',stdout=StringIO()) File "/home/teuthworker/src/git.ceph.com_teuthology_5f3731ee8689ef3b0ef7d571e4cc2cb968c78d73/teuthology/orchestra/remote.py", line 254, in read_file proc = self.run(args=args, stdout=stdout, stderr=StringIO(), File "/home/teuthworker/src/git.ceph.com_teuthology_5f3731ee8689ef3b0ef7d571e4cc2cb968c78d73/teuthology/orchestra/remote.py", line 523, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_5f3731ee8689ef3b0ef7d571e4cc2cb968c78d73/teuthology/orchestra/run.py", line 450, in run r.execute() File "/home/teuthworker/src/git.ceph.com_teuthology_5f3731ee8689ef3b0ef7d571e4cc2cb968c78d73/teuthology/orchestra/run.py", line 100, in execute self.client.exec_command(self.command) File "/home/teuthworker/src/git.ceph.com_teuthology_5f3731ee8689ef3b0ef7d571e4cc2cb968c78d73/virtualenv/lib/python3.8/site-packages/paramiko/client.py", line 525, in exec_command chan = self._transport.open_session(timeout=timeout) File "/home/teuthworker/src/git.ceph.com_teuthology_5f3731ee8689ef3b0ef7d571e4cc2cb968c78d73/virtualenv/lib/python3.8/site-packages/paramiko/transport.py", line 920, in open_session return self.open_channel( File "/home/teuthworker/src/git.ceph.com_teuthology_5f3731ee8689ef3b0ef7d571e4cc2cb968c78d73/virtualenv/lib/python3.8/site-packages/paramiko/transport.py", line 1051, in open_channel raise e File "/home/teuthworker/src/git.ceph.com_teuthology_5f3731ee8689ef3b0ef7d571e4cc2cb968c78d73/virtualenv/lib/python3.8/site-packages/paramiko/transport.py", line 2110, in run ptype, m = self.packetizer.read_message() File "/home/teuthworker/src/git.ceph.com_teuthology_5f3731ee8689ef3b0ef7d571e4cc2cb968c78d73/virtualenv/lib/python3.8/site-packages/paramiko/packet.py", line 459, in read_message header = self.read_all(self.__block_size_in, check_rekey=True) File "/home/teuthworker/src/git.ceph.com_teuthology_5f3731ee8689ef3b0ef7d571e4cc2cb968c78d73/virtualenv/lib/python3.8/site-packages/paramiko/packet.py", line 301, in read_all x = self.__socket.recv(n) File "/home/teuthworker/src/git.ceph.com_teuthology_5f3731ee8689ef3b0ef7d571e4cc2cb968c78d73/virtualenv/lib/python3.8/site-packages/gevent/_socketcommon.py", line 657, in recv return self._sock.recv(*args) OSError: [Errno 113] No route to host During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_5f3731ee8689ef3b0ef7d571e4cc2cb968c78d73/teuthology/run_tasks.py", line 154, in run_tasks suppress = manager.__exit__(*exc_info) File "/usr/lib/python3.8/contextlib.py", line 131, in __exit__ self.gen.throw(type, value, traceback) File "/home/teuthworker/src/git.ceph.com_teuthology_5f3731ee8689ef3b0ef7d571e4cc2cb968c78d73/teuthology/task/internal/__init__.py", line 418, in archive misc.pull_directory(rem, archive_dir, path, maybe_compress) File "/home/teuthworker/src/git.ceph.com_teuthology_5f3731ee8689ef3b0ef7d571e4cc2cb968c78d73/teuthology/misc.py", line 748, in pull_directory r = remote.get_tar_stream(remotedir, sudo=True, compress=False) File "/home/teuthworker/src/git.ceph.com_teuthology_5f3731ee8689ef3b0ef7d571e4cc2cb968c78d73/teuthology/orchestra/remote.py", line 678, in get_tar_stream return self.run(args=args, wait=False, stdout=run.PIPE) File "/home/teuthworker/src/git.ceph.com_teuthology_5f3731ee8689ef3b0ef7d571e4cc2cb968c78d73/teuthology/orchestra/remote.py", line 522, in run raise ConnectionError(f'Failed to reconnect to {self.shortname}') ConnectionError: Failed to reconnect to smithi169 2023-12-16T22:21:08.231 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive_upload 2023-12-16T22:21:08.244 INFO:teuthology.task.internal:Not uploading archives. 2023-12-16T22:21:08.271 DEBUG:teuthology.run_tasks:Unwinding manager internal.base 2023-12-16T22:21:08.320 INFO:teuthology.task.internal:Tidying up after the test... 2023-12-16T22:21:08.321 DEBUG:teuthology.orchestra.run.smithi045:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2023-12-16T22:21:08.332 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi169.front.sepia.ceph.com' 2023-12-16T22:21:08.332 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi169.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-12-16T22:21:08.348 INFO:teuthology.orchestra.run.smithi045.stdout: 262166 4 drwxr-xr-x 3 ubuntu ubuntu 4096 Dec 16 22:17 /home/ubuntu/cephtest 2023-12-16T22:21:08.348 INFO:teuthology.orchestra.run.smithi045.stdout: 262167 4 drwxr-xr-x 4 ubuntu ubuntu 4096 Dec 16 22:20 /home/ubuntu/cephtest/archive 2023-12-16T22:21:08.349 INFO:teuthology.orchestra.run.smithi045.stdout: 262169 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Dec 16 21:23 /home/ubuntu/cephtest/archive/syslog 2023-12-16T22:21:08.349 INFO:teuthology.orchestra.run.smithi045.stdout: 265245 0 -rw-rw-rw- 1 ubuntu ubuntu 0 Dec 16 21:23 /home/ubuntu/cephtest/archive/syslog/misc.log 2023-12-16T22:21:08.349 INFO:teuthology.orchestra.run.smithi045.stdout: 265244 0 -rw-rw-rw- 1 ubuntu ubuntu 0 Dec 16 21:23 /home/ubuntu/cephtest/archive/syslog/kern.log 2023-12-16T22:21:08.349 INFO:teuthology.orchestra.run.smithi045.stdout: 674012 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Dec 16 22:18 /home/ubuntu/cephtest/archive/audit 2023-12-16T22:21:08.349 INFO:teuthology.orchestra.run.smithi045.stdout: 674014 292 -rw------- 1 ubuntu ubuntu 297225 Dec 16 22:18 /home/ubuntu/cephtest/archive/audit/audit.log.gz 2023-12-16T22:21:08.349 INFO:teuthology.orchestra.run.smithi045.stdout: 265281 360 -rwxr-xr-x 1 ubuntu ubuntu 367151 Dec 16 21:31 /home/ubuntu/cephtest/cephadm 2023-12-16T22:21:08.350 INFO:teuthology.orchestra.run.smithi045.stderr:rmdir: failed to remove '/home/ubuntu/cephtest': Directory not empty 2023-12-16T22:21:11.413 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.169 2023-12-16T22:21:20.414 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi169.front.sepia.ceph.com' 2023-12-16T22:21:20.415 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi169.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-12-16T22:21:23.473 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.169 2023-12-16T22:21:35.475 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi169.front.sepia.ceph.com' 2023-12-16T22:21:35.476 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi169.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-12-16T22:21:38.545 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.169 2023-12-16T22:21:41.546 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi169.front.sepia.ceph.com' 2023-12-16T22:21:41.547 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi169.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-12-16T22:21:41.617 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.169 2023-12-16T22:21:41.618 WARNING:teuthology.contextutil:'reconnect to {self.shortname}' reached maximum tries (5) after waiting for 30 seconds 2023-12-16T22:21:41.619 ERROR:teuthology.run_tasks:Manager failed: internal.base Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_5f3731ee8689ef3b0ef7d571e4cc2cb968c78d73/teuthology/task/internal/__init__.py", line 48, in base yield File "/home/teuthworker/src/git.ceph.com_teuthology_5f3731ee8689ef3b0ef7d571e4cc2cb968c78d73/teuthology/run_tasks.py", line 154, in run_tasks suppress = manager.__exit__(*exc_info) File "/usr/lib/python3.8/contextlib.py", line 120, in __exit__ next(self.gen) File "/home/teuthworker/src/github.com_ceph_ceph-c_c7586e95519b7be76f3b6e3a4b333888baf46bc9/qa/tasks/kclient.py", line 139, in task forced = umount_all() File "/home/teuthworker/src/github.com_ceph_ceph-c_c7586e95519b7be76f3b6e3a4b333888baf46bc9/qa/tasks/kclient.py", line 119, in umount_all if mount.is_mounted(): File "/home/teuthworker/src/github.com_ceph_ceph-c_c7586e95519b7be76f3b6e3a4b333888baf46bc9/qa/tasks/cephfs/mount.py", line 220, in is_mounted file = self.client_remote.read_file('/proc/self/mounts',stdout=StringIO()) File "/home/teuthworker/src/git.ceph.com_teuthology_5f3731ee8689ef3b0ef7d571e4cc2cb968c78d73/teuthology/orchestra/remote.py", line 254, in read_file proc = self.run(args=args, stdout=stdout, stderr=StringIO(), File "/home/teuthworker/src/git.ceph.com_teuthology_5f3731ee8689ef3b0ef7d571e4cc2cb968c78d73/teuthology/orchestra/remote.py", line 523, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_5f3731ee8689ef3b0ef7d571e4cc2cb968c78d73/teuthology/orchestra/run.py", line 450, in run r.execute() File "/home/teuthworker/src/git.ceph.com_teuthology_5f3731ee8689ef3b0ef7d571e4cc2cb968c78d73/teuthology/orchestra/run.py", line 100, in execute self.client.exec_command(self.command) File "/home/teuthworker/src/git.ceph.com_teuthology_5f3731ee8689ef3b0ef7d571e4cc2cb968c78d73/virtualenv/lib/python3.8/site-packages/paramiko/client.py", line 525, in exec_command chan = self._transport.open_session(timeout=timeout) File "/home/teuthworker/src/git.ceph.com_teuthology_5f3731ee8689ef3b0ef7d571e4cc2cb968c78d73/virtualenv/lib/python3.8/site-packages/paramiko/transport.py", line 920, in open_session return self.open_channel( File "/home/teuthworker/src/git.ceph.com_teuthology_5f3731ee8689ef3b0ef7d571e4cc2cb968c78d73/virtualenv/lib/python3.8/site-packages/paramiko/transport.py", line 1051, in open_channel raise e File "/home/teuthworker/src/git.ceph.com_teuthology_5f3731ee8689ef3b0ef7d571e4cc2cb968c78d73/virtualenv/lib/python3.8/site-packages/paramiko/transport.py", line 2110, in run ptype, m = self.packetizer.read_message() File "/home/teuthworker/src/git.ceph.com_teuthology_5f3731ee8689ef3b0ef7d571e4cc2cb968c78d73/virtualenv/lib/python3.8/site-packages/paramiko/packet.py", line 459, in read_message header = self.read_all(self.__block_size_in, check_rekey=True) File "/home/teuthworker/src/git.ceph.com_teuthology_5f3731ee8689ef3b0ef7d571e4cc2cb968c78d73/virtualenv/lib/python3.8/site-packages/paramiko/packet.py", line 301, in read_all x = self.__socket.recv(n) File "/home/teuthworker/src/git.ceph.com_teuthology_5f3731ee8689ef3b0ef7d571e4cc2cb968c78d73/virtualenv/lib/python3.8/site-packages/gevent/_socketcommon.py", line 657, in recv return self._sock.recv(*args) OSError: [Errno 113] No route to host During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_5f3731ee8689ef3b0ef7d571e4cc2cb968c78d73/teuthology/run_tasks.py", line 154, in run_tasks suppress = manager.__exit__(*exc_info) File "/usr/lib/python3.8/contextlib.py", line 131, in __exit__ self.gen.throw(type, value, traceback) File "/home/teuthworker/src/git.ceph.com_teuthology_5f3731ee8689ef3b0ef7d571e4cc2cb968c78d73/teuthology/task/internal/__init__.py", line 54, in base ctx.cluster.run( File "/home/teuthworker/src/git.ceph.com_teuthology_5f3731ee8689ef3b0ef7d571e4cc2cb968c78d73/teuthology/orchestra/cluster.py", line 85, in run procs = [remote.run(**kwargs, wait=_wait) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_teuthology_5f3731ee8689ef3b0ef7d571e4cc2cb968c78d73/teuthology/orchestra/cluster.py", line 85, in procs = [remote.run(**kwargs, wait=_wait) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_teuthology_5f3731ee8689ef3b0ef7d571e4cc2cb968c78d73/teuthology/orchestra/remote.py", line 522, in run raise ConnectionError(f'Failed to reconnect to {self.shortname}') ConnectionError: Failed to reconnect to smithi169 2023-12-16T22:21:41.644 DEBUG:teuthology.run_tasks:Unwinding manager kernel 2023-12-16T22:21:41.658 DEBUG:teuthology.run_tasks:Unwinding manager console_log 2023-12-16T22:21:41.688 DEBUG:teuthology.run_tasks:Exception was not quenched, exiting: OSError: [Errno 113] No route to host 2023-12-16T22:21:41.688 INFO:teuthology.nuke:Checking targets against current locks 2023-12-16T22:21:41.719 DEBUG:teuthology.nuke:shortname: smithi045 2023-12-16T22:21:41.720 INFO:teuthology.task.internal.check_lock:Checking locks... 2023-12-16T22:21:41.744 DEBUG:teuthology.nuke:shortname: smithi169 2023-12-16T22:21:41.745 INFO:teuthology.task.internal.check_lock:Checking locks... 2023-12-16T22:21:41.749 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi045.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/yuriw-2023-12-16_15:45:14-fs-wip-yuri8-testing-2023-12-15-0911-distro-default-smithi/7494858', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2023-12-16 21:14:45.499527', 'locked_by': 'scheduled_yuriw@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBPsCK7s1vHn0T4FJqBKnHXIpKBUcg+SA1RW0IVqgduhRtZ+wdrh/8M1i96JDgGgUBfEwf4A9vlTbi2lG9J6V2oE='} 2023-12-16T22:21:41.771 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi169.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/yuriw-2023-12-16_15:45:14-fs-wip-yuri8-testing-2023-12-15-0911-distro-default-smithi/7494858', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2023-12-16 21:14:45.497652', 'locked_by': 'scheduled_yuriw@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBDcLjeTZR2mTfVitxMoIj6Egtg+Gcrvh+9VO54SqCqzI0kSuJl14hyyfN8dF8vxDH8FuhaDPT12sO01w//NOn5s='} 2023-12-16T22:21:41.805 INFO:teuthology.orchestra.console.smithi045:Power off 2023-12-16T22:21:41.805 DEBUG:teuthology.orchestra.console.smithi045:pexpect command: ipmitool -H smithi045.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2023-12-16T22:21:41.832 DEBUG:teuthology.orchestra.console.smithi045:power off output: Chassis Power Control: Down/Off 2023-12-16T22:21:41.832 DEBUG:teuthology.orchestra.console.smithi045:pexpect command: ipmitool -H smithi045.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-12-16T22:21:41.850 INFO:teuthology.orchestra.console.smithi169:Power off 2023-12-16T22:21:41.851 DEBUG:teuthology.orchestra.console.smithi169:pexpect command: ipmitool -H smithi169.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2023-12-16T22:21:41.860 DEBUG:teuthology.orchestra.console.smithi045:check power output: Chassis Power is on 2023-12-16T22:21:41.874 DEBUG:teuthology.orchestra.console.smithi169:power off output: Chassis Power Control: Down/Off 2023-12-16T22:21:41.874 DEBUG:teuthology.orchestra.console.smithi169:pexpect command: ipmitool -H smithi169.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-12-16T22:21:41.915 DEBUG:teuthology.orchestra.console.smithi169:check power output: Chassis Power is on 2023-12-16T22:21:45.861 DEBUG:teuthology.orchestra.console.smithi045:pexpect command: ipmitool -H smithi045.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-12-16T22:21:45.917 DEBUG:teuthology.orchestra.console.smithi169:pexpect command: ipmitool -H smithi169.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-12-16T22:21:45.973 DEBUG:teuthology.orchestra.console.smithi045:check power output: Chassis Power is on 2023-12-16T22:21:46.027 DEBUG:teuthology.orchestra.console.smithi169:check power output: Chassis Power is on 2023-12-16T22:21:49.977 DEBUG:teuthology.orchestra.console.smithi045:pexpect command: ipmitool -H smithi045.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-12-16T22:21:50.028 DEBUG:teuthology.orchestra.console.smithi169:pexpect command: ipmitool -H smithi169.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-12-16T22:21:50.090 DEBUG:teuthology.orchestra.console.smithi045:check power output: Chassis Power is off 2023-12-16T22:21:50.140 DEBUG:teuthology.orchestra.console.smithi169:check power output: Chassis Power is on 2023-12-16T22:21:50.191 INFO:teuthology.orchestra.console.smithi045:Power off completed 2023-12-16T22:21:54.141 DEBUG:teuthology.orchestra.console.smithi169:pexpect command: ipmitool -H smithi169.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-12-16T22:21:54.252 DEBUG:teuthology.orchestra.console.smithi169:check power output: Chassis Power is off 2023-12-16T22:21:54.353 INFO:teuthology.orchestra.console.smithi169:Power off completed 2023-12-16T22:21:54.458 INFO:teuthology.run:Summary data: description: fs/upgrade/mds_upgrade_sequence/{bluestore-bitmap centos_9.stream conf/{client mds mon osd} fail_fs/yes overrides/{ignorelist_health ignorelist_wrongly_marked_down pg-warn syntax} roles tasks/{0-from/quincy 1-volume/{0-create 1-ranks/1 2-allow_standby_replay/yes 3-inline/yes 4-verify} 2-client 3-upgrade-mgr-staggered 4-config-upgrade/{fail_fs} 5-upgrade-with-workload 6-verify}} duration: 3304.5270891189575 failure_reason: 'Command failed on smithi045 with status 1: ''sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 88ff363c-9c5a-11ee-95a5-87774f69a715 -e sha1=c7586e95519b7be76f3b6e3a4b333888baf46bc9 -- bash -c \''ceph versions | jq -e \''"\''"\''.mgr | keys\''"\''"\'' | grep $sha1\''''' flavor: default owner: scheduled_yuriw@teuthology sentry_event: https://sentry.ceph.com/organizations/ceph/?query=c83a120f77f048c9bbb43efdd1f904ee status: fail success: false 2023-12-16T22:21:54.459 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2023-12-16T22:21:54.548 INFO:teuthology.run:FAIL